Automatic splitting of output torrent files

[mr-bo-jangles]

Hey, this is more a feasibility questions than anything.

Currently with large batches of content we're having issues getting anything other than qbittorent to load the generated torrents while preserving the file list, the solution is to split. So if we're to split the torrents up we want to generate the minimum number of these files.

Is there the possibility of generating these automaticly split torrents that cover the same folder structure once the overall .torrent file gets too large?

[casey]

One good first step that we should definitely do is figure out exactly what the limiting factor is, and then add a lint for it. imdl has lints for a few things, like uneven piece length, small piece length, etc, which it will complain about if they're violated, but which you can disable if you want. I think this should definitely be a lint.

Do you know what exactly the limiting factor is? Is it the size of the .torrent file, the size of the pieces list, the number of files, or possibly something else?

I think there are two possible approaches here:

• Make it easy to use another tool to give imdl lists of files to include in a torrent. Currently, imdl doesn't have a good way to pass in a large list of files. There's --glob which allows including or excluding files, but if the list of files you want to include or exclude is large, you might be limited by the number of arguments that are allowed to a single command. This would probably take the form of --whitelist and --blacklist arguments, which would take a text file with one filename per line, and those filenames would be included or excluded from the torrent. Then, you could write a python script or something, which would invoke imdl with the right --whitelists.

• Make imdl split torrents. This would be a bigger change, and depends on what the actual limit is that you're running into. I'm not sure exactly what to call it, but maybe this would come in the form of --split LIMIT:AMOUNT, so you could do things like --split content-size:500GiB or --split file-count:10000.

The former approach is much easier, and I can imagine it being used in other contexts. The latter approach is obviously much nicer, but probably also a more involved change.

In either case though, a lint would be a great place to start. If you don't mind doing some experimenting, and finding the specific limits, and which torrent clients have those issues, that would be a huge help.

[mr-bo-jangles]

So we're likely encountering multiple issues:

The first limit I've found is uTorrent with their limit of 65535 pieces per torrent file. "µTorrent won't open torrents with more than 65535 pieces by design"

The second limiting factor is the number of files per .torrent making it impractical to use the torrent with a webui which is common with torrent server setups. This may be negated by torrentv2?

The third limiting factor is probably just poor memory management for some torrent clients which choke/crash with larger .torrents

Being practical I'd guess that when the final .torrent size is getting over 4MB you're probably close to hitting one of these limits

[casey]

Lints should be relatively objective, so I think having more than 65535 pieces is good lint, since uTorrent is so popular. I'm not sure number of files or torrent size would make good lints, because I'm not sure what they should be set to.

I created #499 to track adding a piece-count lint.

I'm definitely in favor of adding automatic torrent splitting, although I'm not sure when/if I'll actually implement it myself.