Spent a lot of time working on this actually, and it's more or less working.
If you invoke dpb -fN, then it will create n jobs specifically for fetching distfiles. Those are local (should be movable to -DFETCHHOST=value, will happen soon) and don't really consume cpu (apart from the checksum part). I even tested it by wiping all my distfiles and found a few fun surprises in the process. The heuristics to choose which distfile to fetch first are totally rudimentary for now, so it doesn't work all that great with an empty distfiles directory. For "normal" work, it performs great: instead of having a lot of dpb job sitting around waiting for fetch to occur, fetch will happen on the fly, and having a correct progress meter for fetching distfiles is good too. Compared to old mirror-maker, dpb uses ftp -C in a somewhat smart way: it creates temporary files that will get moved after they're checksummed. (that was today's commits, actually, I had to finish the logic of that part). dpb does not perform too great with FETCH_MANUALLY either, it will just log the corresponding messages, and NOT restart the jobs correctly: after fetching manually, you will have to abort and restart dpb, for now. To do list: - better handling of restart on lock removal. - better heuristics for fetch order: should probably start with small distfiles with no dependencies to avoid queue starvation, then switch to an heuristics directly tied to the build engine, then do the largest files first (icky part is to figure out when to switch strategies, no smart idea yet). - clean-up engine some more and extend to do regress. - document the new file names.
