On Sat, Aug 29, 2015 at 12:50 AM, Darshit Shah <dar...@gmail.com> wrote: > Thanking You, > Darshit Shah > Sent from mobile device. Please excuse my brevity > On 29-Aug-2015 1:13 pm, "Tim Rühsen" <tim.rueh...@gmx.de> wrote: >> >> Hi, >> >> normally it makes much more sense when having several download mirrors and >> checksums for each chunk. The perfect technique for such is called > 'Metalink' >> (more on www.metalinker.org). >> Wget has it in branch 'master'. A GSOC project of Hubert Tarasiuk. >> > Sometimes the evil ISPs enforce a per connection bandwidth limit. In such a > case, multi segment downloads from a single server do make sense. > > Since metalink already has support for downloading a file over multiple > connections, it should not be too difficult to reuse the code for use > outside of metalink. The current Metalink impl in Wget will not download from multiple mirrors simultaneously since Wget itself is single-threaded. Adding optional (POSIX) threads support to Wget (especially for the Metalinks) could be perhaps worth discussion. For now the solution might be to start multiple Wget instances using the --start-pos option and somehow limit the length of download (I am not sure if Wget currently has an option to do that).
> > I think it would be a good idea to do so. I'm not sure if all the possible > variations of the range headers are parsed by Wget. >> Additionally, Wget2 is under development, already having the option > --chunk- >> size (e.g. --chunk-size=1M) to start a multi-threaded download of a file. >> >> Regards, Tim >> >> >> Am Freitag, 28. August 2015, 15:41:27 schrieb Random Coder: >> > On Fri, Aug 28, 2015 at 3:06 PM, Ander Juaristi <ajuari...@gmx.es> > wrote: >> > > Hi, >> > > >> > > Would you point us to some potential use cases? How would a Wget user >> > > benefit from such a feature? One of the best regarded feature of > download >> > > managers is the ability to resume paused downloads, and that's already >> > > supported by Wget. Apart from that, I can't come across any other use >> > > case. But that's me, maybe you have a broader overview. >> > One possible feature, described in flowery language from a product >> > description: "... splits files into several sections and downloads >> > them simultaneously, allowing you to use any type of connection at the >> > maximum available speed. With FDM download speed increases, or even >> > more!" >> > >> > And, just show this can help, at least in some situations, here's an >> > example using curl (sorry, I don't know how to do a similar request in >> > wget). First a normal download of the file: >> > >> > curl -o all http://mirror.internode.on.net/pub/test/100meg.test >> > >> > This command takes an average of 48.9 seconds to run on my current >> > network connection. Now, if I split up the download as the download >> > manager will, and run these four commands at the same instant: >> > >> > curl -o part1 -r0-25000000 >> > http://mirror.internode.on.net/pub/test/100meg.test curl -o part2 >> > -r25000001-50000000 >> > http://mirror.internode.on.net/pub/test/100meg.test >> > curl -o part3 -r50000001-75000000 >> > http://mirror.internode.on.net/pub/test/100meg.test >> > curl -o part4 -r75000001- >> > http://mirror.internode.on.net/pub/test/100meg.test >> > >> > The union of time it takes all four commands to run ends up being an >> > average of 19.9 seconds over a few test runs on the same connection. >> > There's some penalty here because I need to spend time combining the >> > files afterwards, but if the command supported this logic internally, >> > no doubt much of that work could be done up front as the file is >> > downloaded.