Re: [Bug-wget] Multi segment download

2015-09-08 Thread Darshit Shah 
Thanking You,
Darshit Shah
Sent from mobile device. Please excuse my brevity
On 09-Sep-2015 8:50 am, "Hubert Tarasiuk"  wrote:
>
> On Sat, Aug 29, 2015 at 12:50 AM, Darshit Shah  wrote:
> > Thanking You,
> > Darshit Shah
> > Sent from mobile device. Please excuse my brevity
> > On 29-Aug-2015 1:13 pm, "Tim Rühsen"  wrote:
> >>
> >> Hi,
> >>
> >> normally it makes much more sense when having several download mirrors
and
> >> checksums for each chunk. The perfect technique for such is called
> > 'Metalink'
> >> (more on www.metalinker.org).
> >> Wget has it in branch 'master'. A GSOC project of Hubert Tarasiuk.
> >>
> > Sometimes the evil ISPs enforce a per connection bandwidth limit. In
such a
> > case, multi segment downloads from a single server do make sense.
> >
> > Since metalink already has support for downloading a file over multiple
> > connections, it should not be too difficult to reuse the code for use
> > outside of metalink.
> The current Metalink impl in Wget will not download from multiple
> mirrors simultaneously since Wget itself is single-threaded.
> Adding optional (POSIX) threads support to Wget (especially for the
> Metalinks) could be perhaps worth discussion.
> For now the solution might be to start multiple Wget instances using
> the --start-pos option and somehow limit the length of download (I am
> not sure if Wget currently has an option to do that).
>
There isn't. And a trivial implementation of the same didn't work. When I
tried it, Wget kept retrying the download on a 206 response since it
believed the server misunderstood the request. It will take a little more
effort to implement this.
> >
> > I think it would be a good idea to do so. I'm not sure if all the
possible
> > variations of the range headers are parsed by Wget.
> >> Additionally, Wget2 is under development, already having the option
> > --chunk-
> >> size (e.g. --chunk-size=1M) to start a multi-threaded download of a
file.
> >>
> >> Regards, Tim
> >>
> >>
> >> Am Freitag, 28. August 2015, 15:41:27 schrieb Random Coder:
> >> > On Fri, Aug 28, 2015 at 3:06 PM, Ander Juaristi 
> > wrote:
> >> > > Hi,
> >> > >
> >> > > Would you point us to some potential use cases? How would a Wget
user
> >> > > benefit from such a feature? One of the best regarded feature of
> > download
> >> > > managers is the ability to resume paused downloads, and that's
already
> >> > > supported by Wget. Apart from that, I can't come across any other
use
> >> > > case. But that's me, maybe you have a broader overview.
> >> > One possible feature, described in flowery language from a product
> >> > description: "... splits files into several sections and downloads
> >> > them simultaneously, allowing you to use any type of connection at
the
> >> > maximum available speed. With FDM download speed increases, or even
> >> > more!"
> >> >
> >> > And, just show this can help, at least in some situations, here's an
> >> > example using curl (sorry, I don't know how to do a similar request
in
> >> > wget).  First a normal download of the file:
> >> >
> >> > curl -o all http://mirror.internode.on.net/pub/test/100meg.test
> >> >
> >> > This command takes an average of 48.9 seconds to run on my current
> >> > network connection.  Now, if I split up the download as the download
> >> > manager will, and run these four commands at the same instant:
> >> >
> >> > curl -o part1 -r0-2500
> >> > http://mirror.internode.on.net/pub/test/100meg.test curl -o part2
> >> > -r2501-5000
> >> > http://mirror.internode.on.net/pub/test/100meg.test
> >> > curl -o part3 -r5001-7500
> >> > http://mirror.internode.on.net/pub/test/100meg.test
> >> > curl -o part4 -r7501-
> >> > http://mirror.internode.on.net/pub/test/100meg.test
> >> >
> >> > The union of time it takes all four commands to run ends up being an
> >> > average of 19.9 seconds over a few test runs on the same connection.
> >> > There's some penalty here because I need to spend time combining the
> >> > files afterwards, but if the command supported this logic internally,
> >> > no doubt much of that work could be done up front as the file is
> >> > downloaded.

Re: [Bug-wget] Multi segment download

2015-09-08 Thread Hubert Tarasiuk 
On Sat, Aug 29, 2015 at 12:50 AM, Darshit Shah  wrote:
> Thanking You,
> Darshit Shah
> Sent from mobile device. Please excuse my brevity
> On 29-Aug-2015 1:13 pm, "Tim Rühsen"  wrote:
>>
>> Hi,
>>
>> normally it makes much more sense when having several download mirrors and
>> checksums for each chunk. The perfect technique for such is called
> 'Metalink'
>> (more on www.metalinker.org).
>> Wget has it in branch 'master'. A GSOC project of Hubert Tarasiuk.
>>
> Sometimes the evil ISPs enforce a per connection bandwidth limit. In such a
> case, multi segment downloads from a single server do make sense.
>
> Since metalink already has support for downloading a file over multiple
> connections, it should not be too difficult to reuse the code for use
> outside of metalink.
The current Metalink impl in Wget will not download from multiple
mirrors simultaneously since Wget itself is single-threaded.
Adding optional (POSIX) threads support to Wget (especially for the
Metalinks) could be perhaps worth discussion.
For now the solution might be to start multiple Wget instances using
the --start-pos option and somehow limit the length of download (I am
not sure if Wget currently has an option to do that).

>
> I think it would be a good idea to do so. I'm not sure if all the possible
> variations of the range headers are parsed by Wget.
>> Additionally, Wget2 is under development, already having the option
> --chunk-
>> size (e.g. --chunk-size=1M) to start a multi-threaded download of a file.
>>
>> Regards, Tim
>>
>>
>> Am Freitag, 28. August 2015, 15:41:27 schrieb Random Coder:
>> > On Fri, Aug 28, 2015 at 3:06 PM, Ander Juaristi 
> wrote:
>> > > Hi,
>> > >
>> > > Would you point us to some potential use cases? How would a Wget user
>> > > benefit from such a feature? One of the best regarded feature of
> download
>> > > managers is the ability to resume paused downloads, and that's already
>> > > supported by Wget. Apart from that, I can't come across any other use
>> > > case. But that's me, maybe you have a broader overview.
>> > One possible feature, described in flowery language from a product
>> > description: "... splits files into several sections and downloads
>> > them simultaneously, allowing you to use any type of connection at the
>> > maximum available speed. With FDM download speed increases, or even
>> > more!"
>> >
>> > And, just show this can help, at least in some situations, here's an
>> > example using curl (sorry, I don't know how to do a similar request in
>> > wget).  First a normal download of the file:
>> >
>> > curl -o all http://mirror.internode.on.net/pub/test/100meg.test
>> >
>> > This command takes an average of 48.9 seconds to run on my current
>> > network connection.  Now, if I split up the download as the download
>> > manager will, and run these four commands at the same instant:
>> >
>> > curl -o part1 -r0-2500
>> > http://mirror.internode.on.net/pub/test/100meg.test curl -o part2
>> > -r2501-5000
>> > http://mirror.internode.on.net/pub/test/100meg.test
>> > curl -o part3 -r5001-7500
>> > http://mirror.internode.on.net/pub/test/100meg.test
>> > curl -o part4 -r7501-
>> > http://mirror.internode.on.net/pub/test/100meg.test
>> >
>> > The union of time it takes all four commands to run ends up being an
>> > average of 19.9 seconds over a few test runs on the same connection.
>> > There's some penalty here because I need to spend time combining the
>> > files afterwards, but if the command supported this logic internally,
>> > no doubt much of that work could be done up front as the file is
>> > downloaded.

Re: [Bug-wget] Multi segment download

2015-09-08 Thread Yousong Zhou 
On 29 August 2015 at 04:04, Abhilash Mhaisne  wrote:
> Hey all. I am new to this mailing list.
> As far as I've used wget, it downloads a specified file as a single segment.
> Can we modify this such that wget will download a file by dividing it into
> multiple
> segments and then combining all at reciever host? Just like some
> proprietary download
> managers do? If work on such a feature is going on, I'd like to be a part
> of it.
>

I guess this can be scripted with --start-pos option of wget?

yousong

[Bug-wget] Feature: Disabling progress bar when wget is backgrounded

2015-09-08 Thread Christian Neukirchen 
Hi,

Sometimes I start wget, but the remote site is too slow, so I rather
want to run it in background, however when I simply use job control
for that, wget will keep spewing the progress bar all over my
terminal.  I have found the SIGHUP/SIGUSR1 feature to redirect output
to a log file, but I think the following small patch is even more
useful, since the progress bar will simply resume when wget is
foregrounded again (also, the final message is still printed to the
terminal in any case):

--- src/progress.c
+++ src/progress.c
@@ -1179,10 +1179,12 @@ create_image (struct bar_progress *bp, double 
dl_total_time, bool done)
 static void
 display_image (char *buf)
 {
-  bool old = log_set_save_context (false);
-  logputs (LOG_PROGRESS, "\r");
-  logputs (LOG_PROGRESS, buf);
-  log_set_save_context (old);
+  if (tcgetpgrp (fileno (stderr)) == getpid ()) {
+bool old = log_set_save_context (false);
+logputs (LOG_PROGRESS, "\r");
+logputs (LOG_PROGRESS, buf);
+log_set_save_context (old);
+  }
 }
 
 static void

This probably needs some guards for portability to all platforms.
Only tested on Linux 4.1 so far.

Opinions?

-- 
Christian Neukirchenhttp://chneukirchen.org

Re: [Bug-wget] Feature: Disabling progress bar when wget is backgrounded

2015-09-08 Thread Ander Juaristi 
I remember this same issue being raised not so much ago, which ended up in a 
wontfix. However, now that we have something to set up and run, something clear 
to test, maybe the results are acceptable :D

Sent from my smartphone. Excuse my brevity.

 Christian Neukirchen escribió 

>Hi,
>
>Sometimes I start wget, but the remote site is too slow, so I rather
>want to run it in background, however when I simply use job control
>for that, wget will keep spewing the progress bar all over my
>terminal.  I have found the SIGHUP/SIGUSR1 feature to redirect output
>to a log file, but I think the following small patch is even more
>useful, since the progress bar will simply resume when wget is
>foregrounded again (also, the final message is still printed to the
>terminal in any case):
>
>--- src/progress.c
>+++ src/progress.c
>@@ -1179,10 +1179,12 @@ create_image (struct bar_progress *bp, double 
>dl_total_time, bool done)
> static void
> display_image (char *buf)
> {
>-  bool old = log_set_save_context (false);
>-  logputs (LOG_PROGRESS, "\r");
>-  logputs (LOG_PROGRESS, buf);
>-  log_set_save_context (old);
>+  if (tcgetpgrp (fileno (stderr)) == getpid ()) {
>+bool old = log_set_save_context (false);
>+logputs (LOG_PROGRESS, "\r");
>+logputs (LOG_PROGRESS, buf);
>+log_set_save_context (old);
>+  }
> }
> 
> static void
>
>This probably needs some guards for portability to all platforms.
>Only tested on Linux 4.1 so far.
>
>Opinions?
>
>-- 
>Christian Neukirchenhttp://chneukirchen.org
>

[Bug-wget] [bug #40426] wget hangs with -r and -O -

2015-09-08 Thread Ander Juaristi 
Follow-up Comment #2, bug #40426 (project wget):

This was eventually merged on 2015-08-15, commit
12bae50b28fa20f9ffa3d5b0e88f4ebf51fa6864.

Please, check whether the issue is effectively fixed, and close this.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/