Re: [Bug-wget] progressive download feature request
28.12.2011 0:34, Keisial wrote / napísal(a): Michal Tausk wrote: If you can put wireshark on it, check to see which "FIN" comes over first. I bet Keisial is right. I bet the server is telling wget "I'm done" by sending the FIN. Hope this helps pedz -- I can try that, but you are both probably right. Even though, it should be doable from the client's side(?). Server can be Apache, but might be some other that does not support this progressive feature (I'm not sure which one does). So it can still be pretty nice feature. What do you think? Michal I think you're confusing something. How do you expect the client to do that? (other than looping checking if the file is bigger and hoping to not query the server faster than the file grow rate) May be easier to trick the server not to add the Content-Length and send everything readable at that point (eg. with a cgi). I was just wondering if this can be done as work-around-stuff. There are few other options in wget that were created similiarly, in order to bypass something that's either missing on the server and can be influenced from client's side (eg.--ignore-length, --no-http-keep-alive, --no-cache). I was asking for something like that. Yes, it can be some clever loop (loop until the filesize matches, or with some threshold), but what would you say if it was something like a buffered reader, ie download the file in optional chunks, eg. 4,8,16...MB/kB... In case the file grows, each chunk would provide new content-length. When it stops growing, content-length matches downloaded size. Of course, "--limit-rate" might help if the file grows slower. I haven't seen such feature in curl, too. I can look around for the cgi trick you are talking about. Though, still, this above can be something -- unless you prove otherwise, since you are in charge, not me. Thanks.
Re: [Bug-wget] progressive download feature request
Michal Tausk wrote: > If you can put wireshark on it, check to see which "FIN" comes over first. I > bet Keisial is right. I bet the server is telling wget "I'm done" by sending > the FIN. > > Hope this helps > pedz > > -- > > I can try that, but you are both probably right. Even though, it should be > doable from the client's side(?). Server can be Apache, but might be some > other that does not support this progressive feature (I'm not sure which one > does). So it can still be pretty nice feature. What do you think? > > Michal I think you're confusing something. How do you expect the client to do that? (other than looping checking if the file is bigger and hoping to not query the server faster than the file grow rate) May be easier to trick the server not to add the Content-Length and send everything readable at that point (eg. with a cgi).
Re: [Bug-wget] progressive download feature request
On Dec 27, 2011, at 4:36 PM, Keisial wrote: > On 27/12/11 17:05, Michal Tausk wrote: >> The "--ignore-length" is not taken into consideration, (logically) >> when using "--continue" as it needs to count the difference in size >> between the downloaded file and the remote file. However, the file is >> downloaded only up to the content-length it retrieved on the last >> invocation of "wget -c". > It doesn't need to use Content-Length with -c, and in fact -c doesn't > seem to inhibit ignore-length, so it should work. > Are you sure it's not the server sending you only up to the length that > was available when you requested the file? If you can put wireshark on it, check to see which "FIN" comes over first. I bet Keisial is right. I bet the server is telling wget "I'm done" by sending the FIN. Hope this helps pedz -- I can try that, but you are both probably right. Even though, it should be doable from the client's side(?). Server can be Apache, but might be some other that does not support this progressive feature (I'm not sure which one does). So it can still be pretty nice feature. What do you think? Michal
Re: [Bug-wget] progressive download feature request
On Dec 27, 2011, at 4:36 PM, Keisial wrote: > On 27/12/11 17:05, Michal Tausk wrote: >> The "--ignore-length" is not taken into consideration, (logically) >> when using "--continue" as it needs to count the difference in size >> between the downloaded file and the remote file. However, the file is >> downloaded only up to the content-length it retrieved on the last >> invocation of "wget -c". > It doesn't need to use Content-Length with -c, and in fact -c doesn't > seem to inhibit ignore-length, so it should work. > Are you sure it's not the server sending you only up to the length that > was available when you requested the file? If you can put wireshark on it, check to see which "FIN" comes over first. I bet Keisial is right. I bet the server is telling wget "I'm done" by sending the FIN. Hope this helps pedz
Re: [Bug-wget] progressive download feature request
On 27/12/11 17:05, Michal Tausk wrote: > The "--ignore-length" is not taken into consideration, (logically) > when using "--continue" as it needs to count the difference in size > between the downloaded file and the remote file. However, the file is > downloaded only up to the content-length it retrieved on the last > invocation of "wget -c". It doesn't need to use Content-Length with -c, and in fact -c doesn't seem to inhibit ignore-length, so it should work. Are you sure it's not the server sending you only up to the length that was available when you requested the file?
[Bug-wget] progressive download feature request
Dear developers of wget, If you find some free time, would you, please, implement a feature that would progressively download a file that is growing on the remote site? Though it might be obvious, let me explain what I mean. In real world, some files might grow on the remote filesystem, eg. files that are being dumped, streams, ongoing download of larger file, etc. If we want to get these, currently, we need to run something like this several times consecutively: wget -c http://www.mydomain.com/file.asf ...until the file has stopped growing (eg. certian tv stream was dumped). The "--ignore-length" is not taken into consideration, (logically) when using "--continue" as it needs to count the difference in size between the downloaded file and the remote file. However, the file is downloaded only up to the content-length it retrieved on the last invocation of "wget -c". Thanks and all the best in 2012! Michal
