Restarting wget after crash ??

[Bazuka]

If I am running Wget overnight to crawl some sites (say about 50,000 URLs)
and it crashes/hangs up for some reason after retrieving half of them, is it
possible to restart it from the point where it crashed (instead of
downloading everything again ) ?

Re: Restarting wget after crash ??

[Hrvoje Niksic]

"Bazuka" <[EMAIL PROTECTED]> writes:

> If I am running Wget overnight to crawl some sites (say about 50,000
> URLs) and it crashes/hangs up for some reason after retrieving half
> of them, is it possible to restart it from the point where it
> crashed (instead of downloading everything again ) ?

The `-nc' option should do what you want.

Re: Restarting wget after crash ??

[Bazuka]

> The `-nc' option should do what you want.

My understanding of the -nc option is that it doesn't overwrite the existing
files. What does it do when downloading "index.html" files ? There might be
many copies of that file...so does it not overwrite the previous copy ?

I have modified Wget slightly so it writes the URL and file info to a
database and then deletes the actual file (with --delete-after). Can I
use -nc option in that case ?


"Hrvoje Niksic" <[EMAIL PROTECTED]> wrote in message
[EMAIL PROTECTED]">news:[EMAIL PROTECTED]...
> "Bazuka" <[EMAIL PROTECTED]> writes:
>
> > If I am running Wget overnight to crawl some sites (say about 50,000
> > URLs) and it crashes/hangs up for some reason after retrieving half
> > of them, is it possible to restart it from the point where it
> > crashed (instead of downloading everything again ) ?
>
> The `-nc' option should do what you want.
>

Re: Restarting wget after crash ??

[Ian Abbott]

On 27 Jun 2001, at 9:43, "Bazuka" <[EMAIL PROTECTED]> wrote:

> I have modified Wget slightly so it writes the URL and file info to a
> database and then deletes the actual file (with --delete-after). Can I
> use -nc option in that case ?

Yes, but it download the deleted files again.