Thanks! I see, but then, how to exclude from being downloaded per file-basis?
-----Original Message----- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED]] Sent: 22 November 2001 14:45 To: Wget List; Tomas Hjelmberg Subject: Re: bug? Tomas Hjelmberg <[EMAIL PROTECTED]> writes: > I want to exclude /var/www/html/tomas.html from being indexed. > It looks like: [...] > <meta name="robots" content="noindex"> > <title>Tomas</title> [...] > > I invoke wget with: > wget -r http://localhost > And tomas.html is unfourtnuately downloaded anyway... Wget doesn't reallly "index" anything, so it pretty much ignores `noindex'. You can specify "nofollow" in which case Wget will refuse to recurse into the document.