Tomas Hjelmberg <[EMAIL PROTECTED]> writes: > I want to exclude /var/www/html/tomas.html from being indexed. > It looks like: [...] > <meta name="robots" content="noindex"> > <title>Tomas</title> [...] > > I invoke wget with: > wget -r http://localhost > And tomas.html is unfourtnuately downloaded anyway...
Wget doesn't reallly "index" anything, so it pretty much ignores `noindex'. You can specify "nofollow" in which case Wget will refuse to recurse into the document.
