Hi-
I'm using wget in some qa scripts to recurse through a site I'm
developing to find 404s and 500s and bad resource references. I'm using
RHEL4's wget:
GNU Wget 1.10.2 (Red Hat modified)
I'm running it as per below:
wget \
-kpSrN \
-F -i $urls \
-B $base \
-D $domain \
-np \
-l 2 \
--user-agent="$ua" \
--header="$lm" \
$urls is an html file of all relative urls the CMS should be producing.
--user-agent is needed to bypass the paywall.
$base is the absolute base for the relative urls.
$domain is to keep from crawling off-site links.
--header sends a special old last-modified header which the web server
echoes back in cases when one is not provided by the app server.
In any event, my problem is that I'm finding that wget is sending a
Referer: along with *most* requests, but not all.
I want Referers sent for everything so I can easily find (at least one
of) the referring pages for problematic URLs.
Is this a bug?
Thanks,
Jonah
