This may be getting off-topic, but FWIW I have found a website (washingtonpost.com IIRC) that returns a completely valid redirect response but with code 404. Browsers happily follow the redirect, but my LWP-based spider saw the 404 and stopped!
-- - - Kingpin [interesting witty quote unavailable, thanks to lack of features in MS Outlook]
