Martin 'Kingpin' Thurn wrote: > This may be getting off-topic, but FWIW I have found a website (washingtonpost.com >IIRC) that returns a completely valid redirect > response but with code 404. Browsers happily follow the redirect, but my LWP-based >spider saw the 404 and stopped!
By default, LWP::UserAgent will not redirect POST requests. So it could be that. You can override that with the requests_redirectable() method. I have not had an issue with redirects except when POSTs were involved and the override was not set. Also, you have to make sure you let LWP handle cookies or they'll be lost on the redirect. -- Steve Sapovits Global Sports Interactive http://www.globalsports.com Work Email: [EMAIL PROTECTED] Home Email: [EMAIL PROTECTED] Work Phone: 610-491-7087 Cell Phone: 610-574-7706
