J,L writes: > i think to fool a server to fetch page,d/l files etc, > cookie, refer,user-agent,authentication are the ones > related most. or all?
There is never a "most" in this case. Here are some problems I've run into besides the one you mentioned: 1) Mismatched SSL versions 2) Expecting Referer: to be something specific 3) Unexpected Authorization: (fixed in a patch to this list) 4) Not retrieving JavaScript or other included files (usually, to set cookies, but you never know). 5) Requests received too fast As somebody else on this list already mentioned, you have to do exactly what the browser does if you're having problems. The reasons for this are usually poorly implemented software on the server side and not a security scheme. Happy scraping, Rob
