Morten Bo Johansen <[EMAIL PROTECTED]> writes:
> I can't seem to fetch pages in batch mode from The Internet
> Movie Database. In the page that is returned to me, following a
> try it seems that the user-agent header has not been passed on
> which is a requirement from IMDb. I have indeed set my
> user-agent header in wwwoffle.config and to be sure I can see
> that it is being passed on when going to
> http://privacy.net/analyze/ both when being online from within
> a browser and when fetching the page in batch mode, so I wonder
> if the problem is with IMDb.com or something in my own setup?
>
> How do I debug if the user-agent header is being passed on to
> IMDb.com? I have tried tcpdump -i lo port 8080 and a lot of
> information is spewed out but I suspect this is not the right
> command.
You can ask WWWOFFLE to give you a detailed log file that shows all of
the headers sent in each direction.
If you use sh/bash as your shell then run
wwwoffled -d 6 -c wwwoffle.conf > wwwoffled.log 2>&1
If you use csh/tcsh as your shell then run
wwwoffled -d 6 -c wwwoffle.conf >& wwwoffled.log
When you examine the wwwoffled.log file you will see all of the
headers that were sent in each direction.
Have you asked WWWOFFLE to give a different header for IMDb.com in the
configuration file? It is possible that you have made a mistake with
the URL-SPECIFICATION syntax.
--
Andrew.
----------------------------------------------------------------------
Andrew M. Bishop [EMAIL PROTECTED]
http://www.gedanken.demon.co.uk/
WWWOFFLE users page:
http://www.gedanken.demon.co.uk/wwwoffle/version-2.7/user.html
