Andrew M. Bishop <[EMAIL PROTECTED]> wrote:
AMB> Morten Bo Johansen <[EMAIL PROTECTED]> writes:
>> I can't seem to fetch pages in batch mode from The Internet
>> Movie Database. In the page that is returned to me, following a
>> try it seems that the user-agent header has not been passed on
>> which is a requirement from IMDb. I have indeed set my
>> user-agent header in wwwoffle.config and to be sure I can see
AMB> You can ask WWWOFFLE to give you a detailed log file that shows all of
AMB> the headers sent in each direction.
AMB> If you use sh/bash as your shell then run
AMB> wwwoffled -d 6 -c wwwoffle.conf > wwwoffled.log 2>&1
Thanks for the advice.
AMB> When you examine the wwwoffled.log file you will see all of the
AMB> headers that were sent in each direction.
AMB> Have you asked WWWOFFLE to give a different header for IMDb.com in the
AMB> configuration file? It is possible that you have made a mistake with
AMB> the URL-SPECIFICATION syntax.
No, I have not referenced IMDb.com anywhere in my main
configuration file. I have blocked their advertisements,
though.
If I place an outgoing offline request with IMDb.com and then
go online to -fetch it, then I get the error page (403) which
contains among other things these lines:
Server: imdb-online-1107.vdc.amazon.com
(us.imdb.com)(us.imdb.com)
Date: Sun Oct 6 11:47:00 2002
IP: 212.54.69.73
Browser:
Cookie:
Url: /Name?wieth,+mogens
Method: GET
Referrer:
As you can see the information about my browser has not been
recorded by their server. The wwwoffled.log file obtained by
the command given by you doesn't contain any information about
the User-Agent either.
Now, if I go online again and refresh the 403-page that I just
got from the previous -fetch then the user-agent header from my
wwwoffle.conf is being passed on by wwwoffled and I get the
information I want.
If you place an outgoing request using the command
$ wwwoffle http://us.imdb.com/Name?finney,+albert
and then -fetch it, does it work for you?
I am using version 2.7e by the way.
Thanks,
Morten
--
"To create man was a quaint and original idea, but to add the sheep
was tautology." (Mark Twain)
