Hi Andy,
Here are some of the problems that I have had which were real brain-teasers along with some thoughts on how to solve them, including the LWP::Debug trick. Feel free to republish, but please cite the source.
Cheers,
Peter
Q: How do I figure out why $mech->get($url) doesn't work, times out, or has other strange problems ?
A: There are many reasons why a get can fail. The server can take you to someplace you didn't expect. It can generate redirects which are not properly handled. You can get time outs. Servers are down more often than you think! Etc. A couple of places to start:
1. Check $mech->status() after each call
2. Check the URL with $mech->uri() to see where you ended up
3. If things are really strange, turn on debugging with
use LWP::Debug qw(+);
Just put this in the main program. This causes LWP to print out a
trace of the HTTP traffic between client and server and can be
used to figure out what is happening at the protocol level.It is also useful to set many traps to verify that processing is proceeding as expected. A $mech program should always have an "I didn't expect to get here" or "I don't recognize the page that I am processing" case and bail out.
Since errors can be transient, by the time you notice that the error has occurred, it might not be possible to reproduce it manually. So for automated processing it is useful to email yourself the following information:
* $CLASS ( or other indication of where processing is taking place ) * An Error Message * $mech->uri * $mech->content
Q: I want to post a form. I filled it out and submitted it with $mech->submit(). But the server ignored everything and resent an empty form. What happened?
A: The post is handled by application software. It is common for PHP programmers to use the same file both to display a form and to process the arguments returned. So the first task of the application programmer is to decide whether there are arguments to processes. The program can check whether a particular parameter has been set, whether a hidden parameter has been set, or whether the submit button has been clicked. (There are probably other ways that I haven't thought of).
In any case, if your form is not setting the parameter (e.g. the submit button) which the web application is keying on (and as an outsider there is no way to know what it is keying on), it will not notice that the form has been submitted. So - try using $mech->click() instead of submit() or vice-versa.
Q: I seemed to have logged in successfully to the server, but when I try to access protected content I get HTTP 500 (Internal Server) or other strange errors. Why?
A: Some web sites use distributed databases for their processing. It can take a few seconds for the login/session information to percolate through to all the servers. For human users with their slow reaction times, this is not a problem, but a Perl script can outrun the server. So try adding a sleep(5) between logging in and actually doing anything (the optimal delay must be determined experimentally).
Q: I am trying to scrape a page that uses Javascript. The links are Javascript functions instead of URLs. What can I do?
A: Since Javascript is completely visible to the client, it cannot be used to prevent a scraper from following links. But it can make life difficult, and until someone writes a Javascript interpreter for Perl or a mechanize clone to control Firefox, there will be no general solution. But if you want to scrape specific pages, then a solution is always possible.
One typical use of Javascript is to perform argument checking before posting to the server. The URL you want is probably just buried in the Javascript function. Do a regular expression match on $mech->content() to find the link that you want and $mech->get it directly (this assumes that you know what your are looking for in advance).
In more difficult cases, the Javascript is used for URL mangling to satisfy the needs of some middleware. In this case you need to figure out what the Javascript is doing (why are these URLs always really long?). There is probably some function with one or more arguments which calculates the new URL. Step one: using your favorite browser, get the before and after URLs and save them to files. Edit each file, converting the the argument separators ('?', '&' or ';') into newlines. Now it is easy to use diff or comm to find out what Javascript did to the URL. Step 2 - find the function call which created the URL - you will need to parse and interpret its argument list. Using the Javascript Debugger Extension for Firefox may help with the analysis. At this point, it is fairly trivial to write your own function which emulates the Javascript for the pages you want to process.
Andy Lester wrote:
On Wed, Feb 23, 2005 at 06:51:09PM +0100, Martin Kos ([EMAIL PROTECTED]) wrote:
One of the best tips I've gotten from this list is to put "use
LWP::Debug qw(+); " in to your code. This turns on a trace so you can
Can one of you guys please write up a paragraph on that LWP::Debug trick so that I can drop it in the FAQ? I didn't even know about it.
Thanks, xoxo, Andy
-- ---------------------------------------------------------------------- Peter Stevens Phone: +41 43 535 8517 www.MinuteWatcher.com Fax: +41 44 544 8392
