Re: LWP discrimination?
On Tue, 8 Oct 2002, Paul Flint wrote:
Magnus Hjorleifsson Jr [EMAIL PROTECTED] wrote:
Are you just going out to the web site to see if it is up and running maybe
using an http get?
No. I'm using a HEAD request. But I admit I have no knowledge of the
http protocol or the transaction beyond the fact that it is there. I've
set some other variables as suggested in the LWP documentation, but I
don't know if they are causing this behavior. Here is what I have.
if ($q-param('offsite')){
# Create a user agent object
my $ua = LWP::UserAgent-new;
$ua-agent(Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90));
$ua-timeout($timeout);
foreach my $link (keys %off_site_links){
# Create a request
my $req = HTTP::Request-new(HEAD = $link);
$req-content_type('application/x-www-form-urlencoded');
$req-content('match=wwwerrors=0');
# Pass request to the user agent and get a response back
my $res = $ua-request($req);
# Check the outcome of the response
if ($res-is_success) {
# make the href link
$live_links .= qq[
p
a href=$link target=_blank$link/a
];
# split the tab separated page paths and join with a br / tag
$live_links .= join('br /', split(/\t/, $off_site_links{$link})) . '/p';
$live_link_count++;
} else {
# make the href link
$dead_links .= qq[
p
a href=$link target=_blank$link/a
];
# split the tab separated page paths and join with a br / tag
$dead_links .= join('br /', split(/\t/, $off_site_links{$link})) . '/p';
$dead_link_count++;
} # if success
} # end foreach
} # offsite
Exactly how much _content_ do you expect to get back with a HEAD
request? Try a GET instead.
[EMAIL PROTECTED] Carl Jolley
All opinions are my own and not necessarily those of my employer
___
Perl-Win32-Users mailing list
[EMAIL PROTECTED]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: LWP discrimination?
Paul Flint wrote:
I use LWP to check for dead external links, but some sites don't seem to
respond. I thought it might be a timeout problem, but a 10 second
timeout does not help, and the sites in question do not take near that
long when accessed manually. I'm wondering, do some servers block access
by LWP somehow?
You could sniff the transaction from browser and LWP and check for differences.
You could try specifying the same browser/agent in your LWP script, eg:
$ua-agent('Mozilla/8.0');
Dump your response object and see what's there, eg:
use Data::Dumper; $Data::Dumper::Indent=1;
...
my $res = $ua-request($req) or die request: $!;
print Data::Dumper-Dump([$res], [qw($res)]);
--
,-/- __ _ _ $Bill Luebkert ICQ=162126130
(_/ / )// // DBE Collectibles Mailto:[EMAIL PROTECTED]
/ ) /-- o // // http://dbecoll.tripod.com/ (Free site for Perl)
-/-' /___/__/_/_ Castle of Medieval Myth Magic http://www.todbe.com/
___
Perl-Win32-Users mailing list
[EMAIL PROTECTED]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
