In response to some off-line requests, here's an
example:
sub execute_request {
my($req) = @_;
$req->header('Accept-encoding' => 'gzip, deflate');
print "\nReq
#$req_nbr.$sub_request:\n",$req->as_string;
my $res = $ua->request($req);
$last_response = $res;
if ($res->is_success) {
print "\nResp Headers #$req_nbr.$sub_request:\n";
print $res->protocol, " ", $res->status_line, "\n";
print $res->headers_as_string;
my $follow_url = undef;
if ( $res->header('Content-encoding')) {
my $encoding = $res->header('Content-encoding');
if ($encoding =~ /deflate/o ) {
# process deflate encoding
print "\ndeflate encoding found\n";
$res->content(inflateBody($res->content_ref));
} elsif ($encoding =~ /gzip/o ) {
# process gzip encoding
print "\ngzip encoding found\n";
die "gzip decompression not yet implemented\n";
} else {
die "unsuported content-encoding $encoding returned
by browser\n";
}
}
if ( $res->header('Refresh') &&
$res->header('Refresh') =~ /url=(.*)/o ) {
print "\nFollowing re-direct via Refresh found in
HTTP headers or HTML <head>\n";
$follow_url = $1;
} elsif ($res->content && $res->content =~
/document\.location\.href = \"(.+)\"/o ) {
print "\nFollowing re-direct via script
document.location.href = \n";
$follow_url = $1;
} elsif ($res->content && $res->content =~
/<meta\s+http-equiv="?Refresh"?\s+content\s*=\s*"\d*;\s*[Uu][Rr][Ll]=(.*?)"/o) {
print "\nFollowing re-direct via http-equiv=Refresh
found in HTML <body> or compressed <head>\n";
$follow_url = $1;
}
if ($follow_url) {
decode_entities($follow_url);
my $base =
$req->uri->scheme."://".$req->uri->authority;
#print "\nbase=$base\n";
$sub_request++;
get_url(URI->new_abs($follow_url,$base));
}
} else {
print "\nFailed: ", $res->status_line;
}
return $res;
}
> --- David Carter <[EMAIL PROTECTED]> wrote:
> > First, you need to determine exactly how the
> > redirect is being done. Look at
> > the response you are receiving, and determine what
> > method it is using to do
> > the redirect. Is it a 302 response? Is it a meta
> > refresh tag in the html
> > <head>? Is it javascript?
> >
> > You can't write code to handle this until you know
> > exactly what you should
> > be looking for in the response. Once you have
> > determined this, you can write
> > code to handle it.
> >
> > Are you doing a POST or a GET on the failing
> > request? LWP will not by
> > default process a redirect of a POST request. Many
> > web sites use this
> > technique, often when going from a login page to the
> > next page inside the
> > secured area of the application. This can be fixed
> > by overriding a method
> > within UserAgent. I think it is called
> > request_redirectable or something
> > close to that. Search the mailing list archives for
> > code that does this.
> >
> > If the page is using a meta refresh or some
> > javascript, you can write a
> > regular expression to search for this in the
> > response content & extract the
> > URL to be redirected to. Fairly simple for a meta
> > refresh, maybe not so
> > simple for more complex javascript.
> >
> >
> >
> > ---
> > David Carter
> > [EMAIL PROTECTED]
> >
> >
> > > >-----Original Message-----
> > > >From: Ben [mailto:[EMAIL PROTECTED]
> > > >Sent: Tuesday, November 04, 2003 7:51 PM
> > > >To: [EMAIL PROTECTED]; [EMAIL PROTECTED];
> > [EMAIL PROTECTED]
> > > >Subject: RE: Perl Redirect doesn't work
> > > >
> > > >Qing, Keary, and Friends,
> > > >
> > > >I am having the same problem you're having.
> > Please
> > > >help.
> > > >
> > > >I have a script that gets data from various
> > sites.
> > > >One of the sites suddenly uses a redirect in the
> > url.
> > > >
> > > >I used to go to www.site.com/123.html. Now
> > > >www.site.com/123.html redirects to
> > > >xyz.site.com/231.html. The data fields and
> > everything
> > > >else is the same, except I am not able to get any
> > > >information.
> > > >
> > > >Keary mentioned:
> > > >"LWP will only redirect on an HTTP redirect (e.g.
> > > >status 302). Web sites may use other methods to
> > > >simulate a redirect, such as JavaScript or META
> > > >refresh, which LWP doesn't parse and handle."
> > > >
> > > >If this is the case, how can I get around a
> > Javascript
> > > >or META refresh?
> > > >
> > > >Any suggestions or information would be
> > appreciated.
> > > >I am fairly new to Perl. Thank you.
> > > >
> > > >
> > > >Ben
> > > >
> > > >
> > > >-----
> > > >
> > > >
> > > >Hi, All
> > > > I have one question about the redirection.
> > The
> > > >following is my code:
> > > >
> > > > use LWP::UserAgent;
> > > > my $ua=LWP::UserAgent->new;
> > > > my $req=HTTP::Request->new
> > >
> >
>
>(GET=>'http://www.somedomainhere.com/redirect/mdm2.htm');
> > > > # this URL should be
> > > >directed to another one
> > > >
> > > > my $res=$ua->request($req);
> > > > my $string = $res->content;
> > > >
> > > > But when I run the program, it didn't
> > > >automatically redirect to another web
> > > >URL. It just show me "The web page has been
> > > >removed.... blah blah". According to
> > > >LWP document, the request "GET" should be in the
> > > >redirect list. Why it doesn't
> > > >work? Could someone help me for this? Thanks!
> > > >
> > > >
> > > > Qing
> > > >
> > >
> >
>
>http://archive.develooper.com/[EMAIL PROTECTED]/msg04787.html?x
> >
> >
> >