RE: Long Time Script Suddenly Failing
Peter,
I'm really pulling my hair out now. I had tried different timeouts (even the default
of 3 minutes) and still it's not working. Even using your simplified version of the
script as a test I'm still getting that error so there must be something going on on
the Linux box I'm using.
I had thought that for some reason our box was no longer recognizing the
http://inventory page and used the ip address instead. I'm no perl or unix expert and
am really frustrated but at least it would appear that the problem is definitely on my
end.
Guy
-Original Message-
From: Peter Scott [mailto:peter@;psdt.com]
Sent: Wednesday, October 23, 2002 11:42 AM
To: [EMAIL PROTECTED]
Subject: Re: Long Time Script Suddenly Failing
In article <[EMAIL PROTECTED]>,
I wrote:
>I predict that if you insert
>
> BEGIN { $|=1; print "Content-type: text/plain\n\n" }
>
>after the #! line of that script, you will see something illuminating when
>you next visit it through a browser.
Boy, do I have egg on my face. I didn't pay attention to the question
and just saw the 500 server error message - thought it was a CGI program,
didn't see that it was browser masquerading. My apologies to Guy.
I took the script, ripped out the database part, and ended up with what's
below. It doesn't have the problem described; it fetches a page just fine.
All I can think is that since you set a 15-second timeout that you were
having network problems that day that made it a bit slow to connect to that
site. Try this again, and if you get the same problem, try accessing the
URL that your program wants to fetch from a browser or with the GET program
that comes with LWP.
The only problem I found was that the script makes a new URI::URL object
and yet there is no "use URI::URL". So I don't understand how it compiled;
it should have said
Can't locate object method "new" via package "URI::URL"
(perhaps you forgot to load "URI::URL"?)
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
use URI::URL;
my $VERBOSE=1;
@MyAgent::ISA = qw(LWP::UserAgent);
sub MyAgent::redirect_ok {
my ($self,$request)=@_;
if ($request->method eq "POST") {
$request->method("GET");
}
return 1;
}
gettraffic("foo");
sub gettraffic {
my ($keyword) = @_;
my $traffic=0;
my $url =
"http://inventory.overture.com/d/searchinventory/suggestion/?term="; . $keyword;
my $ua = new MyAgent;
my $agent = 'Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)';
## Set some User Agent properties.
$ua->agent($agent);
$ua->timeout(15);
## Convert the URL into a url object (with a couple methods we can use).
my $u = new URI::URL( $url );
## Fetch the web page.
my $req = HTTP::Request->new(GET=>$u->as_string);
my $res = $ua->request($req);
my $resstring = $res->as_string;
print "URL: $url\n" if ($VERBOSE);
print "$resstring\n" if ($VERBOSE);
# Check the outcome of the response
if ($res->is_success) {
my $content = $res->content;
$content =~ m/.* ([0-9]+)<\/b>.*/s;
my $traffic = $1;
if (!$traffic) {
$traffic = 0;
}
}
}
--
Peter Scott
http://www.perldebugged.com
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Long Time Script Suddenly Failing
In article <[EMAIL PROTECTED]>,
I wrote:
>I predict that if you insert
>
> BEGIN { $|=1; print "Content-type: text/plain\n\n" }
>
>after the #! line of that script, you will see something illuminating when
>you next visit it through a browser.
Boy, do I have egg on my face. I didn't pay attention to the question
and just saw the 500 server error message - thought it was a CGI program,
didn't see that it was browser masquerading. My apologies to Guy.
I took the script, ripped out the database part, and ended up with what's
below. It doesn't have the problem described; it fetches a page just fine.
All I can think is that since you set a 15-second timeout that you were
having network problems that day that made it a bit slow to connect to that
site. Try this again, and if you get the same problem, try accessing the
URL that your program wants to fetch from a browser or with the GET program
that comes with LWP.
The only problem I found was that the script makes a new URI::URL object
and yet there is no "use URI::URL". So I don't understand how it compiled;
it should have said
Can't locate object method "new" via package "URI::URL"
(perhaps you forgot to load "URI::URL"?)
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
use URI::URL;
my $VERBOSE=1;
@MyAgent::ISA = qw(LWP::UserAgent);
sub MyAgent::redirect_ok {
my ($self,$request)=@_;
if ($request->method eq "POST") {
$request->method("GET");
}
return 1;
}
gettraffic("foo");
sub gettraffic {
my ($keyword) = @_;
my $traffic=0;
my $url =
"http://inventory.overture.com/d/searchinventory/suggestion/?term="; . $keyword;
my $ua = new MyAgent;
my $agent = 'Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)';
## Set some User Agent properties.
$ua->agent($agent);
$ua->timeout(15);
## Convert the URL into a url object (with a couple methods we can use).
my $u = new URI::URL( $url );
## Fetch the web page.
my $req = HTTP::Request->new(GET=>$u->as_string);
my $res = $ua->request($req);
my $resstring = $res->as_string;
print "URL: $url\n" if ($VERBOSE);
print "$resstring\n" if ($VERBOSE);
# Check the outcome of the response
if ($res->is_success) {
my $content = $res->content;
$content =~ m/.* ([0-9]+)<\/b>.*/s;
my $traffic = $1;
if (!$traffic) {
$traffic = 0;
}
}
}
--
Peter Scott
http://www.perldebugged.com
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Long Time Script Suddenly Failing
Hi all,
I have been requested to repost this message to the begginers group to be worked on
there.
Hopefully someone can give me a clue about a problem I'm having. I'm using the script
below to try to retrieve the web page
"http://inventory.overture.com/d/searchinventory/suggestion/?term="; with a term
attached at the end of it. This script has been working for months and is all of a
sudden returning: "500 (Internal Server Error) Can't connect to
inventory.overture.com:80 (Timeout)"
NOTE: *** This script is not run from a browser but is called from a PHP page and run
on a Linux box in the background. I've received the above 500 error while trying to
run it directly from the command line. ***
I can connect to other sites without a problem such as yahoo.com.
Does anyone know why something like this might suddenly be taking place after such a
long time of working flawlessly? Could it be something has been changed at the web
site I'm trying to get?
Thanks so much in advance.
Guy Davis
#!/usr/bin/perl
use strict;
use DBI;
use LWP::UserAgent;
# test to make sure they entered one or two arguments
if($#ARGV != 0)
{
print "\nUsage: getraffic.pl research_run_id\n\n";
exit();
}
# We need to subclass LWP::UserAgent in order to allow redirects on POSTS
# and fix some other problems. In other words, to make it behave like a
# 'real' web browser
@MyAgent::ISA = qw(LWP::UserAgent);
sub MyAgent::redirect_ok {
my ($self,$request)=@_;
# *
# IMPORTANT:
#
# POSTs that get redirected __MUST__ change the request
# method to GET
# *
if ($request->method eq "POST") {
$request->method("GET");
}
return 1;
}
# what client do you want to get? first variable from the command line
my $test_phrase = "";
my $research_id = $ARGV[0];
my $VERBOSE = 1;
my $db = DB_connect();
my ($rp_id, $rp_phrase_name);
my $sql = "";
my $sth;
# Getting started - actively checking for traffic
$sql = "UPDATE research SET research_traffic_updated=1
WHERE research_id=$research_id";
$sth = $db->prepare( $sql );
$sth->execute() || die ( $DBI::errstr );
$sql = "SELECT rp_id, rp_phrase_name FROM researchphrase
WHERE rp_research_id=$research_id AND rp_status<>2 AND rp_clicks IS NULL";
$sth = $db->prepare( $sql );
$sth->execute() || die ( $DBI::errstr );
$sth->bind_columns(undef, \($rp_id, $rp_phrase_name));
while ($sth->fetch()) { # update each phrase
my $kw = $rp_phrase_name;
$kw =~ s/ /+/g;
gettraffic($rp_id, $rp_phrase_name, $kw);
}
# All done - not actively checking for traffic
$sql = "UPDATE research SET research_traffic_updated=0
WHERE research_id=$research_id";
$sth = $db->prepare( $sql );
$sth->execute() || die ( $DBI::errstr );
$sth->finish();
# now disconnect from the database
$db->disconnect();
sub gettraffic {
my ($rp_id, $rp_phrase_name, $keyword) = @_;
my $traffic=0;
my $url =
"http://inventory.overture.com/d/searchinventory/suggestion/?term="; . $keyword;
my $ua = new MyAgent;
my $agent = 'Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)';
## Set some User Agent properties.
$ua->agent($agent);
$ua->timeout(15);
## Convert the URL into a url object (with a couple methods we can use).
my $u = new URI::URL( $url );
## Fetch the web page.
my $req = HTTP::Request->new(GET=>$u->as_string);
my $res = $ua->request($req);
my $resstring = $res->as_string;
print "URL: $url\n" if ($VERBOSE);
print "$resstring\n" if ($VERBOSE);
# Check the outcome of the response
if ($res->is_success) {
my $content = $res->content;
$content =~ m/.* ([0-9]+)<\/b>.*/s;
my $traffic = $1;
if (!$traffic) {
$traffic = 0;
}
update_db($rp_id, $traffic);
}
}
sub update_db {
my ($rphraseid, $traffic) = @_;
my $sql = "UPDATE researchphrase SET rp_clicks=$traffic WHERE
rp_id=$rphraseid";
my $sth = $db->prepare( $sql );
$sth->execute() || die( $DBI::errstr );
$sth->finish();
}
sub DB_connect {
my $table = shift;
my $db = DBI->connect("DBI:mysql:database", "username", "password",
{PrintError=>0, RaiseError=>0});
return $db || "DBERROR: Could not connect to database.";
}
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Long Time Script Suddenly Failing
I predict that if you insert
BEGIN { $|=1; print "Content-type: text/plain\n\n" }
after the #! line of that script, you will see something illuminating when
you next visit it through a browser.
If you still get a 500, then most likely the file permissions are wrong.
--
Peter Scott
http://www.perldebugged.com
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Long Time Script Suddenly Failing
Hi all,
Hopefully someone can give me a clue about a problem I'm having. I'm using the script
below to try to retrieve the web page
"http://inventory.overture.com/d/searchinventory/suggestion/?term="; with a term
attached at the end of it. This script has been working for months and is all of a
sudden returning: "500 (Internal Server Error) Can't connect to
inventory.overture.com:80 (Timeout)"
I can connect to other sites without a problem such as yahoo.com.
Does anyone know why something like this might suddenly be taking place after such a
long time of working flawlessly? Could it be something has been changed at the web
site I'm trying to get?
Thanks so much in advance.
Guy Davis
#!/usr/bin/perl
use strict;
use DBI;
use LWP::UserAgent;
# test to make sure they entered one or two arguments
if($#ARGV != 0)
{
print "\nUsage: getraffic.pl research_run_id\n\n";
exit();
}
# We need to subclass LWP::UserAgent in order to allow redirects on POSTS
# and fix some other problems. In other words, to make it behave like a
# 'real' web browser
@MyAgent::ISA = qw(LWP::UserAgent);
sub MyAgent::redirect_ok {
my ($self,$request)=@_;
# *
# IMPORTANT:
#
# POSTs that get redirected __MUST__ change the request
# method to GET
# *
if ($request->method eq "POST") {
$request->method("GET");
}
return 1;
}
# what client do you want to get? first variable from the command line
my $test_phrase = "";
my $research_id = $ARGV[0];
my $VERBOSE = 1;
my $db = DB_connect();
my ($rp_id, $rp_phrase_name);
my $sql = "";
my $sth;
# Getting started - actively checking for traffic
$sql = "UPDATE research SET research_traffic_updated=1
WHERE research_id=$research_id";
$sth = $db->prepare( $sql );
$sth->execute() || die ( $DBI::errstr );
$sql = "SELECT rp_id, rp_phrase_name FROM researchphrase
WHERE rp_research_id=$research_id AND rp_status<>2 AND rp_clicks IS NULL";
$sth = $db->prepare( $sql );
$sth->execute() || die ( $DBI::errstr );
$sth->bind_columns(undef, \($rp_id, $rp_phrase_name));
while ($sth->fetch()) { # update each phrase
my $kw = $rp_phrase_name;
$kw =~ s/ /+/g;
gettraffic($rp_id, $rp_phrase_name, $kw);
}
# All done - not actively checking for traffic
$sql = "UPDATE research SET research_traffic_updated=0
WHERE research_id=$research_id";
$sth = $db->prepare( $sql );
$sth->execute() || die ( $DBI::errstr );
$sth->finish();
# now disconnect from the database
$db->disconnect();
sub gettraffic {
my ($rp_id, $rp_phrase_name, $keyword) = @_;
my $traffic=0;
my $url =
"http://inventory.overture.com/d/searchinventory/suggestion/?term="; . $keyword;
my $ua = new MyAgent;
my $agent = 'Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)';
## Set some User Agent properties.
$ua->agent($agent);
$ua->timeout(15);
## Convert the URL into a url object (with a couple methods we can use).
my $u = new URI::URL( $url );
## Fetch the web page.
my $req = HTTP::Request->new(GET=>$u->as_string);
my $res = $ua->request($req);
my $resstring = $res->as_string;
print "URL: $url\n" if ($VERBOSE);
print "$resstring\n" if ($VERBOSE);
# Check the outcome of the response
if ($res->is_success) {
my $content = $res->content;
$content =~ m/.* ([0-9]+)<\/b>.*/s;
my $traffic = $1;
if (!$traffic) {
$traffic = 0;
}
update_db($rp_id, $traffic);
}
}
sub update_db {
my ($rphraseid, $traffic) = @_;
my $sql = "UPDATE researchphrase SET rp_clicks=$traffic WHERE
rp_id=$rphraseid";
my $sth = $db->prepare( $sql );
$sth->execute() || die( $DBI::errstr );
$sth->finish();
}
sub DB_connect {
my $table = shift;
my $db = DBI->connect("DBI:mysql:database", "username", "password",
{PrintError=>0, RaiseError=>0});
return $db || "DBERROR: Could not connect to database.";
}
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
