Re: proxy error and using LWP getstore
Thanks for the tip on WWW::Mechanize, Im going to play with it later,
meanwhile I grabbed this code from the lwpcook manpage (large documents section)
interesting thing is when I print out the chunk to my outfile... it prints the entire
document,
I was thinking it might not print the last line or so but it does so I guess the page
does not
send a EOF.
Not sure what pearl considers the EOF to be when getting a html file ?? do you know if
its
/html ?
Because the printed out chunk does have a /html
Code from the lwpcook manpage (large documents section):
my $expected_length;
my $bytes_received = 0;
my $res =
$ua-request(HTTP::Request-new(GET = $url),
sub {
my($chunk, $res) = @_;
$bytes_received += length($chunk);
unless (defined $expected_length) {
$expected_length = $res-content_length || 0;
}
if ($expected_length) {
printf OUT %d%% - ,
100 * $bytes_received / $expected_length;
}
print OUT $bytes_received bytes received\n;
# XXX Should really do something with the chunk itself
print OUT2 $chunk;
});
print $res-status_line, \n;
print OUT $res-status_line, \n;
***
Just in case anyone cares the whole test program:
#!/usr/bin/perl -w
use strict;
use URI::URL;
use LWP;
use LWP::Debug qw(+ -conns);
use HTTP::Cookies;
my $errors_page = proxy_errors.txt;
my $url = url ('http://earthquake.usgs.gov/recenteqsUS/Quakes/quakes_all.html');
#my $url = url
('http://www.nanpa.com/number_resource_info/co_code_assignments1.html/');
my $outdir = C:\\a_perl\\proxy_tests\\ ;
my $src = $outdir .'quakes_all.html';
open OUT, $errors_page or die Create $errors_page: $!;
open (OUT2, $src) or die Cant write on file '$src'\n;
my $PROXY_URL = 'http://proxy-web.dri.edu/'; ### Proxy URL or Address + Port
my $PROXY_FTP = 'http://proxy-ftp.dri.edu/';
my $ua = LWP::UserAgent-new(env_proxy = 1,
timeout = 120,
);
$ua-proxy(http = $PROXY_URL);
$ua-proxy(ftp = $PROXY_FTP);
$ua-cookie_jar();
$ua-cookie_jar(HTTP::Cookies-new(file = 'lwpcookies.txt',
autosave = 1));
#my $req = new HTTP::Request 'GET', $url;
my $expected_length;
my $bytes_received = 0;
my $res =
$ua-request(HTTP::Request-new(GET = $url),
sub {
my($chunk, $res) = @_;
$bytes_received += length($chunk);
unless (defined $expected_length) {
$expected_length = $res-content_length || 0;
}
if ($expected_length) {
printf OUT %d%% - ,
100 * $bytes_received / $expected_length;
}
print OUT $bytes_received bytes received\n;
# XXX Should really do something with the chunk itself
print OUT2 $chunk;
});
print $res-status_line, \n;
print OUT $res-status_line, \n;
**
$Bill Luebkert wrote:
lorid wrote:
Thanks Bill!
Your code worked great, I tested it on a different url and it worked ! yeah, but
it timed out on the page Im trying to get (Im working from home on a 56k modem)
but it
seems to get
other pages just fine. With the debug code I can see that on the ... quakes.all
page it
zips along and then
it hangs. I played with increasing the timeout and it still timesout... must be
something in the source code..
but the main thing is now I can use LWP to get files.
The purpose in reading this file was to learn how to get a file thru LWP (even
through a
proxy) to parse it
and create new files.
Ive got the rest working fine but was stuck on the proxy part... thanks again
Some sites may need to send cookies.
Try using one of these (the third one will allow you to save them
between session):
$ua-cookie_jar();
$ua-cookie_jar(HTTP::Cookies::Netscape-new);
$ua-cookie_jar(HTTP::Cookies-new(file = 'lwpcookies.txt',
autosave = 1));
You can also check out WWW::Mechanize for this sort of thing.
--
,-/- __ _ _ $Bill LuebkertMailto:[EMAIL PROTECTED]
(_/ / )// // DBE CollectiblesMailto:[EMAIL PROTECTED]
/ ) /-- o // // Castle of Medieval Myth Magic http://www.todbe.com/
-/-' /___/__/_/_http://dbecoll.tripod.com/ (My Perl/Lakers stuff)
___
Perl-Win32-Users mailing list
[EMAIL PROTECTED]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: proxy error and using LWP getstore
Thanks Bill!
Your code worked great, I tested it on a different url and it worked ! yeah, but
it timed out on the page Im trying to get (Im working from home on a 56k modem) but it
seems to get
other pages just fine. With the debug code I can see that on the ... quakes.all page
it
zips along and then
it hangs. I played with increasing the timeout and it still timesout... must be
something in the source code..
but the main thing is now I can use LWP to get files.
The purpose in reading this file was to learn how to get a file thru LWP (even through
a
proxy) to parse it
and create new files.
Ive got the rest working fine but was stuck on the proxy part... thanks again
Lori
$Bill Luebkert wrote:
lorid wrote:
I am still having trouble using the proxy, I thought I'd send my new code in which
I hopefully used your code suggestions correctly. I can ping proxy-web.dri.edu ,
and I have added this debug line (use LWP::Debug qw(+ -conns);)
so I can see that it is really proxied to proxy-web.dri.edu
any clues ??
New error:
500 (Internal Server Error) Can't connect to proxy-web.dri.edu:80 (Bad hostname
'proxy-web.dri.edu')
Client-Date: Fri, 14 Nov 2003 19:44:18 GMT
heres my new code:
This works (commented out the $ua-proxy lines to test):
use strict;
use URI::URL;
use LWP;
use LWP::Debug qw(+ -conns);
my $errors_page = proxy_errors.txt;
my $url = url ('http://earthquake.usgs.gov/recenteqsUS/Quakes/quakes_all.html');
my $PROXY_URL = 'http://proxy-web.dri.edu/'; ### Proxy URL or Address + Port
my $PROXY_FTP = 'http://proxy-ftp.dri.edu/';
my $ua = new LWP::UserAgent;
$ua-proxy(http = $PROXY_URL);
$ua-proxy(ftp = $PROXY_FTP);
my $req = new HTTP::Request 'GET', $url;
my $res = $ua-request($req);
open OUT, $errors_page or die Create $errors_page: $!;
if ($res-is_success) {
print OUT $res-as_string;
print Got URL OK\n;
} else {
print OUT Error getting $url, , $res-status_line, \n;
print STDERR Error getting $url, , $res-status_line, \n;
die;
}
close OUT;
# what's the purpose in reading this file ?
#my $outdir = 'C:/a_perl/hw10/sols' ;
#my $src = $outdir/quakes_all.html;
# open IN, $src or die Can't read file $src: $!\n;
print Hi Lori\n;
# close IN;
__END__
--
,-/- __ _ _ $Bill LuebkertMailto:[EMAIL PROTECTED]
(_/ / )// // DBE CollectiblesMailto:[EMAIL PROTECTED]
/ ) /-- o // // Castle of Medieval Myth Magic http://www.todbe.com/
-/-' /___/__/_/_http://dbecoll.tripod.com/ (My Perl/Lakers stuff)
___
Perl-Win32-Users mailing list
[EMAIL PROTECTED]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
___
Perl-Win32-Users mailing list
[EMAIL PROTECTED]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: proxy error and using LWP getstore
lorid wrote: Thanks Bill! Your code worked great, I tested it on a different url and it worked ! yeah, but it timed out on the page Im trying to get (Im working from home on a 56k modem) but it seems to get other pages just fine. With the debug code I can see that on the ... quakes.all page it zips along and then it hangs. I played with increasing the timeout and it still timesout... must be something in the source code.. but the main thing is now I can use LWP to get files. The purpose in reading this file was to learn how to get a file thru LWP (even through a proxy) to parse it and create new files. Ive got the rest working fine but was stuck on the proxy part... thanks again Some sites may need to send cookies. Try using one of these (the third one will allow you to save them between session): $ua-cookie_jar(); $ua-cookie_jar(HTTP::Cookies::Netscape-new); $ua-cookie_jar(HTTP::Cookies-new(file = 'lwpcookies.txt', autosave = 1)); You can also check out WWW::Mechanize for this sort of thing. -- ,-/- __ _ _ $Bill LuebkertMailto:[EMAIL PROTECTED] (_/ / )// // DBE CollectiblesMailto:[EMAIL PROTECTED] / ) /-- o // // Castle of Medieval Myth Magic http://www.todbe.com/ -/-' /___/__/_/_http://dbecoll.tripod.com/ (My Perl/Lakers stuff) ___ Perl-Win32-Users mailing list [EMAIL PROTECTED] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: proxy error and using LWP getstore
I am still having trouble using the proxy, I thought I'd send my new code in which
I hopefully used your code suggestions correctly. I can ping proxy-web.dri.edu ,
and I have added this debug line (use LWP::Debug qw(+ -conns);)
so I can see that it is really proxied to proxy-web.dri.edu
any clues ??
New error:
500 (Internal Server Error) Can't connect to proxy-web.dri.edu:80 (Bad hostname
'proxy-web.dri.edu')
Client-Date: Fri, 14 Nov 2003 19:44:18 GMT
heres my new code:
#!/usr/bin/perl -w
use strict;
use URI::URL;
use LWP; # LWP = Library for World Wide Web access in Perl
use LWP::Debug qw(+ -conns);
require LWP::UserAgent;
#proxy-web.dri.edu
my $PROXY_URL = 'http://proxy-web.dri.edu/'; ### Proxy URL or Address + Port
my $PROXY_FTP = 'http://proxy-ftp.dri.edu/';
my $ua = ;
$ua = new LWP::UserAgent;
$ua-proxy(['http', 'ftp'], $PROXY_URL,$PROXY_FTP);
my $n = ;
my $data = ;
my $response = ;
my $errors_page = proxy_errors.txt;
open (OUT2, $errors_page) or die Can't write to file $errors_page: $!\n;
my $outdir = 'C:a_perl\\hw10\\sols\\' ;
my $url = url('http://earthquake.usgs.gov/recenteqsUS/Quakes/quakes_all.html/');
my $src = $outdir .'quakes_all.html';
my $req = new HTTP::Request 'GET', $url;
my $res = $ua-request($req);
print OUT2 $res-as_string;
$response = $ua-get($url);
die Error while getting , $response-request-uri,
-- , $response-status_line, \nAborting
unless $response-is_success;
open (OUT, $src) or die Can't read file $src: $!\n;
print Hi Lori\n;
close OUT;
close OUT2;
$Bill Luebkert wrote:
lorid wrote:
Any docs or examples on how to use a proxy when using getstore will be
greatly appreciated.!
I am having trouble getting a html file from the web due to a proxy
issue:
I found a debugging tool that helps, and I may just need to ask my boss
for the right proxy name but I think I have it...
anyway here is my simple test case , but first the error msg's I
received:
1st error I got that let me know it really is a proxy problem :
LWP::UserAgent:: _need_proxy: Not proxid .
So I am guessing I can use the same proxy I use for FTP and tried
using this line I got from the UserAgent manpage under LWP
:$ua-proxy(['http', 'ftp'], 'http://proxy.sn.no:8001/');
replacing the line in bold with my ftp proxy from work (which is how
I'm connected to the web)
Which didnt quite work cuz the error I get now is:
Cant call methond proxy on an undefined value at line 30 (the line
entails: my$ua-proxy(['http','ftp'],'http://proxy-ftp.dri.edu/');
my test program:
#!/usr/bin/perl -w
use strict;
use LWP::Simple; # LWP = Library for World Wide Web access in Perl
#use LWP::DebugFile;
use LWP::Debug qw(+ -conns);
my $n = ;
my $data = ;
my $response = ;
print LWP::Debug::conns(read $n bytes: $data);
use URI::URL;
my $outdir = C:\\a_perl\\test\\getwebfiles\\ ;
my $url =
url('http://earthquake.usgs.gov/recenteqsUS/Quakes/quakes_all.html');
my $src = $outdir .'quakes_all.html';
open (OUT, $src) or die Can't read file $src: $!\n;
require LWP::UserAgent;
#$ua-proxy(['http', 'ftp'], 'http://proxy.sn.no:8001/');
my $ua-proxy(['http', 'ftp'],'http://proxy-ftp.dri.edu/');
Either use Simple or use an agent. If you are going to use $ua,
you need to define it:
use LWP;
my $ua = new LWP::UserAgent;
my $ua-proxy(['http', 'ftp'], 'http://proxy-ftp.dri.edu/');
my $req = new HTTP::Request 'GET', $url;
my $res = $ua-request($req);
print $res-as_string;
$response = $ua-get($url);
die Error while getting , $response-request-uri,
-- , $response-status_line, \nAborting
unless $response-is_success;
close OUT;
--
,-/- __ _ _ $Bill LuebkertMailto:[EMAIL PROTECTED]
(_/ / )// // DBE CollectiblesMailto:[EMAIL PROTECTED]
/ ) /-- o // // Castle of Medieval Myth Magic http://www.todbe.com/
-/-' /___/__/_/_http://dbecoll.tripod.com/ (My Perl/Lakers stuff)
___
Perl-Win32-Users mailing list
[EMAIL PROTECTED]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
___
Perl-Win32-Users mailing list
[EMAIL PROTECTED]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
Re: proxy error and using LWP getstore
lorid wrote:
I am still having trouble using the proxy, I thought I'd send my new code in which
I hopefully used your code suggestions correctly. I can ping proxy-web.dri.edu ,
and I have added this debug line (use LWP::Debug qw(+ -conns);)
so I can see that it is really proxied to proxy-web.dri.edu
any clues ??
New error:
500 (Internal Server Error) Can't connect to proxy-web.dri.edu:80 (Bad hostname
'proxy-web.dri.edu')
Client-Date: Fri, 14 Nov 2003 19:44:18 GMT
heres my new code:
This works (commented out the $ua-proxy lines to test):
use strict;
use URI::URL;
use LWP;
use LWP::Debug qw(+ -conns);
my $errors_page = proxy_errors.txt;
my $url = url ('http://earthquake.usgs.gov/recenteqsUS/Quakes/quakes_all.html');
my $PROXY_URL = 'http://proxy-web.dri.edu/'; ### Proxy URL or Address + Port
my $PROXY_FTP = 'http://proxy-ftp.dri.edu/';
my $ua = new LWP::UserAgent;
$ua-proxy(http = $PROXY_URL);
$ua-proxy(ftp = $PROXY_FTP);
my $req = new HTTP::Request 'GET', $url;
my $res = $ua-request($req);
open OUT, $errors_page or die Create $errors_page: $!;
if ($res-is_success) {
print OUT $res-as_string;
print Got URL OK\n;
} else {
print OUT Error getting $url, , $res-status_line, \n;
print STDERR Error getting $url, , $res-status_line, \n;
die;
}
close OUT;
# what's the purpose in reading this file ?
#my $outdir = 'C:/a_perl/hw10/sols' ;
#my $src = $outdir/quakes_all.html;
# open IN, $src or die Can't read file $src: $!\n;
print Hi Lori\n;
# close IN;
__END__
--
,-/- __ _ _ $Bill LuebkertMailto:[EMAIL PROTECTED]
(_/ / )// // DBE CollectiblesMailto:[EMAIL PROTECTED]
/ ) /-- o // // Castle of Medieval Myth Magic http://www.todbe.com/
-/-' /___/__/_/_http://dbecoll.tripod.com/ (My Perl/Lakers stuff)
___
Perl-Win32-Users mailing list
[EMAIL PROTECTED]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
