Perfect! Thank you very much!
Peter
Gisle Aas wrote:
Peter Stevens <[EMAIL PROTECTED]> writes:
I am using mech to scrape data from various websites. I wanted to
collect data about the bytes sent and received by my scraper (I need
this for sizing purposes). I looked though Mech and LWP, but did not
see any methods which give me that information. Is there a way to do
this?
Not directly, but you can replace the protocol handler with your own that counts bytes passed by. This is an example that will count the bytes sent over http:
#!/usr/bin/perl -w
use LWP::UserAgent; use LWP::Protocol;
LWP::Protocol::implementor('http', 'MyHTTP'); my $bytes_in = 0; my $bytes_out = 0;
my $ua = LWP::UserAgent->new(keep_alive => 1);
for (1..3) { my $res = $ua->get("http://www.example.com"); print "$_: ", $res->status_line, "\n"; }
print "received $bytes_in bytes, send $bytes_out bytes\n";
# Overridden protocol handler that counts the bytes transfered package MyHTTP; use base 'LWP::Protocol::http';
package MyHTTP::Socket; use base 'LWP::Protocol::http::Socket';
sub sysread { my $self = shift; my $n = $self->SUPER::sysread(@_); $bytes_in += $n if defined($n) && $n > 0; return $n; }
sub syswrite { my $self = shift; my $n = $self->SUPER::syswrite(@_); $bytes_out += $n if defined($n) && $n > 0; return $n; }
__END__
Regards, Gisle
