Peter Stevens <[EMAIL PROTECTED]> writes:
> I am using mech to scrape data from various websites. I wanted to
> collect data about the bytes sent and received by my scraper (I need
> this for sizing purposes). I looked though Mech and LWP, but did not
> see any methods which give me that information. Is there a way to do
> this?
Not directly, but you can replace the protocol handler with your own
that counts bytes passed by. This is an example that will count the
bytes sent over http:
#!/usr/bin/perl -w
use LWP::UserAgent;
use LWP::Protocol;
LWP::Protocol::implementor('http', 'MyHTTP');
my $bytes_in = 0;
my $bytes_out = 0;
my $ua = LWP::UserAgent->new(keep_alive => 1);
for (1..3) {
my $res = $ua->get("http://www.example.com";);
print "$_: ", $res->status_line, "\n";
}
print "received $bytes_in bytes, send $bytes_out bytes\n";
# Overridden protocol handler that counts the bytes transfered
package MyHTTP;
use base 'LWP::Protocol::http';
package MyHTTP::Socket;
use base 'LWP::Protocol::http::Socket';
sub sysread {
my $self = shift;
my $n = $self->SUPER::sysread(@_);
$bytes_in += $n if defined($n) && $n > 0;
return $n;
}
sub syswrite {
my $self = shift;
my $n = $self->SUPER::syswrite(@_);
$bytes_out += $n if defined($n) && $n > 0;
return $n;
}
__END__
Regards,
Gisle
