You should consult O'Reilly's Perl and LWP for a good explanation of how to
use the toke parser. Here's some code that I wrote.
The important part is that token->[0] refers to the token type.
Token->[1] often holds the text of the token.
Token->[4] has the source code in case of a start tag.
Andrew Johnson
Marketing Writer
Elias/Savion Advertising
Phone: 412.642.7700 Fax 412.642.2277
www.elias-savion.com
[EMAIL PROTECTED]
sub Report
{ open (ARTICLES, "$_[0]");
open (DATA, ">data.csv");
while (<ARTICLES>)
{
my $count=0;
my $numtokens=0;
my $response=$browser->get("$_");
die "Error getting: ", $response->status_line,
$response->headers_as_string
if $response->is_error;
my $content = $response->content;
my $stream = HTML::TokeParser->new(\$content)
|| die "Coulnd't read HTML $content BLAH BLAH LAH";
my $header =$response->header('X-META-PUB-DATE');
$header=~ s/,/;/g;
if (!$header)
{
$header='N/A';
}
print DATA "$header,";
print DATA "BusinessWeek,";
$header=$response->header('X-META-AUTHOR');
$header=~ s/,/;/g;
$header=~s/\n/ /;
if (!$header)
{
$header='N/A';
}
print DATA "$header,";
$header =$response->header('X-META-HEADLINE');
$header=~ s/,/;/g;
if (!$header)
{
$header='N/A';
}
print DATA "$header,";
my %keyfinds;
while(my $token=$stream->get_token)
{
if ($token->[0] eq 'T')
{
if ($token->[1] =~ /\w/)
{
if ($token->[1] =~
/(BUSH|CLINTON)/)
{
$keyfinds{$1}+=1;
$numtokens++;
my @rawdata=$token->[1];
chomp @rawdata;
foreach my $line (@rawdata)
{
$line =~ s/\t/ /g;
my
@array=split(/\s/,$line);
foreach my $word
(@array)
{
unless($word
eq '')
{
$count++;
}
}
}
}
}
}
}
my $value = $count*3885/20;
print DATA
"$count,$numtokens,977128,$value";$count=0;$numtokens=0;
my $highest=0;
my $highstring;
foreach my $key (%keyfinds)
{
if ($keyfinds{$key} > $highest)
{
$highest=$keyfinds{$key};
$highstring=$key;
}
}
if ($highstring)
{
print DATA ",$highstring,$highest,$_"; $highest=0;
}
else
{
print DATA ",,,$_";
}
}
close DATA;
close ARTICLES;
}
-----Original Message-----
From: Malcolm Mill [mailto:[EMAIL PROTECTED]
Sent: Monday, May 02, 2005 11:30 AM
To: [email protected]
Subject: html parsing
Hi,
I'm trying to extract information from html like this...
http://www.rafb.net/paste/results/Ze4RTm27.html
I've tried modifiying examples from the man pages for HTML::TokeParser,
and HTML::TreeBuilder without much success.
I just want to identify such blocks of html by the attributes in the
child nodes; extract the text node under the first '<td>',
extract the text node under the second '<td>' as well as the href
attribute in the enclosed '<a>' node,
store the output in a hash which I can pass to other functions or
print to a csv file.
If anyone can suggest anything while I read the docs and relevant
hacks in "Spidering Hacks" more carefully it would be appreciated.
Regards,
Malcolm.