You should consult O'Reilly's Perl and LWP for a good explanation of how to
use the toke parser. Here's some code that I wrote.

The important part is that token->[0] refers to the token type. 
Token->[1] often holds the text of the token.
Token->[4] has the source code in case of a start tag.

Andrew Johnson
Marketing Writer                 
Elias/Savion Advertising 
Phone: 412.642.7700 Fax 412.642.2277
www.elias-savion.com 
[EMAIL PROTECTED]  

sub Report
{       open (ARTICLES, "$_[0]");
        open (DATA, ">data.csv");
        while (<ARTICLES>)
        {
                my $count=0;
                my $numtokens=0;
                my $response=$browser->get("$_");
                die "Error getting: ", $response->status_line,
$response->headers_as_string
                if $response->is_error;
                my $content = $response->content;
                my $stream = HTML::TokeParser->new(\$content)
                || die "Coulnd't read HTML $content BLAH BLAH LAH";
                my $header =$response->header('X-META-PUB-DATE');
                $header=~ s/,/;/g;
                if (!$header)
                {
                        $header='N/A';
                }
                print DATA "$header,";  
                print DATA "BusinessWeek,";
                $header=$response->header('X-META-AUTHOR');
                $header=~ s/,/;/g;
                $header=~s/\n/ /;
                if (!$header)
                {
                        $header='N/A';
                }
                print DATA "$header,";
                $header =$response->header('X-META-HEADLINE');
                $header=~ s/,/;/g;
                if (!$header)
                {
                        $header='N/A';
                }
                print DATA "$header,";
                my %keyfinds;
                while(my $token=$stream->get_token)
                {
                        if ($token->[0] eq 'T')
                        {
                                if ($token->[1] =~ /\w/)
                                {
                                        if ($token->[1] =~
/(BUSH|CLINTON)/)
                                        {
                                                $keyfinds{$1}+=1;
                                                $numtokens++;
                                                my @rawdata=$token->[1];
                                                chomp @rawdata;
                                                foreach my $line (@rawdata)
                                                {
                                                        $line =~ s/\t/ /g;
                                                        my
@array=split(/\s/,$line);
                                                        foreach my $word
(@array)
                                                        {
                                                                unless($word
eq '')
                                                                {
 
$count++;
                                                                }
                                                        }
                                                }
                                        }
                                }
                        }
                }
                my $value = $count*3885/20;
                print DATA
"$count,$numtokens,977128,$value";$count=0;$numtokens=0;
                my $highest=0;
                my $highstring;
                foreach my $key (%keyfinds)
                {
                        if ($keyfinds{$key} > $highest)
                        {
                                $highest=$keyfinds{$key};
                                $highstring=$key;
                        }
                }
                if ($highstring)
                {
                        print DATA ",$highstring,$highest,$_";  $highest=0;

                }
                else 
                {
                        print DATA ",,,$_";
                } 
        }               
close DATA;
close ARTICLES;
}               

           
-----Original Message-----
From: Malcolm Mill [mailto:[EMAIL PROTECTED] 
Sent: Monday, May 02, 2005 11:30 AM
To: [email protected]
Subject: html parsing

Hi, 
I'm trying to extract information from html like this...

http://www.rafb.net/paste/results/Ze4RTm27.html

I've tried modifiying examples from the man pages for HTML::TokeParser, 
and HTML::TreeBuilder without much success.

I just want to identify such blocks of html by the attributes in the
child nodes; extract the text node under the first '<td>',
extract the text node under the second '<td>' as well as the href
attribute in the enclosed '<a>' node,
store the output in a hash which I can pass to other functions or
print to a csv file.

If anyone can suggest anything while I read the docs and relevant
hacks in "Spidering Hacks" more carefully it would be appreciated.

Regards, 
Malcolm.

Reply via email to