I've found this script on another message board that is close, but still doesn't work with my data. Any ideas on modifications? I think my biggest problem is the regex in the split function, because what this does is match ONLY against the first column in the line, when I need it to match anything in the fourth column. Thanks for your help, and I'll see what I can do about allowing playboy.com (although since I work at a public school district, it might not be a good idea!) The script follows:
#!/bin/perl -w use strict; my %domains; open FILE1, '< domains.txt or die $!; while(<FILE1>) { chomp; $domains{$_}=1; } close FILE1; open OUT, '>> access.out' or die $!; open FILE2, '< access.log' or die $!; while(my $line=<FILE2>) { my($num); ($num, undef)=split /\s+/,$line, 2; if(defined $domains{$num}) { print OUT $line; } else { print OUT "$line not found"; } } close FILE2; close OUT; "Wiggins D'Anconia" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Mike M wrote: > > Hi, > > > > I'm new to Perl and have what I hope is a simple question: I have a Perl > > script that parses a log file from our proxy server and reformats it to a > > more easily readable space-delimited text file. I also have another file > > that has a categorized list of internet domains, also space-delimited. A > > snippet of both text files is below: > > > > Proxy Log > > ----snip---- > > 10/23/2003 4:18:32 192.168.0.100 http://www.squid-cache.org OK > > 10/23/2003 4:18:33 192.168.1.150 http://msn.com OK > > 10/23/2003 4:18:33 192.168.1.150 http://www.playboy.com DENIED > > ----snip---- > > > > Categorized Domains List > > ----snip---- > > msn.com news > > playboy.com porn > > squid-cache.com software > > ----snip---- > > > > What I would like to do is write a script that compares the URL in the proxy > > log with the categorized domains list file and creates a new file that looks > > something like this: > > > > New File > > ----snip---- > > 10/23/2003 4:18:32 192.168.0.100 http://www.squid-cache.org software OK > > 10/23/2003 4:18:33 192.168.1.150 http://msn.com news OK > > 10/23/2003 4:18:33 192.168.1.150 http://www.playboy.com porn DENIED > > ----snip---- > > > > Is this possible with Perl?? I've been trying to do this by importing the > > log files into SQL and then running queries, but it's so much slower than > > Perl (the proxy logs are roughly 1 million lines). Any ideas? > > > > What have you tried, where have you failed? Just about anything is > possible with Perl, and there are hints that will make this more > bearable, but this isn't a one stop shopping place, so give it a try > yourself first.... > > You seem to have a good grasp of what is needed, break it down into > parts and see what you come up with... > > 1. We need a list of domains to match against and what category they are in, > 2. We need a line from the log to get the domain, > 3. We need to look up into the list of domains to see if the domain is > there, > 4. If it is we need to add the category to the end of the domain, > 5. We need a place to store the information back to. > > So you need at the very least: > > perldoc -f open > perldoc -f print > > And you are probably going to want, > > perldoc -f exists > perldoc -f keys > perldoc -f values > perldoc -f grep > > And then probably a while loop....So some pseudo code might look like: > > open file of categorized domains > store categorized domains into an easily accessible data structure > close file > > open file for writing to store updated log to > open file that has log lines in it > while we read the file, do some stuff, where: > if the line has a domain name > pull the domain name > compare it to the data structure > if it is in teh data structure update the line > print the line to the new location > repeat > > close the read file > close the write file > > Have a beer. By the way you should deny msn.com instead of playboy.com > ;-).... > > http://danconia.org > -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]