Re: Storing filehandles(for writing) in hashes doesn't work (Re: whilereach my $variable (FILEHANDLE) )
On 01/15/2007 11:21 PM, Michael Alipio wrote:
Hi,
Ok, seems like a pet logs is not a good example.:-)
Let me revise my story:
I have a logfile which contains different clients firewall's logs.
Let's say the log file is: firewall.log
Now each line in the logfile has a $deviceid string that identifies where or which client it came from.
What I did was to list down all of these clients in a file named deviceid.conf.
Let's say it contains: ('client_name' dash 'device_id')
client1 - 293u0sdfj
client2 - 8325kjsdf
client3 - kjldas8282
.
clientn - sdkfj28350
Having said that, my goals are:
1. read the firewall.log line by line.
if it see a particular device_id in $_, and it knows that that deviceid is for this
particular client (using the information found at deviceid.conf) it will
write that line into /client1/$date.log or /client2/$date.log etc.
By the way, our logs are being rotated such that it contains logs from 6:26 yesterday, to 6:25
today, so $date on the above was obtained by let's say getting the /date=(\S+)/ on the first line
entry of the log, let's say it reads 2007-01-10, so our $date will be 2007-01-10_11, so
the logfile for a particular client will be /client1/2007-01-10_11.log
Here is an example of a line in the logfile:
Jan 10 06:26:17 210.23.194.86 date=2007-01-10
time=06:30:14,devname=sccp_firewall,device_id=FWF60A1234566,log_id=00210100
01,type=traffic,subtype=allowed,pri=notice,vd=root;SN=14435461,duration=139,user=N/A,group=N/A,policyid=11,proto=6,service=7
500/tcp,status=accept,src=192.169.1.70,srcname=192.168.1.3,dst=192.169.1.17,dstname=192.169.1.17,src_int=internal,dst_int
=wan2,sent=144,rcvd=0,sent_pkt=3,rcvd_pkt=0,src_port=2354,dst_port=7500,vpn=N/A,tran_ip=0.0.0.0,tran_port=0,dir_disp=org,tra
n_disp=noop
The device_id in this log entry does not appear in the device id file
you showed above.
So far, I've been trying to use the code that was given to me but I'm still far
from my goal:
#!/usr/bin/perl
use warnings;
use strict;
my $logfile='firewall.log';
my $devices='deviceid.conf';
our %log;
##
# 1ST PART
open DEVICES, '', $devices or die Can't open $devices $!;
while ( my $device = DEVICES){
chomp $device;
($device) = $device =~ /(\S+)$/;
open( my $fh, '', $device.log) or die Can't open $device.log: $!;
$log{$device} = $fh;
}
close DEVICES;
#
So far I can understand that in the first part, the code will read the
deviceid.conf and create a file handle for writing for each device id, and
store these filehandles inside %log. But that is what I wanted to do, I want
to, as I wanted to write my logs into /clientN/date.log instead of
client's_deviceid.log. So I'm still trying to figure out this one.
/(\S+)$/ only matches the device-id; the client-id is thrown away. Don't
throw it away. Create a regular expression that captures the client id
and use the client-id to create the log file name.
Next:
# 2nd PART
my $re = '^\S+\s+(' . join( '|', keys %log ). ')';
open( IN, '', $logfile ) or die Can't open $logfile: $!;
while( my $line = IN ){
if( $line =~ m/$re/ ){
print $log{$1} $line;
}
}
###
The second part is what confuses me, especially the line with my $re, and also the
if($line =~ m/$re/)
As far as I can understand, the $re will contain a regexp with ('device_id1 |
device_id2 | device_id3 | device_idN'), so that whenever it sees any pattern
that match either of those device_ids, it will print it to say $log{device_idN}
which points to the file handle that writes to device_idN.log.
But this is not the case.
The line print $log{$1} $line; doesn't even work as if it cannot decode the
$log{$1}
You are correct, my $re creates a regular expression with all of the
device ids, and you are also correct that print $log{$1} $line does
not work. You must use print { $log{$1} } $line.
Useless use of a constant in void context at extractdevice.pl line 30.
Scalar found where operator expected at extractdevice.pl line 35, near } $line
(Missing operator before $line?)
syntax error at extractdevice.pl line 35, near } $line
If I comment those codes inside the while in that second part, the program will
successfully create emtpy device_id1.log, device_id2.log, etc. etc.)
Any idea what's wrong with this one?
My comments are above.
I wrote the program a completely different way. I think you should read
'deviceid.conf' and place the device-ids and client-ids into a hash
($clientids). Then open and start reading the logfile. Whenever you are
able to capture a device-id that is in the %clientids hash, construct a
filename to the desired output log file and append the input logfile
line to that output log file.
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
http://learn.perl.org/
Re: Storing filehandles(for writing) in hashes doesn't work (Re: whilereach my $variable (FILEHANDLE) )
Hi,
Ok, here's my code:
my $logfile='firewalllog';
my $devicefile='deviceid';
our %log;
open DEVICES, '', $devicefile or die Can't open $devicefile $!;
while (my $device DEVICES){
($device) = $device =~ /(\S+)$/;
open (my $fh, '', $device..log) or die Can't write to $device..log: $!;
$log{device} = $fh;
}
close DEVICES;
So I should be able to print to at least {$log{any_device_id}} right??
But using this:
print {$log{FWF60A1234566}} testing;
gives me:
Use of uninitialized value in ref-to-glob cast at extractdevice.pl line 20.
Can't use string () as a symbol ref while strict refs in use at
extractdevice.pl line 20.
Any idea what does the above means?
- Original Message
From: Mumia W. [EMAIL PROTECTED]
To: Beginners List beginners@perl.org
Sent: Tuesday, January 16, 2007 3:13:38 PM
Subject: Re: Storing filehandles(for writing) in hashes doesn't work (Re:
whilereach my $variable (FILEHANDLE) )
On 01/15/2007 11:21 PM, Michael Alipio wrote:
Hi,
Ok, seems like a pet logs is not a good example.:-)
Let me revise my story:
I have a logfile which contains different clients firewall's logs.
Let's say the log file is: firewall.log
Now each line in the logfile has a $deviceid string that identifies where or
which client it came from.
What I did was to list down all of these clients in a file named
deviceid.conf.
Let's say it contains: ('client_name' dash 'device_id')
client1 - 293u0sdfj
client2 - 8325kjsdf
client3 - kjldas8282
.
clientn - sdkfj28350
Having said that, my goals are:
1. read the firewall.log line by line.
if it see a particular device_id in $_, and it knows that that deviceid
is for this particular client (using the information found at
deviceid.conf) it will write that line into /client1/$date.log or
/client2/$date.log etc.
By the way, our logs are being rotated such that it contains logs from 6:26
yesterday, to 6:25 today, so $date on the above was obtained by let's say
getting the /date=(\S+)/ on the first line entry of the log, let's say it
reads 2007-01-10, so our $date will be 2007-01-10_11, so the logfile for a
particular client will be /client1/2007-01-10_11.log
Here is an example of a line in the logfile:
Jan 10 06:26:17 210.23.194.86 date=2007-01-10
time=06:30:14,devname=sccp_firewall,device_id=FWF60A1234566,log_id=00210100
01,type=traffic,subtype=allowed,pri=notice,vd=root;SN=14435461,duration=139,user=N/A,group=N/A,policyid=11,proto=6,service=7
500/tcp,status=accept,src=192.169.1.70,srcname=192.168.1.3,dst=192.169.1.17,dstname=192.169.1.17,src_int=internal,dst_int
=wan2,sent=144,rcvd=0,sent_pkt=3,rcvd_pkt=0,src_port=2354,dst_port=7500,vpn=N/A,tran_ip=0.0.0.0,tran_port=0,dir_disp=org,tra
n_disp=noop
The device_id in this log entry does not appear in the device id file
you showed above.
So far, I've been trying to use the code that was given to me but I'm still
far from my goal:
#!/usr/bin/perl
use warnings;
use strict;
my $logfile='firewall.log';
my $devices='deviceid.conf';
our %log;
##
# 1ST PART
open DEVICES, '', $devices or die Can't open $devices $!;
while ( my $device = DEVICES){
chomp $device;
($device) = $device =~ /(\S+)$/;
open( my $fh, '', $device.log) or die Can't open $device.log: $!;
$log{$device} = $fh;
}
close DEVICES;
#
So far I can understand that in the first part, the code will read the
deviceid.conf and create a file handle for writing for each device id, and
store these filehandles inside %log. But that is what I wanted to do, I want
to, as I wanted to write my logs into /clientN/date.log instead of
client's_deviceid.log. So I'm still trying to figure out this one.
/(\S+)$/ only matches the device-id; the client-id is thrown away. Don't
throw it away. Create a regular expression that captures the client id
and use the client-id to create the log file name.
Next:
# 2nd PART
my $re = '^\S+\s+(' . join( '|', keys %log ). ')';
open( IN, '', $logfile ) or die Can't open $logfile: $!;
while( my $line = IN ){
if( $line =~ m/$re/ ){
print $log{$1} $line;
}
}
###
The second part is what confuses me, especially the line with my $re, and
also the if($line =~ m/$re/)
As far as I can understand, the $re will contain a regexp with ('device_id1 |
device_id2 | device_id3 | device_idN'), so that whenever it sees any pattern
that match either of those device_ids, it will print it to say
$log{device_idN} which points to the file handle that writes to
device_idN.log.
But this is not the case.
The line print $log{$1} $line; doesn't even work as if it cannot decode the
$log{$1}
You are correct, my $re creates a regular expression with all of the
device ids, and you are also correct that print $log{$1} $line does
not work. You must use print { $log{$1} } $line.
Useless use of a constant in void
Re: Storing filehandles(for writing) in hashes doesn't work (Re: whilereach my $variable (FILEHANDLE) )
On 01/16/2007 04:02 AM, Michael Alipio wrote:
- Original Message
From: Mumia W. [EMAIL PROTECTED]
To: Beginners List beginners@perl.org
Sent: Tuesday, January 16, 2007 3:13:38 PM
Subject: Re: Storing filehandles(for writing) in hashes doesn't work (Re: whilereach
my $variable (FILEHANDLE) )
On 01/15/2007 11:21 PM, Michael Alipio wrote:
Hi,
Ok, seems like a pet logs is not a good example.:-)
Let me revise my story:
I have a logfile which contains different clients firewall's logs.
Let's say the log file is: firewall.log
Now each line in the logfile has a $deviceid string that identifies where or which client it came from.
What I did was to list down all of these clients in a file named deviceid.conf.
Let's say it contains: ('client_name' dash 'device_id')
client1 - 293u0sdfj
client2 - 8325kjsdf
client3 - kjldas8282
.
clientn - sdkfj28350
Having said that, my goals are:
1. read the firewall.log line by line.
if it see a particular device_id in $_, and it knows that that deviceid is for this
particular client (using the information found at deviceid.conf) it will
write that line into /client1/$date.log or /client2/$date.log etc.
By the way, our logs are being rotated such that it contains logs from 6:26 yesterday, to 6:25
today, so $date on the above was obtained by let's say getting the /date=(\S+)/ on the first line
entry of the log, let's say it reads 2007-01-10, so our $date will be 2007-01-10_11, so
the logfile for a particular client will be /client1/2007-01-10_11.log
Here is an example of a line in the logfile:
Jan 10 06:26:17 210.23.194.86 date=2007-01-10
time=06:30:14,devname=sccp_firewall,device_id=FWF60A1234566,log_id=00210100
01,type=traffic,subtype=allowed,pri=notice,vd=root;SN=14435461,duration=139,user=N/A,group=N/A,policyid=11,proto=6,service=7
500/tcp,status=accept,src=192.169.1.70,srcname=192.168.1.3,dst=192.169.1.17,dstname=192.169.1.17,src_int=internal,dst_int
=wan2,sent=144,rcvd=0,sent_pkt=3,rcvd_pkt=0,src_port=2354,dst_port=7500,vpn=N/A,tran_ip=0.0.0.0,tran_port=0,dir_disp=org,tra
n_disp=noop
The device_id in this log entry does not appear in the device id file
you showed above.
So far, I've been trying to use the code that was given to me but I'm still far
from my goal:
#!/usr/bin/perl
use warnings;
use strict;
my $logfile='firewall.log';
my $devices='deviceid.conf';
our %log;
##
# 1ST PART
open DEVICES, '', $devices or die Can't open $devices $!;
while ( my $device = DEVICES){
chomp $device;
($device) = $device =~ /(\S+)$/;
open( my $fh, '', $device.log) or die Can't open $device.log: $!;
$log{$device} = $fh;
}
close DEVICES;
#
So far I can understand that in the first part, the code will read the
deviceid.conf and create a file handle for writing for each device id, and
store these filehandles inside %log. But that is what I wanted to do, I want
to, as I wanted to write my logs into /clientN/date.log instead of
client's_deviceid.log. So I'm still trying to figure out this one.
/(\S+)$/ only matches the device-id; the client-id is thrown away. Don't
throw it away. Create a regular expression that captures the client id
and use the client-id to create the log file name.
Next:
# 2nd PART
my $re = '^\S+\s+(' . join( '|', keys %log ). ')';
open( IN, '', $logfile ) or die Can't open $logfile: $!;
while( my $line = IN ){
if( $line =~ m/$re/ ){
print $log{$1} $line;
}
}
###
The second part is what confuses me, especially the line with my $re, and also the
if($line =~ m/$re/)
As far as I can understand, the $re will contain a regexp with ('device_id1 |
device_id2 | device_id3 | device_idN'), so that whenever it sees any pattern
that match either of those device_ids, it will print it to say $log{device_idN}
which points to the file handle that writes to device_idN.log.
But this is not the case.
The line print $log{$1} $line; doesn't even work as if it cannot decode the
$log{$1}
You are correct, my $re creates a regular expression with all of the
device ids, and you are also correct that print $log{$1} $line does
not work. You must use print { $log{$1} } $line.
Useless use of a constant in void context at extractdevice.pl line 30.
Scalar found where operator expected at extractdevice.pl line 35, near } $line
(Missing operator before $line?)
syntax error at extractdevice.pl line 35, near } $line
If I comment those codes inside the while in that second part, the program will
successfully create emtpy device_id1.log, device_id2.log, etc. etc.)
Any idea what's wrong with this one?
My comments are above.
I wrote the program a completely different way. I think you should read
'deviceid.conf' and place the device-ids and client-ids into a hash
($clientids). Then open and start reading the logfile. Whenever you are
able to capture a device-id that is in the %clientids hash, construct a
filename to the desired output
Storing filehandles(for writing) in hashes doesn't work (Re: whilereach my $variable (FILEHANDLE) )
Hi,
Ok, seems like a pet logs is not a good example.:-)
Let me revise my story:
I have a logfile which contains different clients firewall's logs.
Let's say the log file is: firewall.log
Now each line in the logfile has a $deviceid string that identifies where or
which client it came from.
What I did was to list down all of these clients in a file named
deviceid.conf.
Let's say it contains: ('client_name' dash 'device_id')
client1 - 293u0sdfj
client2 - 8325kjsdf
client3 - kjldas8282
.
clientn - sdkfj28350
Having said that, my goals are:
1. read the firewall.log line by line.
if it see a particular device_id in $_, and it knows that that deviceid is
for this particular client (using the information found at deviceid.conf) it
will write that line into /client1/$date.log or /client2/$date.log etc.
By the way, our logs are being rotated such that it contains logs from 6:26
yesterday, to 6:25 today, so $date on the above was obtained by let's say
getting the /date=(\S+)/ on the first line entry of the log, let's say it reads
2007-01-10, so our $date will be 2007-01-10_11, so the logfile for a
particular client will be /client1/2007-01-10_11.log
Here is an example of a line in the logfile:
Jan 10 06:26:17 210.23.194.86 date=2007-01-10
time=06:30:14,devname=sccp_firewall,device_id=FWF60A1234566,log_id=00210100
01,type=traffic,subtype=allowed,pri=notice,vd=root;SN=14435461,duration=139,user=N/A,group=N/A,policyid=11,proto=6,service=7
500/tcp,status=accept,src=192.169.1.70,srcname=192.168.1.3,dst=192.169.1.17,dstname=192.169.1.17,src_int=internal,dst_int
=wan2,sent=144,rcvd=0,sent_pkt=3,rcvd_pkt=0,src_port=2354,dst_port=7500,vpn=N/A,tran_ip=0.0.0.0,tran_port=0,dir_disp=org,tra
n_disp=noop
So far, I've been trying to use the code that was given to me but I'm still far
from my goal:
#!/usr/bin/perl
use warnings;
use strict;
my $logfile='firewall.log';
my $devices='deviceid.conf';
our %log;
##
# 1ST PART
open DEVICES, '', $devices or die Can't open $devices $!;
while ( my $device = DEVICES){
chomp $device;
($device) = $device =~ /(\S+)$/;
open( my $fh, '', $device.log) or die Can't open $device.log: $!;
$log{$device} = $fh;
}
close DEVICES;
#
So far I can understand that in the first part, the code will read the
deviceid.conf and create a file handle for writing for each device id, and
store these filehandles inside %log. But that is what I wanted to do, I want
to, as I wanted to write my logs into /clientN/date.log instead of
client's_deviceid.log. So I'm still trying to figure out this one.
Next:
# 2nd PART
my $re = '^\S+\s+(' . join( '|', keys %log ). ')';
open( IN, '', $logfile ) or die Can't open $logfile: $!;
while( my $line = IN ){
if( $line =~ m/$re/ ){
print $log{$1} $line;
}
}
###
The second part is what confuses me, especially the line with my $re, and
also the if($line =~ m/$re/)
As far as I can understand, the $re will contain a regexp with ('device_id1 |
device_id2 | device_id3 | device_idN'), so that whenever it sees any pattern
that match either of those device_ids, it will print it to say $log{device_idN}
which points to the file handle that writes to device_idN.log.
But this is not the case.
The line print $log{$1} $line; doesn't even work as if it cannot decode the
$log{$1}
Useless use of a constant in void context at extractdevice.pl line 30.
Scalar found where operator expected at extractdevice.pl line 35, near } $line
(Missing operator before $line?)
syntax error at extractdevice.pl line 35, near } $line
If I comment those codes inside the while in that second part, the program will
successfully create emtpy device_id1.log, device_id2.log, etc. etc.)
Any idea what's wrong with this one?
- Original Message
From: Thomas Bätzler [EMAIL PROTECTED]
To: Perl Beginners beginners@perl.org
Sent: Monday, January 15, 2007 4:24:02 PM
Subject: RE: whilereach my $variable (FILEHANDLE) (sort of like foreach
my $variable (@array)
Hi,
Michael Alipio [EMAIL PROTECTED] asked:
I've been thinking about this for quite some time now.
Suppose I have a text file (mypet.log that contains:
dog
pig
cat
Then I have another text file and I want to read it line by
line and extract each line that has those contained in the
first file. Sort of like running while inside while.
myallpetlogs contains:
blabla dog blah
blabla cat blah
blabla dog blah
blabla pig blah
blabla cat blah
blabla pig blah
Now, the goal is to have 3 files which contains all extracted
lines in myallpetlogs
Assuming that you don't have too many log extracts to create, the easy
way would be to read in mypet.log first and open a new logfile for
each entry using a scalar as the filehandle. That scalar can be stored
in a hash using the pet name as a key.
When reading the logfile you determine the pet name using a regular
expression and write the line to the pets
RE: Storing filehandles(for writing) in hashes doesn't work (Re: whilereach my $variable (FILEHANDLE) )
Hi,
Michael Alipio [EMAIL PROTECTED] wrote:
Ok, seems like a pet logs is not a good example.:-)
Let me revise my story:
I have a logfile which contains different clients firewall's logs.
Let's say the log file is: firewall.log
Now each line in the logfile has a $deviceid string that
identifies where or which client it came from.
What I did was to list down all of these clients in a file
named deviceid.conf.
Let's say it contains: ('client_name' dash 'device_id')
client1 - 293u0sdfj
client2 - 8325kjsdf
client3 - kjldas8282
.
clientn - sdkfj28350
Having said that, my goals are:
1. read the firewall.log line by line.
if it see a particular device_id in $_, and it knows that
that deviceid is for this particular client (using the
information found at deviceid.conf) it will write that line
into /client1/$date.log or /client2/$date.log etc.
By the way, our logs are being rotated such that it contains
logs from 6:26 yesterday, to 6:25 today, so $date on the
above was obtained by let's say getting the /date=(\S+)/ on
the first line entry of the log, let's say it reads
2007-01-10, so our $date will be 2007-01-10_11, so the
logfile for a particular client will be /client1/2007-01-10_11.log
Here is an example of a line in the logfile:
Jan 10 06:26:17 210.23.194.86 date=2007-01-10
time=06:30:14,devname=sccp_firewall,device_id=FWF60A1234566,lo
g_id=00210100
01,type=traffic,subtype=allowed,pri=notice,vd=root;SN=14435461
,duration=139,user=N/A,group=N/A,policyid=11,proto=6,service=7
500/tcp,status=accept,src=192.169.1.70,srcname=192.168.1.3,dst
=192.169.1.17,dstname=192.169.1.17,src_int=internal,dst_int
=wan2,sent=144,rcvd=0,sent_pkt=3,rcvd_pkt=0,src_port=2354,dst_
port=7500,vpn=N/A,tran_ip=0.0.0.0,tran_port=0,dir_disp=org,tra
n_disp=noop
So far, I've been trying to use the code that was given to me
but I'm still far from my goal:
#!/usr/bin/perl
use warnings;
use strict;
my $logfile='firewall.log';
my $devices='deviceid.conf';
our %log;
##
# 1ST PART
open DEVICES, '', $devices or die Can't open $devices $!;
while ( my $device = DEVICES){
chomp $device;
($device) = $device =~ /(\S+)$/;
open( my $fh, '', $device.log) or die Can't open
$device.log: $!;
$log{$device} = $fh;
}
close DEVICES;
#
OK, that is obviously too simple for what you want to do.
use strict;
use warnings;
[...]
open DEVICES, '', $devices or die Can't open $devices $!;
my %log;
my %client;
while ( my $line = DEVICES){
my($clientid, $devid) = ( $line =~ m/^(\S+)\s+-\s+(\S+)/ );
# store device-id = client-id relationship
$client{$devid} = $clientid;
# store device-id = logfile relationship
# we initialize this with a bogus date and no filehandle
@($log{$devid}){'date','fh'} = ('1970-01-01', undef);
}
close DEVICES;
So far I can understand that in the first part, the code will
read the deviceid.conf and create a file handle for writing
for each device id, and store these filehandles inside %log.
But that is what I wanted to do, I want to, as I wanted to
write my logs into /clientN/date.log instead of
client's_deviceid.log. So I'm still trying to figure out this one.
Well, you get what you ask for ;-)
The new idea works on the assumption that your logfile is
chronologically sorted. We can then look at each line in turn
and identify the log entry date and the device id for which
the event was logged.
What you then do is this:
- check $log{$devid}{'date'} if the date matches.
- if it doesn't match, close $log{$devid}{'fh'}
- open a new logfile named $clientid/$newdate.log;
- store the new date and handle in $log{$devid}{'date'} and $log{$devid}{'fh'}
- write the log line to $log{$devid}{'fh'}
Writing the code is left as an exercise for the reader ;-)
HTH,
Thomas
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
http://learn.perl.org/
