Re: Does this script have the efficiency problems?
On 3/9/06, Practical Perl [EMAIL PROTECTED] wrote:
snip
my $dir=/home/datas/loginIP;# here stored about 140 files,which are
totally 1.7G.
snip
next if /^192\.168\./;
next unless /^(\d+\.\d+\.\d+\.)(\d+)/;
$low{$1}{$2}=1 if $2 128;
$total{$1}{$2}=1;
snip
Based on the lines I have singled out it looks like your hash will be
very large. If this is the case then you might consider using a DBM.
Look at dbmopen, DB::File, NDBM::File, ODBM::File, SDBM::File, and
AnyDBM::File.
You might also consider putting in some debug messages to determine
what section of the code is taking the longest to run. There is a
profiler for Perl, but I have never used it.
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
http://learn.perl.org/ http://learn.perl.org/first-response
Does this script have the efficiency problems?
Hello,
I have a script,which run well at most time.This day I use it to analyse the
files of 1.7G,it become very slow and can't get executed continuely
anymore.When I run 'strace -p ' (here is this script's PID),there
is no output,it seems died.
Here is my script:
#!/usr/bin/perl
use strict;
use warnings;
my $date=`date +%y%m%d`;
chomp $date;
my $dir=/home/datas/loginIP;# here stored about 140 files,which are
totally 1.7G.
my @files= glob $date/*.ip;
my (%low, %total);
foreach my $file (@files)
{
open (FILE,$file) or die $!;
while(FILE)
{
next if /^192\.168\./;
next unless /^(\d+\.\d+\.\d+\.)(\d+)/;
$low{$1}{$2}=1 if $2 128;
$total{$1}{$2}=1;
}
close FILE;
}
open (RESULT,,dynamic.$date.re) or die $!;
foreach (sort { scalar keys %{$total{$b}} = scalar keys %{$total{$a}} }
keys %total)
{
my $low = scalar keys %{$low{$_}};
my $high = (scalar keys %{$total{$_}}) - $low;
if ($low 64 and $high 64)
{
printf RESULT %-25s%-20s\n, $_.0, $_.255;
}elsif ($low 64 and $high = 64)
{
printf RESULT %-25s%-20s\n, $_.0, $_.127;
}elsif ($low = 64 and $high 64)
{
printf RESULT %-25s%-20s\n, $_.128, $_.255;
}
}
close RESULT;
The files handled by the script are looked as:
211.139.227.247:3088
220.181.31.245:1134
220.181.31.247:1126
220.181.31.248:1120
220.181.31.246:1071
220.178.47.2:977
221.11.26.219:961
221.11.26.220:934
210.31.76.252:911
...
Why this happen and how to resolve it? Any suggestion is welcome.Thanks.
Re: Does this script have the efficiency problems?
On Thu, 9 Mar 2006, Practical Perl wrote: Here is my script: #!/usr/bin/perl use strict; use warnings; my $date=`date +%y%m%d`; chomp $date; ^^^ Not that this is your problem, but why on earth are you shelling out for the date? Perl can do this just fine, you know, and you don't even have to chomp() the result :-) Why this happen and how to resolve it? Any suggestion is welcome.Thanks. What happened when you benchmarked your script? Where did you determine that it is spending the most time? Figure that out and then you'll know where to focus your attention. -- Chris Devers -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] http://learn.perl.org/ http://learn.perl.org/first-response
Re: Does this script have the efficiency problems?
Practical Perl am Donnerstag, 9. März 2006 09.50:
Hello,
I have a script,which run well at most time.This day I use it to analyse
the files of 1.7G,it become very slow and can't get executed continuely
anymore.When I run 'strace -p ' (here is this script's PID),there
is no output,it seems died.
Here is my script:
#!/usr/bin/perl
use strict;
use warnings;
my $date=`date +%y%m%d`;
chomp $date;
my $dir=/home/datas/loginIP;# here stored about 140 files,which are
totally 1.7G.
my @files= glob $date/*.ip;
my (%low, %total);
foreach my $file (@files)
{
open (FILE,$file) or die $!;
while(FILE)
{
next if /^192\.168\./;
next unless /^(\d+\.\d+\.\d+\.)(\d+)/;
$low{$1}{$2}=1 if $2 128;
$total{$1}{$2}=1;
}
close FILE;
}
open (RESULT,,dynamic.$date.re) or die $!;
foreach (sort { scalar keys %{$total{$b}} = scalar keys %{$total{$a}} }
keys %total)
{
my $low = scalar keys %{$low{$_}};
my $high = (scalar keys %{$total{$_}}) - $low;
if ($low 64 and $high 64)
{
printf RESULT %-25s%-20s\n, $_.0, $_.255;
}elsif ($low 64 and $high = 64)
{
printf RESULT %-25s%-20s\n, $_.0, $_.127;
}elsif ($low = 64 and $high 64)
{
printf RESULT %-25s%-20s\n, $_.128, $_.255;
}
}
close RESULT;
The files handled by the script are looked as:
211.139.227.247:3088
220.181.31.245:1134
220.181.31.247:1126
220.181.31.248:1120
220.181.31.246:1071
220.178.47.2:977
221.11.26.219:961
221.11.26.220:934
210.31.76.252:911
...
Why this happen and how to resolve it? Any suggestion is welcome.Thanks.
Do you have sufficient memory to hold all data in a hash and sort it?
Does the box swap?
Hans
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
http://learn.perl.org/ http://learn.perl.org/first-response
Re: Does this script have the efficiency problems?
Chris Devers wrote: On Thu, 9 Mar 2006, Practical Perl wrote: Here is my script: #!/usr/bin/perl use strict; use warnings; my $date=`date +%y%m%d`; chomp $date; ^^^ Not that this is your problem, but why on earth are you shelling out for the date? Perl can do this just fine, you know, and you don't even have to chomp() the result :-) Try: #!/usr/bin/perl use strict; use warnings; use POSIX; my @now = localtime; print strftime( '%y%m%d', @now ), \n; __END__ The function strftime(3) uses the same format conversion specifications as date(1). See `man strftime` for details. -- Just my 0.0002 million dollars worth, --- Shawn Probability is now one. Any problems that are left are your own. SS Heart of Gold, _The Hitchhiker's Guide to the Galaxy_ This statement is true but unprovable. Kurt Godel * Perl tutorials at http://perlmonks.org/?node=Tutorials * A searchable perldoc is at http://perldoc.perl.org/ -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] http://learn.perl.org/ http://learn.perl.org/first-response
Re: Does this script have the efficiency problems?
Chris == Chris Devers [EMAIL PROTECTED] writes: Chris Not that this is your problem, but why on earth are you shelling out Chris for the date? Perl can do this just fine, you know, and you don't even Chris have to chomp() the result :-) yes, the irony of picking practical perl as your gmail name, but doing something very *impractical* with Perl. I didn't want that to be missed. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 merlyn@stonehenge.com URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training! -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] http://learn.perl.org/ http://learn.perl.org/first-response
