Re: Compare large files memory error
Perhaps the DOS FC (file compare) command will suffice for your application: Compares two files or sets of files and displays the differences between them FC [/A] [/C] [/L] [/LBn] [/N] [/T] [/U] [/W] [/] [drive1:] [path1]filename1 [drive2:][path2]filename2 FC /B [drive1:][path1]filename1 [drive2:][path2]filename2 /A Displays only first and last lines for each set of differences. /B Performs a binary comparison. /C Disregards the case of letters. /L Compares files as ASCII text. /LBn Sets the maximum consecutive mismatches to the specified number of lines. /N Displays the line numbers on an ASCII comparison. /T Does not expand tabs to spaces. /U Compare files as UNICODE text files. /W Compresses white space (tabs and spaces) for comparison. / Specifies the number of consecutive lines that must match after a mismatch. Will -- Original Message -- >No, I am using Win NT. > >Regards, > >Mark >Surrey,UK > >> Does your system have the 'cmp' program on it? IIRC this is a standard >or >> fairly standard Unix utility which does exactly what you want and you >could >> just call it from Perl... > >>Regards >> >>Mark Bedish >>Surrey,UK >> >> >>In a message dated Tue, 7 Aug 2001 13:36:50 British Summer Time, >>[EMAIL PROTECTED] (Randal L. Schwartz) writes: >> >> > > "Jos" == Jos I Boumans <[EMAIL PROTECTED]> writes: >> > >> > Jos> if you're just interested in *EXACTLY* the same files, use MD5 > >> (get it from >> > Jos> search.cpan.org) >> > Jos> even one BIT of difference (literally) will result in a different > >> checksum >> > Jos> thorough and fast >> > >> > MD5 is useful when you don't want to use the original data to compare >> > (or no longer have access to it). But if you've got both big files >> > sitting in front of you, it's a waste to compute md5(one) == md5(two). >> > Instead, just get File::Compare from the CPAN, which does a very >> > efficient comparison: first length, then big-block by big-block >> > equality. >> > >> > print "Just another Perl hacker,"; >> > >> > -- >> > Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 > >0095 >> > <[EMAIL PROTECTED]> http://www.stonehenge.com/merlyn/> >> > Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. >> > See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl > >> training! >> > >> > -- >> > To unsubscribe, e-mail: [EMAIL PROTECTED] >> > For additional commands, e-mail: [EMAIL PROTECTED] >> >> >> >>-- >>To unsubscribe, e-mail: [EMAIL PROTECTED] >>For additional commands, e-mail: [EMAIL PROTECTED] > >-- >Peter Scott >Pacific Systems Design Technologies >http://www.perldebugged.com > > >-- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] > >-- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] > > Visit iWon.com - the Internet's largest guaranteed cash giveaway! Click here now for your "Thank You" gift: http://www.iwon.com/giftcenter/0,2612,,00.html?t_id=20157 -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Compare large files memory error
No, I am using Win NT. Regards, Mark Surrey,UK > Does your system have the 'cmp' program on it? IIRC this is a standard or > fairly standard Unix utility which does exactly what you want and you could > just call it from Perl... >Regards > >Mark Bedish >Surrey,UK > > >In a message dated Tue, 7 Aug 2001 13:36:50 British Summer Time, >[EMAIL PROTECTED] (Randal L. Schwartz) writes: > > > > "Jos" == Jos I Boumans <[EMAIL PROTECTED]> writes: > > > > Jos> if you're just interested in *EXACTLY* the same files, use MD5 > (get it from > > Jos> search.cpan.org) > > Jos> even one BIT of difference (literally) will result in a different > checksum > > Jos> thorough and fast > > > > MD5 is useful when you don't want to use the original data to compare > > (or no longer have access to it). But if you've got both big files > > sitting in front of you, it's a waste to compute md5(one) == md5(two). > > Instead, just get File::Compare from the CPAN, which does a very > > efficient comparison: first length, then big-block by big-block > > equality. > > > > print "Just another Perl hacker,"; > > > > -- > > Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 > > <[EMAIL PROTECTED]> http://www.stonehenge.com/merlyn/> > > Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. > > See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl > training! > > > > -- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > >-- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Compare large files memory error
At 09:59 AM 8/7/01 -0400, [EMAIL PROTECTED] wrote: >Randal, > >Thanks for the file compare tip, it is incredibly fast! However it doesnt >tell me where the difference is. Can I get it to print out the first block >of data that is different? Does your system have the 'cmp' program on it? IIRC this is a standard or fairly standard Unix utility which does exactly what you want and you could just call it from Perl... >Regards > >Mark Bedish >Surrey,UK > > >In a message dated Tue, 7 Aug 2001 13:36:50 British Summer Time, >[EMAIL PROTECTED] (Randal L. Schwartz) writes: > > > > "Jos" == Jos I Boumans <[EMAIL PROTECTED]> writes: > > > > Jos> if you're just interested in *EXACTLY* the same files, use MD5 > (get it from > > Jos> search.cpan.org) > > Jos> even one BIT of difference (literally) will result in a different > checksum > > Jos> thorough and fast > > > > MD5 is useful when you don't want to use the original data to compare > > (or no longer have access to it). But if you've got both big files > > sitting in front of you, it's a waste to compute md5(one) == md5(two). > > Instead, just get File::Compare from the CPAN, which does a very > > efficient comparison: first length, then big-block by big-block > > equality. > > > > print "Just another Perl hacker,"; > > > > -- > > Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 > > <[EMAIL PROTECTED]> http://www.stonehenge.com/merlyn/> > > Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. > > See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl > training! > > > > -- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > >-- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] -- Peter Scott Pacific Systems Design Technologies http://www.perldebugged.com -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Compare large files memory error
if you're just interested in *EXACTLY* the same files, use MD5 (get it from
search.cpan.org)
even one BIT of difference (literally) will result in a different checksum
thorough and fast
hth
Jos Boumans
- Original Message -
From: "Maxim Berlin" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Tuesday, August 07, 2001 1:35 PM
Subject: Re: Compare large files memory error
> Hello Mbedish,
>
> Tuesday, August 07, 2001, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> Mac> Is there a better way to compare large files than this snippet,
> Mac> which runs out of memory if files > 30mb.
>
> Mac> It is also slow, about the same speed as comparing in a text editor!
>
> Mac> Thank you.
>
> Mac> __SNIP__
>
> Mac> @file1 = ();
> Mac> @file2 = ();
> first, it is a very bad practice to read entire file into memory. for
> example, if file size is bigger than free memory, you'll have
unpredictable
> result.
> second, the slowest operation in your example is file I/O. your files
> may have differences in second string, so you do not need to read rest of
> file.
>
>
> Mac> $are_equal = compare_arrays(\@file1, \@file2);
>
> Mac> if ($are_equal) {
>
> Mac> print "Files are IDENTICAL \n";
> Mac> }
>
> Mac> else
>
> Mac> {
>
> Mac> print "Files are DIFFERENT \n";
> Mac> }
>
> Mac> sub compare_arrays {
> Mac>my ($first, $second) = @_;
> Mac>#no warnings; # silence spurious -w undef complaints
> third, you compare your arrays two times
>
> first pass:
> Mac>return 0 unless @$first == @$second;
> second pass:
> Mac>for (my $i = 0; $i < @$first; $i++) {
> Mac>return 0 if $first->[$i] ne $second->[$i];
> Mac>}
>
> Mac>return 1;
> Mac>}
>
> Mac> __SNIP__
> so, i can recommend something like
> $retval=1;
> while( $a= )
> {
> $b=;
> if ($a ne $b)
> {
> $retval=0; last;
> }
> }
> close IN1; close IN2;
> return $retval;
>
> at the beginning, you can compare files size (perldoc -f stat).
>
>
> Best wishes,
> Maximmailto:[EMAIL PROTECTED]
>
>
>
> --
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
>
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Compare large files memory error
Hello Mbedish,
Tuesday, August 07, 2001, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
Mac> Is there a better way to compare large files than this snippet,
Mac> which runs out of memory if files > 30mb.
Mac> It is also slow, about the same speed as comparing in a text editor!
Mac> Thank you.
Mac> __SNIP__
Mac> @file1 = ();
Mac> @file2 = ();
first, it is a very bad practice to read entire file into memory. for
example, if file size is bigger than free memory, you'll have unpredictable
result.
second, the slowest operation in your example is file I/O. your files
may have differences in second string, so you do not need to read rest of
file.
Mac> $are_equal = compare_arrays(\@file1, \@file2);
Mac> if ($are_equal) {
Mac> print "Files are IDENTICAL \n";
Mac> }
Mac> else
Mac> {
Mac> print "Files are DIFFERENT \n";
Mac> }
Mac> sub compare_arrays {
Mac>my ($first, $second) = @_;
Mac>#no warnings; # silence spurious -w undef complaints
third, you compare your arrays two times
first pass:
Mac>return 0 unless @$first == @$second;
second pass:
Mac>for (my $i = 0; $i < @$first; $i++) {
Mac>return 0 if $first->[$i] ne $second->[$i];
Mac>}
Mac>return 1;
Mac>}
Mac> __SNIP__
so, i can recommend something like
$retval=1;
while( $a= )
{
$b=;
if ($a ne $b)
{
$retval=0; last;
}
}
close IN1; close IN2;
return $retval;
at the beginning, you can compare files size (perldoc -f stat).
Best wishes,
Maximmailto:[EMAIL PROTECTED]
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Compare large files memory error
Is there a better way to compare large files than this snippet, which runs out of
memory if files > 30mb.
It is also slow, about the same speed as comparing in a text editor!
Thank you.
__SNIP__
@file1 = ();
@file2 = ();
$are_equal = compare_arrays(\@file1, \@file2);
if ($are_equal) {
print "Files are IDENTICAL \n";
}
else
{
print "Files are DIFFERENT \n";
}
sub compare_arrays {
my ($first, $second) = @_;
#no warnings; # silence spurious -w undef complaints
return 0 unless @$first == @$second;
for (my $i = 0; $i < @$first; $i++) {
return 0 if $first->[$i] ne $second->[$i];
}
return 1;
}
__SNIP__
Regards,
Mark
Surrey,UK
--
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
