Re: (visual) diff for large files
Am 2008-05-01 13:10:56, schrieb michael: > At work we've been discussing (below) 'diff' running out of memory. I've > tried to see if 'rdiff' can help (but no idea how to back out the > differences from the "delta" o/p file). I've seen that freeBSD has a > diff utility (eg 2bsd-diff) that does NOT read in all of the files in > one go and thus has less problems with 'memory exhausted' but I cannot > find a Debian equivalent. Therefore, > > does anybody know of a freeBSD->Debian ports site? > > I did try downloading the 2.11bsd-diff source but it doesn't build on my > system (unsurprisingly!) Errors? Since I am currently kicked off by a wather-damage and have only my old IBM ThinkPad 570 I can not realy help you, but... Thanks, Greetings and nice Day Michelle Konzack Systemadministrator 24V Electronic Engineer Tamay Dogan Network Debian GNU/Linux Consultant -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ # Debian GNU/Linux Consultant # Michelle Konzack Apt. 917 ICQ #328449886 +49/177/935194750, rue de Soultz MSN LinuxMichi +33/6/61925193 67100 Strasbourg/France IRC #Debian (irc.icq.com) signature.pgp Description: Digital signature
Re: (visual) diff for large files
On Sat, May 3, 2008 at 3:06 PM, michael <[EMAIL PROTECTED]> wrote: > Yes now I am in the right dir and apply your patch and compile I get > some execs but it still falls over with large (2.5GB) input files: Hi, When I have to compare two large files, first, I split them into smaller files, then I run diff between these files (with bash help, of course). In my case it worked fine. Regards
Re: (visual) diff for large files
2008/5/3 michael <[EMAIL PROTECTED]>: > Yes now I am in the right dir and apply your patch and compile I get > some execs but it still falls over with large (2.5GB) input files: > > [EMAIL PROTECTED]:/tmp/xxdiff_files > $ /home/michael/src/bsd-diff/bin/diff/diff OLDrun.log.part4 > run.log.part4 > diff: OLDrun.log.part4: Value too large for defined data type Hello, hmmm, I think that is more a Linux issue then. Sorry. But you could install a FreeBSD in a virtual machine =). Best regards, Sergio Cuellar --- "Meine Hoffnung soll mich leiten Durch die Tage ohne Dich Und die Liebe soll mich tragen Wenn der Schmerz die Hoffnung bricht" -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: (visual) diff for large files
On Fri, 2008-05-02 at 18:54 -0500, Sergio Cuéllar Valdés wrote: > 2008/5/2 michael <[EMAIL PROTECTED]>: > > I had a quick try (before above patch) but it fails on headers: > > > > > > [EMAIL PROTECTED]:~/src/bsd-diff/bin$ gcc --version > > gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21) > > Copyright (C) 2006 Free Software Foundation, Inc. > > This is free software; see the source for copying conditions. There is > > NO > > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > > PURPOSE. > > > > > > [EMAIL PROTECTED]:~/src/bsd-diff/bin$ make > > cd adb; make SEPFLAG=-i > > make[1]: Entering directory `/home/michael/src/bsd-diff/bin/adb' > > cc -O -DNUM_SYMS_CACHE=50 -c -o access.o access.c > > In file included from access.c:1: > > defs.h:15:25: error: machine/reg.h: No such file or directory > > In file included from access.c:1: > > defs.h:194: warning: conflicting types for built-in function 'round' > > access.c: In function 'acces': > > access.c:69: error: 'errno' undeclared (first use in this function) > > access.c:69: error: (Each undeclared identifier is reported only once > > access.c:69: error: for each function it appears in.) > > access.c: In function 'setovmap': > > access.c:142: error: 'NOVL' undeclared (first use in this function) > > access.c:148: error: invalid use of undefined type 'struct ovlhdr' > > make[1]: *** [access.o] Error 1 > > make[1]: Leaving directory `/home/michael/src/bsd-diff/bin/adb' > > make: *** [adb] Error 2 > > Hello, > > those errors seem very different of what I get. This are the steps I made: > > 1. I have unpacked the file file8.tar.gz (MD5: > 77397e6d554361c127592b1fea2d776f) in the directory bsd-src. > 2. Go to bin/diff directory, where there are the following files: > * diff.c > * diffdir.c > * diff.h > * diffh.c > * diffreg.c > * Makefile > Yes now I am in the right dir and apply your patch and compile I get some execs but it still falls over with large (2.5GB) input files: [EMAIL PROTECTED]:/tmp/xxdiff_files $ /home/michael/src/bsd-diff/bin/diff/diff OLDrun.log.part4 run.log.part4 diff: OLDrun.log.part4: Value too large for defined data type -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: (visual) diff for large files
2008/5/2 michael <[EMAIL PROTECTED]>: > I had a quick try (before above patch) but it fails on headers: > > > [EMAIL PROTECTED]:~/src/bsd-diff/bin$ gcc --version > gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21) > Copyright (C) 2006 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is > NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > PURPOSE. > > > [EMAIL PROTECTED]:~/src/bsd-diff/bin$ make > cd adb; make SEPFLAG=-i > make[1]: Entering directory `/home/michael/src/bsd-diff/bin/adb' > cc -O -DNUM_SYMS_CACHE=50 -c -o access.o access.c > In file included from access.c:1: > defs.h:15:25: error: machine/reg.h: No such file or directory > In file included from access.c:1: > defs.h:194: warning: conflicting types for built-in function 'round' > access.c: In function 'acces': > access.c:69: error: 'errno' undeclared (first use in this function) > access.c:69: error: (Each undeclared identifier is reported only once > access.c:69: error: for each function it appears in.) > access.c: In function 'setovmap': > access.c:142: error: 'NOVL' undeclared (first use in this function) > access.c:148: error: invalid use of undefined type 'struct ovlhdr' > make[1]: *** [access.o] Error 1 > make[1]: Leaving directory `/home/michael/src/bsd-diff/bin/adb' > make: *** [adb] Error 2 Hello, those errors seem very different of what I get. This are the steps I made: 1. I have unpacked the file file8.tar.gz (MD5: 77397e6d554361c127592b1fea2d776f) in the directory bsd-src. 2. Go to bin/diff directory, where there are the following files: * diff.c * diffdir.c * diff.h * diffh.c * diffreg.c * Makefile 3. I I try to compile using gcc 4.2.3 I got the following errors and warnings: $ make cc -O -DDIFF='"/bin/diff"' -DDIFFH='"/usr/libexec/diffh"' -DPR='"/bin/pr"' -c -o diff.o diff.c In file included from diff.c:5: diff.h:80: warning: conflicting types for built-in function 'malloc' diff.c: In function 'savestr': diff.c:155: warning: incompatible implicit declaration of built-in function 'strlen' diff.c:161: warning: incompatible implicit declaration of built-in function 'strcpy' diff.c: In function 'done': diff.c:183: warning: incompatible implicit declaration of built-in function 'exit' cc -O -DDIFF='"/bin/diff"' -DDIFFH='"/usr/libexec/diffh"' -DPR='"/bin/pr"' -c -o diffdir.o diffdir.c In file included from diffdir.c:5: diff.h:80: warning: conflicting types for built-in function 'malloc' diffdir.c: In function 'diffdir': diffdir.c:43: warning: incompatible implicit declaration of built-in function 'strcpy' diffdir.c:47: warning: incompatible implicit declaration of built-in function 'strcat' diffdir.c: In function 'setfile': diffdir.c:124: warning: incompatible implicit declaration of built-in function 'exit' diffdir.c:126: warning: incompatible implicit declaration of built-in function 'strcpy' diffdir.c: In function 'setupdir': diffdir.c:202: error: 'struct dirent' has no member named 'd_namlen' diffdir.c:211: warning: incompatible implicit declaration of built-in function 'strcpy' diffdir.c: In function 'compare': diffdir.c:243: warning: incompatible implicit declaration of built-in function 'strcpy' make: *** [diffdir.o] Fehler 1 4. Now, if I use gcc ver 3.3.6 (that is what I have now at home), I get the following errors: $ make clean $ CC=/usr/bin/gcc-3.3 make /usr/bin/gcc-3.3 -O -DDIFF='"/bin/diff"' -DDIFFH='"/usr/libexec/diffh"' -DPR='"/bin/pr"' -c -o diffdir.o diffdir.c diffdir.c: In Funktion »setupdir«: diffdir.c:202: error: structure has no member named `d_namlen' make: *** [diffdir.o] Fehler 1 5. Make the modifications I told you to the file diffdir.c 6. Compile again $ make clean $ CC=/usr/bin/gcc-3.3 make You will get some warnings, but you will have the binaries diff and diffh. Where are you compiling diff, and what directory level ? Go directly do the bin/diff directory. Hope this will help you. Best regards, Sergio Cuellar -- "Meine Hoffnung soll mich leiten Durch die Tage ohne Dich Und die Liebe soll mich tragen Wenn der Schmerz die Hoffnung bricht" -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: (visual) diff for large files
On Fri, 2008-05-02 at 14:17 -0500, Sergio Cuéllar Valdés wrote:
> 2008/5/2 michael <[EMAIL PROTECTED]>:
> > On Thu, 2008-05-01 at 13:21 -0500, Sergio Cuéllar Valdés wrote:
> > > 2008/5/1 michael <[EMAIL PROTECTED]>:
> > > > At work we've been discussing (below) 'diff' running out of memory.
> > I've
> > > > tried to see if 'rdiff' can help (but no idea how to back out the
> > > > differences from the "delta" o/p file). I've seen that freeBSD has a
> > > > diff utility (eg 2bsd-diff) that does NOT read in all of the files in
> > > > one go and thus has less problems with 'memory exhausted' but I cannot
> > > > find a Debian equivalent. Therefore,
> > > >
> > > > does anybody know of a freeBSD->Debian ports site?
> > > >
> > > > I did try downloading the 2.11bsd-diff source but it doesn't build on
> > my
> > > > system (unsurprisingly!)
>
> Hello,
>
> I compiled diff. I used gcc-3.4. I had to modify diffdir.c. These are
> the changes:
>
> @@ -18,6 +18,7 @@ struct dir {
> short d_reclen;
> short d_namlen;
> char*d_entry;
> + char*d_name;
> };
>
> struct dir *setupdir();
> @@ -172,7 +173,7 @@ setupdir(cp)
> char *cp;
> {
> register struct dir *dp = 0, *ep;
> - register struct direct *rp;
> + register struct dir *rp;
> register int nitems, n;
> DIR *dirp;
>
> Go to the directory of diff (bin/diff) and $ CC=/usr/bin/gcc-3.4 make
>
> There are some warnings, but you will get 2 ELF binaries: diff and diffh.
>
> I used the compiled diff to see the differences of the file diffdir.c:
>
> $ ./diff /tmp/orig/bin/diff/diffdir.c /tmp/modified/bin/diff/diffdir.c
> 20a21
> > char*d_name;
> 175c176
> < register struct direct *rp;
> ---
> > register struct dir *rp;
>
>
> hope it will help you.
>
> Best regards,
> Sergio Cuellar
>
I had a quick try (before above patch) but it fails on headers:
[EMAIL PROTECTED]:~/src/bsd-diff/bin$ gcc --version
gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is
NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.
[EMAIL PROTECTED]:~/src/bsd-diff/bin$ make
cd adb; make SEPFLAG=-i
make[1]: Entering directory `/home/michael/src/bsd-diff/bin/adb'
cc -O -DNUM_SYMS_CACHE=50 -c -o access.o access.c
In file included from access.c:1:
defs.h:15:25: error: machine/reg.h: No such file or directory
In file included from access.c:1:
defs.h:194: warning: conflicting types for built-in function 'round'
access.c: In function 'acces':
access.c:69: error: 'errno' undeclared (first use in this function)
access.c:69: error: (Each undeclared identifier is reported only once
access.c:69: error: for each function it appears in.)
access.c: In function 'setovmap':
access.c:142: error: 'NOVL' undeclared (first use in this function)
access.c:148: error: invalid use of undefined type 'struct ovlhdr'
make[1]: *** [access.o] Error 1
make[1]: Leaving directory `/home/michael/src/bsd-diff/bin/adb'
make: *** [adb] Error 2
>
> --
> "Meine Hoffnung soll mich leiten
> Durch die Tage ohne Dich
> Und die Liebe soll mich tragen
> Wenn der Schmerz die Hoffnung bricht"
>
>
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: (visual) diff for large files
2008/5/2 michael <[EMAIL PROTECTED]>:
> On Thu, 2008-05-01 at 13:21 -0500, Sergio Cuéllar Valdés wrote:
> > 2008/5/1 michael <[EMAIL PROTECTED]>:
> > > At work we've been discussing (below) 'diff' running out of memory. I've
> > > tried to see if 'rdiff' can help (but no idea how to back out the
> > > differences from the "delta" o/p file). I've seen that freeBSD has a
> > > diff utility (eg 2bsd-diff) that does NOT read in all of the files in
> > > one go and thus has less problems with 'memory exhausted' but I cannot
> > > find a Debian equivalent. Therefore,
> > >
> > > does anybody know of a freeBSD->Debian ports site?
> > >
> > > I did try downloading the 2.11bsd-diff source but it doesn't build on my
> > > system (unsurprisingly!)
Hello,
I compiled diff. I used gcc-3.4. I had to modify diffdir.c. These are
the changes:
@@ -18,6 +18,7 @@ struct dir {
short d_reclen;
short d_namlen;
char*d_entry;
+ char*d_name;
};
struct dir *setupdir();
@@ -172,7 +173,7 @@ setupdir(cp)
char *cp;
{
register struct dir *dp = 0, *ep;
- register struct direct *rp;
+ register struct dir *rp;
register int nitems, n;
DIR *dirp;
Go to the directory of diff (bin/diff) and $ CC=/usr/bin/gcc-3.4 make
There are some warnings, but you will get 2 ELF binaries: diff and diffh.
I used the compiled diff to see the differences of the file diffdir.c:
$ ./diff /tmp/orig/bin/diff/diffdir.c /tmp/modified/bin/diff/diffdir.c
20a21
> char*d_name;
175c176
< register struct direct *rp;
---
> register struct dir *rp;
hope it will help you.
Best regards,
Sergio Cuellar
--
"Meine Hoffnung soll mich leiten
Durch die Tage ohne Dich
Und die Liebe soll mich tragen
Wenn der Schmerz die Hoffnung bricht"
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: (visual) diff for large files
On Thu, 2008-05-01 at 13:21 -0500, Sergio Cuéllar Valdés wrote: > 2008/5/1 michael <[EMAIL PROTECTED]>: > > At work we've been discussing (below) 'diff' running out of memory. I've > > tried to see if 'rdiff' can help (but no idea how to back out the > > differences from the "delta" o/p file). I've seen that freeBSD has a > > diff utility (eg 2bsd-diff) that does NOT read in all of the files in > > one go and thus has less problems with 'memory exhausted' but I cannot > > find a Debian equivalent. Therefore, > > > > does anybody know of a freeBSD->Debian ports site? > > > > I did try downloading the 2.11bsd-diff source but it doesn't build on my > > system (unsurprisingly!) > > Hello, > > >From where have you downloaded the source of the bsd-diff ? I am > searching for it, just to try to compile it. Which are the errors you > get ? http://www.freebsd.org/cgi/pds.cgi?ports/textproc/2bsd-diff -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: (visual) diff for large files
2008/5/1 michael <[EMAIL PROTECTED]>: > At work we've been discussing (below) 'diff' running out of memory. I've > tried to see if 'rdiff' can help (but no idea how to back out the > differences from the "delta" o/p file). I've seen that freeBSD has a > diff utility (eg 2bsd-diff) that does NOT read in all of the files in > one go and thus has less problems with 'memory exhausted' but I cannot > find a Debian equivalent. Therefore, > > does anybody know of a freeBSD->Debian ports site? > > I did try downloading the 2.11bsd-diff source but it doesn't build on my > system (unsurprisingly!) Hello, >From where have you downloaded the source of the bsd-diff ? I am searching for it, just to try to compile it. Which are the errors you get ? Best regards, Sergio Cuellar -- "Meine Hoffnung soll mich leiten Durch die Tage ohne Dich Und die Liebe soll mich tragen Wenn der Schmerz die Hoffnung bricht" -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
(visual) diff for large files
At work we've been discussing (below) 'diff' running out of memory. I've tried to see if 'rdiff' can help (but no idea how to back out the differences from the "delta" o/p file). I've seen that freeBSD has a diff utility (eg 2bsd-diff) that does NOT read in all of the files in one go and thus has less problems with 'memory exhausted' but I cannot find a Debian equivalent. Therefore, does anybody know of a freeBSD->Debian ports site? I did try downloading the 2.11bsd-diff source but it doesn't build on my system (unsurprisingly!) Thanks, Michael Forwarded Message > From: Michael Bane <[EMAIL PROTECTED]> > To: michael <[EMAIL PROTECTED]> > Subject: Re: [MAN-UNIX-GROUP] (visual) diff for large files > Date: Thu, 01 May 2008 12:50:22 +0100 > > On Wed, 2008-04-30 at 17:23 +0100, michael wrote: > > Like many I now generate large files (for purpose of this discussion > > even 1.5GB is large), they are ASCII and I wish to 'diff' them > > visually. > > > > For reasonably sized files (up to a few GB) 'diff' will tell me where > > the differences are for given lines but sometimes it's a bit tricky to > > interpret. (And does it take a while (and SO SO SO much memory!) for > > large files!) > > > > 'sdiff' seems to do the job, producing side-by-side output, againg for > > reasonably sized files... however it seems to show only the first 50 (if > > that) cols of output which isn't much help for me (each of my rows are > > about 150 chars wide) > > > > My favourite tool, to date, has been 'xxdiff' (as per SGI's 'xdiff') but > > that falls over on files over about 1.5GB. It seems as thou xdiff calls > > diff and the memory usage is somehow doubled compared to 'diff' alone... > > > > I've quickly tried 'tkdiff' and that falls over even more often. > > > > So, my question is, given the >>GB files that are in common use today, > > especially on high performance computing machines (with say 16GB RAM and > > 32GB swap), how do people compare their outputs? I'm presuming most of > > the utilities fall over since they try and keep everything in memory and > > once that's full they fall over (with the side effect of bringing that > > machine painfully > > S...L...O...W..L..Y to its knees) > > > > What do people on horace use for large files? Or on your own HPC > > clusters or even desktops? I've spend way too many hours this month > > cutting files into segments in order to pass to xxdiff... > > > > It appears that freeBSD has solved the problem of diff starting off by > reading files (and thus exhausting memory)... > http://www.freebsdsoftware.org/textproc/2bsd-diff.html > but does anybody know of freeBSD to Debian (or Fedora) ports??? > > Thanks, Michael > > > > Many thanks, Michael -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
