Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
Paul Bijnens schrieb: >> Why not implement some check-routine into Amanda that checks for the >> installed tar-release and gives a WARNING if a well-known problematic >> release is found? > > That would give a lot of false positives. > > RedHat for example backports important bugfixes to earlier versions > of programs. And while the "tar --version" here says "1.14" > Redhad has its own numbering scheme found by "rpm -q tar", resulting > in "tar-1.14-9.RHEL4", on my machine. And that version has all the > known critical bugs fixed, included the one above. Ok, then it has to get pointed out more clearly at installation-time that the user has to check for a proper release by himself or something. Users don't check the docs for problems they don't have/see, most of them even don't check the docs if they actually have/see a problem ;-) Stefan.
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On 2006-03-16 18:37, Stefan G. Weichinger wrote: Dave Ewart schrieb: On Wednesday, 08.03.2006 at 10:27 -0500, Ian Turner wrote: Please try upgrading your tar and check if you still see this behaviour. OK, you've hit the nail on the head: Using the existing system tar: $ tar --version tar (GNU tar) 1.14 And using a rebuilt version from a more recent source: $ ~/src/tar-1.15.1/src/tar --version tar (GNU tar) 1.15.1 A new tar is indeed the answer it seems. Suggestion related to the tar-issues: Why not implement some check-routine into Amanda that checks for the installed tar-release and gives a WARNING if a well-known problematic release is found? That would give a lot of false positives. RedHat for example backports important bugfixes to earlier versions of programs. And while the "tar --version" here says "1.14" Redhad has its own numbering scheme found by "rpm -q tar", resulting in "tar-1.14-9.RHEL4", on my machine. And that version has all the known critical bugs fixed, included the one above. -- Paul Bijnens, xplanation Technology ServicesTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, * * F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... * * ... "Are you sure?" ... YES ... Phew ... I'm out * ***
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
Dave Ewart schrieb: > On Wednesday, 08.03.2006 at 10:27 -0500, Ian Turner wrote: > >> Please >> try upgrading your tar and check if you still see this behaviour. > > OK, you've hit the nail on the head: > > Using the existing system tar: > > $ tar --version > tar (GNU tar) 1.14 > And using a rebuilt version from a more recent source: > > $ ~/src/tar-1.15.1/src/tar --version > tar (GNU tar) 1.15.1 > A new tar is indeed the answer it seems. Suggestion related to the tar-issues: Why not implement some check-routine into Amanda that checks for the installed tar-release and gives a WARNING if a well-known problematic release is found? For the wishlist, I assume ;-) Stefan
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday, 08.03.2006 at 12:05 -0500, Jon LaBadie wrote: > So, rather than use "listed-incremental" to recognize > it is a renamed directory, gnutar's behavior is to > consider then entire directory tree "changed" and > back up the entire tree. > > Reading this thread reminded me of another one where > the poster was complaining about this "feature" of > gnutar. That poster has a lot of rotating log dirs > that were quite large. They were rotated by renaming > them each night (log.2 becomes log.3, log.1 -> log.2 etc.) > Their complaint was that the data in the directory WAS > being backed up each night though it was unchanging. > > Be nice if gnutar could deal with both needs. Indeed, that very issue had occurred to me too :-) However, directory renames are fairly rare and IMO if there is an 'error' it should be "back up too much but making sure it's complete" rather than "keep the backup size small but occasionally miss something important". Of course, supporting both Would Be Nice, as you say... Dave. -- Dave Ewart [EMAIL PROTECTED] Computing Manager, Cancer Epidemiology Unit Cancer Research UK / Oxford University PGP: CC70 1883 BD92 E665 B840 118B 6E94 2CFD 694D E370 Get key from http://www.ceu.ox.ac.uk/~davee/davee-ceu-ox-ac-uk.asc N 51.7518, W 1.2016 signature.asc Description: Digital signature
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
--On March 8, 2006 11:34:16 AM + Dave Ewart <[EMAIL PROTECTED]> wrote: Thoughts/opinions here? BTW the AMANDA server runs Debian/Woody (2.4.2p2-4) and the client being backed up above runs Debian/Sarge AMD64 (2.4.4p3-3). It could very well be b -- if it is it's not amanda, it's tar. The backup program is responsible exclusively for what does and does not get backed up. Amanda just communicates a level/last backup date/file list to use. Depending on the dump/backup program. I know that a new tar version in sarge atleast before the security update taht just went out is broken anyway. It doesn't create valid archives a lot of the time, you'd be better off installing a backported tar from unstable, or rebuilding the one in oldstable and using that. Google around for invalid base64 or obsolescent base64 header skipping to next archive (i think that's right) if you haven't seen the error message yet. Cheers, Dave. -- Dave Ewart [EMAIL PROTECTED] Computing Manager, Cancer Epidemiology Unit Cancer Research UK / Oxford University PGP: CC70 1883 BD92 E665 B840 118B 6E94 2CFD 694D E370 Get key from http://www.ceu.ox.ac.uk/~davee/davee-ceu-ox-ac-uk.asc N 51.7518, W 1.2016 -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wed, Mar 08, 2006 at 04:21:42PM +, Dave Ewart wrote: > On Wednesday, 08.03.2006 at 10:27 -0500, Ian Turner wrote: > > > This does not appear to happen with CVS tar in listed-incremental mode. > > Please > > try upgrading your tar and check if you still see this behaviour. > > > > OK, you've hit the nail on the head: > [snip] > > > A new tar is indeed the answer it seems. So, rather than use "listed-incremental" to recognize it is a renamed directory, gnutar's behavior is to consider then entire directory tree "changed" and back up the entire tree. Reading this thread reminded me of another one where the poster was complaining about this "feature" of gnutar. That poster has a lot of rotating log dirs that were quite large. They were rotated by renaming them each night (log.2 becomes log.3, log.1 -> log.2 etc.) Their complaint was that the data in the directory WAS being backed up each night though it was unchanging. Be nice if gnutar could deal with both needs. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday, 08.03.2006 at 10:27 -0500, Ian Turner wrote: > On Wednesday 08 March 2006 06:34, Dave Ewart wrote: > > In summary: when a directory is *renamed* the files underneath it are > > not considered to have changed > > This does not appear to happen with CVS tar in listed-incremental mode. > Please > try upgrading your tar and check if you still see this behaviour. > > For reference, this is the test I performed: > > $ # Build CVS tar, chdir to src/. > $ mkdir -p test/foo > $ touch test/foo/bar > $ ./tar -cvf test1.tar --listed-incremental test1.incr test > ./tar: test/foo: Directory is new > test/ > test/foo/ > test/foo/bar > $ mv test/foo test/baz > $ ./tar -cvf test2.tar --listed-incremental test1.incr test > ./tar: test/baz: Directory is new > test/ > test/baz/ > test/baz/bar OK, you've hit the nail on the head: Using the existing system tar: $ tar --version tar (GNU tar) 1.14 $ find . ./blah ./blah/a ./blah/a/foo ./blah/a/bar $ tar -cvf test1.tar --listed-incremental test1.incr blah tar: blah/a: Directory is new blah/ blah/a/ blah/a/bar blah/a/foo $ mv blah/a blah/b $ tar -cvf test1.tar --listed-incremental test1.incr blah tar: blah/b: Directory is new blah/ blah/b/ And using a rebuilt version from a more recent source: $ ~/src/tar-1.15.1/src/tar --version tar (GNU tar) 1.15.1 $ ~/src/tar-1.15.1/src/tar -cvf test1.tar --listed-incremental test1.incr blah /home/davee/src/tar-1.15.1/src/tar: blah/a: Directory is new blah/ blah/a/ blah/a/bar blah/a/foo $ mv blah/a blah/b $ ~/src/tar-1.15.1/src/tar -cvf test1.tar --listed-incremental test1.incr blah /home/davee/src/tar-1.15.1/src/tar: blah/b: Directory is new blah/ blah/b/ blah/b/bar blah/b/foo A new tar is indeed the answer it seems. Many thanks, people. Dave. -- Dave Ewart [EMAIL PROTECTED] Computing Manager, Cancer Epidemiology Unit Cancer Research UK / Oxford University PGP: CC70 1883 BD92 E665 B840 118B 6E94 2CFD 694D E370 Get key from http://www.ceu.ox.ac.uk/~davee/davee-ceu-ox-ac-uk.asc N 51.7518, W 1.2016 signature.asc Description: Digital signature
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday, 08.03.2006 at 10:02 -0500, Gene Heskett wrote: > >This package of 'tar' is the current default in Debian/Sarge and is > >updated with various security issues without changing the version > >number, so it's not entirely clear what version is really there under > >the hood. > > Which is exactly why I get this stuff (both tar and amanda) from the > src and build my own. It Just Works(TM). :) Well, that works both ways of course. The latest source might introduce new problems, and at least distro-issued packages have been preconfigured to put the files in 'all the right places', as per the distro's usual specifications ... I'll investigate what Ian has posted about listed-incremental... Dave. -- Dave Ewart [EMAIL PROTECTED] Computing Manager, Cancer Epidemiology Unit Cancer Research UK / Oxford University PGP: CC70 1883 BD92 E665 B840 118B 6E94 2CFD 694D E370 Get key from http://www.ceu.ox.ac.uk/~davee/davee-ceu-ox-ac-uk.asc N 51.7518, W 1.2016 signature.asc Description: Digital signature
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wed, Mar 08, 2006 at 10:31:44AM -0500, Gene Heskett enlightened us: > >I hope you build your own RPMs, since I know you use an RPM based > > system ;-) > > Nope, I use checkinstall for some stuff, but this isn't being done for > these. And they are pinned in yum.conf. Yeah, my systems borked. > Funny thing is, it works fine for everything I want to do, which at > times can best be described as 'an eclectic bunch' of apps. > I know we're drifting a bit off-topic here, but usually what I do for non-mission-critical machines (such as a home workstation) is grab stuff from the latest fedora development repository and rebuild them on my system. That's what I did for tar, and what I do for amanda (even on production systems I do that). Just a suggestion :-) Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday 08 March 2006 10:17, Matt Hyclak wrote: >On Wed, Mar 08, 2006 at 10:02:43AM -0500, Gene Heskett enlightened us: >> >On Wednesday, 08.03.2006 at 07:35 -0500, Gene Heskett wrote: >> >> >On the client being backed-up: >> >> > >> >> >$ tar --version >> >> > >> >> >tar (GNU tar) 1.14 >> >> >> >> We believe this version of tar is borked. Please get the older >> >> 1.13-19, 1.13-25, or the newer 1.15-1. For whatever reason, 1.14 >> >> was only visible on gnu.org for about 5 or 6 weeks, being >> >> replaced with 1.15-1, which for gnu.org, considering the speed >> >> they normally run at, is instantainious. I've been running >> >> 1.15-1 since about a week after it became available without >> >> problems. >> > >> >Interesting. Do you think that the behaviour I'm seeing is solely >> > as a result of 'tar' here? >> > >> >Is the behaviour I describe something that you believe *should* >> > *not* happen with AMANDA when using 'tar'? >> >> I do not know this for a fact. ISTR the file headers it made >> weren't correct and recovery failures were the result. Possibly >> someone else can elaborate here? > >It created proper archives, however it failed extracting archives that > had sparse files in them properly. RedHat still uses this version in > RHEL4, but has backported the fix for this problem from 1.15.1. > >> >This package of 'tar' is the current default in Debian/Sarge and is >> >updated with various security issues without changing the version >> >number, so it's not entirely clear what version is really there >> > under the hood. >> >> Which is exactly why I get this stuff (both tar and amanda) from the >> src and build my own. It Just Works(TM). :) > >I hope you build your own RPMs, since I know you use an RPM based > system ;-) Nope, I use checkinstall for some stuff, but this isn't being done for these. And they are pinned in yum.conf. Yeah, my systems borked. Funny thing is, it works fine for everything I want to do, which at times can best be described as 'an eclectic bunch' of apps. >Matt -- Cheers, Gene People having trouble with vz bouncing email to me should add the word 'online' between the 'verizon', and the dot which bypasses vz's stupid bounce rules. I do use spamassassin too. :-) Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2006 by Maurice Eugene Heskett, all rights reserved.
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday 08 March 2006 06:34, Dave Ewart wrote: > In summary: when a directory is *renamed* the files underneath it are > not considered to have changed This does not appear to happen with CVS tar in listed-incremental mode. Please try upgrading your tar and check if you still see this behaviour. For reference, this is the test I performed: $ # Build CVS tar, chdir to src/. $ mkdir -p test/foo $ touch test/foo/bar $ ./tar -cvf test1.tar --listed-incremental test1.incr test ./tar: test/foo: Directory is new test/ test/foo/ test/foo/bar $ mv test/foo test/baz $ ./tar -cvf test2.tar --listed-incremental test1.incr test ./tar: test/baz: Directory is new test/ test/baz/ test/baz/bar Cheers, --Ian -- Forums for Amanda discussion: http://forums.zmanda.com/
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wed, Mar 08, 2006 at 10:02:43AM -0500, Gene Heskett enlightened us: > >On Wednesday, 08.03.2006 at 07:35 -0500, Gene Heskett wrote: > >> >On the client being backed-up: > >> > > >> >$ tar --version > >> > > >> >tar (GNU tar) 1.14 > >> > >> We believe this version of tar is borked. Please get the older > >> 1.13-19, 1.13-25, or the newer 1.15-1. For whatever reason, 1.14 > >> was only visible on gnu.org for about 5 or 6 weeks, being replaced > >> with 1.15-1, which for gnu.org, considering the speed they normally > >> run at, is instantainious. I've been running 1.15-1 since about a > >> week after it became available without problems. > > > >Interesting. Do you think that the behaviour I'm seeing is solely as > > a result of 'tar' here? > > > >Is the behaviour I describe something that you believe *should* *not* > >happen with AMANDA when using 'tar'? > > > I do not know this for a fact. ISTR the file headers it made weren't > correct and recovery failures were the result. Possibly someone else > can elaborate here? > It created proper archives, however it failed extracting archives that had sparse files in them properly. RedHat still uses this version in RHEL4, but has backported the fix for this problem from 1.15.1. > >This package of 'tar' is the current default in Debian/Sarge and is > >updated with various security issues without changing the version > >number, so it's not entirely clear what version is really there under > >the hood. > > Which is exactly why I get this stuff (both tar and amanda) from the src > and build my own. It Just Works(TM). :) > I hope you build your own RPMs, since I know you use an RPM based system ;-) Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday 08 March 2006 08:43, Dave Ewart wrote: >On Wednesday, 08.03.2006 at 07:35 -0500, Gene Heskett wrote: >> >On the client being backed-up: >> > >> >$ tar --version >> > >> >tar (GNU tar) 1.14 >> >> We believe this version of tar is borked. Please get the older >> 1.13-19, 1.13-25, or the newer 1.15-1. For whatever reason, 1.14 >> was only visible on gnu.org for about 5 or 6 weeks, being replaced >> with 1.15-1, which for gnu.org, considering the speed they normally >> run at, is instantainious. I've been running 1.15-1 since about a >> week after it became available without problems. > >Interesting. Do you think that the behaviour I'm seeing is solely as > a result of 'tar' here? > >Is the behaviour I describe something that you believe *should* *not* >happen with AMANDA when using 'tar'? > I do not know this for a fact. ISTR the file headers it made weren't correct and recovery failures were the result. Possibly someone else can elaborate here? >This package of 'tar' is the current default in Debian/Sarge and is >updated with various security issues without changing the version >number, so it's not entirely clear what version is really there under >the hood. Which is exactly why I get this stuff (both tar and amanda) from the src and build my own. It Just Works(TM). :) In this case I put the new version in /usr/local/bin & reconfigured & rebuilt amanda to look for it there, this so I could switch back if I needed to. But I never did need to. >Dave. -- Cheers, Gene People having trouble with vz bouncing email to me should add the word 'online' between the 'verizon', and the dot which bypasses vz's stupid bounce rules. I do use spamassassin too. :-) Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2006 by Maurice Eugene Heskett, all rights reserved.
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday 08 March 2006 06:34, Dave Ewart wrote: > In summary: when a directory is *renamed* the files underneath it are > not considered to have changed Usually, I would say that this is a limitation of the UNIX filesystem. A directory's modification time can change anytime you add or remove a file in that directory. Obviously it doesn't make sense to re-backup every file in a hierarchy just because you created or removed one. However, in this case you are using gnutar, which creates listed incrementals (I assume you did not configure --without-gnutar-listdir) that can be used to detect the renaming of a directory. Therefore I have forwarded your complaint to the GNU tar folks. Cheers, --Ian -- Forums for Amanda discussion: http://forums.zmanda.com/
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday, 08.03.2006 at 07:35 -0500, Gene Heskett wrote: > >On the client being backed-up: > > > >$ tar --version > > > >tar (GNU tar) 1.14 > > We believe this version of tar is borked. Please get the older > 1.13-19, 1.13-25, or the newer 1.15-1. For whatever reason, 1.14 was > only visible on gnu.org for about 5 or 6 weeks, being replaced with > 1.15-1, which for gnu.org, considering the speed they normally run at, > is instantainious. I've been running 1.15-1 since about a week after > it became available without problems. Interesting. Do you think that the behaviour I'm seeing is solely as a result of 'tar' here? Is the behaviour I describe something that you believe *should* *not* happen with AMANDA when using 'tar'? This package of 'tar' is the current default in Debian/Sarge and is updated with various security issues without changing the version number, so it's not entirely clear what version is really there under the hood. Dave. -- Dave Ewart [EMAIL PROTECTED] Computing Manager, Cancer Epidemiology Unit Cancer Research UK / Oxford University PGP: CC70 1883 BD92 E665 B840 118B 6E94 2CFD 694D E370 Get key from http://www.ceu.ox.ac.uk/~davee/davee-ceu-ox-ac-uk.asc N 51.7518, W 1.2016 signature.asc Description: Digital signature
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday 08 March 2006 06:50, Dave Ewart wrote: >On Wednesday, 08.03.2006 at 11:34 +, Dave Ewart wrote: >> I've just been caught out by something which is either >> >> (a) a limitation of AMANDA and/or tar/dump; >> (b) a bug. > >A follow-up, prompted by a reply from Paul, indicating that I had > indeed forgotten a couple of rather important bits of information! > >- The filesystem being discussed is ext3 > >- AMANDA is using 'tar' (i.e. amanda.conf states: program "GNUTAR") > >On the client being backed-up: > >$ tar --version > >tar (GNU tar) 1.14 We believe this version of tar is borked. Please get the older 1.13-19, 1.13-25, or the newer 1.15-1. For whatever reason, 1.14 was only visible on gnu.org for about 5 or 6 weeks, being replaced with 1.15-1, which for gnu.org, considering the speed they normally run at, is instantainious. I've been running 1.15-1 since about a week after it became available without problems. >Copyright (C) 2004 Free Software Foundation, Inc. >... > >Dave. -- Cheers, Gene People having trouble with vz bouncing email to me should add the word 'online' between the 'verizon', and the dot which bypasses vz's stupid bounce rules. I do use spamassassin too. :-) Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2006 by Maurice Eugene Heskett, all rights reserved.
Re: If most recent backup is not level 0, recovery fails to bring back all files when directories have been renamed
On Wednesday, 08.03.2006 at 11:34 +, Dave Ewart wrote: > I've just been caught out by something which is either > > (a) a limitation of AMANDA and/or tar/dump; > (b) a bug. A follow-up, prompted by a reply from Paul, indicating that I had indeed forgotten a couple of rather important bits of information! - The filesystem being discussed is ext3 - AMANDA is using 'tar' (i.e. amanda.conf states: program "GNUTAR") On the client being backed-up: $ tar --version tar (GNU tar) 1.14 Copyright (C) 2004 Free Software Foundation, Inc. ... Dave. -- Dave Ewart [EMAIL PROTECTED] Computing Manager, Cancer Epidemiology Unit Cancer Research UK / Oxford University PGP: CC70 1883 BD92 E665 B840 118B 6E94 2CFD 694D E370 Get key from http://www.ceu.ox.ac.uk/~davee/davee-ceu-ox-ac-uk.asc N 51.7518, W 1.2016 signature.asc Description: Digital signature