Re: testing a patched tar question.
On Sunday 08 April 2007, [EMAIL PROTECTED] wrote: On Sun, Apr 08, 2007 at 08:29:53PM -0500, Marc Mengel wrote: Couldn't someone hack up a utility to do a major number swapout in the file where tar keeps track of what is in the last incremental/full/etc.? That's what I suggested in http://tech.groups.yahoo.com/group/amanda-users/message/62562 I'm a little busy to work on it at the moment, if you're interested in taking up the task! Dustin I'm all for it, as it could be included in GenesAmandaHelper's as a preliminary task to be run in front of the backup itself. Now the question that comes to mind about such a utility is, what if tar is also sensitive to the change-date of these files, would that not defeat the purpose? I rebooted last night, to the same kernel as before but as a new build because I'd wanted to test something else, and the device-mapper thing moved again, so for 3 nights in a row, amanda has been disturbed and told to restart from square one. My vtapes disk is 96% full, so I had to reduce the size of the vtape back to something reasonably sane for a 180GB partition. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) Humor in the Court: Q. And lastly, Gary, all your responses must be oral. O.K.? What school do you go to? A. Oral. Q. How old are you? A. Oral.
Re: testing a patched tar question.
On Sun, Apr 08, 2007 at 09:43:13PM -0500, [EMAIL PROTECTED] wrote:
On Sun, Apr 08, 2007 at 08:29:53PM -0500, Marc Mengel wrote:
Couldn't someone hack up a utility to do a major number swapout in the
file where tar keeps track of what is in the last incremental/full/etc.?
That's what I suggested in
http://tech.groups.yahoo.com/group/amanda-users/message/62562
I'm a little busy to work on it at the moment, if you're interested in
taking up the task!
Dustin
Not a show stopper, but if someone does take up the task it will be important
to consider various OS's and tar versions so as to avoid linux-specific type
assumptions. As an example, here are two gnutar-lists from different OS's
and I think different tar releases:
834 589377 ./jon/.gconf/desktop/gnome/accessibility
7602561 33040 ./jon/.gconf/apps/epiphany
In each case the second column is the inode number. I'm guessing that the
first column is somehow identifying the device, despite their wildly different
format/sizes. It is the same for each entry in the file, though they could
vary if include's in the DLE definition could pull in things from multiple
file systems.
If the current, new device value that tar will use is not known, it could be
obtained with something like this:
cd head of DLE
mkdir -p _foo/foo
tar -cf /dev/null --listed-incremental=_bar _foo
awk 'NR=2 {print $1}' _bar
rm -r _foo _bar
cd -
Assumes you can write to head of DLE and nothing there is named _foo nor _bar.
--
Jon H. LaBadie [EMAIL PROTECTED]
JG Computing
4455 Province Line Road(609) 252-0159
Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: testing a patched tar question.
On Monday 09 April 2007, Jon LaBadie wrote:
On Sun, Apr 08, 2007 at 09:43:13PM -0500, [EMAIL PROTECTED] wrote:
On Sun, Apr 08, 2007 at 08:29:53PM -0500, Marc Mengel wrote:
Couldn't someone hack up a utility to do a major number swapout in
the file where tar keeps track of what is in the last
incremental/full/etc.?
That's what I suggested in
http://tech.groups.yahoo.com/group/amanda-users/message/62562
I'm a little busy to work on it at the moment, if you're interested in
taking up the task!
Dustin
Not a show stopper, but if someone does take up the task it will be
important to consider various OS's and tar versions so as to avoid
linux-specific type assumptions. As an example, here are two
gnutar-lists from different OS's and I think different tar releases:
834 589377 ./jon/.gconf/desktop/gnome/accessibility
7602561 33040 ./jon/.gconf/apps/epiphany
In each case the second column is the inode number. I'm guessing that
the first column is somehow identifying the device, despite their
wildly different format/sizes. It is the same for each entry in the
file, though they could vary if include's in the DLE definition could
pull in things from multiple file systems.
If the current, new device value that tar will use is not known, it
could be obtained with something like this:
cd head of DLE
mkdir -p _foo/foo
tar -cf /dev/null --listed-incremental=_bar _foo
awk 'NR=2 {print $1}' _bar
rm -r _foo _bar
cd -
Assumes you can write to head of DLE and nothing there is named _foo
nor _bar.
For linux at any rate this is derivable from a simple 'stat /' command.
[EMAIL PROTECTED] linux-2.6.21-rc6]# stat /
File: `/'
Size: 4096Blocks: 16 IO Block: 4096 directory
Device: fe00h/65024dInode: 2 Links: 29
Access: (0755/drwxr-xr-x) Uid: (0/root) Gid: (0/root)
Access: 2007-04-09 11:09:13.0 -0400
Modify: 2007-04-08 14:11:57.0 -0400
Change: 2007-04-08 14:11:57.0 -0400
Subtract the 00h, and you have the 254 its currently sitting at,
apparently subject to the whims of an individual boot. If I build md as
a module, then md will get 254 and device-mapper (dm_mod) will get 253,
and if I build pktcdvd, the packet writing util for burning cd/dvd's, it
will get one of those numbers, shoving device-mapper down to 252. This
number does not change AFAIK, for any file/directory on that particular
disk. To demo:
[EMAIL PROTECTED] linux-2.6.21-rc6]#
stat /usr/local/etc/amanda/Daily/amanda.conf
File: `/usr/local/etc/amanda/Daily/amanda.conf'
Size: 22454 Blocks: 48 IO Block: 4096 regular file
Device: fe00h/65024dInode: 11534709Links: 1
Access: (0644/-rw-r--r--) Uid: ( 501/ amanda) Gid: (6/disk)
Access: 2007-04-09 11:35:51.0 -0400
Modify: 2007-04-09 10:09:02.0 -0400
Change: 2007-04-09 10:09:02.0 -0400
The Device: number is the same.
Question then: Does tar use the expanded hex value given first, or the
decimal version, which isn't quite a 1/1 translation according to mental
hex calculator. Humm, wet ram must need refreshed, kcalc says it does.
And an inspection of the revelant files says its the decimal version,
present as the first number of every line in the file AFTER line 1. So
this is doable by a bash script I believe. But I'll need a nap and 3
more cups before I start that project.
--
Cheers, Gene
There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order.
-Ed Howdershelt (Author)
I don't know, said the voice on the PA, apathetic
bloody planet, I've no sympathy at all.
Re: testing a patched tar question.
On Mon, Apr 09, 2007 at 12:05:01PM -0400, Gene Heskett wrote: Subtract the 00h, and you have the 254 its currently sitting at, apparently subject to the whims of an individual boot. If I build md as a module, then md will get 254 and device-mapper (dm_mod) will get 253, and if I build pktcdvd, the packet writing util for burning cd/dvd's, it will get one of those numbers, shoving device-mapper down to 252. This number does not change AFAIK, for any file/directory on that particular disk. To demo: Is it the case, then, that any block device loaded as a module will get a dynamically generated major device number? Or just devices which don't have a fixed major yet? Device: fe00h/65024dInode: 11534709Links: 1 And an inspection of the revelant files says its the decimal version, present as the first number of every line in the file AFTER line 1. So this is doable by a bash script I believe. But I'll need a nap and 3 more cups before I start that project. Could you send me a copy of your incremental file? AFAICT, they're binary files, so I'm not sure how you're speaking of lines.. If there's private info in there, please use tar to make a new incremental of a non-private directory, rather than munging the file. Dustin -- Dustin J. Mitchell Storage Software Engineer, Zmanda, Inc. http://www.zmanda.com/
Re: testing a patched tar question.
On Monday 09 April 2007, [EMAIL PROTECTED] wrote: On Mon, Apr 09, 2007 at 12:05:01PM -0400, Gene Heskett wrote: Subtract the 00h, and you have the 254 its currently sitting at, apparently subject to the whims of an individual boot. If I build md as a module, then md will get 254 and device-mapper (dm_mod) will get 253, and if I build pktcdvd, the packet writing util for burning cd/dvd's, it will get one of those numbers, shoving device-mapper down to 252. This number does not change AFAIK, for any file/directory on that particular disk. To demo: Is it the case, then, that any block device loaded as a module will get a dynamically generated major device number? Or just devices which don't have a fixed major yet? This is Logical Volume Manager stuffs. I don't believe it effects partitions normally mounted from /etc/fstab, but I could be wrong. Device: fe00h/65024dInode: 11534709Links: 1 And an inspection of the revelant files says its the decimal version, present as the first number of every line in the file AFTER line 1. So this is doable by a bash script I believe. But I'll need a nap and 3 more cups before I start that project. Could you send me a copy of your incremental file? AFAICT, they're binary files, so I'm not sure how you're speaking of lines.. I don't think you are looking at the right file family then. Here, I find a file that points at a directory with no subdirs at all, apparently has only a unix style timestamp in a single line similar to the top line of this snip from /usr/local/var/amanda/gnutar-lists: [EMAIL PROTECTED] gnutar-lists]$ cat coyote_var_0 1175582239 60928 19300353 ./cache/gstreamer-0.8 60928 19169380 ./run/cups 60928 19169448 ./lib/scrollkeeper/az 60928 19595604 ./tmp/RealPlayer/share/locale/ko 60928 19300370 ./cache/yum/updates-source 60928 19235048 ./tmp/kdecache-root/background 60928 19169358 ./cache/man/local However there is in this particular file one anomaly, 60928 19170711 ./lib/pgsql/data/base 18 9355 ./lib/nfs/rpc_pipefs --- 60928 19169412 ./lib/scrollkeeper/am Which is actually a directory with other files in it according to an ls -l. So that's a puzzle, and probably best if not kicked to see if its alive :) Now a cat of the _5 version of that looks like this: [EMAIL PROTECTED] gnutar-lists]$ cat coyote_var_5 1175495769 60928 19169380 ./run/cups 60928 19300353 ./cache/gstreamer-0.8 60928 19169448 ./lib/scrollkeeper/az 60928 19595604 ./tmp/RealPlayer/share/locale/ko 60928 19300370 ./cache/yum/updates-source 60928 19169358 ./cache/man/local And I've NDI why the top two entries are swapped. Another of those sleeping dogs I suspect :) And these are the files given to tar as reference files according to the runtar.$date.debug files I have. If there's private info in there, please use tar to make a new incremental of a non-private directory, rather than munging the file. Dustin The above should give you an idea, nothing precious there that I know of. :) FWIW, those 60928 numbers are the correct ones for a LANANA assigned #238, hex 1238=EEh EE00h=60928 Now I just need a utility that will fix these that don't have that number there and I should be good to go. Hint... -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) Electromagnetic energy loss
Re: testing a patched tar question.
On Monday 09 April 2007, Marc W. Mengel wrote: [EMAIL PROTECTED] wrote: Could you send me a copy of your incremental file? AFAICT, they're binary files, so I'm not sure how you're speaking of lines.. They're 512 byte records with a text line in each. Really. What I am looking are are not, file length is variable and the format is: [EMAIL PROTECTED] gnutar-lists]$ cat coyote_var_5 1175495769 60928 19169380 ./run/cups 60928 19300353 ./cache/gstreamer-0.8 60928 19169448 ./lib/scrollkeeper/az 60928 19595604 ./tmp/RealPlayer/share/locale/ko 60928 19300370 ./cache/yum/updates-source 60928 19169358 ./cache/man/local 60928 19235048 ./tmp/kdecache-root/background 60928 19988737 ./tmp/kdecache-root/kio_help/root/.kde/share/doc 60928 19791895 ./spool/cups-pdf The 60928 is the Device: decimal number from a stat of any file on the drive. see tar/src/incremen.c read_directory_file() and write_directory_file() It looks like one could glue those two routines together and get a routine that would read one file and write another, and, say, change the dev field. the device numbers are the ones returned by stat() in the st_dev field. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) May I ask a question?
Re: testing a patched tar question.
On Fri, Apr 06, 2007 at 08:41:56PM -0400, Gene Heskett wrote: Greetings; I don't know if everyone has been following the trail of tears regarding doing backups with the newer kernel versions, but there is a gotcha coming up. There needs to be a patch applied to the linux kernel that takes the hard drive device major numbers out of the experimental area as defined by a document called LANANA. This move the ide device number from the major 253 to the major 238 in the /dev/major,minor numbers scheme. This *should* not have an effect on tar, but it does. Rebooting to one of these newer kernels, or back to an older kernel if the newer one has been running for a few backup sessions results in the Device: hex/decimal values changing when doing a 'stat' on a file. Tar treats this as an indicator that the data is brand new and must therefore be given a level 0 backup. Of course our typical tape size, affordable by home users such as I, doesn't approach being able to handle a single backup of nearly 50GB, the best we can do is raise heck with our schedule by doing 2 or 3 backup sessions with as large a tapetype size setting as we have combined vtape and holding disk area to handle. My view is that since the inode data doesn't change, and the files timestamps don't change just because I've rebooted to a different kernel, that tar is miss-firing on this when asked to do a --listed-incremental and the reference file furnished was built under a different kernel with a different major device number for the ide/atapi drives. Ok, that's the background as I try to head off what could be an upgrade disaster that I now know about, but which will scare the bajezus out of someone who isn't ready for it. As of 2.6.21-rc6, that patch has been reverted while the details of how to do this with less disturbance are being worked out. I've also queried the guys listed in the tarballs ChangeLog just this afternoon after it became obvious that [EMAIL PROTECTED], a moderated list, wasn't being moderated in a timely manner. I've downloaded, from gnu.org, the latest tarball snapshot, 1.16.2-20070123, and installed a 1 line patch that removes this Device: sensitivity. I also don't know but what I may have opened that proverbial can of worms either. Since nothing in the file or filesystem has actually changed, I view this sensitivity to the device major numbers as a bug, but there may in fact be a very good reason for it that I'm not aware of. If there is, then someone should make me aware of the reasons that this is a part of tars logic. I suspect that a file is defined by the combination of its inode # and device id (maj,min). If you take away consideration of the device id, then lots of files would be defined by the same, non-unique inode #. If all you took away were consideration for the maj device number, then systems with unique maj numbers for ide, sata, scsi, usb, ... could have files with the same remaining identifier (min, inode #). Unless there was a switch to something like the uuid (universal, unique id) scheme (which is hardly universal ;) I suspect we are stuck with device inode # to define a file. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: testing a patched tar question.
On Saturday 07 April 2007, Jon LaBadie wrote: [...] I suspect that a file is defined by the combination of its inode # and device id (maj,min). If you take away consideration of the device id, then lots of files would be defined by the same, non-unique inode #. If all you took away were consideration for the maj device number, then systems with unique maj numbers for ide, sata, scsi, usb, ... could have files with the same remaining identifier (min, inode #). Unless there was a switch to something like the uuid (universal, unique id) scheme (which is hardly universal ;) I suspect we are stuck with device inode # to define a file. Well, my test run wasn't what I expected, so this effort is probably best abandoned. What we are left with, when the linux kernel finally gets this patch for good (its been reverted for now based on my findings, but its a good patch, and really needs to be done at some point even if we do have to play a little 52 pickup by doing an amanda restart from square one, which will fix it when the patch hits production status) we will have to treat amanda as if its a restart from square one, slowly feeding it the big dle's, one per run. This is a case of being forewarned might be forearmed. Not much else we can do at this point. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) Data for intranet got routed through the extranet and landed on the internet.
Re: testing a patched tar question.
On Sat, Apr 07, 2007 at 01:38:12PM -0400, Gene Heskett wrote: What we are left with, when the linux kernel finally gets this patch for good (its been reverted for now based on my findings, but its a good patch, and really needs to be done at some point even if we do have to play a little 52 pickup by doing an amanda restart from square one, which will fix it when the patch hits production status) we will have to treat amanda as if its a restart from square one, slowly feeding it the big dle's, one per run. This is a case of being forewarned might be forearmed. Not much else we can do at this point. One possibility might be to work with the GNU Tar folks to write a script/app to convert the data stored in the list files. Then the Amanda docs could warn about whatever kernel upgrade will trigger this, instructing users to run this script or app against their /gnutar-lists directories simultaneously with the upgrade. I'm still far from having a setup where I could test this sort of thing, but if their response to the idea is that it'd be too much trouble, I may be able to take a first swing at the script, if you (Gene) are willing to test it out. I'm assuming the list file format is a fairly straightforward packed-record format. Dustin -- Dustin J. Mitchell Storage Software Engineer, Zmanda, Inc. http://www.zmanda.com/
Re: testing a patched tar question.
On Saturday 07 April 2007, [EMAIL PROTECTED] wrote: On Sat, Apr 07, 2007 at 01:38:12PM -0400, Gene Heskett wrote: What we are left with, when the linux kernel finally gets this patch for good (its been reverted for now based on my findings, but its a good patch, and really needs to be done at some point even if we do have to play a little 52 pickup by doing an amanda restart from square one, which will fix it when the patch hits production status) we will have to treat amanda as if its a restart from square one, slowly feeding it the big dle's, one per run. This is a case of being forewarned might be forearmed. Not much else we can do at this point. One possibility might be to work with the GNU Tar folks to write a script/app to convert the data stored in the list files. Then the Amanda docs could warn about whatever kernel upgrade will trigger this, instructing users to run this script or app against their /gnutar-lists directories simultaneously with the upgrade. I'm still far from having a setup where I could test this sort of thing, but if their response to the idea is that it'd be too much trouble, I may be able to take a first swing at the script, if you (Gene) are willing to test it out. I'm assuming the list file format is a fairly straightforward packed-record format. Dustin I would be willing to try some of that, provided there was a 'leave it as it found it' option after the testing was done. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) Why do so many foods come packaged in plastic? It's quite uncanny.
testing a patched tar question.
Greetings; I don't know if everyone has been following the trail of tears regarding doing backups with the newer kernel versions, but there is a gotcha coming up. There needs to be a patch applied to the linux kernel that takes the hard drive device major numbers out of the experimental area as defined by a document called LANANA. This move the ide device number from the major 253 to the major 238 in the /dev/major,minor numbers scheme. This *should* not have an effect on tar, but it does. Rebooting to one of these newer kernels, or back to an older kernel if the newer one has been running for a few backup sessions results in the Device: hex/decimal values changing when doing a 'stat' on a file. Tar treats this as an indicator that the data is brand new and must therefore be given a level 0 backup. Of course our typical tape size, affordable by home users such as I, doesn't approach being able to handle a single backup of nearly 50GB, the best we can do is raise heck with our schedule by doing 2 or 3 backup sessions with as large a tapetype size setting as we have combined vtape and holding disk area to handle. My view is that since the inode data doesn't change, and the files timestamps don't change just because I've rebooted to a different kernel, that tar is miss-firing on this when asked to do a --listed-incremental and the reference file furnished was built under a different kernel with a different major device number for the ide/atapi drives. Ok, that's the background as I try to head off what could be an upgrade disaster that I now know about, but which will scare the bajezus out of someone who isn't ready for it. As of 2.6.21-rc6, that patch has been reverted while the details of how to do this with less disturbance are being worked out. I've also queried the guys listed in the tarballs ChangeLog just this afternoon after it became obvious that [EMAIL PROTECTED], a moderated list, wasn't being moderated in a timely manner. I've downloaded, from gnu.org, the latest tarball snapshot, 1.16.2-20070123, and installed a 1 line patch that removes this Device: sensitivity. I also don't know but what I may have opened that proverbial can of worms either. Since nothing in the file or filesystem has actually changed, I view this sensitivity to the device major numbers as a bug, but there may in fact be a very good reason for it that I'm not aware of. If there is, then someone should make me aware of the reasons that this is a part of tars logic. At any rate, this new tar works with my testing scripts when executed insitu in its build tree, and the only fuss when doing a 'make test' in the src tree is a claim that the exclude function failed. That I could tolerate I think, at least long enough to test. To that end, I modified my configure script to point at this new version of tar, and did the usual install here. That appears to have succeeded. What I want to know from all of you, is do I push to get this patch into tar, or how else should we attempt to reduce the pain here? Let me emphasize that once amanda has been able to make all new level 0's of your data, then amanda will settle down and this will be just another bump in the road. So lets talk, please. In the meantime I have a test backup running for effect... -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) At the end of your life there'll be a good rest, and no further activities are scheduled.
Re: testing a patched tar question.
Gene Heskett wrote: Greetings; I don't know if everyone has been following the trail of tears regarding doing backups with the newer kernel versions, but there is a gotcha coming up. There needs to be a patch applied to the linux kernel that takes the hard drive device major numbers out of the experimental area as defined by a document called LANANA. This move the ide device number from the major 253 to the major 238 in the /dev/major,minor numbers scheme. This *should* not have an effect on tar, but it does. Rebooting to one of these newer kernels, or back to an older kernel if the newer one has been running for a few backup sessions results in the Device: hex/decimal values changing when doing a 'stat' on a file. Tar treats this as an indicator that the data is brand new and must therefore be given a level 0 backup. Of course our typical tape size, affordable by home users such as I, doesn't approach being able to handle a single backup of nearly 50GB, the best we can do is raise heck with our schedule by doing 2 or 3 backup sessions with as large a tapetype size setting as we have combined vtape and holding disk area to handle. My view is that since the inode data doesn't change, and the files timestamps don't change just because I've rebooted to a different kernel, that tar is miss-firing on this when asked to do a --listed-incremental and the reference file furnished was built under a different kernel with a different major device number for the ide/atapi drives. Ok, that's the background as I try to head off what could be an upgrade disaster that I now know about, but which will scare the bajezus out of someone who isn't ready for it. As of 2.6.21-rc6, that patch has been reverted while the details of how to do this with less disturbance are being worked out. I've also queried the guys listed in the tarballs ChangeLog just this afternoon after it became obvious that [EMAIL PROTECTED], a moderated list, wasn't being moderated in a timely manner. I've downloaded, from gnu.org, the latest tarball snapshot, 1.16.2-20070123, and installed a 1 line patch that removes this Device: sensitivity. I also don't know but what I may have opened that proverbial can of worms either. Since nothing in the file or filesystem has actually changed, I view this sensitivity to the device major numbers as a bug, but there may in fact be a very good reason for it that I'm not aware of. If there is, then someone should make me aware of the reasons that this is a part of tars logic. At any rate, this new tar works with my testing scripts when executed insitu in its build tree, and the only fuss when doing a 'make test' in the src tree is a claim that the exclude function failed. That I could tolerate I think, at least long enough to test. To that end, I modified my configure script to point at this new version of tar, and did the usual install here. That appears to have succeeded. What I want to know from all of you, is do I push to get this patch into tar, or how else should we attempt to reduce the pain here? Let me emphasize that once amanda has been able to make all new level 0's of your data, then amanda will settle down and this will be just another bump in the road. I would be hesitant to push for that change. However inconvenient it might be as a result of a kernel upgrade, a new major number is a new device, and to ignore it changing is inviting disaster. You would need to look at tar's code to see how it's using the device number, it might have an effect on the --one-file-system option. Frank So lets talk, please. In the meantime I have a test backup running for effect... -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: testing a patched tar question.
On Saturday 07 April 2007, Frank Smith wrote: Gene Heskett wrote: Greetings; I don't know if everyone has been following the trail of tears regarding doing backups with the newer kernel versions, but there is a gotcha coming up. There needs to be a patch applied to the linux kernel that takes the hard drive device major numbers out of the experimental area as defined by a document called LANANA. This move the ide device number from the major 253 to the major 238 in the /dev/major,minor numbers scheme. This *should* not have an effect on tar, but it does. Rebooting to one of these newer kernels, or back to an older kernel if the newer one has been running for a few backup sessions results in the Device: hex/decimal values changing when doing a 'stat' on a file. Tar treats this as an indicator that the data is brand new and must therefore be given a level 0 backup. Of course our typical tape size, affordable by home users such as I, doesn't approach being able to handle a single backup of nearly 50GB, the best we can do is raise heck with our schedule by doing 2 or 3 backup sessions with as large a tapetype size setting as we have combined vtape and holding disk area to handle. My view is that since the inode data doesn't change, and the files timestamps don't change just because I've rebooted to a different kernel, that tar is miss-firing on this when asked to do a --listed-incremental and the reference file furnished was built under a different kernel with a different major device number for the ide/atapi drives. Ok, that's the background as I try to head off what could be an upgrade disaster that I now know about, but which will scare the bajezus out of someone who isn't ready for it. As of 2.6.21-rc6, that patch has been reverted while the details of how to do this with less disturbance are being worked out. I've also queried the guys listed in the tarballs ChangeLog just this afternoon after it became obvious that [EMAIL PROTECTED], a moderated list, wasn't being moderated in a timely manner. I've downloaded, from gnu.org, the latest tarball snapshot, 1.16.2-20070123, and installed a 1 line patch that removes this Device: sensitivity. I also don't know but what I may have opened that proverbial can of worms either. Since nothing in the file or filesystem has actually changed, I view this sensitivity to the device major numbers as a bug, but there may in fact be a very good reason for it that I'm not aware of. If there is, then someone should make me aware of the reasons that this is a part of tars logic. At any rate, this new tar works with my testing scripts when executed insitu in its build tree, and the only fuss when doing a 'make test' in the src tree is a claim that the exclude function failed. That I could tolerate I think, at least long enough to test. To that end, I modified my configure script to point at this new version of tar, and did the usual install here. That appears to have succeeded. What I want to know from all of you, is do I push to get this patch into tar, or how else should we attempt to reduce the pain here? Let me emphasize that once amanda has been able to make all new level 0's of your data, then amanda will settle down and this will be just another bump in the road. I would be hesitant to push for that change. However inconvenient it might be as a result of a kernel upgrade, a new major number is a new device, and to ignore it changing is inviting disaster. You would need to look at tar's code to see how it's using the device number, it might have an effect on the --one-file-system option. Frank So lets talk, please. In the meantime I have a test backup running for effect... And the test run wasn't as I anticipated, leaving behind some timestamps that were invalid when I reverted to the usual tar. It also appeared to want to back it all up in one swell foop. So maybe this trail is best abandoned after all. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) If I could drop dead right now, I'd be the happiest man alive! -- Samuel Goldwyn
Re: tar question
On Fri, 9 Jan 2004 at 9:52am, s97sa4 wrote Hi Please can you help! Please email the list itself ([EMAIL PROTECTED]) and not members directly -- you'll get more responses and not annoy people. I would like to configure Amanda to use Sun tar rather than gnutar to preserve ACL on all files that are backed up. I was gnutar does not preserve all info for ACL. I highly doubt Sun's tar is compatible with all the command line options amanda throws at tar. If you can (i.e. you are backing up partitions), use ufsdump instead. If not, what I do on my Linux systems is run a script that dumps all the ACLs to a file, from which they can be restored (getfacl -R on Linux). -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: tar question
On Fri, Jan 09, 2004 at 06:52:37AM -0500, Joshua Baker-LePain wrote: On Fri, 9 Jan 2004 at 9:52am, s97sa4 wrote Hi Please can you help! Please email the list itself ([EMAIL PROTECTED]) and not members directly -- you'll get more responses and not annoy people. I would like to configure Amanda to use Sun tar rather than gnutar to preserve ACL on all files that are backed up. I was gnutar does not preserve all info for ACL. I highly doubt Sun's tar is compatible with all the command line options amanda throws at tar. If you can (i.e. you are backing up partitions), use ufsdump instead. If not, what I do on my Linux systems is run a script that dumps all the ACLs to a file, from which they can be restored (getfacl -R on Linux). A big problem would be amanad's use of gnu-style options rather than single letter options. I.e. --file foobar rather than -f foobar. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
tar question
We've been using amanda 2.4.4 for a while now, but we are still having a few problems with tar and solaris. We had been getting this error on a few of solaris machines: --snip-- /-- homes01 /disk1/homes lev 0 FAILED [/usr/local/sfw/bin/gtar returned 2] sendbackup: start [homes01:/disk1/homes level 0] sendbackup: info BACKUP=/usr/local/sfw/bin/gtar sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/local/sfw/bin/gtar -f... - sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end ? gtar: time_t value 18446744073616021196 too large (max=68719476735) | Total bytes written: 1442283520 ? gtar: Error exit delayed from previous errors sendbackup: error [/usr/local/sfw/bin/gtar returned 2] \ --/snip-- We were using a slightly older version of gtar, so we installed 1.13.92. Now we are getting this for nearly all of our solaris machines: --snip-- /-- homes01/disk1/homes lev 0 STRANGE sendbackup: start [homes01:/disk1/homes level 0] sendbackup: info BACKUP=/usr/local/sfw/bin/gtar sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/local/sfw/bin/gtar -f... - sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end ? gtar: Removing leading `./' from member names | Total bytes written: 10240 (10KiB, 81KiB/s) sendbackup: size 10 sendbackup: end \ --/snip-- So I went looking around to see what version of tar I should be using with Amanda 2.4.4, but I couldn't find a definite answer anywhere. Have I overlooked it? Either way, could someone point me in the right direction? Thanks, --Charlie Wiseman System Administrator Computer Science Dept. University of Tennessee, Knoxville
Re: tar question
On Mon, 5 Jan 2004 at 2:37pm, Charlie Wiseman wrote We were using a slightly older version of gtar, so we installed 1.13.92. Now we are getting this for nearly all of our solaris machines: 1.13.25 is what you want. The newer versions are not compatible with amanda. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
gnu tar question?
hi everyone again, Does anybody know how can i use gnu tar to make incremental backups? Is it posible? Thanks. --- Javier Fernández Pérez Servicio de Informática Corporativa D.G. de Telecomunicaciones y Transportes Consejería de Fomento - Junta de Castilla y León Rigoberto Cortejoso, 14. 47014 Valladolid (Spain) e-mail: [EMAIL PROTECTED] ---
Re: gnu tar question?
On 15 Jan 2002 at 1:43pm, [EMAIL PROTECTED] wrote Does anybody know how can i use gnu tar to make incremental backups? Is it posible? Just tell amanda to use 'program GNUTAR' in your dumptype. amanda will take care of the rest. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: gnu tar question?
[EMAIL PROTECTED] wrote: hi everyone again, Does anybody know how can i use gnu tar to make incremental backups? Is it posible? In the amanda.conf file there is a section called 'dumptype'. All the different kind of dumps are defined. GNUTAR dumps usually have names that include 'tar'. In the disklist file, select a dumptype that includes tar in its name: client.localnet /home/users comp-user-tar That should do it. Thanks. --- Javier Fernández Pérez Servicio de Informática Corporativa D.G. de Telecomunicaciones y Transportes Consejería de Fomento - Junta de Castilla y León Rigoberto Cortejoso, 14. 47014 Valladolid (Spain) e-mail: [EMAIL PROTECTED] --- -- David T. Smith PGP Fingerprint: 7B01 0086 BC4E C092 5348 B9AE E79A 07F2 9E59 29C2 ph: 1 203 364 1796 fax: 1 203 364 1795 cell: 1 203 770 1685 E-mail [EMAIL PROTECTED]
Re: tar question
On Fri, May 11, 2001 at 12:51:43AM -0400, Jon LaBadie wrote: On Thu, May 10, 2001 at 08:41:24PM -0400, Ray Shaw wrote: On that note, it is annoying that if I have an exclude file, say /etc/amanda/exclude.home, running amanda with that file produces different results than: tar cvf /dev/null /home -X /etc/amanda/exclude.home I don't think amanda tar's /home, I think it cd's to /home and tar's . (current dir). Thus your exclude file should be relative to ., not /home. Ah, yes, that's correct. Thanks for pointing that out; it was the first step in creating the Tower of Globbing in my last post :) For testing, I now use: cd /home tar cvf /dev/null . -X /root/exclude moo and then look at the contents of moo. -- --Ray - Sotto la panca la capra crepa sopra la panca la capra campa
Re: tar question
On Thu, May 10, 2001 at 08:36:07PM -0500, John R. Jackson wrote: ... I'd like to back up at least their mail, and probably web directories. ... I've been trying to do this with exclude lists, but I haven't hit the solution yet. ... Wow! I wouldn't have had the guts to try this with exclude lists. They give me (real :-) headaches just trying to do normal things :-). Brave or stupid, I've gotten it working with some help from a LUG member on my sh globbing. If you have a line like so in your disklist: myhost /home myhost-home and the myhost-home dumptype is calling /etc/amanda/exclude.myhost-home, which contains this: ./*/? ./*/?? ./*/??? ./*/?* ./*/[!mM]??? ./*/[Mm][!a]?? ./*/[Mm]a[!i]? ./*/[Mm]ai[!l] Then /home/*/Mail and /home/*/mail will be backed up. This creation of evil could easily be extended to back up /home/*/public_html, too, or whatever else you wanted. Maybe you could do it with inclusion instead of exclusion? Take a look at: ftp://gandalf.cc.purdue.edu/pub/amanda/gtar-wrapper.* During the pre_estimate you could run find to gather a list of what you want to back up, squirrel that away and then pass it to gtar for both the estimate and the real run. Yes, but that's a compile-time option, right? As I'm a university student and will be replaced in a few years, I don't want my successor to be confused when apt-get dist-upgrade breaks the /home backup :) Of course, I hope we have a bigger tape drive by then... -- --Ray - Sotto la panca la capra crepa sopra la panca la capra campa
tar question
I'm using amanda 2.4.1p1 (the version included in Debian potato). Currently, the /home partition on our main user system is too large to back up fully (we have only a small tape drive). Rather than completely leave users out in the cold, I'd like to back up at least their mail, and probably web directories. So: /home/*/[M,m]ail /home/*/public_html I've been trying to do this with exclude lists, but I haven't hit the solution yet. This: [^*/public_html] at least, does not work. On that note, it is annoying that if I have an exclude file, say /etc/amanda/exclude.home, running amanda with that file produces different results than: tar cvf /dev/null /home -X /etc/amanda/exclude.home Why is this? Thanks for any help, -- --Ray - Sotto la panca la capra crepa sopra la panca la capra campa
Re: tar question
... I'd like to back up at least their mail, and probably web directories. ... I've been trying to do this with exclude lists, but I haven't hit the solution yet. ... Wow! I wouldn't have had the guts to try this with exclude lists. They give me (real :-) headaches just trying to do normal things :-). Maybe you could do it with inclusion instead of exclusion? Take a look at: ftp://gandalf.cc.purdue.edu/pub/amanda/gtar-wrapper.* During the pre_estimate you could run find to gather a list of what you want to back up, squirrel that away and then pass it to gtar for both the estimate and the real run. FYI, there is also an exclusion test script there: ftp://gandalf.cc.purdue.edu/pub/amanda/gtartest-exclude ... that might make figuring out the exclude list easier if you want to continue down that path. On that note, it is annoying that if I have an exclude file, say /etc/amanda/exclude.home, running amanda with that file produces different results than: tar cvf /dev/null /home -X /etc/amanda/exclude.home Why is this? What's different? I assume different files are listed/backed up? Who does more, Amanda or cvf? Is Amanda a subset of cvf, or are they all mixed up? Was Amanda doing a level 0? Are you sure you used the same tar that Amanda does (look at a /tmp/amanda/amandad*debug file for the tar path Amanda uses on that client). Amanda runs GNU tar in incremental backup mode. Take a look at a /tmp/amanda/sendbackup*debug file on the client to see the extra arguments involved. I would expect the lists to be the same if you do a full dump with Amanda (in which case the listed increment file will be empty) and with cvf, but otherwise all bets are off as it would also depend on what files had been modified. --Ray John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: tar question
On Thu, May 10, 2001 at 08:41:24PM -0400, Ray Shaw wrote: I'm using amanda 2.4.1p1 (the version included in Debian potato). Currently, the /home partition on our main user system is too large to back up fully (we have only a small tape drive). Rather than completely leave users out in the cold, I'd like to back up at least their mail, and probably web directories. So: /home/*/[M,m]ail /home/*/public_html I've been trying to do this with exclude lists, but I haven't hit the solution yet. This: [^*/public_html] at least, does not work. On that note, it is annoying that if I have an exclude file, say /etc/amanda/exclude.home, running amanda with that file produces different results than: tar cvf /dev/null /home -X /etc/amanda/exclude.home I don't think amanda tar's /home, I think it cd's to /home and tar's . (current dir). Thus your exclude file should be relative to ., not /home. jl -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing [EMAIL PROTECTED] 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
