Re: tar backups not doing incrementals
On Tue, Jul 02, 2019 at 18:22:20 -0700, Jim Kusznir wrote: > It appears that after commenting out most of the DLEs and forcing a full on > a set that can actually fit on a single tape, following backups are > correctly incrementing. I'm still working my way through the DLEs Okay, that lends support to the theory that the problem is something related to rebooting (as opposed to something more dynamic/real-time). > Perhaps it was a device number changing or somethingAlthough the > confusing thing that comes up as I think about it more...The client is a > "FreeBSD Jail" running on a FreeNAS system. That means that the devices > are hidden from the jail; the filesystem is "just there" (due to jail > settings), so it doesn't see any /dev entries or similar. To the jail, > everything is just a single filesystem with no device backing. So I don't > know if it could be device minor changing, etc. I don't know what else it > could be, though As I mentioned before I am not familiar with FreeNAS and FreeBSD Jails, but all Unix filesystem have a device number assigned to them as part of the basic "API" of being a filesystem, and by default GNU tar checks that number as part of the scanning-for-changes it does when preforming an incremental backup. So, even though the "physical" device is hidden by the "jail" infrastructure, there must be some device number assigned to the filesystem as it is seen from within the jail. In my June 18 email I asked if the "stat" command was available with the jail... and I still think that's the easiest way to find out what's going on, if it is available. If not, I'm curious if the jail really does not have any /dev directory at all? Or is it rather that there are virtualized entries found there? (For example, what does "df /path/to/a/directory/" show when run from within the jail... and assuming a device-like name showes up in the output, what does "ls -l /dev/" say?) > > The hardware (mac address) of the amanda SERVER did change, but it kept the > same IP address (after an initial snafu), as the hard drive was > transplanted into a system with an identical motherboard/cpu. The first > run attempt failed due to security checks, which I traced down to an IP > change (DHCP), but for which was corrected before the first full run. I > don't expect this to happen again. Unless that first failed attempt actually managed to do a partial run and corrupt the GNU tar snapshot files on the client (which seems unlikely), I don't think these networking problems on the server side are related to the incremental problems on the client side. Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: tar backups not doing incrementals
So backups are moving along. They are large, so it takes time (one run is 2-3 days, for example -- That's 1 tape!). It appears that after commenting out most of the DLEs and forcing a full on a set that can actually fit on a single tape, following backups are correctly incrementing. I'm still working my way through the DLEs (probably about 1/3 the way through what I was backing up before the glitch). It will literally take me 2+ months to correct / recover from, so I really cannot afford for this to happen again Perhaps it was a device number changing or somethingAlthough the confusing thing that comes up as I think about it more...The client is a "FreeBSD Jail" running on a FreeNAS system. That means that the devices are hidden from the jail; the filesystem is "just there" (due to jail settings), so it doesn't see any /dev entries or similar. To the jail, everything is just a single filesystem with no device backing. So I don't know if it could be device minor changing, etc. I don't know what else it could be, though The hardware (mac address) of the amanda SERVER did change, but it kept the same IP address (after an initial snafu), as the hard drive was transplanted into a system with an identical motherboard/cpu. The first run attempt failed due to security checks, which I traced down to an IP change (DHCP), but for which was corrected before the first full run. I don't expect this to happen again. --Jim On Sun, Jun 16, 2019 at 3:40 PM Nathan Stratton Treadway wrote: > On Sat, Jun 15, 2019 at 20:35:30 +0200, Uwe Menges wrote: > > On 6/15/19 6:43 PM, Jim Kusznir wrote: > > > Ever since, any backup is run as a level 0 (even though it reports > doing > > > a level 1, 2, etc). Everything runs as 0. > > > > > > How do I fix this? > > > > I fixed a similar symptom on my system with > > property "CHECK-DEVICE" "NO" > > (using amgtar), which hands "--no-check-device" to GNU tar, > > which makes it no longer think the toplevel dir has been renamed, > > so it doesn't store all files in its incremental. > > Yes, if the device for the filesystems in question are changing from > boot to boot, then using the --no-check-device option is necessary to > allow the incrementals to work as expected across reboots. But I > believe that option is not available for the GNUTAR application, so > assuming Jim is indeed using GNUTAR then it seems worth making sure > that's actually his problem before trying to implement that particular > solution > > > I think what happened to me is that some LVM volumes got mounted with a > > different minor number, triggering that. > > > > Given that LVM is really common nowadays, I wonder if that option should > > be default.. I can't think of a use case where the current default is > > preferable, at the moment. > > Yes. In particular, as long as --one-file-system is passed to GNU tar > (the default situation in Amanda) then every file backed up by that run > of tar is going to be on one single device, and the check-device > operation when building the list of incremental changes is never going > to be useful. (And obviously, in the case of changed device numbers for > the filesystem, the check is actually harmful). > > I don't know about changing the actual default in the amgtar application > itself at this point in time, but perhaps it at least makes sense to > recommend disabling check-device in the example amanda.conf and other > documentation > > Nathan > > > > Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region > Ray Ontko & Co. - Software consulting services - > http://www.ontko.com/ > GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 > Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239 >
Re: tar backups not doing incrementals
On Tue, Jun 18, 2019 at 00:13:40 -0400, Nathan Stratton Treadway wrote: > Okay, I'm thinking a way forward from here is to try running a dump run > with just a few of your DLEs, so that they actually fit on a tape and > those dumps run to completion. You can either do this by commenting out Jim, just curious if you had had a chance to investigate your GNU-tar-incremental problems any further? Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: tar backups not doing incrementals
I am sorry, did not follow this thread but have had similar problems (level 1 backup behave as level 0 and thus back-upped all files), so my suggestion might be mentioned already. My amanda problem only happened with an ntfs formatted disk partition and at the end I found that adding the option noatime in the mount cured it. It looked as if atime did modify although it should not (but my computer has daily reboots) /dev/sdc1 /home/xxx/yyy ntfs-3g \ defaults,uid=1000,gid=100,umask=0022,noatime,nofail,rw 0 0 Charles On Tue, 18 Jun 2019 00:13:40 -0400 Nathan Stratton Treadway wrote: > On Mon, Jun 17, 2019 at 18:13:30 -0700, Jim Kusznir wrote: > > The backups are not completing because the level 0 problem: planner > > thinks its running a backup that can complete, but ends up with 10x > > the data that a single run can handle, and it fills up the tape and > > the holding disk (which is a little more than a full tape), and > > then the run suspends until I kill it off with amcleankup -k. So > > that explains the new files. > > [...] > > I guess its possible the minor number is changing, but that too > > seems unlikely. > > > > At this point, I also am not sure what is the best way to recover. > > I have a full holding disk (mostly of incomplete backups), and > > the .new gtar lists, etc. It seems like a waste of a tape (and 26 > > hours to fill said tape) to put a bunch of partial backups > > Okay, I'm thinking a way forward from here is to try running a dump > run with just a few of your DLEs, so that they actually fit on a tape > and those dumps run to completion. You can either do this by > commenting out all but a few DLEs in your disklist file, or by > passing a few host/disk expressions on the amdump command line. > > I'd probably start with just one or two DLEs, and be sure to do a > "amadmin ... force" on those DLEs beforehand so they run as level 0 > dumps. > > I don't believe the .new files left out there in the snapshot > directory will cause a problem for you -- I"m pretty sure Amanda will > just overwrite them the next time it tries to use the level in > question. > > You probably do need to clear room in your holding disk so that future > dump runs have space to work. Can you run amflush to clear them > complete ones to tape? After that, it's probably safe to just delete > the incomplete ones that remain (on the theory that you will getting > complete backups of the DLEs in question before too long...) > > > Once you get a DLE or two to complete successfully, then you can try > running those same DLEs again, and see if those incremental are the > expected size > > (Obviously with this approach the next two tapes you use will be > mostly empty, but the advantage is that the tests will run pretty > quickly, and hopefully they will go a long way toward narrowing down > the source of the problem) > > > Meanwhile, just to start investigating the possibility that it is a > device number problem: does the "stat" command on the FreeNAS box show > you device-number info? On my Linux box the stat output looks like: > $ stat .profile >File: .profile > Size: 675 Blocks: 8 IO Block: 4096 regular > file Device: 901h/2305d Inode: 83583 Links: 1 > Access: (0640/-rw-r-) Uid: ( 1001/nathanst) Gid: > ( 1001/nathanst) Access: 2019-06-17 10:22:00.630973405 -0400 > Modify: 2016-05-20 17:58:08.467463415 -0400 > Change: 2017-06-07 10:19:38.467463415 -0400 >Birth: - > > If you run "stat" on some file that's included in one of your DLEs, > and the output has the "Device: 901h/2305d" part (or something > similar), then we should be able to use that to check to see if the > device number is causing the problem. > > Along those lines, does the FreNAS shell environment have Perl > installed? Or can you easily copy a snapshot file over to a machine > that has Perl? > > Nathan > > > Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic > region Ray Ontko & Co. - Software consulting services - > http://www.ontko.com/ GPG Key: > http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key > fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239 On Sun, 16 Jun 2019 18:35:20 -0400 Nathan Stratton Treadway wrote: > On Sat, Jun 15, 2019 at 20:35:30 +0200, Uwe Menges wrote: > > On 6/15/19 6:43 PM, Jim Kusznir wrote: > > > Ever since, any backup is run as a level 0 (even though it > > > reports doing a level 1, 2, etc). Everything runs as 0. > > > > > > How do I fix this? > > > > I fixed a similar symptom on my system with > > property "CHECK-DEVICE" "NO" > > (using amgtar), which hands "--no-check-device" to GNU tar, > > which makes it no longer think the toplevel dir has been renamed, > > so it doesn't store all files in its
Re: tar backups not doing incrementals
On Mon, Jun 17, 2019 at 18:13:30 -0700, Jim Kusznir wrote: > The backups are not completing because the level 0 problem: planner thinks > its running a backup that can complete, but ends up with 10x the data that > a single run can handle, and it fills up the tape and the holding disk > (which is a little more than a full tape), and then the run suspends until > I kill it off with amcleankup -k. So that explains the new files. [...] > I guess its possible the minor number is changing, but that too seems > unlikely. > > At this point, I also am not sure what is the best way to recover. I have > a full holding disk (mostly of incomplete backups), and the .new gtar > lists, etc. It seems like a waste of a tape (and 26 hours to fill said > tape) to put a bunch of partial backups Okay, I'm thinking a way forward from here is to try running a dump run with just a few of your DLEs, so that they actually fit on a tape and those dumps run to completion. You can either do this by commenting out all but a few DLEs in your disklist file, or by passing a few host/disk expressions on the amdump command line. I'd probably start with just one or two DLEs, and be sure to do a "amadmin ... force" on those DLEs beforehand so they run as level 0 dumps. I don't believe the .new files left out there in the snapshot directory will cause a problem for you -- I"m pretty sure Amanda will just overwrite them the next time it tries to use the level in question. You probably do need to clear room in your holding disk so that future dump runs have space to work. Can you run amflush to clear them complete ones to tape? After that, it's probably safe to just delete the incomplete ones that remain (on the theory that you will getting complete backups of the DLEs in question before too long...) Once you get a DLE or two to complete successfully, then you can try running those same DLEs again, and see if those incremental are the expected size (Obviously with this approach the next two tapes you use will be mostly empty, but the advantage is that the tests will run pretty quickly, and hopefully they will go a long way toward narrowing down the source of the problem) Meanwhile, just to start investigating the possibility that it is a device number problem: does the "stat" command on the FreeNAS box show you device-number info? On my Linux box the stat output looks like: $ stat .profile File: .profile Size: 675 Blocks: 8 IO Block: 4096 regular file Device: 901h/2305d Inode: 83583 Links: 1 Access: (0640/-rw-r-) Uid: ( 1001/nathanst) Gid: ( 1001/nathanst) Access: 2019-06-17 10:22:00.630973405 -0400 Modify: 2016-05-20 17:58:08.467463415 -0400 Change: 2017-06-07 10:19:38.467463415 -0400 Birth: - If you run "stat" on some file that's included in one of your DLEs, and the output has the "Device: 901h/2305d" part (or something similar), then we should be able to use that to check to see if the device number is causing the problem. Along those lines, does the FreNAS shell environment have Perl installed? Or can you easily copy a snapshot file over to a machine that has Perl? Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: tar backups not doing incrementals
Thanks for the reply. The backups are not completing because the level 0 problem: planner thinks its running a backup that can complete, but ends up with 10x the data that a single run can handle, and it fills up the tape and the holding disk (which is a little more than a full tape), and then the run suspends until I kill it off with amcleankup -k. So that explains the new files. The big question still remains is why is a level 1 or 2 backup (which by all accounts, amreport / amstatus show that it IS actually doing a level 1 or 2, it just happens to be 1%+ of what it expected) is actually grabbing ALL files and not changed files like it should... I guess its possible the minor number is changing, but that too seems unlikely. The amclient is technically running in a FreeBSD "jail" (essentially a "ultra-lightweight VM") as required by FreeNAS, as anything installed in the main FreeNAS applaince is wiped next upgrade. Jails are how you install additional packages in a long-term way. The jail does have access to the underlying filesystem from FreeNAS. Maybe I do need that option to tell it not to pay attention to devices...but I thought tar would go by file path anyway, and any changes in underlying devices wouldn't have a difference... At this point, I also am not sure what is the best way to recover. I have a full holding disk (mostly of incomplete backups), and the .new gtar lists, etc. It seems like a waste of a tape (and 26 hours to fill said tape) to put a bunch of partial backups If it properly does the levels, then the backup run will fit on a single tape. If it does not honor the levels, then it would take like 5 or 7 tapes to do the run, and I do not have a tape changer (nor do I have the luxury to do unneeded fulls of everything anytime Amanda pukes at something). My data is too big (but not big enough to have a large library, multiple tape drives, and such...Big data on a budget backup system :( ) --Jim On Mon, Jun 17, 2019 at 11:35 AM Nathan Stratton Treadway < natha...@ontko.com> wrote: > On Mon, Jun 17, 2019 at 10:22:10 -0700, Jim Kusznir wrote: > > The client I'm trying to back up is a FreeNAS appliance, so its not > LVM. I > > You are running the Amanda client software directly on the FreeNAS > appliance, right? And also have shell access on the box to produce your > "ls" output, etc? > > > don't think the devices are changing boot to boot, but that's possible. > > Actually, now that I think about it, I don't think it is > possiblethere > > is only one device, and I'm sub-dividing that with a bunch of > > I am not familiar with FreeNAS so I don't know how it is set up, but > note that the fact there is only one large filesystem doesn't actually > mean that the block device contianing that filesystem necessarily gets > the same (minor) device number each boot. However, this particular > issue should only cause problems for Amanda across reboots or other > occations when the block device would be re-detected, so at the very > least it would appear you ahve something else going on as well... and in > particular tracking down the .new files definitely seems the > higher-priority track of investigation at the moment... > > > -rw--- 1 amanda amanda 3695323 May 7 02:04 > amclient-tdriveCYS-General_0 > > -rw--- 1 amanda amanda 3767257 May 31 23:46 > amclient-tdriveCYS-General_1 > > -rw--- 1 amanda amanda 3695323 Jun 12 20:13 > amclient-tdriveCYS-General_1.new > > -rw--- 1 amanda amanda 9061561 May 8 04:43 > amclient-tdriveCreativeImage_0 > > -rw--- 1 amanda amanda 9061649 May 31 22:47 > amclient-tdriveCreativeImage_1 > > -rw--- 1 amanda amanda 9061561 Jun 13 20:51 > amclient-tdriveCreativeImage_1.new > > > > I'm bothered by the .new extensions, and I don't know why they are there. > > But those are in fact volumes that I've been getting level 0's of even > > though it claims its a level 1 or 2. > > Yes, that's not a good sign. GNU tar updates the snapshot file passed > in to it as it runs, so before invoking tar Amanda makes a copy of the > previous level's snapshot file (over to the file with the .new > extension), and when tar is finished Amanda renames the file to the > proper final name. So the fact that you have .new files still out there > means the dump is not runnning to completion somehow. > > (Am I correct you last ran a dump on Jun 12 or 13 or so, and the job was > no longer running at the point you took that directory listing?) > > So probably the next step is to look for the Amanda client debug files > (e.g. /var/log/amanda/client// on my Ubuntu boxes) and look > through the sendbackup.201906*.debug and runtar.201906*.debug files to > see if you can find any indication of what's causing the jobs to fail. > > Also check the ...//amdump.2019* file (on the *Amanda server* > side) to see if there are any messages in there regarding these DLEs. > > The Amanda Mail Report email doesn't list any error messages?
Re: tar backups not doing incrementals
On Mon, Jun 17, 2019 at 10:22:10 -0700, Jim Kusznir wrote: > The client I'm trying to back up is a FreeNAS appliance, so its not LVM. I You are running the Amanda client software directly on the FreeNAS appliance, right? And also have shell access on the box to produce your "ls" output, etc? > don't think the devices are changing boot to boot, but that's possible. > Actually, now that I think about it, I don't think it is possiblethere > is only one device, and I'm sub-dividing that with a bunch of I am not familiar with FreeNAS so I don't know how it is set up, but note that the fact there is only one large filesystem doesn't actually mean that the block device contianing that filesystem necessarily gets the same (minor) device number each boot. However, this particular issue should only cause problems for Amanda across reboots or other occations when the block device would be re-detected, so at the very least it would appear you ahve something else going on as well... and in particular tracking down the .new files definitely seems the higher-priority track of investigation at the moment... > -rw--- 1 amanda amanda 3695323 May 7 02:04 > amclient-tdriveCYS-General_0 > -rw--- 1 amanda amanda 3767257 May 31 23:46 > amclient-tdriveCYS-General_1 > -rw--- 1 amanda amanda 3695323 Jun 12 20:13 > amclient-tdriveCYS-General_1.new > -rw--- 1 amanda amanda 9061561 May 8 04:43 > amclient-tdriveCreativeImage_0 > -rw--- 1 amanda amanda 9061649 May 31 22:47 > amclient-tdriveCreativeImage_1 > -rw--- 1 amanda amanda 9061561 Jun 13 20:51 > amclient-tdriveCreativeImage_1.new > > I'm bothered by the .new extensions, and I don't know why they are there. > But those are in fact volumes that I've been getting level 0's of even > though it claims its a level 1 or 2. Yes, that's not a good sign. GNU tar updates the snapshot file passed in to it as it runs, so before invoking tar Amanda makes a copy of the previous level's snapshot file (over to the file with the .new extension), and when tar is finished Amanda renames the file to the proper final name. So the fact that you have .new files still out there means the dump is not runnning to completion somehow. (Am I correct you last ran a dump on Jun 12 or 13 or so, and the job was no longer running at the point you took that directory listing?) So probably the next step is to look for the Amanda client debug files (e.g. /var/log/amanda/client// on my Ubuntu boxes) and look through the sendbackup.201906*.debug and runtar.201906*.debug files to see if you can find any indication of what's causing the jobs to fail. Also check the ...//amdump.2019* file (on the *Amanda server* side) to see if there are any messages in there regarding these DLEs. The Amanda Mail Report email doesn't list any error messages? Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: tar backups not doing incrementals
The client I'm trying to back up is a FreeNAS appliance, so its not LVM. I don't think the devices are changing boot to boot, but that's possible. Actually, now that I think about it, I don't think it is possiblethere is only one device, and I'm sub-dividing that with a bunch of include/exclude lists. Here's my ls of my gnutar-lists: -rw--- 1 amanda amanda 0 May 10 08:28 amclient-tdriveCYS-2017-Masters-A-D_0.new -rw--- 1 amanda amanda 0 May 10 08:05 amclient-tdriveCYS-2017-Masters-E-M_0.new -rw--- 1 amanda amanda 0 May 10 08:07 amclient-tdriveCYS-2017-Masters-N-Q_0.new -rw--- 1 amanda amanda 0 May 11 12:06 amclient-tdriveCYS-2017-Masters-R-S_0.new -rw--- 1 amanda amanda778788 May 9 22:46 amclient-tdriveCYS-2017-Masters-rest_0 -rw--- 1 amanda amanda 9893530 May 12 13:22 amclient-tdriveCYS-2017-Sessions-A-D_0 -rw--- 1 amanda amanda 9886842 May 31 21:57 amclient-tdriveCYS-2017-Sessions-A-D_1 -rw--- 1 amanda amanda 9893530 Jun 13 20:52 amclient-tdriveCYS-2017-Sessions-A-D_1.new -rw--- 1 amanda amanda 8867217 May 12 05:55 amclient-tdriveCYS-2017-Sessions-E-J_0 -rw--- 1 amanda amanda 8867312 May 31 22:07 amclient-tdriveCYS-2017-Sessions-E-J_1 -rw--- 1 amanda amanda 7292999 May 11 21:09 amclient-tdriveCYS-2017-Sessions-K-M_0 -rw--- 1 amanda amanda 7287876 May 31 21:58 amclient-tdriveCYS-2017-Sessions-K-M_1 -rw--- 1 amanda amanda 7292999 Jun 13 20:51 amclient-tdriveCYS-2017-Sessions-K-M_1.new -rw--- 1 amanda amanda 3883757 May 11 03:02 amclient-tdriveCYS-2017-Sessions-N-Q_0 -rw--- 1 amanda amanda 3883757 Jun 11 19:19 amclient-tdriveCYS-2017-Sessions-N-Q_1 -rw--- 1 amanda amanda 6103596 May 12 01:24 amclient-tdriveCYS-2017-Sessions-R-S_0 -rw--- 1 amanda amanda 6103596 May 31 21:51 amclient-tdriveCYS-2017-Sessions-R-S_1 -rw--- 1 amanda amanda 6103596 Jun 13 20:52 amclient-tdriveCYS-2017-Sessions-R-S_1.new -rw--- 1 amanda amanda 4692267 May 11 18:34 amclient-tdriveCYS-2017-Sessions-T-Z_0 -rw--- 1 amanda amanda 4692267 Jun 12 02:02 amclient-tdriveCYS-2017-Sessions-T-Z_1 -rw--- 1 amanda amanda119633 May 9 15:49 amclient-tdriveCYS-2017-Sessions-rest_0 -rw--- 1 amanda amanda119586 Jun 13 20:51 amclient-tdriveCYS-2017-Sessions-rest_1 -rw--- 1 amanda amanda 7528145 Jun 3 03:46 amclient-tdriveCYS-2018-Session-A-B_0 -rw--- 1 amanda amanda 7528145 Jun 13 20:51 amclient-tdriveCYS-2018-Session-A-B_1.new -rw--- 1 amanda amanda 15204819 May 22 14:07 amclient-tdriveCYS-2018-Session-A-D_0 -rw--- 1 amanda amanda 4490122 Jun 12 05:28 amclient-tdriveCYS-2018-Session-C_0 -rw--- 1 amanda amanda 4490122 Jun 13 21:00 amclient-tdriveCYS-2018-Session-C_1 -rw--- 1 amanda amanda 6695009 Jun 12 23:33 amclient-tdriveCYS-2018-Session-D-F_0 -rw--- 1 amanda amanda 6689636 Jun 13 20:58 amclient-tdriveCYS-2018-Session-D-F_1 -rw--- 1 amanda amanda 0 May 21 01:56 amclient-tdriveCYS-2018-Session-E-J_0.new -rw--- 1 amanda amanda 10302105 May 21 06:27 amclient-tdriveCYS-2018-Session-K-M_0 -rw--- 1 amanda amanda174474 May 16 10:55 amclient-tdriveCYS-2018-Session-rest_0 -rw--- 1 amanda amanda174234 Jun 13 20:52 amclient-tdriveCYS-2018-Session-rest_1 -rw--- 1 amanda amanda 3625170 Jun 2 03:20 amclient-tdriveCYS-2019-Session-A-B_0 -rw--- 1 amanda amanda 6248803 May 15 11:30 amclient-tdriveCYS-2019-Session-A-D_0 -rw--- 1 amanda amanda 7005138 May 28 18:59 amclient-tdriveCYS-2019-Session-A-D_1 -rw--- 1 amanda amanda 2076372 Jun 2 00:37 amclient-tdriveCYS-2019-Session-C_0 -rw--- 1 amanda amanda 3091423 Jun 1 22:39 amclient-tdriveCYS-2019-Session-D-F_0 -rw--- 1 amanda amanda 6217405 May 30 08:30 amclient-tdriveCYS-2019-Session-E-J_0 -rw--- 1 amanda amanda 5489291 May 15 06:34 amclient-tdriveCYS-2019-Session-E-J_1 -rw--- 1 amanda amanda 5644929 May 16 12:08 amclient-tdriveCYS-2019-Session-E-J_2 -rw--- 1 amanda amanda 3878808 Jun 2 05:26 amclient-tdriveCYS-2019-Session-G-H_0 -rw--- 1 amanda amanda 3878808 Jun 12 23:33 amclient-tdriveCYS-2019-Session-G-H_1.new -rw--- 1 amanda amanda 3418950 Jun 11 23:20 amclient-tdriveCYS-2019-Session-I-L_0 -rw--- 1 amanda amanda 3663681 May 8 05:07 amclient-tdriveCYS-2019-Session-K-M_0 -rw--- 1 amanda amanda 4062806 May 14 19:44 amclient-tdriveCYS-2019-Session-K-M_1 -rw--- 1 amanda amanda 4795279 Jun 1 10:32 amclient-tdriveCYS-2019-Session-K-M_2 -rw--- 1 amanda amanda 2792045 Jun 12 20:13 amclient-tdriveCYS-2019-Session-M_0 -rw--- 1 amanda amanda 1814850 May 7 02:52 amclient-tdriveCYS-2019-Session-N-Q_0 -rw--- 1 amanda amanda 2244093 May 14 06:02 amclient-tdriveCYS-2019-Session-N-Q_1 -rw--- 1 amanda amanda 2649037 Jun 1 03:44 amclient-tdriveCYS-2019-Session-N-Q_2
Re: tar backups not doing incrementals
On Sat, Jun 15, 2019 at 20:35:30 +0200, Uwe Menges wrote: > On 6/15/19 6:43 PM, Jim Kusznir wrote: > > Ever since, any backup is run as a level 0 (even though it reports doing > > a level 1, 2, etc). Everything runs as 0. > > > > How do I fix this? > > I fixed a similar symptom on my system with > property "CHECK-DEVICE" "NO" > (using amgtar), which hands "--no-check-device" to GNU tar, > which makes it no longer think the toplevel dir has been renamed, > so it doesn't store all files in its incremental. Yes, if the device for the filesystems in question are changing from boot to boot, then using the --no-check-device option is necessary to allow the incrementals to work as expected across reboots. But I believe that option is not available for the GNUTAR application, so assuming Jim is indeed using GNUTAR then it seems worth making sure that's actually his problem before trying to implement that particular solution > I think what happened to me is that some LVM volumes got mounted with a > different minor number, triggering that. > > Given that LVM is really common nowadays, I wonder if that option should > be default.. I can't think of a use case where the current default is > preferable, at the moment. Yes. In particular, as long as --one-file-system is passed to GNU tar (the default situation in Amanda) then every file backed up by that run of tar is going to be on one single device, and the check-device operation when building the list of incremental changes is never going to be useful. (And obviously, in the case of changed device numbers for the filesystem, the check is actually harmful). I don't know about changing the actual default in the amgtar application itself at this point in time, but perhaps it at least makes sense to recommend disabling check-device in the example amanda.conf and other documentation Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: tar backups not doing incrementals
On 6/15/19 6:43 PM, Jim Kusznir wrote: > Ever since, any backup is run as a level 0 (even though it reports doing > a level 1, 2, etc). Everything runs as 0. > > How do I fix this? I fixed a similar symptom on my system with property "CHECK-DEVICE" "NO" (using amgtar), which hands "--no-check-device" to GNU tar, which makes it no longer think the toplevel dir has been renamed, so it doesn't store all files in its incremental. I think what happened to me is that some LVM volumes got mounted with a different minor number, triggering that. Given that LVM is really common nowadays, I wonder if that option should be default.. I can't think of a use case where the current default is preferable, at the moment. Yours, Uwe
Re: tar backups not doing incrementals
On Sat, Jun 15, 2019 at 09:43:42 -0700, Jim Kusznir wrote: > Ever since, any backup is run as a level 0 (even though it reports doing a > level 1, 2, etc). Everything runs as 0. You mean that the Amanda Mail Report shows the various incremental levels, but the actual size of the dump is always the same as a level dump? Are you backing up anything on the Amanda server, or are all your DLEs on the BSD client? Nothing changed on the BSD client, except that it was rebooted, right? How long has it been continuing to do this? Does the behavior continue after a given DLE performs an actual level 0 backup? > dumptype is based on comp-tar-user, and has exclude lists (I'm backing up a > 100TB storage array by subdividing using exclude lists as indicated in the > wiki). > > What could have been going on here? How do I get it working again? Assuming you are referring to the comp-user-tar dumptype from the example amanda.conf file distributed with Amanda, that would be using the GNUTAR program, and thus it would be using tar's "snapshot" files to manage the incrementals. So one thing to check is to find where the snapshot files are being stored and make sure they are getting updated correctly each run -- post a before and after output of "ls -l" on that directory. (On my Ubuntu Linux box the snapshot files are put under /var/lib/amanda/gnutar-lists/ by default, and the names of the files are constructed from the host name, mount point, and incremental level, e.g.: -rw--- 1 backup backup 10083684 Nov 7 2018 myhost.example.com__0 -rw--- 1 backup backup 10085466 Nov 2 2018 myhost.example.com__1 -rw--- 1 backup backup 95 Nov 7 2018 myhost.example.com_mnt_0 -rw--- 1 backup backup 74 Nov 9 2018 myhost.exmaple.com_mnt_1 ) Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: tar backups not doing incrementals
Post your conf files on GDrive or something and send us a link. On 2019-06-15 12:43 p.m., Jim Kusznir wrote: > Hi all: > > I have a system with a linux amanda server and a FreeBSD amanda > client. All was working well. > > I transplanted the amanda server into a new mobo/cpu (but same HDD, > etc, and mobo was exactly the same model -- just moved it into a rack > case from a desktop case), and of course rebooted it in the process. > I also rebooted the BSD client. > > Ever since, any backup is run as a level 0 (even though it reports > doing a level 1, 2, etc). Everything runs as 0. > > How do I fix this? > > dumptype is based on comp-tar-user, and has exclude lists (I'm backing > up a 100TB storage array by subdividing using exclude lists as > indicated in the wiki). > > What could have been going on here? How do I get it working again? > > Thanks! > --Jim
tar backups not doing incrementals
Hi all: I have a system with a linux amanda server and a FreeBSD amanda client. All was working well. I transplanted the amanda server into a new mobo/cpu (but same HDD, etc, and mobo was exactly the same model -- just moved it into a rack case from a desktop case), and of course rebooted it in the process. I also rebooted the BSD client. Ever since, any backup is run as a level 0 (even though it reports doing a level 1, 2, etc). Everything runs as 0. How do I fix this? dumptype is based on comp-tar-user, and has exclude lists (I'm backing up a 100TB storage array by subdividing using exclude lists as indicated in the wiki). What could have been going on here? How do I get it working again? Thanks! --Jim
