I recently did a do-release-upgrade on one of my Amanda servers, taking it from Ubuntu 14.04 Server
LTS to Ubuntu 16.04 Server LTS. With the update of all the libraries, etc., I had to recompile
Amanda. So, I took the opportunity to upgrade Amanda from 3.3.6 to 3.5.1 (`make uninstall` the one
before starting with the next). It went relatively smoothly (used`sudo ./configure
--with-user=amanda --with-group=amanda --without-ipv6 --with-ssh-security`), with a few glitches
that I'm used to that had to be fixed after the install. Had to fix permissions at and beneath
/usr/local/share/perl/5.22.1/, install the libjson-perl package, etc.
However, I'm stuck on this one. It was last Thursday that I did the upgrade. Backups ran Thursday
evening, and all the other servers it was backing up were successfully backed up, but localhost was
not. I checked some of the usual culprits such as the location and permissions of
amanda-security.conf, etc.
Now, I'm getting most of the DLEs for localhost backed up, but three are repeatedly failing. The
Amanda email report starts with
localhost /data lev 0 FAILED [data timeout]
localhost /data lev 0 FAILED [Got empty header]
It turns out that the three failing DLEs are all on "/", that is, they don't have separate mount
points and/or partitions. There are several other DLEs whose full path is in /data, but they are
separate volumes with mount points in /data, and they get backed up. I checked the kern.log thinking
maybe there was a problem with the disk that is "/", but the kern.log has almost nothing in it and
no mention of any problems anywhere. Also, /etc is backed up successfully, and it is on that same
"/", which is a mirrored root drive.
amcheck claims no problems:
amanda@wahoo:~$ amcheck daily
Amanda Tape Server Host Check
-
NOTE: Holding disk '/amanda1': 437825536 KB disk space available, using
427339776 KB
NOTE: Holding disk '/amanda2': 437825536 KB disk space available, using
427339776 KB
slot 6: volume 'ISB-Daily-006'
Will write to volume 'ISB-Daily-006' in slot 6.
NOTE: skipping tape-writable test
Server check took 0.594 seconds
Amanda Backup Client Hosts Check
Client check: 4 hosts checked in 3.059 seconds. 0 problems found.
(brought to you by Amanda 3.5.1)
amanda@wahoo:~$
tar is a locally built 1.29 at /usr/local/bin/tar. The version that came in with Ubuntu 16.04 is
/bin/tar and is 1.28. The backup is using local-comp-user-tar. Should I try switching everything to
amgtar? I'm using that on at least one other server that has much much larger and more complex drive
assemblies and LVMs.
I've copied some of the debug log files below.
--
---
Chris Hoogendyk
-
O__ Systems Administrator
c/ /'_ --- Biology & Geosciences Departments
(*) \(*) -- 315 Morrill Science Center
~~ - University of Massachusetts, Amherst
---
Erdös 4
--- Section of
/tmp/amanda/server/daily/dumper.20190114224002000.debug -
SERVICE sendbackup
OPTIONS
features=9efefbfff3fffbf71f;maxdumps=1;hostname=localhost;config=daily;timestamp=20190114224001;data-shm-control-name=/amanda_shm_control-30920-0;
GNUTAR
/data
0
local
FAST
YES
YES
AMANDA
Mon Jan 14 22:52:08.876122021 2019: pid 30449: thd-0x839e00: dumper:
security_getdriver(name=local) returns 0x7fc5bdd60a00
Mon Jan 14 22:52:08.877094986 2019: pid 30923: thd-0x839e00: dumper: Executing:
/usr/bin/lsb_release '--id' '-s'
Mon Jan 14 22:52:08.939296138 2019: pid 30924: thd-0x839e00: dumper: Executing:
/usr/bin/lsb_release '--description' '-s'
Mon Jan 14 22:52:08.997940727 2019: pid 30449: thd-0x86ed40: dumper:
security_handleinit(handle=0x7fc5b40246a0, driver=0x7fc5bdd60a00 (LOCAL))
Mon Jan 14 22:52:08.998212439 2019: pid 30449: thd-0x86ed40: dumper:
security_streaminit(stream=0x7fc5b4056e30, driver=0x7fc5bdd60a00 (LOCAL))
Mon Jan 14 22:52:09.998095734 2019: pid 30449: thd-0x839e00: dumper:
tcpm_send_token: data is still flowing
Mon Jan 14 22:52:10.008113852 2019: pid 30449: thd-0x839e00: dumper: got
response:
CONNECT DATA 49 MESG 48 INDEX 47 STATE 46
OPTIONS features=9efefbfff3fffbf71f;
Mon Jan 14 22:52:10.008207510 2019: pid 30449: thd-0x839e00: dumper:
security_streaminit(stream=0x9729c0, driver=0x7fc5bdd60a00 (LOCAL))
Mon Jan 14 22:52:10.008260895 2019: pid 30449: thd-0x839e00: dumper:
security_streaminit(stream=0x97aa60, driver=0x7fc5bdd60a00 (LOCAL))
Mon Jan 14 22:52:10.008340144 2019: pid 30449: thd-0x839e00: dumper:
security_streaminit(stream=0x982b00, driver=0x7fc5bdd60a00 (LOCAL))
Mon Jan 14 22:52:10.008414203 2019: pid 30449: thd-0x839e00: dumper:
security_streaminit(stream=0x98aba0, driver=0x7fc5bdd60a00 (LOCAL))
Mon Jan 14 22:52:10.008437799 2019: pid 30449: thd-0x839e00: dumper:
security_close(handle=0x7fc5b40246a0,
