I'm running Amanda 2.4.2p2 (modified from the Debian 2.4.2p2-4 packages)
on Linux with a Seagate STT20000A IDE tape (via the scsi-emulation
driver) and 10GB native Travan tapes. I'm using software compression -
hardware compression is (or should be) off.
I recently added another host with about 9GB of data to my backup
routine. Most of the data is mounted from a NetWare box. I've split the
volumes into 1-2 GB chunks with tar.
Some nights, to some tapes, my backups run fine. I get about 8 GB dumped
to tape:
STATISTICS:
Total Full Daily
-------- -------- --------
Estimate Time (hrs:min) 0:17
Run Time (hrs:min) 7:26
Dump Time (hrs:min) 6:59 5:49 1:10
Output Size (meg) 8297.0 7495.6 801.4
Original Size (meg) 14403.4 12933.6 1469.7
Avg Compressed Size (%) 57.6 58.0 54.1 (level:#disks ...)
Filesystems Dumped 30 6 24 (1:23 2:1)
Avg Dump Rate (k/s) 337.6 366.5 194.1
Tape Time (hrs:min) 5:15 4:21 0:54
Tape Size (meg) 8298.0 7495.8 802.2
Tape Used (%) 87.8 79.3 8.5 (level:#disks ...)
Filesystems Taped 30 6 24 (1:23 2:1)
Avg Tp Write Rate (k/s) 449.3 489.8 253.2
DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
-------------------------- --------------------------------- ------------
bluebird / 1 250 32 12.8 0:14 2.3 0:07 9.2
bluebird /etc 1 210 32 15.2 0:06 5.3 0:02 27.3
bluebird /users1 0 64691703377568 52.2 87:48 641.1 87:49 641.0
bluebird /usr 1 4450 384 8.6 0:07 56.4 0:04 117.6
bluebird /var 1 91860 8832 9.6 0:27 328.2 0:24 363.1
goldfinch / 1 2930 480 16.4 1:17 6.2 0:05 109.7
goldfinch /etc 1 100 32 32.0 0:12 2.8 0:04 14.3
goldfinch /home 1 10 32 320.0 0:05 6.0 0:03 23.5
phoenix / 1 2220 224 10.1 1:37 2.3 0:03 96.2
phoenix /etc 1 96 96 -- 0:08 11.8 0:03 44.0
phoenix /var 1 13344 13344 -- 0:49 274.9 0:34 394.6
pokey / 1 200 32 16.0 0:02 17.0 0:02 26.6
pokey /boot 1 10 32 320.0 0:01 43.0 0:15 4.2
pokey /etc 1 170 32 18.8 0:02 19.8 0:03 21.6
pokey -etware/SYS 1 959090 523904 54.6 35:33 245.7 21:54 398.7
pokey -tware/VOL1 1 27100 9024 33.3 0:57 157.0 0:22 405.5
pokey -1/SOFTWARE 1 302190 252224 83.5 13:54 302.5 28:46 146.1
pokey -VOL1/users 0 1957080 981888 50.2 78:18 209.0 57:32 284.4
pokey -rs/COLLECT 0 1019770 448800 44.0 35:13 212.4 23:40 316.1
pokey -ers/COMMON 0 1840 736 40.0 0:05 160.0 0:05 148.9
pokey -s/JMCGOWAN 0 20643401596320 77.3 92:07 288.8 44:57 592.0
pokey /usr 1 3380 320 9.5 0:50 6.4 0:03 109.3
pokey /var 1 7350 2528 34.4 0:09 285.1 0:07 376.7
raven / 1 120 32 26.7 0:01 41.2 0:03 22.3
raven /boot 1 10 32 320.0 0:00 120.8 0:02 26.9
raven /etc 1 140 32 22.9 0:02 16.1 0:06 10.9
raven /users1 0 17318401270176 73.3 55:31 381.3 47:06 449.4
raven /usr 2 7870 672 8.5 7:31 1.5 0:05 133.8
raven /var 1 68290 7456 10.9 1:20 93.4 0:38 199.4
raven /www 1 13610 864 6.3 5:05 2.8 0:08 107.6
umnhpc87 /Users 0 FAILED ---------------------------------------
umnhpc87 /etc 0 FAILED ---------------------------------------
umnhpc87 /var 0 FAILED ---------------------------------------
(Ignore umnhpc87 - someone turned it off)
But other days, to other tapes, I run out of tape at about 5 GB:
These dumps were to tape umnh-20011112-1.
*** A TAPE ERROR OCCURRED: [[writing file: No space left on device]].
Some dumps may have been left in the holding disk.
Run amflush to flush them to tape.
The next tape Amanda expects to use is: umnh-20011129-1.
FAILURE AND STRANGE DUMP SUMMARY:
pokey /netware/VOL1/users lev 1 FAILED [out of tape]
pokey /netware/VOL1/users lev 1 FAILED ["data write: Broken pipe"]
pokey /netware/VOL1/users lev 1 FAILED [dump to tape failed]
STATISTICS:
Total Full Daily
-------- -------- --------
Estimate Time (hrs:min) 0:17
Run Time (hrs:min) 9:55
Dump Time (hrs:min) 5:29 5:17 0:11
Output Size (meg) 5167.4 5094.2 73.2
Original Size (meg) 8768.4 8539.0 229.4
Avg Compressed Size (%) 58.9 59.7 30.8 (level:#disks ...)
Filesystems Dumped 32 21 11 (1:10 3:1)
Avg Dump Rate (k/s) 268.2 274.0 108.8
Tape Time (hrs:min) 2:55 2:45 0:09
Tape Size (meg) 5168.4 5094.8 73.6
Tape Used (%) 54.7 53.9 0.8 (level:#disks ...)
Filesystems Taped 32 21 11 (1:10 3:1)
Avg Tp Write Rate (k/s) 504.4 525.7 133.0
FAILED AND STRANGE DUMP DETAILS:
/-- pokey /netware/VOL1/users lev 1 FAILED ["data write: Broken pipe"]
sendbackup: start [pokey:/netware/VOL1/users level 1]
sendbackup: info BACKUP=/bin/tar
sendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/tar -f... -
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
\--------
^L
NOTES:
[snip]
taper: tape umnh-20011112-1 kb 9590432 fm 33 writing file: No space left on device
DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
-------------------------- --------------------------------- ------------
bluebird / 0 55720 26336 47.3 0:38 702.0 1:03 421.2
bluebird /etc 0 5140 1184 23.0 0:07 177.6 0:07 175.0
bluebird /users1 3 207460 68512 33.0 4:44 240.9 7:44 147.6
bluebird /usr 1 4620 384 8.3 0:07 52.8 0:03 120.7
bluebird /var 0 448850 246976 55.0 6:21 648.1 15:36 263.9
goldfinch / 1 2770 416 15.0 1:17 5.4 0:04 114.2
goldfinch /etc 0 2590 544 21.0 0:15 36.0 0:03 174.1
goldfinch /home 1 10 32 320.0 0:06 5.3 0:07 9.0
phoenix / 0 395480 173856 44.0 65:19 44.4 8:48 329.2
phoenix /etc 0 1152 1152 -- 1:36 12.0 0:09 127.4
phoenix /var 1 3776 3776 -- 0:40 94.9 0:39 96.7
pokey / 0 39440 16192 41.1 0:30 534.6 1:22 197.5
pokey /boot 0 4660 3776 81.0 0:05 748.8 0:11 344.4
pokey /etc 0 3220 640 19.9 0:06 111.3 0:06 121.4
pokey -etware/SYS 0 960360 524480 54.6 35:07 248.9 13:55 628.1
pokey -tware/VOL1 0 27550 9248 33.6 1:25 109.2 0:27 341.5
pokey -1/SOFTWARE 0 302190 252224 83.5 13:58 301.1 10:52 386.9
pokey -VOL1/users 1 FAILED ---------------------------------------
pokey -rs/COLLECT 0 943800 379392 40.2 30:02 210.6 20:29 308.7
pokey -ers/COMMON 0 1840 736 40.0 0:06 131.0 0:07 106.6
pokey -s/JMCGOWAN 0 20655701596896 77.3 82:17 323.5 40:17 660.6
pokey /usr 1 3380 320 9.5 0:58 5.5 0:03 140.3
pokey /var 1 1810 544 30.1 0:18 31.0 0:04 152.6
raven / 1 120 32 26.7 0:01 23.9 0:08 8.2
raven /boot 1 10 32 320.0 0:01 39.7 0:19 3.4
raven /etc 0 29290 9792 33.4 0:39 250.2 0:44 222.7
raven /users1 1 3140 256 8.2 0:58 4.4 0:07 42.4
raven /usr 1 7820 672 8.6 2:19 4.8 0:08 84.7
raven /var 0 129340 30752 23.8 2:20 220.4 1:07 460.3
raven /www 0 31231001890848 60.5 74:51 421.1 47:37 661.9
umnhpc87 /Users 0 92930 33536 36.1 0:53 628.6 1:39 339.4
umnhpc87 /etc 0 760 192 25.3 0:01 196.2 0:05 47.9
umnhpc87 /var 0 110920 17696 16.0 0:45 391.2 0:41 434.0
I've verified that hardware compression is NOT being used on the tapes
I'm having problems with, and that I can fill them with ~10GB of data
before I hit EOT, using this method:
[ after dumps have completed ]
root@phoenix:/# mt tell
At block 19209506.
[ see if I can put more data to the tape ]
root@phoenix:/# dd if=/dev/zero of=/dev/nst0 bs=1k count=5
5+0 records in
5+0 records out
dd: closing output file `/dev/nst0': No space left on device
[ no ]
root@phoenix:/# mt rewind
root@phoenix:/# mt tell
At block 0.
# su backup -c '/usr/sbin/amrmtape -v umnh-set umnh-20011112-1'
amrmtape: preserving original database in curinfo.orig.26305 (exported).
[ snip list of hosts discarded ]
# mt tell
At block 0.
# dd if=/dev/zero of=/dev/nst0 bs=1024k count=10000
dd: writing `/dev/nst0': No space left on device
9550+0 records in
9549+0 records out
# mt tell
At block 19556993.
#
[ snip rewind & relabel ]
'dd' fits about 10GB of data to the tape, approximately equal to its
native capacity. If hardware compression were on, I'd expect more, so I
infer that hardware compression is off. But, when I re-amlabel the tape
and run dumps to it again, I still only get 5 GB.
I can think of these possibilities:
1 Amanda is not correctly reporting the amount of data it dumped to
tape. (But tape time seems to increase with reported data taped,
making this less likely.)
2 Something is switching compression on and off behind my back.
3 Hardware compression is on, but is so poor that it can't compress
a stream of 0x0s, making my test for compression fail.
4 On some, but not all, of my supposedly-identical tapes, my drive
uses 100MB filemarks.
5 The tape gnomes are switching out my tapes at night while I'm
asleep.
Has anyone else seen behavior like this? Does anyone have ideas about
what's going wrong?
Configs follow:
--- begin amanda.conf ---
#
# amanda.conf - sample Amanda configuration file.
#
# If your configuration is called, say, "DailySet1", then this file
# normally goes in /etc/amanda/DailySet1/amanda.conf.
#
# for explanation of the parameters refer to amanda(8) and
# /usr/doc/amanda/WHATS.NEW.gz
org "UMNH" # your organization name for reports
mailto "waoki jmcgowan" # space separated list of operators at your site
dumpuser "backup" # the user to run dumps under
#
inparallel 2 # maximum dumpers that will run in parallel
netusage 2048 # maximum net bandwidth for Amanda, in KB per sec
# a filesystem is due for a full backup once every <dumpcycle> days
dumpcycle 1 weeks # the number of days in the normal dump cycle
tapecycle 13 tapes # the number of tapes in rotation
runspercycle 3 # 3 runs per cycle
bumpsize 20 MB # minimum savings (threshold) to bump level 1 -> 2
bumpdays 1 # minimum days at each level
bumpmult 4 # threshold = bumpsize * (level-1)**bumpmult
#runtapes 9 # explained in WHATS.NEW
#tpchanger "no-changer" # the tape-changer glue script, see TAPE.CHANGERS
tapedev "/dev/nst0" # Linux @ tuck, important: norewinding
# tapedev "/dev/nrst8" # or use the (no-rewind!) tape device directly
tapetype Travan-20 # what kind of tape it is (see tapetypes below)
#labelstr "^HISS[0-9][0-9]*$" # label constraint regex: all tapes must match
labelstr "^umnh-[0-9]{8}-[0-9][0-9]*$" # label constraint regex: all tapes must match
diskdir "/amspace" # where the holding disk is
disksize -512 KB
#disksize 290 MB # how much space can we use on it
#diskdir "/dumps/amanda/work" # additionaly holding disks can be specified
#diskdir "/mnt/disk4"
#disksize 1000 MB # they are used round-robin
# reserve 60% for incrementals in degraded mode
reserve 60
# Amanda needs a few MB of diskspace for the log and debug files,
# as well as a database. This stuff can grow large, so the conf directory
# isn't usually appropriate.
infofile "/var/lib/amanda/umnh-set/curinfo" # database filename
logfile "/var/log/amanda/umnh-set/log" # log filename
# where the index files live
indexdir "/var/lib/amanda/umnh-set/index"
# tapetypes
#
# Define the type of tape you use here, and use it in "tapetype" above.
# Some typical types of tapes are included here. The tapetype tells amanda
# how many MB will fit on the tape, how big the filemarks are, and how
# fast the tape device is.
#
# For completeness Amanda should calculate the inter-record gaps too, but it
# doesn't. For EXABYTE and DAT tapes this is ok. Anyone using 9 tracks for
# amanda and need IRG calculations? Drop me a note if so.
# lbl-templ "/etc/amanda/bluebird-set/8.5x11.ps"
printer "copier-plain"
define tapetype Travan-20 {
comment "just produced by tapetype program"
length 9450 mbytes
filemark 36 kbytes
speed 913 kbytes
lbl-templ "/etc/amanda/umnh-set/8.5x11.ps"
}
# snipped sample tapetypes I don't use
# dumptypes
#
# These are referred to by the disklist file. The dumptype specifies
# certain "options" for dumping including:
# index - keep an index of the files backed up
# compress-fast - (default) compress on the client using fast algorithm
# compress-best - compress using the best (and slowww) algorithm
# no-compress - don't compress the dump output
# srvcompress - Compress dumps on the tape host instead of client
# machines. This may be useful when a fast tape host
# is backing up slow clients.
# record - (default) record the dump in /etc/dumpdates
# no-record - don't record the dump, for testing
# no-hold - don't go to the holding disk, good for dumping
# the holding disk partition itself.
# skip-full - Skip the disk when a level 0 is due, to allow
# full backups outside Amanda, eg when the machine
# is in single-user mode.
# skip-incr - Skip the disk when the level 0 is NOT due. This
# is used in archive configurations, where only full
# dumps are done and the tapes saved.
# no-full - Do a level 1 every night. This can be used, for
# example, for small root filesystems that only change
# slightly relative to a site-wide prototype. Amanda
# then backs up just the changes.
#
# Also, the dumptype specifies the priority level, where "low", "medium" and
# "high" are the allowed levels. These are only really used when Amanda has
# no tape to write to because of some error. In that "degraded mode", as
# many incrementals as will fit on the holding disk are done, higher priority
# first, to insure the important disks are dumped first.
define dumptype always-full {
comment "Full dump of this filesystem always"
options no-compress
priority high
dumpcycle 0
maxcycle 0
}
# changed all /etc/amanda/exclude.gtar to .amandaexclude
define dumptype bins {
program "GNUTAR"
comment "Binaries (compressed)"
options compress-fast, index, exclude-list ".amandaexclude"
priority low
}
define dumptype bins-nocomp {
program "GNUTAR"
comment "Binaries (not compressed)"
options index, exclude-list ".amandaexclude"
priority low
}
define dumptype sys-data {
program "GNUTAR"
comment "System-generated data (e.g. /var)"
options compress-fast, index, exclude-list ".amandaexclude"
priority medium
}
define dumptype sys-data-nocomp {
program "GNUTAR"
comment "System-generated data (e.g. /var) without compression"
options no-compress, index, exclude-list ".amandaexclude"
priority medium
}
define dumptype configs {
program "GNUTAR"
comment "Configuration files, such as /etc"
options compress-fast, index, exclude-list ".amandaexclude"
priority high
}
define dumptype configs-nocomp {
program "GNUTAR"
comment "Configuration files, such as /etc, without compression"
options no-compress, index, exclude-list ".amandaexclude"
priority high
}
define dumptype user-data {
program "GNUTAR"
comment "user files"
options compress-fast, index, exclude-list ".amandaexclude"
priority medium
}
define dumptype user-important-data {
program "GNUTAR"
comment "very important user files"
options compress-fast, index, exclude-list ".amandaexclude"
priority high
}
define dumptype user-lowvalue-data {
program "GNUTAR"
comment "not important user files"
options compress-fast, index, exclude-list ".amandaexclude"
priority low
}
# snipped sample dumptypes I don't use
--- end amanda.conf ---
--- begin disklist ---
# sample Amanda2 disklist file, derived from CS.UMD.EDU's disklist
#
# If your configuration is called, say, "DailySet1", then this file
# normally goes in /etc/amanda/DailySet1/disklist.
#
# File format is:
#
# hostname diskdev dumptype
#
# where the dumptypes are defined by you in amanda.conf.
# note! at our site compression includes encryption
# note! all dumptypes use an exclude file called ".amandaexclude"
# note:
# FULL SYSTEM means everything needed to restore is on tape
# PARTIAL means some binaries are on tape, but aren't enough
# for full restore
# NO SYS means only configs or data are on tape - do install,
# then restore data + configs
#
# Package manager data == NO SYS unless bins are there too or
# PM data can be easily fed to something to recreate bin
# install
### backup server - no compression, no encryption
## FULL SYSTEM
# NOTE! this exposes the unencrypted GPG key!
# MAKE SURE that the /var/backups/.gnupg is excluded!
phoenix /etc configs-nocomp
phoenix /var sys-data-nocomp
#phoenix /amspace holding-disk
# uses exclude
phoenix / bins-nocomp
### mail
## FULL SYSTEM
bluebird /etc configs
bluebird /users1 user-data
bluebird /var sys-data
bluebird /usr bins
# Uses exclude file! /etc is same fs, but separate
bluebird / bins
### web
## FULL SYS
raven /etc configs
raven /www user-data
raven /var sys-data
raven /users1 user-lowvalue-data
raven / bins
raven /boot bins
raven /usr bins
### bridge
## FULL SYS
goldfinch /etc configs
# what the hell, I'll backup home too
# (but at low priority)
goldfinch /home user-lowvalue-data
# uses exclude file
goldfinch / bins
### loghost
## FULL SYS
# temporarily deactivated
#owl /etc configs
#owl /var sys-data
#owl / sys-data
### edu workstation tag 136
## NO SYS
# OSX test
umnhpc87 /Users user-data
# not much config crap in /etc...
umnhpc87 /etc configs
umnhpc87 /var sys-data
### pokey
## FULL SYS
pokey / bins
pokey /boot bins
pokey /etc configs
pokey /var sys-data
pokey /usr bins
### umnh_fs
## ???
# Netware courtesy of pokey. Volume is mounted and dismounted by a cron
# job on pokey. HOORAY!
pokey /netware/SYS sys-data
# vol1 is big (8.5 gb used). It'll fit on a tape, but I'd rather split it up a bit.
# (rather, it SHOULD fit on a tape...)
# This does not preserve ownership or ACLs. This is also slow.
pokey /netware/VOL1 user-lowvalue-data
pokey /netware/VOL1/SOFTWARE bins
pokey /netware/VOL1/users user-data
pokey /netware/VOL1/users/COMMON user-data
pokey /netware/VOL1/users/COLLECT user-data
pokey /netware/VOL1/users/JMCGOWAN user-data
# why are people dumping 3.1 GB of important stuff into a temp directory?
# oh well, not my problem if they come looking for users/common/temp
# below we find the examples
# snipped examples
--- end disklist ---
--
William Aoki [EMAIL PROTECTED] /"\ ASCII Ribbon Campaign
B1FB C169 C7A6 238B 280B <- key change \ / No HTML in mail or news!
99AF A093 29AE 0AE1 9734 prev. expired X
/ \
