RE: Amcheck crashing on CentOS 5.1 x86_64
Running 2.5.0. The debug file is in-line below. It's short. Looking at it, it seems that it had a timeout waiting for the tape array to respond. It says that it sent a mail message, but I haven't received any messages when amcheck has crashed. Thanks! amcheck: debug 1 pid 1257 ruid 33 euid 0: start at Tue May 6 15:24:46 2008 amcheck-clients: time 0.012: bind_portrange2: trying port=921 amcheck-clients: time 0.012: dgram_bind: socket bound to 0.0.0.0.921 changer: got exit: 0 str: 7 8 1 changer_query: changer return was 8 1 changer_query: searchable = 0 changer_find: looking for NULL changer is searchable = 0 changer: got exit: 0 str: 7 /dev/nst0 changer: got exit: 2 str: 8 Drive not ready after 120 seconds, rewind said /dev/nst0: Input/output error amcheck: spawning /usr/bin/Mail in pipeline amcheck: argument list: /usr/bin/Mail -s BackupDailies AMANDA PROBLEM: FIX BEFORE RUN, IF POSSIBLE [EMAIL PROTECTED] amcheck: pid 1257 finish time Tue May 6 15:31:30 2008 steve jacobson * cozi * IT Manager * [EMAIL PROTECTED] * m: 206.310.7760 * www.cozi.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jean-Louis Martineau Sent: Tuesday, May 06, 2008 6:31 PM To: Steve Jacobson Cc: amanda-users@amanda.org Subject: Re: Amcheck crashing on CentOS 5.1 x86_64 Which amanda release? Post the amcheck.*.debug file Jean-Louis Steve Jacobson wrote: All A couple of weeks ago after working for about two months amcheck started crashing on my system. It dumps the following trace to stderr. Any thoughts on where to begin looking to track this down would be greatly appreciated. Thanks! -Steve J. -Begin Trace-- *** glibc detected *** /usr/sbin/amcheck: double free or corruption (fasttop): 0x6feb8e70 *** === Backtrace: = /lib64/libc.so.6[0x2bc6b4f4] /lib64/libc.so.6(cfree+0x8c)[0x2bc6eb1c] /usr/sbin/amcheck[0x8ef0] /usr/sbin/amcheck(main+0xa2f)[0xa48f] /lib64/libc.so.6(__libc_start_main+0xf4)[0x2bc198a4] /usr/sbin/amcheck[0x65e9] === Memory map: 2aaab000-2aac5000 r-xp 08:02 2867273 /lib64/ld-2.5.so 2aac5000-2aac8000 rw-p 2aac5000 00:00 0 2acc4000-2acc5000 r--p 00019000 08:02 2867273 /lib64/ld-2.5.so 2acc5000-2acc6000 rw-p 0001a000 08:02 2867273 /lib64/ld-2.5.so 2acc6000-2ace7000 r-xp 08:02 8753372 /usr/lib64/libamserver-2.5.0p2.so 2ace7000-2aee7000 ---p 00021000 08:02 8753372 /usr/lib64/libamserver-2.5.0p2.so 2aee7000-2aee9000 rw-p 00021000 08:02 8753372 /usr/lib64/libamserver-2.5.0p2.so 2aee9000-2aeec000 rw-p 2aee9000 00:00 0 2aeec000-2aef6000 r-xp 08:02 8753371 /usr/lib64/libamtape-2.5.0p2.so 2aef6000-2b0f6000 ---p a000 08:02 8753371 /usr/lib64/libamtape-2.5.0p2.so 2b0f6000-2b0f7000 rw-p a000 08:02 8753371 /usr/lib64/libamtape-2.5.0p2.so 2b0f7000-2b119000 r-xp 08:02 8753373 /usr/lib64/libamanda-2.5.0p2.so 2b119000-2b319000 ---p 00022000 08:02 8753373 /usr/lib64/libamanda-2.5.0p2.so 2b319000-2b31b000 rw-p 00022000 08:02 8753373 /usr/lib64/libamanda-2.5.0p2.so 2b31b000-2b33b000 rw-p 2b31b000 00:00 0 2b347000-2b3c9000 r-xp 08:02 2867307 /lib64/libm-2.5.so 2b3c9000-2b5c8000 ---p 00082000 08:02 2867307 /lib64/libm-2.5.so 2b5c8000-2b5c9000 r--p 00081000 08:02 2867307 /lib64/libm-2.5.so 2b5c9000-2b5ca000 rw-p 00082000 08:02 2867307 /lib64/libm-2.5.so 2b5ca000-2b5cb000 rw-p 2b5ca000 00:00 0 2b5cb000-2b5ce000 r-xp 08:02 2867332 /lib64/libtermcap.so.2.0.8 2b5ce000-2b7cd000 ---p 3000 08:02 2867332 /lib64/libtermcap.so.2.0.8 2b7cd000-2b7ce000 rw-p 2000 08:02 2867332 /lib64/libtermcap.so.2.0.8 2b7ce000-2b7e3000 r-xp 08:02 2867308 /lib64/libnsl-2.5.so 2b7e3000-2b9e2000 ---p 00015000 08:02 2867308 /lib64/libnsl-2.5.so 2b9e2000-2b9e3000 r--p 00014000 08:02 2867308 /lib64/libnsl-2.5.so 2b9e3000-2b9e4000 rw-p 00015000 08:02 2867308 /lib64/libnsl-2.5.so 2b9e4000-2b9e6000 rw-p 2b9e4000 00:00 0 2b9e6000-2b9f7000 r-xp 08:02 2867325 /lib64/libresolv-2.5.so 2b9f7000-2bbf7000 ---p 00011000 08:02 2867325 /lib64/libresolv-2.5.so 2bbf7000-2bbf8000 r--p 00011000 08:02 2867325 /lib64/libresolv-2.5.so 2bbf8000-2bbf9000 rw-p 00012000 08:02 2867325 /lib64/libresolv-2.5.so 2bbf9000-2bbfc000 rw-p 2bbf9000 00:00 0 2bbfc000-2bd42000 r-xp 08:02 2867283 /lib64/libc-2.5.so 2bd42000-2bf42000 ---p 00146000 08:02 2867283 /lib64/libc-2.5.so 2bf42000-2bf46000 r--p 00146000 08:02 2867283 /lib64/libc-2.5.so 2bf46000-2bf47000 rw-p 0014a000 08:02 2867283 /lib64/libc-2.5.so 2bf47000-2bf4e000 rw-p
Re: Out of space problem
Op di 06mei08 om 10:28 schreef Paul Bijnens:
Or better, use amtapetape -e 35g /dev/yourtapedrive to get
an estimate (takes about 3-5 hours normally).
I got/use:
define tapetype HP-DAT72 {
comment HP-DAT72x10 Autoloader (AE313A)
# data provided by Gerrit A. Smit [EMAIL PROTECTED]
length 35480 mbytes
filemark 0 kbytes
speed 2992 kbytes
}
Re: Out of space problem
Thanks to everyone for the input - both serious and sarcastic.
Happy to report that hardware compression was enabled. Turned it off and
got a level 0 done last night.
Muy Gracias
N/
On 6/05/2008 10:30 AM, Nigel Allen wrote:
Hi All
I'm experiencing an odd problem with a USB DAT drive that keeps
running out of space. Apologies for the length of the post.
The drive is supposed to be 36 / 72 GB.
Here's the kind of thing I see when I run a level 0 dump.
These dumps were to tape DailySet1-18.
*** A TAPE ERROR OCCURRED: [No more writable valid tape found].
Some dumps may have been left in the holding disk.
Run amflush to flush them to tape.
The next tape Amanda expects to use is: DailySet1-19.
FAILURE AND STRANGE DUMP SUMMARY:
mail..com.au mapper/VolGroup00-LogVol00 lev 0 FAILED
[out of tape]
STATISTICS:
Total Full Incr.
Estimate Time (hrs:min)0:04
Run Time (hrs:min)14:44
Dump Time (hrs:min) 11:22 11:22 0:00
Output Size (meg) 33927.133927.10.0
Original Size (meg) 50710.150710.00.0
Avg Compressed Size (%)66.9 66.9 10.0 (level:#disks
...)
Filesystems Dumped2 1 1 (1:1)
Avg Dump Rate (k/s) 849.1 849.18.5
Tape Time (hrs:min)0:00 0:00 0:00
Tape Size (meg) 0.00.00.0
Tape Used (%) 0.00.00.0 (level:#disks
...)
Filesystems Taped 1 0 1 (1:1)
Chunks Taped 0 0 0
Avg Tp Write Rate (k/s) 7.4-- 7.4
USAGE BY TAPE:
Label Time Size %NbNc
DailySet1-18 0:00 32K0.0 1 0
NOTES:
planner: Forcing full dump of
mail..com.au:mapper/VolGroup00-LogVol00 as directed.
taper: tape DailySet1-18 kb 31022240 fm 2 writing file: No space
left on device
driver: Taper error: [writing file: No space left on device]
driver: going into degraded mode because of taper component error.
big estimate: mail..com.au sda1 1
est: 64Kout 32K
Here is the amanda.conf file with everything not used snipped out:
org DailySet1 # your organization name for reports
mailto amandabackup # space separated list of operators at your site
dumpuser amandabackup # the user to run dumps under
inparallel 4# maximum dumpers that will run in parallel
(max 63)
dumporder sssS# specify the priority order of each dumper
taperalgo first # The algorithm used to choose which dump
image to send
displayunit k # Possible values: k|m|g|t
netusage 600 Kbps # maximum net bandwidth for Amanda, in KB per
sec
dumpcycle 4 weeks # the number of days in the normal dump cycle
runspercycle 20 # the number of amdump runs in dumpcycle days
tapecycle 21 tapes # the number of tapes in rotation
bumpsize 20 Mb # minimum savings (threshold) to bump level 1
- 2
bumppercent 20 # minimum savings (threshold) to bump level 1
- 2
bumpdays 1 # minimum days at each level
bumpmult 4 # threshold = bumpsize * bumpmult^(level-1)
etimeout 300# number of seconds per filesystem for
estimates.
dtimeout 1800 # number of idle seconds before a dump is
aborted.
ctimeout 30 # maximum number of seconds that amcheck waits
tapebufs 20
runtapes 1 # number of tapes to be used in a single run
of amdump
tapedev /dev/nst0 # the no-rewind tape device to be used
changerfile /etc/amanda/DailySet1/changer.conf
changerdev /dev/null
maxdumpsize -1 # Maximum number of bytes the planner
will schedule
tapetype HP-DAT72 # what kind of tape it is (see
tapetypes below)
labelstr ^DailySet1-[0-9][0-9]*$ # label constraint regex: all
tapes must match
amrecover_do_fsf yes# amrecover will call amrestore with the
amrecover_check_label yes # amrecover will call amrestore with the
amrecover_changer null: # amrecover will use the changer if
you restore
holdingdisk hd1 {
comment main holding disk
directory /dumps/amanda # where the holding disk is
use -10 Gb # how much space can we use on it
chunksize 1Gb # size of chunk if you want big dump to be
}
autoflush no #
infofile /etc/amanda/DailySet1/curinfo# database DIRECTORY
logdir /etc/amanda/DailySet1# log directory
indexdir /etc/amanda/DailySet1/index # index directory
define tapetype HP-DAT72 {
comment HP DAT72 USB with hardware compression on
length 72 G
}
define dumptype global {
comment Global definitions
}
define dumptype custom-compress {
global
program GNUTAR
comment Dump with custom client compression
exclude list /etc/amanda/exclude.gtar
Windows Server 2003 cant execute amandad.exe via inetd
I got it working. I chose to use xinetd rather than inetd, and ran everything under the same user, amandabackup. Rather than using the uer sshd_server to run xinetd. Which is what caused the problems. Thanks, Chad E. Kotil Global Research NOC
/var/lib/amanda/gnutar-lists : huge !
Hello, Is it safe to do some clean up in this folder ? Regards,
Re: /var/lib/amanda/gnutar-lists : huge !
On 2008-05-07 17:44, FM wrote: Hello, Is it safe to do some clean up in this folder ? No. It's the list of statefiles, which gnutar uses to decide if a file should be in the incremental backup. Erasing them (or getting them out of sync) will result in incrementals be much larger than normal. -- Paul Bijnens, xplanation Technology ServicesTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, * * F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... * * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: /var/lib/amanda/gnutar-lists : huge !
Tx for the reply, What is strange is that some files are several months old. Does Amanda also need these old ones ? Regards, Paul Bijnens wrote: On 2008-05-07 17:44, FM wrote: Hello, Is it safe to do some clean up in this folder ? No. It's the list of statefiles, which gnutar uses to decide if a file should be in the incremental backup. Erasing them (or getting them out of sync) will result in incrementals be much larger than normal.
Re: /var/lib/amanda/gnutar-lists : huge !
On Wednesday 07 May 2008, FM wrote: Hello, Is it safe to do some clean up in this folder ? Regards, That *should* be auto-cleaning as amanda will/should remove the listings for tapes that have been over-written. Is this not the case at your site? Humm, why are they in /var/lib? Is this an rpm installation? :( That said, I note I had some old ones laying around that were dated prior to my having a drive crash, and when I re-installed on another drive, the directory tree's got re-arranged ( LVM's run summarily canceled, that was the second crash while using it) and the disklist edited to reflect that, so it could no longer find the ones to delete. They can go, and just did as they are no longer of any use. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) F u cn rd ths u cnt spl wrth a dm!
confusing problem with NO-NEW-TAPE
Please forgive my n00bness. This is probably a simple configuration error on
my part, but I am having a problem getting Amanda to flush to tape. Everything
else seems to work fine. The report when I run amflush looks like this (and is
similar to the report from amdump):
Hostname: backup.merp.com
Org : Main merp Amanda
Config : Main
Date: May 7, 2008
There are 7051204K of dumps left in the holding disk.
They will be flushed on the next run.
The next tape Amanda expects to use is: a new tape.
The next new tape already labelled is: merpMain-001.
FAILURE DUMP SUMMARY:
172.16.1.134 /test lev 0 FAILED No new tape.
172.16.1.134 /test lev 0 FAILED No new tape.
172.16.1.134 /test lev 0 FAILED [too many taper retries]
172.16.1.23/backup lev 0 FAILED No new tape.
172.16.1.23/backup lev 0 FAILED No new tape.
172.16.1.23/backup lev 0 FAILED [too many taper retries]
backup.merp.com/etc/amanda lev 0 FAILED No new tape.
backup.merp.com/etc/amanda lev 0 FAILED No new tape.
backup.merp.com/etc/amanda lev 0 FAILED [too many taper retries]
backup.merp.com/var/lib/amanda lev 0 FAILED No new tape.
backup.merp.com/var/lib/amanda lev 0 FAILED No new tape.
backup.merp.com/var/lib/amanda lev 0 FAILED [too many taper retries]
STATISTICS:
Total Full Incr.
Estimate Time (hrs:min)0:00
Run Time (hrs:min) 0:00
Dump Time (hrs:min)0:00 0:00 0:00
Output Size (meg) 0.00.00.0
Original Size (meg) 0.00.00.0
Avg Compressed Size (%) -- -- --
Filesystems Dumped0 0 0
Avg Dump Rate (k/s) -- -- --
Tape Time (hrs:min)0:00 0:00 0:00
Tape Size (meg) 0.00.00.0
Tape Used (%) 0.00.00.0
Filesystems Taped 0 0 0
Chunks Taped 0 0 0
Avg Tp Write Rate (k/s) -- -- --
?
DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISKL ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
-- - -
172.16.1.134 /test 0 FAILED
172.16.1.23 /backup 0 FAILED
backup.merp /etc/amanda 0 FAILED
backup.merp -lib/amanda 0 FAILED
(brought to you by Amanda version 2.6.0)
The next new tape is available, and the changer seems to be working fine.
amcheck works without issue, and the tapes were labeled with amlabel.
Information about the system:
CentOS 5.1 x86
tpchanger chg-zd-mtx
tapedev tape:/dev/tape-nst0
changerfile /etc/amanda/Main/changer.conf
changerdev /dev/changer
Amanda 2.6.0 from zmanda rpm
/proc/scsi/scsi entries for my changer, which is an Exabyte Magnum 1x7
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: EXABYTE Model: EXB-210 Rev: B00E
Type: Medium Changer ANSI SCSI revision: 04
Host: scsi1 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: ULTRIUM-TD2 Rev: 4C60
Type: Sequential-AccessANSI SCSI revision: 03
The emulation on the changer can be changed to others, and I have experimented
with them as well. The behavior doesn't seem to change.
Here is the cut from the amflush log:
amflush: start at Wed May 7 11:17:37 MDT 2008
amflush: datestamp 20080507111733
amflush: starttime 20080507111733
amflush: starttime-locale-independent 2008-05-07 11:17:37 MDT
FLUSH 172.16.1.134 /test 20080430161259 0
/var/dumps/20080430161259/172.16.1.134._test.0
FLUSH 172.16.1.23 /backup 20080430161259 0
/var/dumps/20080430161259/172.16.1.23._backup.0
FLUSH backup.merp.com /etc/amanda 20080430161259 0
/var/dumps/20080430161259/backup.merp.com._etc_amanda.0
FLUSH backup.merp.com /var/lib/amanda 20080430161259 0
/var/dumps/20080430161259/backup.merp.com._var_lib_amanda.0
ENDFLUSH
driver: pid 17097 executable driver version 2.6.0
driver: adding holding disk 0 dir /var/dumps size 183439360 chunksize 1048576
reserving 183439360 out of 183439360 for degraded-mode dumps
driver: send-cmd time 0.004 to taper: START-TAPER 20080507111733
driver: start time 0.006 inparallel 4 bandwidth 8000 diskspace 183439360 dir
OBSOLETE datestamp 20080507111733 driver: drain-ends tapeq FIRST big-dumpers
sssS
taper: pid 17098 executable taper version 2.6.0
taper: using label `merpMain-005' date `20080507111733'
driver: result time 1.582 from taper: TAPER-OK
driver: send-cmd time 1.583 to taper: FILE-WRITE 00-1
Re: /var/lib/amanda/gnutar-lists : huge !
On Wednesday 07 May 2008, FM wrote: Tx for the reply, What is strange is that some files are several months old. Does Amanda also need these old ones ? If the tape they go with has been re-used due to reaching the end of the 'tapecycle', then they can go. If you have some labeled 'no-reuse' that might correspond to an archived tape, then hang onto them. Regards, Paul Bijnens wrote: On 2008-05-07 17:44, FM wrote: Hello, Is it safe to do some clean up in this folder ? No. It's the list of statefiles, which gnutar uses to decide if a file should be in the incremental backup. Erasing them (or getting them out of sync) will result in incrementals be much larger than normal. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) Man who falls in vat of molten optical glass makes spectacle of self.
