Hi Jean-Louis,
It seems that I realized why I got this error...but definitely I didn't have
this with amanda 2.5.
I have a temporary disk location configured and the size of it available for
amanda is 90GB.
I started monitoring during the backup and found out strange behaviour. It
was running OK, taped all filesystem except the last one - 90GB to backup.
During the backup of this filesystem /amdumps was consumed gradually
creating 2GB chunks of data in it and the interesting thing happened when it
reached the limit:
(iteration of one minute)
# amstatus DailySet1 | egrep " asp-al-backup:/backup|tape"
asp-al-backup:/backup 1 83444m dumping 91726m (109.93%)
(12:59:
51)
dumping to tape : 0 0m ( 0.00%)
writing to tape : 0 0m 0m ( 0.00%) ( 0.00%)
failed to tape : 0 0m 0m ( 0.00%) ( 0.00%)
taped : 35 65978m 65974m (100.01%) ( 44.16%)
tape 1 : 35 65978m 65974m ( 2.75%) DailySet1-04 (64
chunks)
taper idle
taper busy : 0:17:17 ( 7.08%)
# df -h /amdumps
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aspalbackup-lv_amdumps
106G 90G 11G 90% /amdumps
After one minute
# amstatus DailySet1 | egrep " asp-al-backup:/backup|tape"
asp-al-backup:/backup 1 83444m dumping to tape (14:41:09)
dumping to tape : 1 83444m ( 55.85%)
writing to tape : 0 0m 0m ( 0.00%) ( 0.00%)
failed to tape : 0 0m 0m ( 0.00%) ( 0.00%)
taped : 35 65978m 65974m (100.01%) ( 44.16%)
tape 1 : 35 65978m 65974m ( 2.75%) DailySet1-04 (64
chunks)
taper writing, tapeq: 0
taper busy : 0:17:17 ( 6.94%)
# df -h /amdumps
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aspalbackup-lv_amdumps
106G 60M 101G 1% /amdumps
After this amanda process taper is got stuck waiting for children that don't
exist anymore:
[root@asp-al-backup ~]# ps -ef | grep taper
33 1553 1552 6 10:32 pts/2 00:22:07 taper DailySet1
root 18413 18300 0 15:57 pts/3 00:00:00 grep taper
[root@asp-al-backup ~]# strace -p 1553
Process 1553 attached - interrupt to quit
futex(0x7fb04f2279d0, FUTEX_WAIT, 4068, NULL^C <unfinished ...>
Process 1553 detached
[root@asp-al-backup ~]# pstree -p 1553
taper(1553)-T-{taper}(4067)
L-{taper}(4068)
[root@asp-al-backup ~]# ps -ef | grep taper
33 1553 1552 6 10:32 pts/2 00:22:54 taper DailySet1
root 18446 18300 0 15:59 pts/3 00:00:00 grep taper
[root@asp-al-backup ~]# ps -ef | grep 4068
root 18454 18300 0 16:00 pts/3 00:00:00 grep 4068
[root@asp-al-backup ~]# ps -ef | grep 4067
root 18456 18300 0 16:00 pts/3 00:00:00 grep 4067
and finished after a long time with the error:
FAILURE DUMP SUMMARY:
asp-al-backup /backup lev 1 FAILED [Not enough holding disk space].
It strange where amanda puts 90GB of data in 1 minute - definitely not on
tape, it just deleted it I presume. And why it doesn't progress with the
backup after flushing the data from temporary disk location to the tape?
I'm going to run this on 3.3.7 version and check how it goes.
Kind Regards,
Dmitry Lozitskiy
-----Original Message-----
From: Jean-Louis Martineau [mailto:martin...@zmanda.com]
Sent: Wednesday, February 04, 2015 10:19 AM
To: dmitry.lozits...@asparona.com; amanda-users@amanda.org
Subject: Re: amanda 2.6.1p2-8 issue
From the report you received, how much data it wrote to the tape?
USAGE BY TAPE:
Label Time Size % Nb Nc
DailySet1-03 0:17 70110807K 26.7 35 35
I don't know why you get a 'no space left' error after only 70GB.
It could be a broken tape or tape drive.
Look at the tape debug file, you get a better error message.
Jean-Louis
On 02/03/2015 03:47 PM, dmitry.lozits...@asparona.com wrote:
> Hi Jean-Louis,
>
> There is only Amanda 2.6.1p1 is available in public repository for
> Oracle Linux 6.
> Are you suggesting to build a fresh version of Amanda from there:
> http://www.amanda.org/download.php?
>
> I thought it was a space issue after migration and I replaced LTO5
> tapedrive and LTO-4 tapes that we used with Amanda 2.5 to LTO-6
> tapedrive and LTO-6 tapes.
> Usually 800GB was enough, but now I use 2.5TB tapes and still have the
> same issues, I doubt that it is a space issue as I calculated all
> space required for backup manually and even if count that there are
> all Leve 0 backups they require less than 1 TB space.
>
> I attached amdump trace and backup log from the last Amanda run. May
> be you can say something looking at them?
>
> Kind Regards,
> Dmitry Lozitskiy
>
>
> -----Original Message-----
> From: Jean-Louis Martineau [mailto:martin...@zmanda.com]
> Sent: Wednesday, February 04, 2015 9:38 AM
> To: dmitry.lozits...@asparona.com; amanda-users@amanda.org
> Subject: Re: amanda 2.6.1p2-8 issue
>
> Why upgrade to an old version? You should upgrade again.
>
> It's hard to say something because you didn't posted enough information.
> I will guess is that amanda need more tape space to do the backup
> according to your configuration, you should increase runtapes.
>
> Jean-Louis
>
> On 02/03/2015 03:07 PM, dmitry.lozits...@asparona.com wrote:
>> Hello,
>>
>> I just recently migrated amanda software from version 2.5.0p2:
>>
>> # uname -a
>>
>> Linux asp-al-backup 2.6.18-128.2.1.4.25.el5xen #1 SMP Tue Mar 23
>> 12:43:27 EDT 2010 i686 i686 i386 GNU/Linux
>>
>> # rpm -qa | grep amanda
>>
>> amanda-server-2.5.0p2-8.el5
>>
>> amanda-client-2.5.0p2-8.el5
>>
>> amanda-2.5.0p2-9.el5
>>
>> to a 2.6.1p1 version:
>>
>> # uname -a
>>
>> Linux asp-al-backup 3.8.13-55.1.5.el6uek.x86_64 #2 SMP Wed Jan 28
>> 17:03:28 PST 2015 x86_64 x86_64 x86_64 GNU/Linux
>>
>> # rpm -qa | grep amanda
>>
>> amanda-server-2.6.1p2-8.el6.x86_64
>>
>> amanda-client-2.6.1p2-8.el6.x86_64
>>
>> amanda-2.6.1p2-8.el6.x86_64
>>
>> Before the upgrade my backups were running smoothly without a fail,
>> but after the upgrade I'm facinig partially backups every amanda run.
>>
>> The error it finishes with:
>>
>> Hostname: asp-al-backup
>>
>> Org : PMKRS Daily
>>
>> Config : DailySet1
>>
>> Date : February 3, 2015
>>
>> These dumps were to tape DailySet1-03.
>>
>> *** A TAPE ERROR OCCURRED: [runtapes volumes already written].
>>
>> There are 76826M of dumps left in the holding disk.
>>
>> Run amflush to flush them to tape.
>>
>> The next tape Amanda expects to use is: 1 new tape.
>>
>> FAILURE DUMP SUMMARY:
>>
>> alfresco /apps/alfresco/backups lev 0 FAILED "runtapes volumes
>> already written"
>>
>> alfresco /apps/alfresco/backups lev 0 FAILED [too many taper
>> retries]
>>
>> asp-al-backup /backup lev 1 FAILED [out of holding
>> space in degraded mode]
>>
>> alfresco /apps/alfresco/backups lev 0: partial taper: No space
>> left on device
>>
>> Does anyone have a clue what these messages are talking about.
>>
>> Regards,
>>
>> Dmitry.
>>
>>
>> Disclaimer: Asparona is part of the Deloitte group of companies.
>> Deloitte refers to the New Zealand member firm of Deloitte Touche
>> Tohmatsu
> Limited.
>> The information contained in this document is confidential to the
>> addressee(s) and may be legally privileged. Any view or opinions
>> expressed are those of the author and may not be those of the
>> organisation to which the author belongs. No guarantee or
>> representation is made that this communication is free of errors,
>> viruses or interference. If you have received this e-mail message in
>> error
> please delete it and notify me.
>
>
--
Disclaimer: Asparona is part of the Deloitte group of companies. Deloitte
refers to the New Zealand member firm of Deloitte Touche Tohmatsu Limited.
The information contained in this document is confidential to the
addressee(s) and may be legally privileged. Any view or opinions expressed
are those of the author and may not be those of the organisation to which
the author belongs. No guarantee or representation is made that this
communication is free of errors, viruses or interference. If you have
received this e-mail message in error please delete it and notify me.
