amflush run while amdump underway tries to flush .tmp files

[Nathan Stratton Treadway]

(Amanda v3.5)

I noticed that Amanda 3.5 no longer aborts amflush if amdump is
currently running (as older versions of Amanda do). 

So out of curiousity I kicked of "amflush TestBackup" while amdump was
busy dumping to the holding disk... and I discovered that amflush
actually tries to go ahead and flush the ".tmp" file files that it finds
in the holding directory:

= From /var/log/amanda/server/TestBackup/amflush.20171116200510.debug:
Thu Nov 16 20:05:17.590062176 2017: pid 26860: thd-0x2f07e00: amflush: flushing 
/amanda/TestBackup-holding/2017111622/client1._.0.tmp
Thu Nov 16 20:05:17.590096226 2017: pid 26860: thd-0x2f07e00: amflush: flushing 
/amanda/TestBackup-holding/2017111622/client2._.1.tmp
=

In this case the taper failed (and thus the amflush didn't actually do
anything with the .tmp files):

= From /var/log/amanda/TestBackup/amdump.1:
driver: send-cmd time 14.132 to taper0: FILE-WRITE worker0-0 00-2 
/amanda/TestBackup-holding/2017111622/client1._.0.tmp client1 / 0 
2017111622 "" "" "" "" "" "" "" "" 0
writing taper command 'FILE-WRITE worker0-0 00-2 
/amanda/TestBackup-holding/2017111622/client1._.0.tmp client1 / 0 
2017111622 "" "" "" "" "" "" "" "" 0
' failed: Broken pipe
=
(There is a line break in the log file just before "' failed".)


= From /var/log/amanda/server/TestBackup/taper.20171116200517.debug:
Thu Nov 16 20:05:18.502419601 2017: pid 26862: thd-0x4232000: taper: Building 
type SPLIT_FILE header of 32768-32768 bytes with name='client1' disk='/' 
dumplevel=0 and blocksize=32768
Thu Nov 16 20:05:22.427709157 2017: pid 26862: thd-0x4232050: taper: no 
next_filename
Thu Nov 16 20:05:22.427743969 2017: pid 26862: thd-0x4232050: taper: sending 
XMSG_CRC message
Thu Nov 16 20:05:22.427748905 2017: pid 26862: thd-0x4232050: taper: 
xfer-source-holding CRC: 2e4f7128 size: 249856000
Thu Nov 16 20:05:22.427757739 2017: pid 26862: thd-0x4232050: taper: 
xfer_queue_message: MSG:  version=0>
Thu Nov 16 20:05:22.427767783 2017: pid 26862: thd-0x4232050: taper: 
xfer-source-holding sending XMSG_DONE message
Thu Nov 16 20:05:22.427773216 2017: pid 26862: thd-0x4232050: taper: 
xfer_queue_message: MSG:  version=0>
[ *** file ends abruptly here ***]
=


 but whether or not that indicates a bug in the taper, it seems like
amflush should not ever try to flush .tmp files from the holding disk...
(right?)



Finally, after this testing I notice that the command_file still has
FLUSH commands for those .tmp files (even though neither the files nor
the containing holding directory now exist).  I've run both "amdump" and
"amflush" since then, and tried "amcleanup" as well.  Is there any
(good) way to clean up these orphan commands?

= From /etc/amanda/TestBackup/command_file:
ID 1633
1603 FLUSH TestBackup /amanda/TestBackup-holding/2017111622/client1._.0.tmp 
client1 / 2017111622 0 TestBackup WORKING:17072 TODO
1604 FLUSH TestBackup /amanda/TestBackup-holding/2017111622/client2._.1.tmp 
client2 / 2017111622 0 TestBackup WORKING:17072 TODO
=

=
# ls -l /amanda/TestBackup-holding/
total 0
=


Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues

[Nathan Stratton Treadway]

On Thu, Nov 16, 2017 at 09:43:15 -0500, Jean-Louis Martineau wrote:
> On 16/11/17 08:31 AM, Nathan Stratton Treadway wrote:
> > DUMP SUMMARY:
> > DUMPER STATS TAPER STATS
> > HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s
> > --- --- -- 
> > --
> It is a bug if they are not listed, can you send me the 
> log..0 file
> 

I have just sent this log file in a separate (off-list) message.

Let me know if you need any other info.

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues

[Jean-Louis Martineau]

Nathan,

Try this patch.

run: amdump --no-dump --no-flush CONFIG
Will do all scheduled vault to the vault-storage

Jean-Louis

On 16/11/17 08:31 AM, Nathan Stratton Treadway wrote:
> On Thu, Oct 19, 2017 at 11:06:40 -0400, Jean-Louis Martineau wrote:
> > Since you want to vault all full, I would set 'vault' in the local
> > storage, set 'dump-selection' in the cloud storage, but will not set
> > 'vault-storage'
> > That way the vault are scheduled but are not executed because
> > vault-storage is not set. Amanda know they must be vaulted.
> > Every month, you can run: amdump CONF BADHOST -ovault-storage="cloud"
> > to do the vaulting.
>
> I am testing a vaulting setup where I have "vault TestOffsite 0" set on
> the main storage and a global "vault-storage TestOffsite".
>
> If the TestOffsite vtapes are not available during the nightly amdump
> run, dumps are written to the main storage and I see COPY commands queue
> up command_file for the vault storage.
>
> If I then mount the Offsite vtape partition and run an "amdump
> TestBackup BADHOST" command, the dumps are successfully copied to the
> vault, and amdump sends a report that looks like this:
>
> =
> Hostname: backupserver
> Org : TestBackup
> Config : TestBackup
> Date : November 16, 2017
>
> The next tape Amanda expects to use is: TESTBACKUP-06.
>
>
> FAILURE DUMP SUMMARY:
> planner: FATAL no DLE to backup
>
>
> STATISTICS:
> Total Full Incr. Level:#
>    
> Estimate Time (hrs:min) 0:00
> Run Time (hrs:min) 0:01
> Dump Time (hrs:min) 0:00 0:00 0:00
> Output Size (meg) 0.0 0.0 0.0
> Original Size (meg) 0.0 0.0 0.0
> Avg Compressed Size (%) -- -- --
> DLEs Dumped 0 0 0
> Avg Dump Rate (k/s) -- -- --
>
> Tape Time (hrs:min) 0:00 0:00 0:00
> Tape Size (meg) 0.0 0.0 0.0
> Tape Used (%) 0.0 0.0 0.0
> DLEs Taped 0 0 0
> Parts Taped 0 0 0
> Avg Tp Write Rate (k/s) -- -- --
>
>
> USAGE BY TAPE:
> Label Time Size % DLEs Parts
> TESTBACKUP-102 0:01 7489M 3.7 9 9
>
>
> NOTES:
> planner: Argument 'BADHOST' matches neither a host nor a disk.
> driver: WARNING: got empty schedule from planner
> taper: Slot 15 with label TESTBACKUP-15 is usable
> taper: Slot 2 with label TESTBACKUP-102 is usable
> taper: tape TESTBACKUP-102 kb 7669248 fm 9 [OK]
>
>
> DUMP SUMMARY:
> DUMPER STATS TAPER STATS
> HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s
> --- --- -- 
> --
>
> (brought to you by Amanda version 3.5)
> =
>
>
> The main thing I notice is that nothing is listed in the Dump Summary
> section. As discussed in earlier emails, if the vault storage is
> available during the nightly amdump run, the dumps written to the vault
> show up as VAULT lines in the Dump Summary section so I'm wondering
> if the equivalent dumps performed during a second run should be showing
> up here in this report?
>
>
> A second thing I noticed was the three error/warning messages coming
> from the planner and driver caused by the use of "BADHOST". Unless
> those messages are actually causing the empty Dump Summary section they
> aren't a terrible problem... but especially for sites where the
> recommended configuration is to regularly run a separate vaulting run of
> amdump (such as the one discussed above), I wonder how difficult it
> would be to add support to amdump for a recognized option to turn on a
> "do pending vault operations but don't try to back up any new DLEs"
> mode?
>
> Nathan
>
> 
> Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ 
> 
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt 
>  
> ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail
diff --git a/man/xml-source/amdump.8.xml b/man/xml-source/amdump.8.xml
index ceb12f3..42ee341 100644
--- a/man/xml-source/amdump.8.xml
+++ b/man/xml-source/amdump.8.xml
@@ -30,6 +30,9 @@
 
   amdump
 --no-taper
+--no-dump
+--no-flush
+--no-vault
 --exact-match
 &configoverride.synopsis;
 config
@@ -78,22 +81,51 @@ man page for more details about Amanda.
 
 
   
-  host [disk]*
+  --no-taper
   
-Specify the host and disk on which the command will work -- see
-the description of DLE specifications in . 
+Disable writting to tape, including flush and vault.
+  
+  
+
+  
+  --no-dump
+  
+Disable dump operations.
+  
+  
+
+  
+  --no-flush
+  
+Disable flush, like setting autoflush to NO for all storages.
+Dump done in this run will be flushed.
+  
+  
+
+ 

Re: "vaulting run" amdump email report issues

[Jean-Louis Martineau]

On 16/11/17 08:31 AM, Nathan Stratton Treadway wrote:
> On Thu, Oct 19, 2017 at 11:06:40 -0400, Jean-Louis Martineau wrote:
> > Since you want to vault all full, I would set 'vault' in the local
> > storage, set 'dump-selection' in the cloud storage, but will not set
> > 'vault-storage'
> > That way the vault are scheduled but are not executed because
> > vault-storage is not set. Amanda know they must be vaulted.
> > Every month, you can run: amdump CONF BADHOST -ovault-storage="cloud"
> > to do the vaulting.
>
> I am testing a vaulting setup where I have "vault TestOffsite 0" set on
> the main storage and a global "vault-storage TestOffsite".
>
> If the TestOffsite vtapes are not available during the nightly amdump
> run, dumps are written to the main storage and I see COPY commands queue
> up command_file for the vault storage.
>
> If I then mount the Offsite vtape partition and run an "amdump
> TestBackup BADHOST" command, the dumps are successfully copied to the
> vault, and amdump sends a report that looks like this:
>
> =
> Hostname: backupserver
> Org : TestBackup
> Config : TestBackup
> Date : November 16, 2017
>
> The next tape Amanda expects to use is: TESTBACKUP-06.
>
>
> FAILURE DUMP SUMMARY:
> planner: FATAL no DLE to backup
This is expected since there is no dump in this run
>
>
> STATISTICS:
> Total Full Incr. Level:#
>    
> Estimate Time (hrs:min) 0:00
> Run Time (hrs:min) 0:01
> Dump Time (hrs:min) 0:00 0:00 0:00
> Output Size (meg) 0.0 0.0 0.0
> Original Size (meg) 0.0 0.0 0.0
> Avg Compressed Size (%) -- -- --
> DLEs Dumped 0 0 0
> Avg Dump Rate (k/s) -- -- --
>
> Tape Time (hrs:min) 0:00 0:00 0:00
> Tape Size (meg) 0.0 0.0 0.0
> Tape Used (%) 0.0 0.0 0.0
> DLEs Taped 0 0 0
> Parts Taped 0 0 0
> Avg Tp Write Rate (k/s) -- -- --
>
>
> USAGE BY TAPE:
> Label Time Size % DLEs Parts
> TESTBACKUP-102 0:01 7489M 3.7 9 9
9 dle was vaulted, so I think it is a success
>
>
> NOTES:
> planner: Argument 'BADHOST' matches neither a host nor a disk.
> driver: WARNING: got empty schedule from planner
> taper: Slot 15 with label TESTBACKUP-15 is usable
> taper: Slot 2 with label TESTBACKUP-102 is usable
> taper: tape TESTBACKUP-102 kb 7669248 fm 9 [OK]
>
>
> DUMP SUMMARY:
> DUMPER STATS TAPER STATS
> HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s
> --- --- -- 
> --
It is a bug if they are not listed, can you send me the 
log..0 file

>
> (brought to you by Amanda version 3.5)
> =
>
>
> The main thing I notice is that nothing is listed in the Dump Summary
> section. As discussed in earlier emails, if the vault storage is
> available during the nightly amdump run, the dumps written to the vault
> show up as VAULT lines in the Dump Summary section so I'm wondering
> if the equivalent dumps performed during a second run should be showing
> up here in this report?
>
>
> A second thing I noticed was the three error/warning messages coming
> from the planner and driver caused by the use of "BADHOST". Unless
> those messages are actually causing the empty Dump Summary section they
> aren't a terrible problem... but especially for sites where the
> recommended configuration is to regularly run a separate vaulting run of
> amdump (such as the one discussed above), I wonder how difficult it
> would be to add support to amdump for a recognized option to turn on a
> "do pending vault operations but don't try to back up any new DLEs"
> mode?
I will add --no-dump, --no-flush and --no-vault option.

And amvault without argument could run 'driver --no-dump --no-flush'

Jean-Louis
>
> Nathan
>
> 
> Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ 
> 
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt 
>  
> ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

Re: Odd non-fatal errors in amdump reports.

[Austin S. Hemmelgarn]

On 2017-11-14 14:37, Austin S. Hemmelgarn wrote:

On 2017-11-14 07:43, Austin S. Hemmelgarn wrote:

On 2017-11-14 07:34, Austin S. Hemmelgarn wrote:

On 2017-11-13 16:42, Jean-Louis Martineau wrote:

On 13/11/17 02:53 PM, Austin S. Hemmelgarn wrote:

driver: send-cmd time 9300.326 to taper1: VAULT-WRITE worker1-0 
00-00120 local-vtl local-vtl Home-0001 client0 /home/1D 0 
20171113073255 "" "" "" "" 1073741824 memory "" "" 0


 > FAIL taper "ST:cloud" "POOL:cloud" client0 /home/1D 
20171113073255 0 error "File 0 not found"


Do that dump still exists on tape Home-0001? Find it with amfetchdump.

If yes, send me the taper debug file.
amfetchdump does not see it, but looking directly at the virtual tape 
directories, I can see it there.


Just tried an amcheckdump on everything, it looks like some of the 
dump files are corrupted, but I can't for the life of me figure out 
why (I test our network regularly and it has no problems, and any 
problems with a particular system should show up as more than just 
corrupted tar files).  I'm going to try disabling compression and see 
if that helps at all, as that's the only processing other than the 
default that we're doing on the dumps (long term, it's not really a 
viable option, but if it fixes things at least we know what's broken).
No luck changing compression.  I would suspect some issue with NFS, but 
I've started seeing the same symptoms on my laptop as well now (which is 
completely unrelated to any of the sets at work other than having an 
almost identical configuration other than paths and the total number of 
tapes).


So, I finally got things working by switching from:

storage "local-vtl"
vault-storage "cloud"

To:

storage: "local-vtl" "cloud"

And removing the "vault" option from the local-vtl storage definition. 
Strictly speaking, this is working around the issue instead of fixing 
it, but it fits within what we need for our usage, and actually makes 
the amdump runs complete faster (since dumps get taped to S3 in parallel 
with getting taped to the local vtapes).


Based on this, and the fact that the issues I was seeing with corrupted 
dumps being reported by amcheckdump, I think the issue is probably an 
interaction between the vaulting code and the regular taping code, but 
I'm not certain.


Thanks for the help.

"vaulting run" amdump email report issues

[Nathan Stratton Treadway]

On Thu, Oct 19, 2017 at 11:06:40 -0400, Jean-Louis Martineau wrote:
> Since you want to vault all full, I would set 'vault' in the local
> storage, set 'dump-selection' in the cloud storage, but will not set
> 'vault-storage'
> That way the vault are scheduled but are not executed because
> vault-storage is not set. Amanda know they must be vaulted.
> Every month, you can run: amdump CONF BADHOST -ovault-storage="cloud"
> to do the vaulting.

I am testing a vaulting setup where I have "vault TestOffsite 0" set on
the main storage and a global "vault-storage TestOffsite".

If the TestOffsite vtapes are not available during the nightly amdump
run, dumps are written to the main storage and I see COPY commands queue
up command_file for the vault storage.

If I then mount the Offsite vtape partition and run an "amdump
TestBackup BADHOST" command, the dumps are successfully copied to the
vault, and amdump sends a report that looks like this:

=
Hostname: backupserver
Org : TestBackup
Config  : TestBackup
Date: November 16, 2017

The next tape Amanda expects to use is: TESTBACKUP-06.


FAILURE DUMP SUMMARY:
  planner: FATAL no DLE to backup


STATISTICS:
  Total   Full  Incr.   Level:#
        
Estimate Time (hrs:min) 0:00
Run Time (hrs:min)  0:01
Dump Time (hrs:min) 0:00   0:00   0:00
Output Size (meg)0.00.00.0
Original Size (meg)  0.00.00.0
Avg Compressed Size (%)  -- -- --
DLEs Dumped0  0  0
Avg Dump Rate (k/s)  -- -- --

Tape Time (hrs:min) 0:00   0:00   0:00
Tape Size (meg)  0.00.00.0
Tape Used (%)0.00.00.0
DLEs Taped 0  0  0
Parts Taped0  0  0
Avg Tp Write Rate (k/s)  -- -- --


USAGE BY TAPE:
  Label Time Size  %  DLEs Parts
  TESTBACKUP-1020:017489M3.7 9 9


NOTES:
  planner: Argument 'BADHOST' matches neither a host nor a disk.
  driver: WARNING: got empty schedule from planner
  taper: Slot 15 with label TESTBACKUP-15 is usable
  taper: Slot 2 with label TESTBACKUP-102 is usable
  taper: tape TESTBACKUP-102 kb 7669248 fm 9 [OK]


DUMP SUMMARY:
 DUMPER STATS   TAPER STATS
HOSTNAME   DISK   L  ORIG-MB   OUT-MB COMP% MMM:SSKB/s MMM:SSKB/s
--- --- -- --

(brought to you by Amanda version 3.5)
=


The main thing I notice is that nothing is listed in the Dump Summary
section.  As discussed in earlier emails, if the vault storage is
available during the nightly amdump run, the dumps written to the vault
show up as VAULT lines in the Dump Summary section so I'm wondering
if the equivalent dumps performed during a second run should be showing
up here in this report?  


A second thing I noticed was the three error/warning messages coming
from the planner and driver caused by the use of "BADHOST".  Unless
those messages are actually causing the empty Dump Summary section they
aren't a terrible problem... but especially for sites where the
recommended configuration is to regularly run a separate vaulting run of
amdump (such as the one discussed above), I wonder how difficult it
would be to add support to amdump for a recognized option to turn on a
"do pending vault operations but don't try to back up any new DLEs"
mode?

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239