Re: "vaulting run" amdump email report issues -- "amdump --no-dump --no-flush" has MISSING in subject line

[Nathan Stratton Treadway]

On Wed, Nov 22, 2017 at 11:58:12 -0500, Jean-Louis Martineau wrote:
> On 22/11/17 12:56 AM, Nathan Stratton Treadway wrote:
> >> I applied this patch and ran the test again, and "amdump --no-dump 
> >> 
> > --no-flush" worked just as well as with the previous patch, but showed  
> >
> > no "dumper lines" in the log.*.0 file.  
> >
> > (Of course the email report did still have the unexpected RESULTS
> > MISSING line for each DLE.)
> I already committed a fix for that, use SVN or GIT.
> 

Sorry to revive this old thread, but I'm finally getting a chance to
look at this carefully under v3.5.1.


The context of this discussion is a configuration where "vault-storage"
is set, and the main storage had a "vault" line pointing to that vault
storage.  If I then run a "normal" amdump without any vault vtapes being
available, this leaves a situation where there is a pending "vault"
operation for each affected DLE..


When I mounted a suitable vault vtape and ran the v3.5.1 "amdump
-no-dump --noflush" command to perform the vaulting, the copy proceeded
as expected... and the Amanda mail report message did not have the
RESULTS MISSING line for each DLE that used to appear, and generally
looked correct.

However, the one thing that did not quite seem correct is that the
Subject line of the email message _did_ have the word "MISSING" in it...

Included below is a (shorted) version of the email, in case that's
helpful in tracking down what is causing that "MISSING".

Nathan

=
To: natha...@ontko.com
Subject: TestBackup MISSING: AMANDA MAIL REPORT FOR February 21, 2018
Date: Wed, 21 Feb 2018 11:00:40 -0500

Hostname: tumhalad
Org : TestBackup
Config  : TestBackup
Date: February 21, 2018

These dumps to storage 'TestOffsite' were to tape TESTBACKUP-108.
The next tape Amanda expects to use for storage 'TestBackup' is: TESTBACKUP-06.
The next tape Amanda expects to use for storage 'TestOffsite' is: 
TESTBACKUP-109.


STATISTICS:
  Total   Full  Incr.   Level:#
        
Estimate Time (hrs:min) 0:00
Run Time (hrs:min)  0:01
Dump Time (hrs:min) 0:00   0:00   0:00
Output Size (meg)0.00.00.0
Original Size (meg)  0.00.00.0
Avg Compressed Size (%)  -- -- --
DLEs Dumped0  0  0
Avg Dump Rate (k/s)  -- -- --

Tape Time (hrs:min) 0:01   0:01   0:00
Tape Size (meg)   7106.0 6813.9  292.1
Tape Used (%)3.53.30.1
DLEs Taped 9  3  6  1:6
Parts Taped9  3  6  1:6
Avg Tp Write Rate (k/s)  92812.991808.4 124623


USAGE BY TAPE:
  Label Time Size  %  DLEs Parts
  TESTBACKUP-1080:017106M3.5 9 9


NOTES:
  taper: Slot 3 with label TESTBACKUP-03 is usable
  taper: Slot 8 with label TESTBACKUP-108 is usable
  taper: tape TESTBACKUP-108 kb 7276536 fm 9 [OK]


DUMP SUMMARY:
 DUMPER STATS   TAPER STATS
HOSTNAME   DISK   L  ORIG-MB   OUT-MB COMP% MMM:SSKB/s MMM:SSKB/s
--- --- -- --
client1/  0  2093   --  VAULT0:22 97401.6
client2/  1 4   --  VAULT0:00 41540.0
[... similar lines for each DLE ...]
=




Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues -- driver.c "nodump" option

[Nathan Stratton Treadway]

On Thu, Nov 23, 2017 at 14:32:38 -0500, Jean-Louis Martineau wrote:
> I forgot to commit it.
> It is now committed.

Yep, looks good now.  Thanks again.

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues -- driver.c "nodump" option

[Jean-Louis Martineau]

On 23/11/17 02:00 PM, Nathan Stratton Treadway wrote:
> On Wed, Nov 22, 2017 at 11:58:12 -0500, Jean-Louis Martineau wrote:
> > On 22/11/17 12:56 AM, Nathan Stratton Treadway wrote:
> > > On Mon, Nov 20, 2017 at 11:45:07 -0500, Jean-Louis Martineau wrote:
> > > > Nathan,
> > > >
> > > > I was too fast to write the first patch, and you are completely 
> right.
> > > >
> > > > I committed the attached patch,now amflush use the --no-dump 
> argument
> > > > and driver merge both 'nodump' and '--no-dump'.
> > >
> > > Yeah, looks good.
> > >
> > > I applied this patch and ran the test again, and "amdump --no-dump
> > > --no-flush" worked just as well as with the previous patch, but showed
> > > no "dumper lines" in the log.*.0 file.
> > >
> > > (Of course the email report did still have the unexpected RESULTS
> > > MISSING line for each DLE.)
> > I already committed a fix for that, use SVN or GIT.
>
> My next step will be to rebuild the latest version from git and use that
> for further tests, but while I'm working on that:
>
> The code used in the above test did include the Report-3.5.diff patch
> from Nov 17, which eliminated the RESULTS MISSING lines in the report
> when amdump is run with a BADHOST argument.
>
> Did you also push a fix that eliminates the RESULTS MISSING lines which
> were being generated when amdump is run with "--no-dump" instead of
> BADHOST? (This was the case I was referring to above, following up from
> my original discussion in an email from Sat, 18 Nov 2017 00:43:32
> -0500.)
>
> I looked through the git log (for the 3_5 branch) but didn't notice any
> changes that seemed to apply to that problem (either on the
> Report.pm/human.pm 
>  
> side, or on the planner.c side). If you did fix that
> case, could you point me to the commit date or commit id?

I forgot to commit it.
It is now committed.

Jean-Louis
>
> Thanks.
>
> Nathan
>
> 
> Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ 
> 
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt 
>  
> ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

Re: "vaulting run" amdump email report issues -- driver.c "nodump" option

[Nathan Stratton Treadway]

On Wed, Nov 22, 2017 at 11:58:12 -0500, Jean-Louis Martineau wrote:
> On 22/11/17 12:56 AM, Nathan Stratton Treadway wrote:
> > On Mon, Nov 20, 2017 at 11:45:07 -0500, Jean-Louis Martineau wrote:
> > > Nathan,
> > >
> > > I was too fast to write the first patch, and you are completely right.
> > >
> > > I committed the attached patch,now amflush use the --no-dump argument
> > > and driver merge both 'nodump' and '--no-dump'.
> >
> > Yeah, looks good.
> >
> > I applied this patch and ran the test again, and "amdump --no-dump
> > --no-flush" worked just as well as with the previous patch, but showed
> > no "dumper lines" in the log.*.0 file.
> >
> > (Of course the email report did still have the unexpected RESULTS
> > MISSING line for each DLE.)
> I already committed a fix for that, use SVN or GIT.

My next step will be to rebuild the latest version from git and use that
for further tests, but while I'm working on that:

The code used in the above test did include the Report-3.5.diff patch
from Nov 17, which eliminated the RESULTS MISSING lines in the report
when amdump is run with a BADHOST argument.  

Did you also push a fix that eliminates the RESULTS MISSING lines which
were being generated when amdump is run with "--no-dump" instead of
BADHOST?  (This was the case I was referring to above, following up from
my original discussion in an email from Sat, 18 Nov 2017 00:43:32
-0500.)

I looked through the git log (for the 3_5 branch) but didn't notice any
changes that seemed to apply to that problem (either on the
Report.pm/human.pm side, or on the planner.c side).  If you did fix that
case, could you point me to the commit date or commit id?

Thanks.

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues -- driver.c "nodump" option

[Jean-Louis Martineau]

On 22/11/17 12:56 AM, Nathan Stratton Treadway wrote:
> On Mon, Nov 20, 2017 at 11:45:07 -0500, Jean-Louis Martineau wrote:
> > Nathan,
> >
> > I was too fast to write the first patch, and you are completely right.
> >
> > I committed the attached patch,now amflush use the --no-dump argument
> > and driver merge both 'nodump' and '--no-dump'.
>
> Yeah, looks good.
>
> I applied this patch and ran the test again, and "amdump --no-dump
> --no-flush" worked just as well as with the previous patch, but showed
> no "dumper lines" in the log.*.0 file.
>
> (Of course the email report did still have the unexpected RESULTS
> MISSING line for each DLE.)
I already committed a fix for that, use SVN or GIT.

Jean-Louis
>
> I also tried the amflush command with this patch applied, and it too
> started up the driver and flushed files to the target storage without
> any trouble.
>
>
> Nathan
>
> 
> Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ 
> 
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt 
>  
> ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

Re: "vaulting run" amdump email report issues -- driver.c "nodump" option

[Nathan Stratton Treadway]

On Mon, Nov 20, 2017 at 11:45:07 -0500, Jean-Louis Martineau wrote:
> Nathan,
> 
> I was too fast to write the first patch, and you are completely right.
> 
> I committed the attached patch,now amflush use the --no-dump argument 
> and driver merge both 'nodump' and '--no-dump'.

Yeah, looks good.

I applied this patch and ran the test again, and "amdump --no-dump
--no-flush" worked just as well as with the previous patch, but showed
no "dumper lines" in the log.*.0 file.

(Of course the email report did still have the unexpected RESULTS
MISSING line for each DLE.)

I also tried the amflush command with this patch applied, and it too
started up the driver and flushed files to the target storage without
any trouble.


Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues -- driver.c "nodump" option

[Jean-Louis Martineau]

Nathan,

I was too fast to write the first patch, and you are completely right.

I committed the attached patch,now amflush use the --no-dump argument 
and driver merge both 'nodump' and '--no-dump'.

Jean-Louis

On 18/11/17 01:11 AM, Nathan Stratton Treadway wrote:
> On Thu, Nov 16, 2017 at 10:15:28 -0500, Jean-Louis Martineau wrote:
> > Nathan,
> >
> > Try this patch.
> >
> > run: amdump --no-dump --no-flush CONFIG
> > Will do all scheduled vault to the vault-storage
>
> [...]
>
> > diff --git a/server-src/driver.c b/server-src/driver.c
> > index 351494a..fd01296 100644
> > --- a/server-src/driver.c
> > +++ b/server-src/driver.c
> > @@ -107,6 +107,10 @@ static int taper_started = 0;
> > static int nb_storage;
> > static cmddatas_t *cmddatas = NULL;
> >
> > +static int no_dump = FALSE;
> > +//static int no_flush = FALSE;
> > +static int no_vault = FALSE;
> > +
> > static int wait_children(int count);
> > static void wait_for_children(void);
> > static void allocate_bandwidth(netif_t *ip, unsigned long kps);
> > @@ -305,6 +309,30 @@ main(
> > }
> >
> > if (argc > 2) {
> > + if (g_str_equal(argv[2], "--no-dump")) {
> > + no_dump = TRUE;
> > + argv++;
> > + argc--;
> > + }
> > + }
>
> I noticed when reviewing the log file from my "amdump --no-dump" run
> that there were still "INFO dumper" lines in there, from when the
> dumpers start up at the beginning of the run and when they terminate at
> the end (with no activity in between).
>
> I wondered if the dumpers should be getting started at all when no_dump
> is true, and took a look at driver.c to see how the dumpers were getting
> kicked off -- and discovered that driver.c already implements a "nodump"
> option, and that indeed if that option is enabled, the dumper processes
> are not spawned.
>
> (I see that this "nodump" option is used by Amflush.pm 
> .)
>
> Anyway, I guess the question is whether it makes sense to merge the new
> --no-dump and original nodump options for driver.c, at least as far as
> having one internal boolean variable to handle both -- or, if not,
> if there are any places where the code currently checks for !nodump
> where !no_dump should be treated equivalently?
>
> Nathan
>
>
>
> 
> Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ 
> 
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt 
>  
> ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail
diff --git a/perl/Amanda/Amflush.pm b/perl/Amanda/Amflush.pm
index f2f8aa2..8323733 100644
--- a/perl/Amanda/Amflush.pm
+++ b/perl/Amanda/Amflush.pm
@@ -317,10 +317,9 @@ sub run {
 	POSIX::close($driver_pipe);
 	POSIX::dup2(fileno($self->{'amdump_log'}), 1);
 	POSIX::dup2(fileno($self->{'amdump_log'}), 2);
-	debug("exec: " . join(' ', $driver, $self->{'config'}, "nodump", "--no-vault",
-   @log_filename, @config_overrides));
+	debug("exec: " . join(' ', $driver, $self->{'config'}, @log_filename, "--no-dump", "--no-vault", @config_overrides));
 	close($self->{'amdump_log'});
-	exec $driver, $self->{'config'}, "nodump", "--no-vault", @log_filename,
+	exec $driver, $self->{'config'}, @log_filename, "--no-dump", "--no-vault", 
 		  @config_overrides;
 	die "Could not exec $driver: $!";
 }
diff --git a/server-src/driver.c b/server-src/driver.c
index fd01296..1625d06 100644
--- a/server-src/driver.c
+++ b/server-src/driver.c
@@ -79,8 +79,6 @@ static off_t total_disksize;
 static char *dumper_program;
 static char *chunker_program;
 static int  inparallel;
-static int nodump = 0;
-static int novault = 0;
 static storage_t *storage;
 static int conf_max_dle_by_volume;
 static int conf_taperalgo;
@@ -274,22 +272,6 @@ main(
 set_config_overrides(cfg_ovr);
 argv0 = argv[0];
 
-if(argc > 2) {
-if(g_str_equal(argv[2], "nodump")) {
-nodump = 1;
-	argv++;
-	argc--;
-}
-}
-
-if(argc > 2) {
-if(g_str_equal(argv[2], "--no-vault")) {
-novault = 1;
-	argv++;
-	argc--;
-}
-}
-
 log_filename = NULL;
 if (argc > 3) {
 	if (g_str_equal(argv[2], "--log-filename")) {
@@ -393,7 +375,7 @@ main(
 
 /* check that we don't do many dump in a day and usetimestamps is off */
 if(strlen(driver_timestamp) == 8) {
-	if (!nodump) {
+	if (!no_dump) {
 	char *conf_logdir = config_dir_relative(getconf_str(CNF_LOGDIR));
 	char *logfile= g_strjoin(NULL, conf_logdir, "/log.",
 	 driver_timestamp, ".0", NULL);
@@ -533,7 +515,7 @@ main(
 nb_storage 

Re: "vaulting run" amdump email report issues -- driver.c "nodump" option

[Nathan Stratton Treadway]

On Thu, Nov 16, 2017 at 10:15:28 -0500, Jean-Louis Martineau wrote:
> Nathan,
> 
> Try this patch.
> 
> run: amdump --no-dump --no-flush CONFIG
> Will do all scheduled vault to the vault-storage

[...]

> diff --git a/server-src/driver.c b/server-src/driver.c
> index 351494a..fd01296 100644
> --- a/server-src/driver.c
> +++ b/server-src/driver.c
> @@ -107,6 +107,10 @@ static int taper_started = 0;
>  static int nb_storage;
>  static cmddatas_t *cmddatas = NULL;
>  
> +static int no_dump = FALSE;
> +//static int no_flush = FALSE;
> +static int no_vault = FALSE;
> +
>  static int wait_children(int count);
>  static void wait_for_children(void);
>  static void allocate_bandwidth(netif_t *ip, unsigned long kps);
> @@ -305,6 +309,30 @@ main(
>  }
>  
>  if (argc > 2) {
> + if (g_str_equal(argv[2], "--no-dump")) {
> + no_dump = TRUE;
> + argv++;
> + argc--;
> + }
> +}

I noticed when reviewing the log file from my "amdump --no-dump" run
that there were still "INFO dumper" lines in there, from when the
dumpers start up at the beginning of the run and when they terminate at
the end (with no activity in between).

I wondered if the dumpers should be getting started at all when no_dump
is true, and took a look at driver.c to see how the dumpers were getting
kicked off -- and discovered that driver.c already implements a "nodump"
option, and that indeed if that option is enabled, the dumper processes
are not spawned.

(I see that this "nodump" option is used by Amflush.pm.)

Anyway, I guess the question is whether it makes sense to merge the new
--no-dump and original nodump options for driver.c, at least as far as
having one internal boolean variable to handle both -- or, if not,
if there are any places where the code currently checks for !nodump
where !no_dump should be treated equivalently?

Nathan




Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues

[Nathan Stratton Treadway]

On Thu, Nov 16, 2017 at 10:15:28 -0500, Jean-Louis Martineau wrote:
> Nathan,
> 
> Try this patch.
> 
> run: amdump --no-dump --no-flush CONFIG
> Will do all scheduled vault to the vault-storage

I recompiled with this patch applied and repeated the test.  Sure
enough, the above command did do the vaulting for all the dumps saved in
the first phase, and the report it generated did not have the 
  planner: FATAL no DLE to backup
or
  planner: Argument 'BADHOST' matches neither a host nor a disk.
  driver: WARNING: got empty schedule from planner
lines... 

(Also, the DUMP SUMMARY section did included the VAULT operation for
each dump, as expected after the other Report-3.5.diff patch.)


The one thing that wasn't quite correct this time around was that the
FAILURE DUMP SUMMARY section of the report *did* have a RESULTS MISSING
line for each DLE.

After comparing the log files for the run from the original email and
this one, I believe these messages are triggered because using --no-dump
means the log file has a "DISK planner" line for each DLE in the
disklist (lines which don't exist when BADHOST is given).  

(But I'm not sure if the correct solution is for  no_dump = TRUE to also
also suppress those "DISK planner" lines from being generated in the
first place, or if Report.pm should treat a VAULT line in the DUMP
SUMMARY as a valid "result" for that DLE...or something else.)


Nathan



Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues

[Nathan Stratton Treadway]

On Fri, Nov 17, 2017 at 09:10:05 -0500, Jean-Louis Martineau wrote:
> I committed the attached patch to fix the issue.

Yep, with that patch the report now shows the VAULT lines as expected
(when run against the log file from that original "amdump" run).

Thanks.

Nathan




Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues

[Jean-Louis Martineau]

Thanks,

I committed the attached patch to fix the issue.

Jean-Louis


On 16/11/17 11:08 AM, Nathan Stratton Treadway wrote:
> On Thu, Nov 16, 2017 at 09:43:15 -0500, Jean-Louis Martineau wrote:
> > On 16/11/17 08:31 AM, Nathan Stratton Treadway wrote:
> > > DUMP SUMMARY:
> > > DUMPER STATS TAPER STATS
> > > HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s
> > > --- --- --
> > > --
> > It is a bug if they are not listed, can you send me the
> > log..0 file
> >
>
> I have just sent this log file in a separate (off-list) message.
>
> Let me know if you need any other info.
>
> Nathan
>
> 
> Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ 
> 
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt 
>  
> ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail
diff --git a/installcheck/Amanda_Report.pl b/installcheck/Amanda_Report.pl
index bea9f05..ceef57d 100755
--- a/installcheck/Amanda_Report.pl
+++ b/installcheck/Amanda_Report.pl
@@ -28,6 +28,7 @@ use Data::Dumper;
 use lib '@amperldir@';
 
 use Installcheck;
+use Installcheck::Run;
 use Amanda::Config qw( :init :getconf );
 use Amanda::Report;
 use Amanda::Debug;
@@ -41,6 +42,9 @@ my %LogfileData;
 my $logCount = 0;
 my $log_filename = "$Installcheck::TMP/Amanda_Report_test.log";
 
+my $testconf = Installcheck::Run::setup();;
+$testconf->write();
+
 # copy/pasted from installcheck/Amanda_Logfile.pl .  Maybe move this
 # to a module?
 sub write_logfile
diff --git a/perl/Amanda/Report.pm b/perl/Amanda/Report.pm
index 643ea07..5f4ed40 100644
--- a/perl/Amanda/Report.pm
+++ b/perl/Amanda/Report.pm
@@ -932,7 +932,6 @@ sub _handle_dumper_line
 my $self = shift @_;
 my ( $type, $str ) = @_;
 my $data = $self->{data};
-my $disklist = $data->{disklist};
 my $programs = $data->{programs};
 my $dumper_p = $programs->{dumper} ||= {};
 
@@ -953,7 +952,7 @@ sub _handle_dumper_line
 	$kb = int($kb/1024) if $info[$x+1] eq 'bytes';
 $orig_kb =~ s{\]$}{};
 
-my $dle= $disklist->{$hostname}->{$disk};
+my $dle= $self->_get_disklist($hostname, $disk);
 my $try= $self->_get_try( $dle, "dumper", $self->{'run_timestamp'});
 my $dumper = $try->{dumper} ||= {};
 	$dumper->{level} = $level;
@@ -992,7 +991,7 @@ sub _handle_dumper_line
 	$kb = int($kb/1024) if $info[$x+1] eq 'bytes';
 $orig_kb =~ s{\]$}{};
 
-my $dle= $disklist->{$hostname}->{$disk};
+my $dle= $self->_get_disklist($hostname, $disk);
 my $try= $self->_get_try( $dle, "dumper", $timestamp );
 my $dumper = $try->{dumper} ||= {};
 
@@ -1009,7 +1008,7 @@ sub _handle_dumper_line
 my @info = Amanda::Util::split_quoted_strings($str);
 my ( $hostname, $disk, $timestamp, $level ) = @info[ 0 .. 3 ];
 
-my $dle= $disklist->{$hostname}->{$disk};
+my $dle= $self->_get_disklist($hostname, $disk);
 my $try= $self->_get_try( $dle, "dumper", $timestamp );
 my $dumper = $try->{dumper} ||= {};
 	$dumper->{date}  = $timestamp;
@@ -1040,7 +1039,6 @@ sub _handle_chunker_line
 my $self = shift @_;
 my ( $type, $str ) = @_;
 my $data  = $self->{data};
-my $disklist  = $data->{disklist};
 my $programs  = $data->{programs};
 my $chunker_p = $programs->{chunker} ||= {};
 
@@ -1061,7 +1059,7 @@ sub _handle_chunker_line
 	$kb = int($kb/1024) if $info[$x+1] eq 'bytes';
 $kps =~ s{\]$}{};
 
-my $dle = $disklist->{$hostname}->{$disk};
+my $dle = $self->_get_disklist($hostname, $disk);
 my $try = $self->_get_try( $dle, "chunker", $timestamp );
 my $chunker = $try->{chunker} ||= {};
 
@@ -1094,7 +1092,6 @@ sub _handle_taper_line
 my $self = shift @_;
 my ( $type, $str ) = @_;
 my $data = $self->{data};
-my $disklist = $data->{disklist};
 my $programs = $data->{programs};
 my $taper_p  = $programs->{taper} ||= {};
 
@@ -1161,7 +1158,7 @@ sub _handle_taper_line
 $kps =~ s{\]$}{};
 $orig_kb =~ s{\]$}{} if defined($orig_kb);
 
-my $dle   = $disklist->{$hostname}{$disk};
+my $dle   = $self->_get_disklist($hostname, $disk);
 my $try   = $self->_get_try($dle, "taper", $timestamp);
 my $taper = $try->{taper} ||= {};
 my $parts = $taper->{parts} ||= [];
@@ -1232,7 +1229,7 @@ sub _handle_taper_line
 $kps 

Re: "vaulting run" amdump email report issues

[Nathan Stratton Treadway]

On Thu, Nov 16, 2017 at 09:43:15 -0500, Jean-Louis Martineau wrote:
> On 16/11/17 08:31 AM, Nathan Stratton Treadway wrote:
> > DUMP SUMMARY:
> > DUMPER STATS TAPER STATS
> > HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s
> > --- --- -- 
> > --
> It is a bug if they are not listed, can you send me the 
> log..0 file
> 

I have just sent this log file in a separate (off-list) message.

Let me know if you need any other info.

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: "vaulting run" amdump email report issues

[Jean-Louis Martineau]

Nathan,

Try this patch.

run: amdump --no-dump --no-flush CONFIG
Will do all scheduled vault to the vault-storage

Jean-Louis

On 16/11/17 08:31 AM, Nathan Stratton Treadway wrote:
> On Thu, Oct 19, 2017 at 11:06:40 -0400, Jean-Louis Martineau wrote:
> > Since you want to vault all full, I would set 'vault' in the local
> > storage, set 'dump-selection' in the cloud storage, but will not set
> > 'vault-storage'
> > That way the vault are scheduled but are not executed because
> > vault-storage is not set. Amanda know they must be vaulted.
> > Every month, you can run: amdump CONF BADHOST -ovault-storage="cloud"
> > to do the vaulting.
>
> I am testing a vaulting setup where I have "vault TestOffsite 0" set on
> the main storage and a global "vault-storage TestOffsite".
>
> If the TestOffsite vtapes are not available during the nightly amdump
> run, dumps are written to the main storage and I see COPY commands queue
> up command_file for the vault storage.
>
> If I then mount the Offsite vtape partition and run an "amdump
> TestBackup BADHOST" command, the dumps are successfully copied to the
> vault, and amdump sends a report that looks like this:
>
> =
> Hostname: backupserver
> Org : TestBackup
> Config : TestBackup
> Date : November 16, 2017
>
> The next tape Amanda expects to use is: TESTBACKUP-06.
>
>
> FAILURE DUMP SUMMARY:
> planner: FATAL no DLE to backup
>
>
> STATISTICS:
> Total Full Incr. Level:#
>    
> Estimate Time (hrs:min) 0:00
> Run Time (hrs:min) 0:01
> Dump Time (hrs:min) 0:00 0:00 0:00
> Output Size (meg) 0.0 0.0 0.0
> Original Size (meg) 0.0 0.0 0.0
> Avg Compressed Size (%) -- -- --
> DLEs Dumped 0 0 0
> Avg Dump Rate (k/s) -- -- --
>
> Tape Time (hrs:min) 0:00 0:00 0:00
> Tape Size (meg) 0.0 0.0 0.0
> Tape Used (%) 0.0 0.0 0.0
> DLEs Taped 0 0 0
> Parts Taped 0 0 0
> Avg Tp Write Rate (k/s) -- -- --
>
>
> USAGE BY TAPE:
> Label Time Size % DLEs Parts
> TESTBACKUP-102 0:01 7489M 3.7 9 9
>
>
> NOTES:
> planner: Argument 'BADHOST' matches neither a host nor a disk.
> driver: WARNING: got empty schedule from planner
> taper: Slot 15 with label TESTBACKUP-15 is usable
> taper: Slot 2 with label TESTBACKUP-102 is usable
> taper: tape TESTBACKUP-102 kb 7669248 fm 9 [OK]
>
>
> DUMP SUMMARY:
> DUMPER STATS TAPER STATS
> HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s
> --- --- -- 
> --
>
> (brought to you by Amanda version 3.5)
> =
>
>
> The main thing I notice is that nothing is listed in the Dump Summary
> section. As discussed in earlier emails, if the vault storage is
> available during the nightly amdump run, the dumps written to the vault
> show up as VAULT lines in the Dump Summary section so I'm wondering
> if the equivalent dumps performed during a second run should be showing
> up here in this report?
>
>
> A second thing I noticed was the three error/warning messages coming
> from the planner and driver caused by the use of "BADHOST". Unless
> those messages are actually causing the empty Dump Summary section they
> aren't a terrible problem... but especially for sites where the
> recommended configuration is to regularly run a separate vaulting run of
> amdump (such as the one discussed above), I wonder how difficult it
> would be to add support to amdump for a recognized option to turn on a
> "do pending vault operations but don't try to back up any new DLEs"
> mode?
>
> Nathan
>
> 
> Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ 
> 
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt 
>  
> ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail
diff --git a/man/xml-source/amdump.8.xml b/man/xml-source/amdump.8.xml
index ceb12f3..42ee341 100644
--- a/man/xml-source/amdump.8.xml
+++ b/man/xml-source/amdump.8.xml
@@ -30,6 +30,9 @@
 
   amdump
 --no-taper
+--no-dump
+--no-flush
+--no-vault
 --exact-match
 
 config
@@ -78,22 +81,51 @@ man page for more details about Amanda.
 
 
   
-  host [disk]*
+  --no-taper
   
-Specify the host and disk on which the command will work -- see
-the description of DLE specifications in . 
+Disable writting to tape, including flush and vault.
+  
+  
+
+  
+  --no-dump
+  
+Disable dump operations.
+  
+  
+
+  
+  --no-flush
+  
+Disable flush, like setting autoflush to NO for all storages.
+Dump done in this run will be flushed.
+  
+  
+
+  
+  --no-vault
+  

Re: "vaulting run" amdump email report issues

[Jean-Louis Martineau]

On 16/11/17 08:31 AM, Nathan Stratton Treadway wrote:
> On Thu, Oct 19, 2017 at 11:06:40 -0400, Jean-Louis Martineau wrote:
> > Since you want to vault all full, I would set 'vault' in the local
> > storage, set 'dump-selection' in the cloud storage, but will not set
> > 'vault-storage'
> > That way the vault are scheduled but are not executed because
> > vault-storage is not set. Amanda know they must be vaulted.
> > Every month, you can run: amdump CONF BADHOST -ovault-storage="cloud"
> > to do the vaulting.
>
> I am testing a vaulting setup where I have "vault TestOffsite 0" set on
> the main storage and a global "vault-storage TestOffsite".
>
> If the TestOffsite vtapes are not available during the nightly amdump
> run, dumps are written to the main storage and I see COPY commands queue
> up command_file for the vault storage.
>
> If I then mount the Offsite vtape partition and run an "amdump
> TestBackup BADHOST" command, the dumps are successfully copied to the
> vault, and amdump sends a report that looks like this:
>
> =
> Hostname: backupserver
> Org : TestBackup
> Config : TestBackup
> Date : November 16, 2017
>
> The next tape Amanda expects to use is: TESTBACKUP-06.
>
>
> FAILURE DUMP SUMMARY:
> planner: FATAL no DLE to backup
This is expected since there is no dump in this run
>
>
> STATISTICS:
> Total Full Incr. Level:#
>    
> Estimate Time (hrs:min) 0:00
> Run Time (hrs:min) 0:01
> Dump Time (hrs:min) 0:00 0:00 0:00
> Output Size (meg) 0.0 0.0 0.0
> Original Size (meg) 0.0 0.0 0.0
> Avg Compressed Size (%) -- -- --
> DLEs Dumped 0 0 0
> Avg Dump Rate (k/s) -- -- --
>
> Tape Time (hrs:min) 0:00 0:00 0:00
> Tape Size (meg) 0.0 0.0 0.0
> Tape Used (%) 0.0 0.0 0.0
> DLEs Taped 0 0 0
> Parts Taped 0 0 0
> Avg Tp Write Rate (k/s) -- -- --
>
>
> USAGE BY TAPE:
> Label Time Size % DLEs Parts
> TESTBACKUP-102 0:01 7489M 3.7 9 9
9 dle was vaulted, so I think it is a success
>
>
> NOTES:
> planner: Argument 'BADHOST' matches neither a host nor a disk.
> driver: WARNING: got empty schedule from planner
> taper: Slot 15 with label TESTBACKUP-15 is usable
> taper: Slot 2 with label TESTBACKUP-102 is usable
> taper: tape TESTBACKUP-102 kb 7669248 fm 9 [OK]
>
>
> DUMP SUMMARY:
> DUMPER STATS TAPER STATS
> HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s
> --- --- -- 
> --
It is a bug if they are not listed, can you send me the 
log..0 file

>
> (brought to you by Amanda version 3.5)
> =
>
>
> The main thing I notice is that nothing is listed in the Dump Summary
> section. As discussed in earlier emails, if the vault storage is
> available during the nightly amdump run, the dumps written to the vault
> show up as VAULT lines in the Dump Summary section so I'm wondering
> if the equivalent dumps performed during a second run should be showing
> up here in this report?
>
>
> A second thing I noticed was the three error/warning messages coming
> from the planner and driver caused by the use of "BADHOST". Unless
> those messages are actually causing the empty Dump Summary section they
> aren't a terrible problem... but especially for sites where the
> recommended configuration is to regularly run a separate vaulting run of
> amdump (such as the one discussed above), I wonder how difficult it
> would be to add support to amdump for a recognized option to turn on a
> "do pending vault operations but don't try to back up any new DLEs"
> mode?
I will add --no-dump, --no-flush and --no-vault option.

And amvault without argument could run 'driver --no-dump --no-flush'

Jean-Louis
>
> Nathan
>
> 
> Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ 
> 
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt 
>  
> ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail