Re: Amanda "info on small DLEs not saved" bug -- proposed patch

[Gene Heskett]

On Friday 09 November 2018 20:46:03 Gene Heskett wrote:

> On Friday 09 November 2018 18:00:41 Nathan Stratton Treadway wrote:
> > On Fri, Nov 02, 2018 at 13:00:38 -0400, Nathan Stratton Treadway 
wrote:
> > > On Fri, Nov 02, 2018 at 12:35:06 -0400, Gene Heskett wrote:
> > > > Fri Nov 02 03:42:06.085962877 2018: pid 13139: thd-0x9824e00:
> > > > driver: not updating because origsize or dumpsize is 0 Fri Nov
> > > > 02 03:42:06.086158866 2018: pid 13139: thd-0x9824e00: driver:
> > > > Building type FILE header of 32768-32768 bytes with
> > > > name='coyote' disk='/usr/games' dumplevel=1 and blocksize=0
> > > > Fri Nov 02 03:42:06.086317001 2018: pid 13139: thd-0x9824e00:
> > > > driver: Building type FILE header of 32768-32768 bytes with
> > > > name='coyote' disk='/usr/games' dumplevel=1 and blocksize=0
> > > > Fri Nov 02 03:42:06.092807051 2018: pid 13139: thd-0x9824e00:
> > > > driver: driver: send-cmd time 2461.017 to taper0: FILE-WRITE
> > > > worker0-0 00-00130 /usr/dumps/20181102030105/coyote._usr_games.1
> > > > coyote /usr/games 1 20181102030105 "" "" "" 1 "" "" "" "" 10
> > >
> > > Ah, interesting, looks like this applies to level 1 dumps as well.
> > > What does "amadmin CONFIG info coyote /usr/games shop
> > > /usr/lib/amanda" output?
> >
> > Okay, I believe I have tracked down this "info on small DLEs not
> > saved" bug...
> >
> >
> > Summary:
> >
> > The short-ish summary is that in Amanda 3.4 and 3.5, any dump which
> > ends but being shorter than 1024 bytes long (after compression) is
> > treated as not having happened at all, as far as Amanda's recording
> > of "info" statistics goes.
> >
> > This is most significant for full dumps (i.e. of very small
> > partitions), because it causes Amanda to think that DLE has never
> > been dumped (or, that the last time it was dumped was on the final
> > run made using Amanda 3.3 or older), and thus it schedules that DLE
> > for a full dump on every run.
> >
> > However, the bug actually also affects incremental dumps (as we
> > discussed in the above-quoted message) -- so DLEs that don't change
> > much on a particular day and thus end up with very tiny incrementals
> > end up recording those dumps as having taken place on 1969/12/31
> > rather than the day they actually occurred.
> >
> >
> > Neither of the above situations is "fatal" as far as preventing
> > Amanda from actually backup up your data, but for people (such as
> > Gene) who are effected, a workaround workaround is simply to make
> > sure that the dump on a particular day is at least 1024 bytes.
> >
> > For full dumps, you can do this just by creating a "dummy" file on
> > the otherwise-very-empty partition in question, using data that's
> > already compressed so that the dump file is still big enough after
> > Amanda compresses it.  (In my tests, I just used the bytes off the
> > front of the amanda_3.4.2.orig.tar.gz file I happened to have
> > sitting around.)
> >
> > (For incrementals the trick is to make sure there is enough changing
> > on the partition that the incremental dump is over the cutoff size;
> > the best way to do that will depend on what data is on that
> > partition, etc.)
> >
> >
> >
> > Internal details and history:
> >
> > The bug happens because the messages that the chunker and driver
> > processes use to communicate with each other specify the size of the
> > files transferred in integral units of "kb" (=1024 bytes), and thus
> > the size given for very small files is "0" -- but the code in the
> > driver that handles updating the info database has a specific check
> > for zero-length dumps and refuses to update the database in that
> > case. (This check is found in driverio.c:update_info_dumper() .)
> >
> > It appears that the bug has existed since Amanda 3.4, when the old
> > chunker.c version of the chunker program was replaced with a new
> > Perl version.
> >
> > Both server-src/chunker.c as found in Amanda 3.3 and
> > server-src/dumper.c as it exists in 3.5 take special care to round
> > "kb" value passed for files which are short-but-not-empty files up
> > to "1" -- but that logic was not implemented in the Perl chunker
> > code when it was created..
> >
> > (Interestingly, if I am reading the old chunker.c code correctly, it
> > used to round up not just very-small-but-not-empty files to 1 kb,
> > but actually rounded the kb figure up to count any trailing partial
> > kilobytes at the end of the file... while the new program seems to
> > just ignore those trailing partial kilobytes.  Presumably this
> > difference simply doesn't matter -- except for the when the size is
> > rounded down to zero.)
> >
> >
> > Proposed patch:
> >
> > This is where having an actual Amanda developer would be very
> > handy... but given that planner.c and old-chunker.c both have
> > special handling for small-but-not-empty files, I figured that
> > adding a similar check to the new Chunker implementation is probably
> > the best fix for this situation, and that hopefully doing 

Re: reviewing contents of dump file (from vtape)

[Nathan Stratton Treadway]

On Sat, Nov 10, 2018 at 05:08:01 -0500, Gene Heskett wrote:
> But what is the correct gzip syntax to get a listing of the contents of 
> one of these smaller files?
> 
> All I can get out of last nights /usr/local file from the unpack by:
> 
> dd if=00043.coyote._usr_local.1 bs=32k skip=1 | /bin/gzip -dcl --name
> is:
>  compresseduncompressed  ratio uncompressed_name
>  -1  -1   0.0% stdout
> 
> as if there is nothing in the file, and it should have had in this case, 
> all the new sbin/am* files from the rebuilt amanda install, but that 
> file for /usr/local is just short of 47 megs.  So its probably all 
> there, and I'm not holding my mouth right with the gzip invocation.
> 
> okteta says there is data starting at the 32k offset, another 46.9 or so 
> megs of it, but no names visible due to the compression.  Call me 
> puzzled.

The Amanda does compression by piping the entire dump-archive on-the-fly
to a gzip process, so that's why the gzip header doesn't include an
explicit file name.

You use tar for your dumps so you just need to pipe to tar at the end of
your command chain in order to see the contents of the tar archive:

  dd if=00043.coyote._usr_local.1 bs=32k skip=1 | zcat | tar -tvf -

(Or, if you don't mind departing a bit from exactly reversing Amanda's
pipeline, to save a few keystrokes you can let tar invoke gzip rather
than calling it explicitly:
  dd if=00043.coyote._usr_local.1 bs=32k skip=1 | tar -tzvf -
)

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: Upgrade woes and eternal hanging of dumps

[Gene Heskett]

On Friday 25 September 2015 14:05:00 Seann wrote:

> On 9/25/2015 11:44 AM, Joi L. Ellis wrote:
> > Yes, an Amanda 3.3.3 server can properly backup an Amanda 2.5
> > client.  I have a number of such in my universe right now.
> >
> > I need to reinforce something here: Just because your iptables/ufw
> > firewall allows port 10080 UDP to reach into your client and server,
> > that doesn't allow for the TCP connection amdump is going to use to
> > actually transfer the data.  Amcheck doesn't check for TCP
> > connectivity so it will claim the configuration is correct.  On my
> > Centos 5.11 client, I do this:
> >
> > [root@copper ~]# lsmod | grep amanda
> > ip_conntrack_amanda 8901  0
> > ip_conntrack   53665  3
> > ip_conntrack_amanda,ip_conntrack_netbios_ns,xt_state
> >
> > [root@copper sysconfig]# grep amanda /etc/sysconfig/iptables-config
> > IPTABLES_MODULES="ip_conntrack_netbios_ns ip_conntrack_amanda"
> >
> > If you don't have ip_conntrack_amanda in your list of loaded kernel
> > modules, the amdump is probably going to fail.  Copper is one of my
> > mission-critical internal servers with public exposure, so it has a
> > very tight iptables firewall configured.  I had issues with it
> > failing to backup until I forcibly configured it to load
> > ip_conntrack_amanda.
> >
> > It would be a very useful addition to amcheck if it could somehow
> > test that the client and server can actually create an amdump TCP
> > connection, because it would illuminate the need to configure
> > ip_conntrack_amanda or lf_conntrack_amanda.
> >
> >> -Original Message-
> >> From: Seann [mailto:nombran...@tsukinokage.net]
> >> Sent: Thursday, September 24, 2015 06:35 PM
> >> To: Debra S Baddorf ; Joi L. Ellis
> >> 
> >> Cc: amanda-users@amanda.org
> >> Subject: Re: Upgrade woes and eternal hanging of dumps
> >>
> >> On 9/21/2015 11:36 AM, Debra S Baddorf wrote:
> >>> YES!I agree with the first and third of these tidbits.  I just
> >>
> >> couldn't remember them.  I've had issues with both of them. 
> >> Including the tricky firewall timeout part,  in Idea Three.
> >>
> >>> Here's hoping you have a network person who can add some skills or
> >>> ideas
> >>
> >> at that level.   Or,  just don't do client estimates,  as in the
> >> first suggested fix.
> >>
> >>> I think we had to allow trusted clients to initiate their OWN
> >>
> >> connections back to the server  (via a firewall rule),  so that
> >> they could still talk to the server even after that server-created
> >> conversation had timed out.   That might count as fix #3,  but it
> >> takes firewall skills. That might be a slightly different
> >> problem/situation  (it sounds a little different)  but I think it's
> >> in this same category, somewhere.  Network savvy people, can you
> >> translate my "generic English" description into what we actually
> >> did?
> >>
> >>> Deb Baddorf,  Fermilab
> >>>
> >>>
> >>> On Sep 21, 2015, at 10:25 AM, Joi L. Ellis
> >>> 
> >>
> >> wrote:
>  I've just read through the long thread prompted by this
>  particular
> >>
> >> post.  I'd like to offer a few points I didn't see mentioned
> >> before...
> >>
>  Idea one: You upgraded from 2.5 to 3.3.  2.5 amdump only spoke
>  UDP with
> >>
> >> a 'bsd' auth protocol, so that was the only action available. 
> >> Thus, inetd.conf didn't specify an -auth=bsd parameter.  3.3
> >> defaults to - auth=bsdtcp if you don't provide it.  Does your new
> >> configuration specify that those clients must be reached with
> >> -auth=bsd from the new server, rather than the server's new default
> >> of -auth=bsdctp?
> >>
>  Idea two: If any of the involved machines are running iptables or
>  ufw
> >>
> >> firewalls, verify the new configuration is still loading the
> >> correct kernel modules. At one point the /etc/default/ufw.conf file
> >> named kernel modules incorrectly after an upgrade, and/or the
> >> nf_conntrack_amanda module itself went missing.  (Some kernels
> >> change the name of this module, usually it's the first two
> >> characters.)  The symptom here is that amcheck thinks everything is
> >> fine, yet the actual amdump process fails because the UDP control
> >> conversation between the server and the client is allowed, but the
> >> TCP data stream amdump uses with -auth=bsdtcp is blocked.
> >>
>  Idea Three: I run an Amanda 3.3.3 server, and I have experienced
>  a
> >>
> >> similar problem to your own.  I've tried posting about it here in
> >> the past and got null response, so I gave up asking for help and
> >> figured out my own workarounds.
> >>
>  My amanda server is behind a corporate firewall, and some of the
> >>
> >> clients are in the DMZ, outside the firewall... and they are
> >> running amanda 2.5 due to the age of the client hosts.   I've had
> >> repeated issues with the corporate firewall interfering with the
> >> planner.
> >>
>  The issue I see is that the amanda server planner fires off a UDP
> >>
> >> "connect

Re: All level 0 on the same run?

[Nathan Stratton Treadway]

On Tue, Oct 30, 2018 at 15:51:36 -0400, Gene Heskett wrote:
> I just changed the length of the dumpcycle and runs percycle up to 10, 
> about last friday while I was makeing the bump* stuff more attractive, 
> but the above command returns that the are 5 filesystens out of date:
> su amanda -c "/usr/local/sbin/amadmin Daily balance"
> 
>  due-date  #fsorig MB out MB   balance
> --
> 10/30 Tue5  0  0  ---
> 10/31 Wed1  17355   8958-45.3%
> 11/01 Thu2  10896  10887-33.5%
> 11/02 Fri4  35944   9298-43.2%
> 11/03 Sat4  14122  10835-33.8%
> 11/04 Sun3  57736  57736   +252.7%
> 11/05 Mon2  39947  30635+87.1%
> 11/06 Tue8   4235   4215-74.3%
> 11/07 Wed4  19503  14732-10.0%
> 11/08 Thu   32  31783  16408 +0.2%
> --
> TOTAL   65 231521 163704 16370

Okay, now that the small-DLE distraction is out of the way, we can get
back to the original question regarding the scheduling of dumps over
your dumpcycle. 

What does your "balance" output show now?  

(In particular, I'm curious if there is still one day with a huge surge
like shown for 11/04 in the listing above.)


Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: reviewing contents of dump file (from vtape)

[Gene Heskett]

On Saturday 10 November 2018 10:38:36 Nathan Stratton Treadway wrote:

> On Sat, Nov 10, 2018 at 05:08:01 -0500, Gene Heskett wrote:
> > But what is the correct gzip syntax to get a listing of the contents
> > of one of these smaller files?
> >
> > All I can get out of last nights /usr/local file from the unpack by:
> >
> > dd if=00043.coyote._usr_local.1 bs=32k skip=1 | /bin/gzip -dcl
> > --name is:
> >  compresseduncompressed  ratio uncompressed_name
> >  -1  -1   0.0% stdout
> >
> > as if there is nothing in the file, and it should have had in this
> > case, all the new sbin/am* files from the rebuilt amanda install,
> > but that file for /usr/local is just short of 47 megs.  So its
> > probably all there, and I'm not holding my mouth right with the gzip
> > invocation.
> >
> > okteta says there is data starting at the 32k offset, another 46.9
> > or so megs of it, but no names visible due to the compression.  Call
> > me puzzled.
>
> The Amanda does compression by piping the entire dump-archive
> on-the-fly to a gzip process, so that's why the gzip header doesn't
> include an explicit file name.
>
> You use tar for your dumps so you just need to pipe to tar at the end
> of your command chain in order to see the contents of the tar archive:
>
>   dd if=00043.coyote._usr_local.1 bs=32k skip=1 | zcat | tar -tvf -
>
> (Or, if you don't mind departing a bit from exactly reversing Amanda's
> pipeline, to save a few keystrokes you can let tar invoke gzip rather
> than calling it explicitly:
>   dd if=00043.coyote._usr_local.1 bs=32k skip=1 | tar -tzvf -
> )
>
>   Nathan
And that last example worked "purr"fectly. Can I blame it on oldtimers?

Thanks Nathan.

Copyright 2018 by Maurice E. Heskett
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: reviewing contents of dump file (from vtape)

[Gene Heskett]

On Saturday 10 November 2018 10:38:36 Nathan Stratton Treadway wrote:

> On Sat, Nov 10, 2018 at 05:08:01 -0500, Gene Heskett wrote:
> > But what is the correct gzip syntax to get a listing of the contents
> > of one of these smaller files?
> >
> > All I can get out of last nights /usr/local file from the unpack by:
> >
> > dd if=00043.coyote._usr_local.1 bs=32k skip=1 | /bin/gzip -dcl
> > --name is:
> >  compresseduncompressed  ratio uncompressed_name
> >  -1  -1   0.0% stdout
> >
> > as if there is nothing in the file, and it should have had in this
> > case, all the new sbin/am* files from the rebuilt amanda install,
> > but that file for /usr/local is just short of 47 megs.  So its
> > probably all there, and I'm not holding my mouth right with the gzip
> > invocation.
> >
> > okteta says there is data starting at the 32k offset, another 46.9
> > or so megs of it, but no names visible due to the compression.  Call
> > me puzzled.
>
> The Amanda does compression by piping the entire dump-archive
> on-the-fly to a gzip process, so that's why the gzip header doesn't
> include an explicit file name.
>
> You use tar for your dumps so you just need to pipe to tar at the end
> of your command chain in order to see the contents of the tar archive:
>
>   dd if=00043.coyote._usr_local.1 bs=32k skip=1 | zcat | tar -tvf -
>
> (Or, if you don't mind departing a bit from exactly reversing Amanda's
> pipeline, to save a few keystrokes you can let tar invoke gzip rather
> than calling it explicitly:
>   dd if=00043.coyote._usr_local.1 bs=32k skip=1 | tar -tzvf -
> )
>
>   Nathan
I ran this last, discovering that I was also backing up the day old 
indices, then added a wc -l to the end of that, getting  1777 for a file 
count. So its all there. And in a bare metal re-install, I really should 
recover this first, then over-write that with the indices file which 
does contain todays indices when doing a bare metal re-install, This 
file only contains yesterdays indices. So I really ought to compose a 
full recovery script to make sure its all done in the correct sequence.

So one should edit his disklist to put this important stuff first, and 
change the sequence string in ones amanda conf to tell amanda to do it 
in the disklist order. But I'm unsure of how to do that.  Hummm. There 
are probably at least 9 ways to skin this cat, and its one I haven't 
given a huge consideration to, been concentrating on just getting all 
the data.  The cat has a slight resemblance to Shrodinger's famous 
feline...

Sometimes it seems to fade away, leaving only the grin that says "aha, I 
gotcha". :)

Yeah, we really should put up a 3.5.1p1 and fix everybody's install.

Copyright 2018 by Maurice E. Heskett
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: All level 0 on the same run?

[Chris Nighswonger]

On Sat, Nov 10, 2018 at 10:57 AM Nathan Stratton Treadway <
natha...@ontko.com> wrote:

> On Tue, Oct 30, 2018 at 15:51:36 -0400, Gene Heskett wrote:
> > I just changed the length of the dumpcycle and runs percycle up to 10,
> > about last friday while I was makeing the bump* stuff more attractive,
> > but the above command returns that the are 5 filesystens out of date:
> > su amanda -c "/usr/local/sbin/amadmin Daily balance"
> >
> >  due-date  #fsorig MB out MB   balance
> > --
> > 10/30 Tue5  0  0  ---
> > 10/31 Wed1  17355   8958-45.3%
> > 11/01 Thu2  10896  10887-33.5%
> > 11/02 Fri4  35944   9298-43.2%
> > 11/03 Sat4  14122  10835-33.8%
> > 11/04 Sun3  57736  57736   +252.7%
> > 11/05 Mon2  39947  30635+87.1%
> > 11/06 Tue8   4235   4215-74.3%
> > 11/07 Wed4  19503  14732-10.0%
> > 11/08 Thu   32  31783  16408 +0.2%
> > --
> > TOTAL   65 231521 163704 16370
>
> Okay, now that the small-DLE distraction is out of the way, we can get
> back to the original question regarding the scheduling of dumps over
> your dumpcycle.
>
> What does your "balance" output show now?
>

I think that is output from one of Gene's systems, but here is the latest
from mine after DLE balancing has been through one successful run. (The
next run will take place Monday morning @ 0200).

backup@scriptor:~ amadmin campus balance

 due-date  #fsorig kB out kB   balance
--
11/10 Sat4  676705513  447515666   +178.0%
11/11 Sun889438595250400-96.7%
11/12 Mon   12  127984592   84074623-47.8%
11/13 Tue   19  304110025  267932333+66.5%
11/14 Wed0  0  0  ---
--
TOTAL   43 1117743989  804773022 160954604
  (estimated 5 runs per dumpcycle)
 (3 filesystems overdue. The most being overdue 17841 days.)

Not sure what's up with the overdues. There were none prior to breaking up
the DLEs. It may just be an artifact.

Kind regards,
Chris

>
> (In particular, I'm curious if there is still one day with a huge surge
> like shown for 11/04 in the listing above.)
>
>
> Nathan
>
>
> 
> Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
> Ray Ontko & Co.  -  Software consulting services  -
> http://www.ontko.com/
>  GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
>  Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239
>

Re: All level 0 on the same run?

[Gene Heskett]

On Saturday 10 November 2018 10:47:03 Nathan Stratton Treadway wrote:

> On Tue, Oct 30, 2018 at 15:51:36 -0400, Gene Heskett wrote:
> > I just changed the length of the dumpcycle and runs percycle up to
> > 10, about last friday while I was makeing the bump* stuff more
> > attractive, but the above command returns that the are 5 filesystens
> > out of date: su amanda -c "/usr/local/sbin/amadmin Daily balance"
> >
> >  due-date  #fsorig MB out MB   balance
> > --
> > 10/30 Tue5  0  0  ---
> > 10/31 Wed1  17355   8958-45.3%
> > 11/01 Thu2  10896  10887-33.5%
> > 11/02 Fri4  35944   9298-43.2%
> > 11/03 Sat4  14122  10835-33.8%
> > 11/04 Sun3  57736  57736   +252.7%
> > 11/05 Mon2  39947  30635+87.1%
> > 11/06 Tue8   4235   4215-74.3%
> > 11/07 Wed4  19503  14732-10.0%
> > 11/08 Thu   32  31783  16408 +0.2%
> > --
> > TOTAL   65 231521 163704 16370
>
> Okay, now that the small-DLE distraction is out of the way, we can get
> back to the original question regarding the scheduling of dumps over
> your dumpcycle.
>
> What does your "balance" output show now?
>
> (In particular, I'm curious if there is still one day with a huge
> surge like shown for 11/04 in the listing above.)
>
>
>   Nathan
Its some better:
amanda@coyote:/amandatapes/Dailys/data$ /usr/local/sbin/amadmin Daily 
balance

 due-date  #fsorig MB out MB   balance
--
11/10 Sat1   7912   3145-78.7%
11/11 Sun1  10886  10886-26.1%
11/12 Mon1  32963   7875-46.6%
11/13 Tue1   7688   7688-47.8%
11/14 Wed2  22109  22109+50.0%
11/15 Thu4  75027  46623   +216.3%
11/16 Fri6   8257   6109-58.6%
11/17 Sat   29  14034   8932-39.4%
11/18 Sun4  21281  16842+14.3%
11/19 Mon   18  34599  17188+16.6%
--
TOTAL   67 234756 147397 14739

It will be interesting to see if it continues to get "better".
I should think it will be under 150% by the 15nth if so. If the planner 
behaves itself.

I'm of the opinion now that this bug has been tickling it wrong for much 
more than the last 30 days or so that its been visible with the update 
from 3.3.7p1 I'd been running forever. balance reports weren't all that 
encouraging and the planner was half out of it mind trying to shuffle 
things to help, without ever getting in "balance".

We shall see. Perhaps I could add a balance report to the end of 
backup.sh so I get it emailed to me every morning? I'll take a look 
after ingesting enough caffeine to get both eyes open simultaneously. 

I've been laying around & taking care of the missus while waiting on 
stuff from China to rebuild the interfaces on my Grizzly g0704 milling 
machine, destroyed by a failure of a $2 buck regulator that failed 
shorted, putting 35 volts on the 5 volt vcc line of the breakout board. 
That obviously gave up the ghost too. The rebuild will include a 
dedicated 5 volt supply and the 35 volt parts to supply some of the 
other line power controlling ice cube relays that control jigs and 
vacuums to suck up the swarf when I'm making furniture parts on it. I'm 
rather partial to the Green & Green huge box joint, which lends itself 
to being carved on cnc machinery.  I wrote that code too.

Obviously I have way to many "hobbies" ;-) I bought 2 years ago, a lathe 
big enough to do some gunsmithing work, and since I bought a 70 yo 
Sheldon model that had been badly abused, rebuilt it to be cnc 
controlled, which works well considering I didn't use a regular pc to do 
it, but an r-pi-3b, breaking new ground. Good enough I used it to put a 
new barrel on old meat in the pot last fall which I also reload for it, 
was a 30-06 Ackley Improved, now a 6.5 Creedmoor thats quite a bit 
easier of the old mans shoulder. Shoots well again, the old Douglas 
barrel was getting rusty, with accuracy to match. ;-)

Thank you Nathan.

Copyright 2018 by Maurice E. Heskett
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: All level 0 on the same run?

[Nathan Stratton Treadway]

On Sat, Nov 10, 2018 at 12:48:15 -0500, Gene Heskett wrote:
> On Saturday 10 November 2018 10:47:03 Nathan Stratton Treadway wrote:
> 
> > On Tue, Oct 30, 2018 at 15:51:36 -0400, Gene Heskett wrote:
> > > I just changed the length of the dumpcycle and runs percycle up to
> > > 10, about last friday while I was makeing the bump* stuff more
> > > attractive, but the above command returns that the are 5 filesystens
> > > out of date: su amanda -c "/usr/local/sbin/amadmin Daily balance"
> > >
> > >  due-date  #fsorig MB out MB   balance
> > > --
> > > 10/30 Tue5  0  0  ---
> > > 10/31 Wed1  17355   8958-45.3%
> > > 11/01 Thu2  10896  10887-33.5%
> > > 11/02 Fri4  35944   9298-43.2%
> > > 11/03 Sat4  14122  10835-33.8%
> > > 11/04 Sun3  57736  57736   +252.7%
> > > 11/05 Mon2  39947  30635+87.1%
> > > 11/06 Tue8   4235   4215-74.3%
> > > 11/07 Wed4  19503  14732-10.0%
> > > 11/08 Thu   32  31783  16408 +0.2%
> > > --
> > > TOTAL   65 231521 163704 16370
> >
> > Okay, now that the small-DLE distraction is out of the way, we can get
> > back to the original question regarding the scheduling of dumps over
> > your dumpcycle.
> >
> > What does your "balance" output show now?
> >
> > (In particular, I'm curious if there is still one day with a huge
> > surge like shown for 11/04 in the listing above.)
> >
> >
> > Nathan
> Its some better:
> amanda@coyote:/amandatapes/Dailys/data$ /usr/local/sbin/amadmin Daily 
> balance
> 
>  due-date  #fsorig MB out MB   balance
> --
> 11/10 Sat1   7912   3145-78.7%
> 11/11 Sun1  10886  10886-26.1%
> 11/12 Mon1  32963   7875-46.6%
> 11/13 Tue1   7688   7688-47.8%
> 11/14 Wed2  22109  22109+50.0%
> 11/15 Thu4  75027  46623   +216.3%
> 11/16 Fri6   8257   6109-58.6%
> 11/17 Sat   29  14034   8932-39.4%
> 11/18 Sun4  21281  16842+14.3%
> 11/19 Mon   18  34599  17188+16.6%
> --
> TOTAL   67 234756 147397 14739
> 
> It will be interesting to see if it continues to get "better".
> I should think it will be under 150% by the 15nth if so. If the planner 
> behaves itself.

Off hand I am still suspicious of the entry here for 11/15, both because
the data size is very high for only 4 DLEs and because you've gone
through a full cycle since your previous listing and the huge surge
hasn't evened out very much.  So I'm guessing that one of those DLEs is
probably very large compared to all your other ones

Anyway, my next step would be to figure out which 4 DLEs are the ones in
that group, which you should be able to do by looking through the output
of "/usr/local/sbin/amadmin Daily due".  For example, try
  /usr/local/sbin/amadmin Daily due | grep "5 day" 
and see if you get 4 DLEs listed (and adjust the count by a day if that
shows you the wrong group of DLEs).

Once you see which four are in that group, you can cross reference with
your Amanda mail reports to figure out the relative sizes of those
particular DLEs.

> 
> I'm of the opinion now that this bug has been tickling it wrong for much 
> more than the last 30 days or so that its been visible with the update 
> from 3.3.7p1 I'd been running forever. balance reports weren't all that 
> encouraging and the planner was half out of it mind trying to shuffle 
> things to help, without ever getting in "balance".

(I am pretty sure that the small-DLE bug didn't affect the overall
balance.  You can see from the "balance" output on 10/30 that the 5 DLEs
in question all show up as needing to be full-dumped that day -- but the
total size for that group is still zero.  So I suspect that there is/are
some other factor(s) behind the single-day surge and whatever
"shuffling" has been going on ...)

> 
> We shall see. Perhaps I could add a balance report to the end of 
> backup.sh so I get it emailed to me every morning? I'll take a look 
> after ingesting enough caffeine to get both eyes open simultaneously. 

Yes, if you are trying to really understand what's going on with the
scheduling it can certainly be useful to be able to watch the day-to-day
changes to the balance listing.

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0

Re: reviewing contents of dump file (from vtape)

[Nathan Stratton Treadway]

On Sat, Nov 10, 2018 at 11:40:09 -0500, Gene Heskett wrote:
> So one should edit his disklist to put this important stuff first, and 
> change the sequence string in ones amanda conf to tell amanda to do it 
> in the disklist order. But I'm unsure of how to do that.  Hummm. There 

You can get Amanda to order the dumps in a given run based on various
criteria, but the order they are listed in the disklist is not one of
those options...  And no matter what criteria you put in your config,
you really can't predict the specific order the dumps will be written
because that depends on the actual time it takes each dump to happen
(relative to all the other dumps that were underway in parallel at dump
time).

But as far a doing a correct bare-metal recovery, it really doesn't
matter at all what order the dump files appear in the vtape directory --
all that matters is the order you restore them onto the
newly-reconstituted system


Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: All level 0 on the same run?

[Gene Heskett]

On Saturday 10 November 2018 13:55:30 Nathan Stratton Treadway wrote:

> On Sat, Nov 10, 2018 at 12:48:15 -0500, Gene Heskett wrote:
> > On Saturday 10 November 2018 10:47:03 Nathan Stratton Treadway wrote:
> > > On Tue, Oct 30, 2018 at 15:51:36 -0400, Gene Heskett wrote:
> > > > I just changed the length of the dumpcycle and runs percycle up
> > > > to 10, about last friday while I was makeing the bump* stuff
> > > > more attractive, but the above command returns that the are 5
> > > > filesystens out of date: su amanda -c "/usr/local/sbin/amadmin
> > > > Daily balance"
> > > >
> > > >  due-date  #fsorig MB out MB   balance
> > > > --
> > > > 10/30 Tue5  0  0  ---
> > > > 10/31 Wed1  17355   8958-45.3%
> > > > 11/01 Thu2  10896  10887-33.5%
> > > > 11/02 Fri4  35944   9298-43.2%
> > > > 11/03 Sat4  14122  10835-33.8%
> > > > 11/04 Sun3  57736  57736   +252.7%
> > > > 11/05 Mon2  39947  30635+87.1%
> > > > 11/06 Tue8   4235   4215-74.3%
> > > > 11/07 Wed4  19503  14732-10.0%
> > > > 11/08 Thu   32  31783  16408 +0.2%
> > > > --
> > > > TOTAL   65 231521 163704 16370
> > >
> > > Okay, now that the small-DLE distraction is out of the way, we can
> > > get back to the original question regarding the scheduling of
> > > dumps over your dumpcycle.
> > >
> > > What does your "balance" output show now?
> > >
> > > (In particular, I'm curious if there is still one day with a huge
> > > surge like shown for 11/04 in the listing above.)
> > >
> > >
> > >   Nathan
> >
> > Its some better:
> > amanda@coyote:/amandatapes/Dailys/data$ /usr/local/sbin/amadmin
> > Daily balance
> >
> >  due-date  #fsorig MB out MB   balance
> > --
> > 11/10 Sat1   7912   3145-78.7%
> > 11/11 Sun1  10886  10886-26.1%
> > 11/12 Mon1  32963   7875-46.6%
> > 11/13 Tue1   7688   7688-47.8%
> > 11/14 Wed2  22109  22109+50.0%
> > 11/15 Thu4  75027  46623   +216.3%
> > 11/16 Fri6   8257   6109-58.6%
> > 11/17 Sat   29  14034   8932-39.4%
> > 11/18 Sun4  21281  16842+14.3%
> > 11/19 Mon   18  34599  17188+16.6%
> > --
> > TOTAL   67 234756 147397 14739
> >
> > It will be interesting to see if it continues to get "better".
> > I should think it will be under 150% by the 15nth if so. If the
> > planner behaves itself.
>
> Off hand I am still suspicious of the entry here for 11/15, both
> because the data size is very high for only 4 DLEs and because you've
> gone through a full cycle since your previous listing and the huge
> surge hasn't evened out very much.  So I'm guessing that one of those
> DLEs is probably very large compared to all your other ones
>
> Anyway, my next step would be to figure out which 4 DLEs are the ones
> in that group, which you should be able to do by looking through the
> output of "/usr/local/sbin/amadmin Daily due".  For example, try
>   /usr/local/sbin/amadmin Daily due | grep "5 day"
> and see if you get 4 DLEs listed (and adjust the count by a day if
> that shows you the wrong group of DLEs).
>
> Once you see which four are in that group, you can cross reference
> with your Amanda mail reports to figure out the relative sizes of
> those particular DLEs.
>
> > I'm of the opinion now that this bug has been tickling it wrong for
> > much more than the last 30 days or so that its been visible with the
> > update from 3.3.7p1 I'd been running forever. balance reports
> > weren't all that encouraging and the planner was half out of it mind
> > trying to shuffle things to help, without ever getting in "balance".
>
> (I am pretty sure that the small-DLE bug didn't affect the overall
> balance.  You can see from the "balance" output on 10/30 that the 5
> DLEs in question all show up as needing to be full-dumped that day --
> but the total size for that group is still zero.  So I suspect that
> there is/are some other factor(s) behind the single-day surge and
> whatever "shuffling" has been going on ...)
>
> > We shall see. Perhaps I could add a balance report to the end of
> > backup.sh so I get it emailed to me every morning? I'll take a look
> > after ingesting enough caffeine to get both eyes open
> > simultaneously.
>
> Yes, if you are trying to really understand what's going on with the
> scheduling it can certainly be useful to be able to watch the
> day-to-day changes to the balance listing.
>
>   Nathan
The biggest problem is where do I put the stuff I have downloaded, which 
includes several multi-gig

Re: reviewing contents of dump file (from vtape)

[Gene Heskett]

On Saturday 10 November 2018 14:13:17 Nathan Stratton Treadway wrote:

> On Sat, Nov 10, 2018 at 11:40:09 -0500, Gene Heskett wrote:
> > So one should edit his disklist to put this important stuff first,
> > and change the sequence string in ones amanda conf to tell amanda to
> > do it in the disklist order. But I'm unsure of how to do that. 
> > Hummm. There
>
> You can get Amanda to order the dumps in a given run based on various
> criteria, but the order they are listed in the disklist is not one of
> those options...  And no matter what criteria you put in your config,
> you really can't predict the specific order the dumps will be written
> because that depends on the actual time it takes each dump to happen
> (relative to all the other dumps that were underway in parallel at
> dump time).
>
> But as far a doing a correct bare-metal recovery, it really doesn't
> matter at all what order the dump files appear in the vtape directory
> -- all that matters is the order you restore them onto the
> newly-reconstituted system
>
Drives seem to be in good health even tho one of them is well into 
100,000 hours of spin time here.  Next big disturbance will probably be 
a jump from wheezy to stretch on all the PC based machines. Since the pi 
is running fairly well on jessie, and its much harder to change, it will 
probably stay on jessie at least till both jessie is out of support AND 
the next version has at least a preempt-rt kernel. Unfortunately, 
neither of the CC sized boards uses grub to boot, so its 20x harder to 
install a new kernel just to test it.

Take care.
>   Nathan
>
> --
>-- Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic
> region Ray Ontko & Co.  -  Software consulting services  -  
> http://www.ontko.com/ GPG Key:
> http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239 Key
> fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239



Copyright 2018 by Maurice E. Heskett
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: All level 0 on the same run?

[Nathan Stratton Treadway]

On Sat, Nov 10, 2018 at 11:39:58 -0500, Chris Nighswonger wrote:
> I think that is output from one of Gene's systems, but here is the latest
> from mine after DLE balancing has been through one successful run. (The
> next run will take place Monday morning @ 0200).
> 
> backup@scriptor:~ amadmin campus balance
> 
>  due-date  #fsorig kB out kB   balance
> --
> 11/10 Sat4  676705513  447515666   +178.0%
> 11/11 Sun889438595250400-96.7%
> 11/12 Mon   12  127984592   84074623-47.8%
> 11/13 Tue   19  304110025  267932333+66.5%
> 11/14 Wed0  0  0  ---
> --
> TOTAL   43 1117743989  804773022 160954604
>   (estimated 5 runs per dumpcycle)
>  (3 filesystems overdue. The most being overdue 17841 days.)
> 
> Not sure what's up with the overdues. There were none prior to breaking up
> the DLEs. It may just be an artifact.

With a 5-day dumpcycle that would mean Amanda thinks the last dump took
place 17846-ish days ago:
  $ date --date="17846 days ago"
  Wed Dec 31 14:24:39 EST 1969
... and the date of 1969/12/13 is the "no data saved" placeholder date
within Amanda's info database.

Anyway, you should be able to identify the three DLEs in question with
  amadmin campus due | grep "Overdue"
and then use "amadmin campus info [...]" to see what amanda has recorded
about them.

I guess there should also be one DLE listed in the "amadmin ... due"
output as being due "today".  It would be interesting to see the info
for that one as well (in order to understand the components of the very
large total size shown on the 11/10 line).

Because Amanda won't postpone any of those for DLEs unless it really has
to, the balance will presumably still have the +175-ish% "surge" after
your run on Monday... but in the runs after that it might try to spread
things out som emore (for example, by promoting a few of the DLEs
currently included in the 11/13 line).

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239