date:20181115

On Thursday 15 November 2018 14:17:29 Austin S. Hemmelgarn wrote:

> On 2018-11-15 13:36, Gene Heskett wrote:
> > On Thursday 15 November 2018 12:57:54 Austin S. Hemmelgarn wrote:
> >> On 2018-11-15 11:53, Gene Heskett wrote:
> >>> On Thursday 15 November 2018 07:36:37 Austin S. Hemmelgarn wrote:
>  On 2018-11-15 06:16, Gene Heskett wrote:
> > I ask because after last nights run it showed one huge and 3
> > teeny level 0's for the 4 new dle's.  So I just re-adjusted the
> > locations of some categories and broke the big one up into 2
> > pieces. "./[A-P]*" and ./[Q-Z]*", so the next run will have 5
> > new dle's.
> >
> > But an estimate does not show the new names that results in.
> > I've even took the estimate assignment calcsize back out of the
> > global dumptype, which ack the manpage, forces the estimates to
> > be derived from a dummy run of tar, didn't help.
> >
> > Clues? Having this info from an estimate query might take a
> > couple hours, but it sure would be helpfull when redesigning
> > ones dle's.I'm fairly certain you can't, because it specifically
> > shows server-side
> 
>  estimates, which have no data to work from if there has never
>  been a dump run for the DLE.
> >>>
> >>> Even if you told it to user tar for the estimate phase? That has
> >>> enough legs to be called a bug. IMO anyway.
> >>
> >> As mentioned in one of my other responses, I can kind of see the
> >> value in this not bothering the client systems.  Keep in mind that
> >> server estimates cost nothing on the client, while calcsize or
> >> client estimates may use a significant amount of resources.
> >
> > My default has been calcsize for three or 4 years, changed because
> > tar was changed & was screwing up the estimates. I can remember 15+
> > years ago when I was using real tar estimates, on a much smaller
> > machine, and it could come within 50 megabytes of filling a DDS-2
> > tape (4 GB compressed) for weeks at a time. So that part of amanda
> > worked a lot better than it does today. And its slowly gone to the
> > dogs as my system grew in complexity.  And went in a handbasket when
> > I had to change to calcsize during the tar churn.
>
> I've not been using AMANDA anywhere near as long as you have, but I've
> actually not seen any issues with accuracy of 'estimate client' mode
> estimates with current versions of GNU tar, except when the estimate
> ran while data in the DLE was being modified (and in that case, it
> makes sense that it would be bogus).

Absolutely, but I don't make it a habit to start downloading install 
iso's when amanda is about to run.

> I generally don't 'estimate 
> client' on my own systems though because it consistently takes far
> longer than 'estimate calcsize', and I'm not picky about the estimates
> being perfect.
>
> >> In this case, I do think the documentation should be a bit clearer,
> >
> > Yes, but who is to rewrite it?  He should know a heck of a lot more
> > than I do about the amanda innards than I do even after 2 decades,
> > and better defined words here and there too. diakdevice is a very
> > poor substitute for the far more common slanguage of "/path/to/"
> >
> >> and it would be useful to be able to get regular (calcsize and/or
> >> client) estimates on-demand, but I do think that the default is
> >> reasonably sane.
> >
> > It may well be sane, we'll see how it works in the morning. AIUI,
> > calcsize runs only on old history. so that should not impinge a load
> > on the client, even when the client is itself.
>
> Unless I'm mistaken:
>
> * 'estimate server' runs only on historical data, and doesn't even
> talk to the client systems.  It's good at limiting the impact the
> estimate has on the client, but reliably gives bogus estimates if your
> DLEs don't show consistent behavior (that is, each backup of a given
> level is roughly the same size as every other backup at that level).
> * 'estimate client' relies on the backup program being used to give it
> info about how big it will be.  It gives estimates that are close to
> 100% accurate, but currently essentially requires running the backup
> process twice (once for the estimate, once for the actual backup) and
> imposes a non-negligible amount of load on the client.
> * 'estimate calcsize' does something kind of in-between.  AIUI, it
> looks at some historical data, and also looks at the on-disk size of
> the data, then factors in compression ratios and such to give an
> estimate that's usually reasonably accurate without needing the DLEs
> to be consistent or imposing significant load on the clients.



Copyright 2018 by Maurice E. Heskett
-- 
Cheers everybody, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page

Re: Does anyone know how to make an amadmin $config estimate work for new dle's?

On Thursday 15 November 2018 14:17:29 Austin S. Hemmelgarn wrote:

> On 2018-11-15 13:36, Gene Heskett wrote:
> > On Thursday 15 November 2018 12:57:54 Austin S. Hemmelgarn wrote:
> >> On 2018-11-15 11:53, Gene Heskett wrote:
> >>> On Thursday 15 November 2018 07:36:37 Austin S. Hemmelgarn wrote:
>  On 2018-11-15 06:16, Gene Heskett wrote:
> > I ask because after last nights run it showed one huge and 3
> > teeny level 0's for the 4 new dle's.  So I just re-adjusted the
> > locations of some categories and broke the big one up into 2
> > pieces. "./[A-P]*" and ./[Q-Z]*", so the next run will have 5
> > new dle's.
> >
> > But an estimate does not show the new names that results in.
> > I've even took the estimate assignment calcsize back out of the
> > global dumptype, which ack the manpage, forces the estimates to
> > be derived from a dummy run of tar, didn't help.
> >
> > Clues? Having this info from an estimate query might take a
> > couple hours, but it sure would be helpfull when redesigning
> > ones dle's.I'm fairly certain you can't, because it specifically
> > shows server-side
> 
>  estimates, which have no data to work from if there has never
>  been a dump run for the DLE.
> >>>
> >>> Even if you told it to user tar for the estimate phase? That has
> >>> enough legs to be called a bug. IMO anyway.
> >>
> >> As mentioned in one of my other responses, I can kind of see the
> >> value in this not bothering the client systems.  Keep in mind that
> >> server estimates cost nothing on the client, while calcsize or
> >> client estimates may use a significant amount of resources.
> >
> > My default has been calcsize for three or 4 years, changed because
> > tar was changed & was screwing up the estimates. I can remember 15+
> > years ago when I was using real tar estimates, on a much smaller
> > machine, and it could come within 50 megabytes of filling a DDS-2
> > tape (4 GB compressed) for weeks at a time. So that part of amanda
> > worked a lot better than it does today. And its slowly gone to the
> > dogs as my system grew in complexity.  And went in a handbasket when
> > I had to change to calcsize during the tar churn.
>
> I've not been using AMANDA anywhere near as long as you have, but I've
> actually not seen any issues with accuracy of 'estimate client' mode
> estimates with current versions of GNU tar, except when the estimate
> ran while data in the DLE was being modified (and in that case, it
> makes sense that it would be bogus).  I generally don't 'estimate
> client' on my own systems though because it consistently takes far
> longer than 'estimate calcsize', and I'm not picky about the estimates
> being perfect.
>
> >> In this case, I do think the documentation should be a bit clearer,
> >
> > Yes, but who is to rewrite it?  He should know a heck of a lot more
> > than I do about the amanda innards than I do even after 2 decades,
> > and better defined words here and there too. diakdevice is a very
> > poor substitute for the far more common slanguage of "/path/to/"
> >
> >> and it would be useful to be able to get regular (calcsize and/or
> >> client) estimates on-demand, but I do think that the default is
> >> reasonably sane.
> >
> > It may well be sane, we'll see how it works in the morning. AIUI,
> > calcsize runs only on old history. so that should not impinge a load
> > on the client, even when the client is itself.
>
> Unless I'm mistaken:
>
> * 'estimate server' runs only on historical data, and doesn't even
> talk to the client systems.  It's good at limiting the impact the
> estimate has on the client, but reliably gives bogus estimates if your
> DLEs don't show consistent behavior (that is, each backup of a given
> level is roughly the same size as every other backup at that level).
> * 'estimate client' relies on the backup program being used to give it
> info about how big it will be.  It gives estimates that are close to
> 100% accurate, but currently essentially requires running the backup
> process twice (once for the estimate, once for the actual backup) and
> imposes a non-negligible amount of load on the client.

That depends on the clients instant duty's. I have backed up a milling 
machine while it was running a 90 lines of code, 3 days to finish while 
sharpening a saw blade, with no apparent interaction on a dual core atom 
powered box. One core was locked away for LCNC, (isolcpus at work) the 
other was free to do the backup client. Didn't bither it a bot. :)

> * 'estimate calcsize' does something kind of in-between.  AIUI, it
> looks at some historical data, and also looks at the on-disk size of
> the data,

That would take time to access the dle's, and the answer is effectively 
instant, ergo it is not questioning the client(s), it has to be working 
only from the history in its own logs.

> then factors in compression ratios and such to give an 
> estimate that's usually reasonably

Re: tape pools, wrapping my mind around them

2018-11-15 Thread Jean-Louis Martineau

You want 3 storage
A vtape can only contains backup from one storage.
You use the storage 'dump_selection' and the dumptype 'tag' to specify
which dle go to that storage

You could use a different pool for each storage.

But using the same pool for all storage have a big benefits:

   - no need to allocate a specific range of slot for each storage

sales use vtape-001, after a few days it is no longer needed (from the
policy), then research or support could use vtape-001

Jean-Louis


Le jeu. 15 nov. 2018, à 15 h 30, Jon LaBadie  a écrit :

> I currently have 240 Vtapes in a single changer.
> Suppose I have 3 "departments", sales, research,
> and support.
>
> Could I create 3 dumptype definitions so that
> all "sales" DLEs are backed up to a tape pool
> consisting of vtapes 1-80, "research" to vtapes
> 81-160, and "support" to vtapes 161-240?
>
> If so, then a single config could handle keeping
> the various data separated while still maintaining
> balance etc.  Of course, is the balance on a tape
> pool basis or on a total configuration basis?
>
> Jon
> --
> Jon H. LaBadie j...@jgcomp.com
>  11226 South Shore Rd.  (703) 787-0688 (H)
>  Reston, VA  20190  (703) 935-6720 (C)
>

RE: tape pools, wrapping my mind around them

I believe that you would define tape pool of 80 for each, with regular 
expressions you may be able to have the same label string format but different 
tape number ranges.
Maybe like this?

Sales POOL1[0-9][0-9]
Support POOL2[0-9][0-9]
Research POOL3[0-9][0-9]

For sanity I think I'd define 3 tape pools in the one changer. You spend time 
loading and unloading tapes to find the pool and tapes you want, but since this 
is done without having to access a physical tape drive or robot it will take 
virtually no time at all.

-Original Message-
From: owner-amanda-us...@amanda.org  On Behalf 
Of Jon LaBadie
Sent: Thursday, November 15, 2018 3:25 PM
To: amanda-users@amanda.org
Subject: tape pools, wrapping my mind around them

ATTENTION: This email came from an external source. Do not open attachments or 
click on links from unknown senders or unexpected emails.


I currently have 240 Vtapes in a single changer.
Suppose I have 3 "departments", sales, research, and support.

Could I create 3 dumptype definitions so that all "sales" DLEs are backed up to 
a tape pool consisting of vtapes 1-80, "research" to vtapes 81-160, and 
"support" to vtapes 161-240?

If so, then a single config could handle keeping the various data separated 
while still maintaining balance etc.  Of course, is the balance on a tape pool 
basis or on a total configuration basis?

Jon
--
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

RE: Clients that return something else

I think if you want a disk image you want the native OS disk image utility, 
(file-system-type)dump. Dump, ufsdump, xfsdump, etc. Amanda supports this.

Do you ever need a disk image other than the boot volume? I realize some 
special applications may have file types that are not recognized for backup by 
tar, but that would be pretty rare.

From: owner-amanda-us...@amanda.org  On Behalf 
Of Chris Miller
Sent: Thursday, November 15, 2018 2:23 PM
To: amanda-users 
Subject: Re: Clients that return something else


ATTENTION: This email came from an external source. Do not open attachments or 
click on links from unknown senders or unexpected emails.

Hi Brian
From: "Cuttler, Brian R (HEALTH)" 
mailto:brian.cutt...@health.ny.gov>>
To: "Chris Miller" mailto:c...@tryx.org>>, "amanda-users" 
mailto:amanda-users@amanda.org>>
Sent: Thursday, November 15, 2018 10:54:49 AM
Subject: RE: Clients that return something else
Why pipe dd to tar when you can just run tar?
Good question. tar works at the filesystem level but dd works at the disk block 
level and I'm not aware of any way that tar can create a disk image, so I need 
to read the disk with dd. AMANDA expects a tar saveset, so I need to pipe 
anything I create to tar.



Er – I think the answer is “yes”, but you may have to roll your own.
Yeah, so do I. I'm just not exactly sure how I tell the client what to do. It 
appears that the dumptype uses something symbolic, and leaves the client up to 
its own devices to determine what it means. I could also do this, but I'd 
really like to be able to define the script on the server. Also, it's not 
exactly clear to me how the client understands what "GNUTAR" or "DUMP" means 
locally -- something must see "GNUTAR" and conclude, "Oh, he wants to run 
/usr/sbin/tar". For example, if I could put "BASH" in my dumptype definition 
for "program", and include that code somehow, that would be perfect! Ever hear 
of anything like that?

Thanks for the help, Brian.
--
Chris.

V:916.974.0424
F:916.974.0428

tape pools, wrapping my mind around them

2018-11-15 Thread Jon LaBadie

I currently have 240 Vtapes in a single changer.
Suppose I have 3 "departments", sales, research,
and support.

Could I create 3 dumptype definitions so that
all "sales" DLEs are backed up to a tape pool
consisting of vtapes 1-80, "research" to vtapes
81-160, and "support" to vtapes 161-240?

If so, then a single config could handle keeping
the various data separated while still maintaining
balance etc.  Of course, is the balance on a tape
pool basis or on a total configuration basis?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Clients that return something else

2018-11-15 Thread Chris Hoogendyk

This is a bit
dated, but may give you an idea of how to attack it. However, it could be that with the api work
that Dustin did there is a better way of doing this with the newer versions of Amanda. I'm not on
Solaris anymore, and I've updated most of my Amanda installations, so I haven't been using this
wrapper for some time.

On 11/15/18 2:22 PM, Chris Miller wrote:

Hi Brian

*From: *"Cuttler, Brian R (HEALTH)"
*To: *"Chris Miller" , "amanda-users"

*Sent: *Thursday, November 15, 2018 10:54:49 AM
*Subject: *RE: Clients that return something else

Why pipe dd to tar when you can just run tar?

Good question. tar works at the filesystem level but dd works at the disk block level and I'm not
aware of any way that tar can create a disk image, so I need to read the disk with dd. AMANDA
expects a tar saveset, so I need to pipe anything I create to tar.

Er – I think the answer is “yes”, but you may have to roll your own.

Yeah, so do I. I'm just not exactly sure how I tell the client what to do. It appears that the
dumptype uses something symbolic, and leaves the client up to its own devices to determine what it
means. I could also do this, but I'd really like to be able to define the script on the server.
Also, it's not exactly clear to me how the client understands what "GNUTAR" or "DUMP" means
locally -- something must see "GNUTAR" and conclude, "Oh, he wants to run /usr/sbin/tar". For
example, if I could put "BASH" in my dumptype definition for "program", and include that code
somehow, that would be perfect! Ever hear of anything like that?

Thanks for the help, Brian.
--
Chris.

V:916.974.0424
F:916.974.0428

--
---

Chris Hoogendyk

-
O__ Systems Administrator
c/ /'_ --- Biology & Geosciences Departments
(*) \(*) -- 315 Morrill Science Center
~~ - University of Massachusetts, Amherst

---

Erdös 4

Re: Clients that return something else

2018-11-15 Thread Debra S Baddorf




> On Nov 15, 2018, at 1:22 PM, Chris Miller  wrote:
> 
> Hi Brian
> From: "Cuttler, Brian R (HEALTH)" 
> To: "Chris Miller" , "amanda-users" 
> Sent: Thursday, November 15, 2018 10:54:49 AM
> Subject: RE: Clients that return something else
> Why pipe dd to tar when you can just run tar?
> Good question. tar works at the filesystem level but dd works at the disk 
> block level and I'm not aware of any way that tar can create a disk image, so 
> I need to read the disk with dd. AMANDA expects a tar saveset, so I need to 
> pipe anything I create to tar.
> 
> 
> 
> Er – I think the answer is “yes”, but you may have to roll your own.
> Yeah, so do I. I'm just not exactly sure how I tell the client what to do. It 
> appears that the dumptype uses something symbolic, and leaves the client up 
> to its own devices to determine what it means. I could also do this, but I'd 
> really like to be able to define the script on the server. Also, it's not 
> exactly clear to me how the client understands what "GNUTAR" or "DUMP" means 
> locally -- something must see "GNUTAR" and conclude, "Oh, he wants to run 
> /usr/sbin/tar". For example, if I could put "BASH" in my dumptype definition 
> for "program", and include that code somehow, that would be perfect! Ever 
> hear of anything like that?
> 
> Thanks for the help, Brian.
> --
> Chris.

Well, you do have to point amanda to tar  and also to dump,  when you compile 
amanda.
Just yesterday had to recompile amanda, since dump and restore had been 
installed AFTER
the first compilation. And there’s a flag you use to point to tar.

This would be on each client.

Deb Baddorf

Re: Clients that return something else

Hi Brian 

> From: "Cuttler, Brian R (HEALTH)" 
> To: "Chris Miller" , "amanda-users" 
> Sent: Thursday, November 15, 2018 10:54:49 AM
> Subject: RE: Clients that return something else
> Why pipe dd to tar when you can just run tar?

Good question. tar works at the filesystem level but dd works at the disk block 
level and I'm not aware of any way that tar can create a disk image, so I need 
to read the disk with dd. AMANDA expects a tar saveset, so I need to pipe 
anything I create to tar. 

> Er – I think the answer is “yes”, but you may have to roll your own.

Yeah, so do I. I'm just not exactly sure how I tell the client what to do. It 
appears that the dumptype uses something symbolic, and leaves the client up to 
its own devices to determine what it means. I could also do this, but I'd 
really like to be able to define the script on the server. Also, it's not 
exactly clear to me how the client understands what "GNUTAR" or "DUMP" means 
locally -- something must see "GNUTAR" and conclude, "Oh, he wants to run 
/usr/sbin/tar". For example, if I could put "BASH" in my dumptype definition 
for "program", and include that code somehow, that would be perfect! Ever hear 
of anything like that? 

Thanks for the help, Brian. 
-- 
Chris. 

V:916.974.0424 
F:916.974.0428

Re: Does anyone know how to make an amadmin $config estimate work for new dle's?


On 2018-11-15 13:36, Gene Heskett wrote:

On Thursday 15 November 2018 12:57:54 Austin S. Hemmelgarn wrote:


On 2018-11-15 11:53, Gene Heskett wrote:

On Thursday 15 November 2018 07:36:37 Austin S. Hemmelgarn wrote:

On 2018-11-15 06:16, Gene Heskett wrote:

I ask because after last nights run it showed one huge and 3 teeny
level 0's for the 4 new dle's.  So I just re-adjusted the
locations of some categories and broke the big one up into 2
pieces. "./[A-P]*" and ./[Q-Z]*", so the next run will have 5 new
dle's.

But an estimate does not show the new names that results in. I've
even took the estimate assignment calcsize back out of the global
dumptype, which ack the manpage, forces the estimates to be
derived from a dummy run of tar, didn't help.

Clues? Having this info from an estimate query might take a couple
hours, but it sure would be helpfull when redesigning ones
dle's.I'm fairly certain you can't, because it specifically shows
server-side


estimates, which have no data to work from if there has never been
a dump run for the DLE.


Even if you told it to user tar for the estimate phase? That has
enough legs to be called a bug. IMO anyway.


As mentioned in one of my other responses, I can kind of see the value
in this not bothering the client systems.  Keep in mind that server
estimates cost nothing on the client, while calcsize or client
estimates may use a significant amount of resources.


My default has been calcsize for three or 4 years, changed because tar
was changed & was screwing up the estimates. I can remember 15+ years
ago when I was using real tar estimates, on a much smaller machine, and
it could come within 50 megabytes of filling a DDS-2 tape (4 GB
compressed) for weeks at a time. So that part of amanda worked a lot
better than it does today. And its slowly gone to the dogs as my system
grew in complexity.  And went in a handbasket when I had to change to
calcsize during the tar churn.
I've not been using AMANDA anywhere near as long as you have, but I've 
actually not seen any issues with accuracy of 'estimate client' mode 
estimates with current versions of GNU tar, except when the estimate ran 
while data in the DLE was being modified (and in that case, it makes 
sense that it would be bogus).  I generally don't 'estimate client' on 
my own systems though because it consistently takes far longer than 
'estimate calcsize', and I'm not picky about the estimates being perfect.
  

In this case, I do think the documentation should be a bit clearer,


Yes, but who is to rewrite it?  He should know a heck of a lot more than
I do about the amanda innards than I do even after 2 decades, and better
defined words here and there too. diakdevice is a very poor substitute
for the far more common slanguage of "/path/to/"

and it would be useful to be able to get regular (calcsize and/or
client) estimates on-demand, but I do think that the default is
reasonably sane.


It may well be sane, we'll see how it works in the morning. AIUI,
calcsize runs only on old history. so that should not impinge a load on
the client, even when the client is itself.

Unless I'm mistaken:

* 'estimate server' runs only on historical data, and doesn't even talk 
to the client systems.  It's good at limiting the impact the estimate 
has on the client, but reliably gives bogus estimates if your DLEs don't 
show consistent behavior (that is, each backup of a given level is 
roughly the same size as every other backup at that level).
* 'estimate client' relies on the backup program being used to give it 
info about how big it will be.  It gives estimates that are close to 
100% accurate, but currently essentially requires running the backup 
process twice (once for the estimate, once for the actual backup) and 
imposes a non-negligible amount of load on the client.
* 'estimate calcsize' does something kind of in-between.  AIUI, it looks 
at some historical data, and also looks at the on-disk size of the data, 
then factors in compression ratios and such to give an estimate that's 
usually reasonably accurate without needing the DLEs to be consistent or 
imposing significant load on the clients.

RE: Clients that retunr something else


I’m configured amanda to run native client utilities gnutar, dump, star (shilly 
tar), er something else.

There are certain known and available/approved things you can do, but the 
bottom line is always – Amanda is an intelligent, scheduling, wrapper, 
client-server using OS native tools.

I think I was thinking pigzip (name?) rather than gzip.

Why pipe dd to tar when you can just run tar?

Note – some versions of gtar are considered broken for Amanda, don’t recall the 
versions or the specific reason (might have had to do with argument handling) 
but that is also likely ancient history.

For zfs file systems we’ve also utilized the snapshot option, perhaps out of 
reach because of the way you’ve globbed your DLEs together.

Er – I think the answer is “yes”, but you may have to roll your own.

From: owner-amanda-us...@amanda.org  On Behalf 
Of Chris Miller
Sent: Thursday, November 15, 2018 1:31 PM
To: amanda-users 
Subject: Clients that retunr something else


ATTENTION: This email came from an external source. Do not open attachments or 
click on links from unknown senders or unexpected emails.

Hi Folks,

Is it possible to have the client run something else besides "tar", like, for 
example, "dd | tar"? Can I specify this from the server? Of course, it goes 
without saying that I need to be able to do the same thing on a Windows client, 
and now I've just said it, so maybe it needed to be said.

Thanks for the help,
--
Chris.

V:916.974.0424
F:916.974.0428

RE: Configuration confusion

I suppose you could have a small modification to your config file every day.

(scripted to run dynamically, # date has very useful output formatting)

# amdump config-type-1-day-${dayname}

With 7 dumps/week and three configs he can have 21 different backup files, 
completely defeating the scheduling but getting exactly the result he wants.

-Original Message-
From: Debra S Baddorf  
Sent: Thursday, November 15, 2018 1:39 PM
To: Cuttler, Brian R (HEALTH) 
Cc: Debra S Baddorf ; Charles Curley 
; amanda-users 
Subject: Re: Configuration confusion

ATTENTION: This email came from an external source. Do not open attachments or 
click on links from unknown senders or unexpected emails.


If you’ve disable level 0 backups in the setup,  *WILL*  it do a level 0,  even 
if you try to force it?
I have one hge config that is set up this way.   I have to edit the 
amanda.conf  and REMOVE the  “incr-only” lines
when I want to force a level 0.   Specially since you HAVE to start with a 
level 0, initially.
I’m pretty sure I’ve tried to force some level 0’s and found they failed,  till 
I removed the  “incr-only”  bits.

Deb Baddorf

> On Nov 15, 2018, at 12:16 PM, Cuttler, Brian R (HEALTH) 
>  wrote:
>
> Alternatively - configure all DLE to be non-fulls, disable level 0 backups 
> entirely and run cron jobs to force level 0 dumps on particular DLEs.
> That way you can get level 0 when you want it to occur an no other DLE will 
> advance to a level 0 on its own.
>
> # amadmin config force client DLE
>
> -Original Message-
> From: owner-amanda-us...@amanda.org  On 
> Behalf Of Charles Curley
> Sent: Thursday, November 15, 2018 12:31 PM
> To: amanda-users 
> Subject: Re: Configuration confusion
>
> ATTENTION: This email came from an external source. Do not open attachments 
> or click on links from unknown senders or unexpected emails.
>
>
> On Thu, 15 Nov 2018 09:03:00 -0800 (PST) Chris Miller  wrote:
>
>> If I run three backups, serial or otherwise, then do they know about 
>> each other? Meaning, is AMANDA smart enough to know not to run more 
>> than one level 0 dump per night? The problem is that level 0 backups 
>> take several hours and if I run multiple then I will still be 
>> completing last nights backup when everybody comes in the next 
>> morning. That would be embarrassing. "Sorry, I didn't complete my 
>> work last night, so you can't continue yours."
>
> Ah, that helps. My experience is in a SOHO environment, so take with the salt 
> shaker handy.
>
> The different configurations don't know about each other at all. So you could 
> in theory have a night in which all three run level 0 backups.
> The only way I know to get that kind of co-ordination is to have one 
> configuration which then backs up all three machines. Unfortunately your 
> requirement not to mix the backups due to custodial and security requirements 
> may kill that idea.
>
> Another thing to look at is to break your DLEs up into lots of smaller DLEs. 
> You'll get more level 0 backups, but they'll be spread around the week more 
> evenly.
>
> Or consider having a longer tape cycle. That means fewer level 0 backups in 
> any one week.
>
> --
> "When we talk of civilization, we are too apt to limit the meaning of the 
> word to its mere embellishments, such as arts and sciences; but the true 
> distinction between it and barbarism is, that the one presents a state of 
> society under the protection of just and well-administered law, and the other 
> is left to the chance government of brute force."
> - The Rev. James White, Eighteen Christian Centuries, 1889 Key 
> fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__charlescurley.com
> =DwIDAg=gRgGjJ3BkIsb5y6s49QqsA=HMrKaRiCv4jddln9fLPIOw=fnbC4Yi7
> 6_5ky_32Tf9A5Geluildi-avCP3JC0hU2RA=oia-zSsMzlRk8WRCNW_-9vpn6VWUX7Vm
> XSyeRVdmgcM=
>

Re: Configuration confusion

2018-11-15 Thread Debra S Baddorf

If you’ve disable level 0 backups in the setup,  *WILL*  it do a level 0,  even 
if you try to force it?   
I have one hge config that is set up this way.   I have to edit the 
amanda.conf  and REMOVE the  “incr-only” lines
when I want to force a level 0.   Specially since you HAVE to start with a 
level 0, initially.
I’m pretty sure I’ve tried to force some level 0’s and found they failed,  till 
I removed the  “incr-only”  bits.

Deb Baddorf

> On Nov 15, 2018, at 12:16 PM, Cuttler, Brian R (HEALTH) 
>  wrote:
> 
> Alternatively - configure all DLE to be non-fulls, disable level 0 backups 
> entirely and run cron jobs to force level 0 dumps on particular DLEs.
> That way you can get level 0 when you want it to occur an no other DLE will 
> advance to a level 0 on its own.
> 
> # amadmin config force client DLE
> 
> -Original Message-
> From: owner-amanda-us...@amanda.org  On Behalf 
> Of Charles Curley
> Sent: Thursday, November 15, 2018 12:31 PM
> To: amanda-users 
> Subject: Re: Configuration confusion
> 
> ATTENTION: This email came from an external source. Do not open attachments 
> or click on links from unknown senders or unexpected emails.
> 
> 
> On Thu, 15 Nov 2018 09:03:00 -0800 (PST) Chris Miller  wrote:
> 
>> If I run three backups, serial or otherwise, then do they know about 
>> each other? Meaning, is AMANDA smart enough to know not to run more 
>> than one level 0 dump per night? The problem is that level 0 backups 
>> take several hours and if I run multiple then I will still be 
>> completing last nights backup when everybody comes in the next 
>> morning. That would be embarrassing. "Sorry, I didn't complete my work 
>> last night, so you can't continue yours."
> 
> Ah, that helps. My experience is in a SOHO environment, so take with the salt 
> shaker handy.
> 
> The different configurations don't know about each other at all. So you could 
> in theory have a night in which all three run level 0 backups.
> The only way I know to get that kind of co-ordination is to have one 
> configuration which then backs up all three machines. Unfortunately your 
> requirement not to mix the backups due to custodial and security requirements 
> may kill that idea.
> 
> Another thing to look at is to break your DLEs up into lots of smaller DLEs. 
> You'll get more level 0 backups, but they'll be spread around the week more 
> evenly.
> 
> Or consider having a longer tape cycle. That means fewer level 0 backups in 
> any one week.
> 
> --
> "When we talk of civilization, we are too apt to limit the meaning of the 
> word to its mere embellishments, such as arts and sciences; but the true 
> distinction between it and barbarism is, that the one presents a state of 
> society under the protection of just and well-administered law, and the other 
> is left to the chance government of brute force."
> - The Rev. James White, Eighteen Christian Centuries, 1889 Key fingerprint = 
> CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__charlescurley.com=DwIDAg=gRgGjJ3BkIsb5y6s49QqsA=HMrKaRiCv4jddln9fLPIOw=fnbC4Yi76_5ky_32Tf9A5Geluildi-avCP3JC0hU2RA=oia-zSsMzlRk8WRCNW_-9vpn6VWUX7VmXSyeRVdmgcM=
>

Re: Does anyone know how to make an amadmin $config estimate work for new dle's?

On Thursday 15 November 2018 12:57:54 Austin S. Hemmelgarn wrote:

> On 2018-11-15 11:53, Gene Heskett wrote:
> > On Thursday 15 November 2018 07:36:37 Austin S. Hemmelgarn wrote:
> >> On 2018-11-15 06:16, Gene Heskett wrote:
> >>> I ask because after last nights run it showed one huge and 3 teeny
> >>> level 0's for the 4 new dle's.  So I just re-adjusted the
> >>> locations of some categories and broke the big one up into 2
> >>> pieces. "./[A-P]*" and ./[Q-Z]*", so the next run will have 5 new
> >>> dle's.
> >>>
> >>> But an estimate does not show the new names that results in. I've
> >>> even took the estimate assignment calcsize back out of the global
> >>> dumptype, which ack the manpage, forces the estimates to be
> >>> derived from a dummy run of tar, didn't help.
> >>>
> >>> Clues? Having this info from an estimate query might take a couple
> >>> hours, but it sure would be helpfull when redesigning ones
> >>> dle's.I'm fairly certain you can't, because it specifically shows
> >>> server-side
> >>
> >> estimates, which have no data to work from if there has never been
> >> a dump run for the DLE.
> >
> > Even if you told it to user tar for the estimate phase? That has
> > enough legs to be called a bug. IMO anyway.
>
> As mentioned in one of my other responses, I can kind of see the value
> in this not bothering the client systems.  Keep in mind that server
> estimates cost nothing on the client, while calcsize or client
> estimates may use a significant amount of resources.
>
My default has been calcsize for three or 4 years, changed because tar 
was changed & was screwing up the estimates. I can remember 15+ years 
ago when I was using real tar estimates, on a much smaller machine, and 
it could come within 50 megabytes of filling a DDS-2 tape (4 GB 
compressed) for weeks at a time. So that part of amanda worked a lot 
better than it does today. And its slowly gone to the dogs as my system 
grew in complexity.  And went in a handbasket when I had to change to 
calcsize during the tar churn.
 
> In this case, I do think the documentation should be a bit clearer,

Yes, but who is to rewrite it?  He should know a heck of a lot more than 
I do about the amanda innards than I do even after 2 decades, and better 
defined words here and there too. diakdevice is a very poor substitute 
for the far more common slanguage of "/path/to/"
> and it would be useful to be able to get regular (calcsize and/or
> client) estimates on-demand, but I do think that the default is
> reasonably sane.

It may well be sane, we'll see how it works in the morning. AIUI, 
calcsize runs only on old history. so that should not impinge a load on 
the client, even when the client is itself.

Copyright 2018 by Maurice E. Heskett
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page

Clients that retunr something else

Hi Folks, 

Is it possible to have the client run something else besides "tar", like, for 
example, "dd | tar"? Can I specify this from the server? Of course, it goes 
without saying that I need to be able to do the same thing on a Windows client, 
and now I've just said it, so maybe it needed to be said. 

Thanks for the help, 
-- 
Chris. 

V:916.974.0424 
F:916.974.0428

Re: Configuration confusion

Hi Brian and Deb,

> Alternatively - configure all DLE to be non-fulls, disable level 0 backups
> entirely and run cron jobs to force level 0 dumps on particular DLEs.
> That way you can get level 0 when you want it to occur an no other DLE will
> advance to a level 0 on its own.

This is a very good solution, I think. I already have the requirement that I 
collect a disk image once per cycle, which AMANDA can't schedule, so I am 
already in "self-scheduling" territory. Can you tell me how I disable level 0 
backups?

Thanks for the help, 
-- 
Chris. 

V:916.974.0424 
F:916.974.0428

RE: Configuration confusion

From: owner-amanda-us...@amanda.org  On Behalf 
Of Chris Miller
Sent: Thursday, November 15, 2018 12:29 PM
To: amanda-users 
Subject: Re: Configuration confusion

ATTENTION: This email came from an external source. Do not open attachments or 
click on links from unknown senders or unexpected emails.

Hi Brian,
From: "Cuttler, Brian R (HEALTH)" 
mailto:brian.cutt...@health.ny.gov>>
To: "Cuttler, Brian R (HEALTH)" 
mailto:brian.cutt...@health.ny.gov>>, "Chris 
Miller" mailto:c...@tryx.org>>, "amanda-users" 
mailto:amanda-users@amanda.org>>
Sent: Wednesday, November 14, 2018 9:08:26 AM
Subject: RE: Configuration confusion
Tape custody – means what, retention policy or storage of the tape when not in 
the drive/juke?
Yes. Simplest is local custody. Off site custody comes in two media flavors -- 
local media and cloud. Off site costs, so we want to minimize this, as well as 
increases the response time for restorations. You get the idea. I don't use 
tapes; I use removable disks, optical media,and usb keydrives. General this is 
determined by the client, so when I plan this, I simply consider which client 
backups need to be sequestered where. This gives me a configuration where I can 
think in terms of "client  gets backedup to NAS ", where NAS  has 
different properties and different dispositions.

  *   Yes, I understand. I have worked at sites where offsite was someone’s 
house (tapes never came back in the right cycle, seems if you use them as hills 
under your train set you might not return the oldest tapes but bring back a 
mix). Other sites had tapes stored in the fire protected computer room, still 
others had them in a room in another part of the building, but it is a very big 
building.

Amanda is not an archiver in the sense that the tapes are cycled on a regular 
basis. You are able to tape a tape out of rotation and replace it, or create a 
unique tape label and perform level 0 backups to it and then mark it as 
no-reuse in the tapelist, but the primary function is not long term archiving, 
though the tools exist to do that very well.

You can use the same tape pool for all three Amanda configs, but they will need 
to have a common tapelist file. But if you are doing that then you are 
selecting a single set of standards for your data-at-rest security, in which 
case there is little reason to maintain 3 different configs.

Each Amanda config will look to level nightly data, but you will have nights 
with relatively little and nights with relatively large data volume swings, er 
think wave interference from physics. You eliminate a lot of that by combining 
disklists into a single configuration.
Yes! "Each Amanda config will look to level nightly data ..."  This is my 
principle question, and I seek to level the nightly data across all configs on 
a given night, which I recognize can't be done, so I seek to combine my 
multiple configs into a single config which specifies multiple sets of DLEs 
being mapped to multiple tapedev, if that can be done. For example, if the 
definition of tapedev had a "DLE ..." argument, and AMANDA were capable of this 
additional dimension of scheduling.

  *   See reply I put in subsequent email by Charles Curley that was a response 
to you.

Amanda will backup a new DLE at level 0 the first time it sees it. If you are 
worried about running long you will want to phase in the DLEs across several 
evenings. You may want to add the largest on Friday night, assuming that no one 
cares how late Amanda runs into Saturday. You will want to avoid adding 
multiple large DLEs on a single night, add a large and a small each night until 
they are all added.
"Phasing" my backup jobs may be my only choice, as that is exactly the problem 
I seek to avoid and I have been admonished to not try to subvert the scheduler 
by forcing level 0 backups to happen on my schedule. As I continue to discuss 
this, I am more and more convinced that AMANDA cannot schedule in two 
dimensions {(backups / night) X (nights / cycle)}

So, suppose I wanted to force my level 0 backups to happen at my discretion, so 
I can level my nightly run times. How would I do that?

Thanks for the help, Brian.
--
Chris.

V:916.974.0424
F:916.974.0428

RE: Configuration confusion

Alternatively - configure all DLE to be non-fulls, disable level 0 backups 
entirely and run cron jobs to force level 0 dumps on particular DLEs.
That way you can get level 0 when you want it to occur an no other DLE will 
advance to a level 0 on its own.

# amadmin config force client DLE

-Original Message-
From: owner-amanda-us...@amanda.org  On Behalf 
Of Charles Curley
Sent: Thursday, November 15, 2018 12:31 PM
To: amanda-users 
Subject: Re: Configuration confusion

ATTENTION: This email came from an external source. Do not open attachments or 
click on links from unknown senders or unexpected emails.

On Thu, 15 Nov 2018 09:03:00 -0800 (PST) Chris Miller  wrote:

> If I run three backups, serial or otherwise, then do they know about 
> each other? Meaning, is AMANDA smart enough to know not to run more 
> than one level 0 dump per night? The problem is that level 0 backups 
> take several hours and if I run multiple then I will still be 
> completing last nights backup when everybody comes in the next 
> morning. That would be embarrassing. "Sorry, I didn't complete my work 
> last night, so you can't continue yours."

Ah, that helps. My experience is in a SOHO environment, so take with the salt 
shaker handy.

The different configurations don't know about each other at all. So you could 
in theory have a night in which all three run level 0 backups.
The only way I know to get that kind of co-ordination is to have one 
configuration which then backs up all three machines. Unfortunately your 
requirement not to mix the backups due to custodial and security requirements 
may kill that idea.

Another thing to look at is to break your DLEs up into lots of smaller DLEs. 
You'll get more level 0 backups, but they'll be spread around the week more 
evenly.

Or consider having a longer tape cycle. That means fewer level 0 backups in any 
one week.

--
"When we talk of civilization, we are too apt to limit the meaning of the word 
to its mere embellishments, such as arts and sciences; but the true distinction 
between it and barbarism is, that the one presents a state of society under the 
protection of just and well-administered law, and the other is left to the 
chance government of brute force."
- The Rev. James White, Eighteen Christian Centuries, 1889 Key fingerprint = 
CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB https://charlescurley.com

Re: Configuration confusion

2018-11-15 Thread Debra S Baddorf




> On Nov 15, 2018, at 11:29 AM, Chris Miller  wrote:
> 
> Hi Brian,
> From: "Cuttler, Brian R (HEALTH)" 
> To: "Cuttler, Brian R (HEALTH)" , "Chris Miller" 
> , "amanda-users" 
> Sent: Wednesday, November 14, 2018 9:08:26 AM
> Subject: RE: Configuration confusion
> Tape custody – means what, retention policy or storage of the tape when not 
> in the drive/juke?
> Yes. Simplest is local custody. Off site custody comes in two media flavors 
> -- local media and cloud. Off site costs, so we want to minimize this, as 
> well as increases the response time for restorations. You get the idea. I 
> don't use tapes; I use removable disks, optical media,and usb keydrives. 
> General this is determined by the client, so when I plan this, I simply 
> consider which client backups need to be sequestered where. This gives me a 
> configuration where I can think in terms of "client  gets backedup to NAS 
> ", where NAS  has different properties and different dispositions.
> 
> 
> 
> Amanda is not an archiver in the sense that the tapes are cycled on a regular 
> basis. You are able to tape a tape out of rotation and replace it, or create 
> a unique tape label and perform level 0 backups to it and then mark it as 
> no-reuse in the tapelist, but the primary function is not long term 
> archiving, though the tools exist to do that very well.
>  
> You can use the same tape pool for all three Amanda configs, but they will 
> need to have a common tapelist file. But if you are doing that then you are 
> selecting a single set of standards for your data-at-rest security, in which 
> case there is little reason to maintain 3 different configs.
>  
> Each Amanda config will look to level nightly data, but you will have nights 
> with relatively little and nights with relatively large data volume swings, 
> er think wave interference from physics. You eliminate a lot of that by 
> combining disklists into a single configuration.
> Yes! "Each Amanda config will look to level nightly data ..."  This is my 
> principle question, and I seek to level the nightly data across all configs 
> on a given night, which I recognize can't be done, so I seek to combine my 
> multiple configs into a single config which specifies multiple sets of DLEs 
> being mapped to multiple tapedev, if that can be done. For example, if the 
> definition of tapedev had a "DLE ..." argument, and AMANDA were capable of 
> this additional dimension of scheduling.
> 
> 
> 
> Amanda will backup a new DLE at level 0 the first time it sees it. If you are 
> worried about running long you will want to phase in the DLEs across several 
> evenings. You may want to add the largest on Friday night, assuming that no 
> one cares how late Amanda runs into Saturday. You will want to avoid adding 
> multiple large DLEs on a single night, add a large and a small each night 
> until they are all added.
> "Phasing" my backup jobs may be my only choice, as that is exactly the 
> problem I seek to avoid and I have been admonished to not try to subvert the 
> scheduler by forcing level 0 backups to happen on my schedule. As I continue 
> to discuss this, I am more and more convinced that AMANDA cannot schedule in 
> two dimensions {(backups / night) X (nights / cycle)}
> 
> So, suppose I wanted to force my level 0 backups to happen at my discretion, 
> so I can level my nightly run times. How would I do that?
> 
> Thanks for the help, Brian.
> --
> Chris.
> 


Well, you *can* do
   amadminforce  node  [dles]
Do man amadmin to get details on the FORCE  command.
I have a separate configuration,  using the same disklist,   which I run once a 
month to get an archive tape,
with all the level 0’s on the same tape.   I have it set to do no-incr  but I 
also tell it   “amadmin archive  force *.full.ipname “
to make sure they are only level 0.  You know - belt *and* suspenders.

PS the phasing in (adding a few disks each night)  is only at the beginning,  
when they all need to do a level 0 backup.
If they’re ALL in the disklist  file, then they’ll ALL get a level 0 backup the 
first time you run amdump.
If you only add a few DLEs  each night,  then you are manually balancing it — 
but only until all are added.  After that,
amanda uses it’s leveling to proceed.

Deb Baddorf

Re: Does anyone know how to make an amadmin $config estimate work for new dle's?


On 2018-11-15 11:53, Gene Heskett wrote:

On Thursday 15 November 2018 07:36:37 Austin S. Hemmelgarn wrote:


On 2018-11-15 06:16, Gene Heskett wrote:

I ask because after last nights run it showed one huge and 3 teeny
level 0's for the 4 new dle's.  So I just re-adjusted the locations
of some categories and broke the big one up into 2 pieces.
"./[A-P]*" and ./[Q-Z]*", so the next run will have 5 new dle's.

But an estimate does not show the new names that results in. I've
even took the estimate assignment calcsize back out of the global
dumptype, which ack the manpage, forces the estimates to be derived
from a dummy run of tar, didn't help.

Clues? Having this info from an estimate query might take a couple
hours, but it sure would be helpfull when redesigning ones dle's.I'm
fairly certain you can't, because it specifically shows server-side


estimates, which have no data to work from if there has never been a
dump run for the DLE.


Even if you told it to user tar for the estimate phase? That has enough
legs to be called a bug. IMO anyway.
As mentioned in one of my other responses, I can kind of see the value 
in this not bothering the client systems.  Keep in mind that server 
estimates cost nothing on the client, while calcsize or client estimates 
may use a significant amount of resources.


In this case, I do think the documentation should be a bit clearer, and 
it would be useful to be able to get regular (calcsize and/or client) 
estimates on-demand, but I do think that the default is reasonably sane.

Re: Configuration confusion

2018-11-15 Thread Charles Curley

On Thu, 15 Nov 2018 09:03:00 -0800 (PST)
Chris Miller  wrote:

> If I run three backups, serial or otherwise, then do they know about
> each other? Meaning, is AMANDA smart enough to know not to run more
> than one level 0 dump per night? The problem is that level 0 backups
> take several hours and if I run multiple then I will still be
> completing last nights backup when everybody comes in the next
> morning. That would be embarrassing. "Sorry, I didn't complete my
> work last night, so you can't continue yours."

Ah, that helps. My experience is in a SOHO environment, so take with
the salt shaker handy.

The different configurations don't know about each other at all. So you
could in theory have a night in which all three run level 0 backups.
The only way I know to get that kind of co-ordination is to have one
configuration which then backs up all three machines. Unfortunately
your requirement not to mix the backups due to custodial and security
requirements may kill that idea.

Another thing to look at is to break your DLEs up into lots of smaller
DLEs. You'll get more level 0 backups, but they'll be spread around the
week more evenly.

Or consider having a longer tape cycle. That means fewer level 0
backups in any one week.

-- 
"When we talk of civilization, we are too apt to limit the meaning of
the word to its mere embellishments, such as arts and sciences; but
the true distinction between it and barbarism is, that the one
presents a state of society under the protection of just and
well-administered law, and the other is left to the chance government
of brute force."
- The Rev. James White, Eighteen Christian Centuries, 1889
Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
https://charlescurley.com

Re: Configuration confusion

Hi Brian, 

> From: "Cuttler, Brian R (HEALTH)" 
> To: "Cuttler, Brian R (HEALTH)" , "Chris Miller"
> , "amanda-users" 
> Sent: Wednesday, November 14, 2018 9:08:26 AM
> Subject: RE: Configuration confusion

> Tape custody – means what, retention policy or storage of the tape when not in
> the drive/juke?

Yes. Simplest is local custody. Off site custody comes in two media flavors -- 
local media and cloud. Off site costs, so we want to minimize this, as well as 
increases the response time for restorations. You get the idea. I don't use 
tapes; I use removable disks, optical media,and usb keydrives. General this is 
determined by the client, so when I plan this, I simply consider which client 
backups need to be sequestered where. This gives me a configuration where I can 
think in terms of "client  gets backedup to NAS ", where NAS  has 
different properties and different dispositions. 

> Amanda is not an archiver in the sense that the tapes are cycled on a regular
> basis. You are able to tape a tape out of rotation and replace it, or create a
> unique tape label and perform level 0 backups to it and then mark it as
> no-reuse in the tapelist, but the primary function is not long term archiving,
> though the tools exist to do that very well.

> You can use the same tape pool for all three Amanda configs, but they will 
> need
> to have a common tapelist file. But if you are doing that then you are
> selecting a single set of standards for your data-at-rest security, in which
> case there is little reason to maintain 3 different configs.

> Each Amanda config will look to level nightly data, but you will have nights
> with relatively little and nights with relatively large data volume swings, er
> think wave interference from physics. You eliminate a lot of that by combining
> disklists into a single configuration.

Yes! " Each Amanda config will look to level nightly data ..." This is my 
principle question, and I seek to level the nightly data across all configs on 
a given night, which I recognize can't be done, so I seek to combine my 
multiple configs into a single config which specifies multiple sets of DLEs 
being mapped to multiple tapedev, if that can be done. For example, if the 
definition of tapedev had a "DLE ..." argument, and AMANDA were capable of this 
additional dimension of scheduling. 

> Amanda will backup a new DLE at level 0 the first time it sees it. If you are
> worried about running long you will want to phase in the DLEs across several
> evenings. You may want to add the largest on Friday night, assuming that no 
> one
> cares how late Amanda runs into Saturday. You will want to avoid adding
> multiple large DLEs on a single night, add a large and a small each night 
> until
> they are all added.

"Phasing" my backup jobs may be my only choice, as that is exactly the problem 
I seek to avoid and I have been admonished to not try to subvert the scheduler 
by forcing level 0 backups to happen on my schedule. As I continue to discuss 
this, I am more and more convinced that AMANDA cannot schedule in two 
dimensions {(backups / night) X (nights / cycle)} 

So, suppose I wanted to force my level 0 backups to happen at my discretion, so 
I can level my nightly run times. How would I do that? 

Thanks for the help, Brian. 
-- 
Chris. 

V:916.974.0424 
F:916.974.0428

Re: Amanda's DB

On Thursday 15 November 2018 08:12:19 Chris Nighswonger wrote:

> On Wed, Nov 14, 2018 at 7:08 PM Nathan Stratton Treadway
> 
>
> wrote:
> > On Wed, Nov 14, 2018 at 15:35:50 -0500, Chris Nighswonger wrote:
> > > Can anyone point me to "good" documentation that describes
> > > Amanda's DB? Things like schema, field descriptions and/or uses,
> > > etc.
>
> [snip]
>
> > > I notice that the later versions of ZWC use SQL.
> >
> > I saw that the Amanda source repository master branch includes the
> > following commit:
> >
> > commit ea4dd29d265b44a914e794d5150e53bce9d9435d
> > Author: Jean-Louis Martineau 
> > Date:   Thu Jan 4 12:52:28 2018 +
> >
> > Use a database (SQLite, MySQL or ProstgreSQL) to replace the
> > tapelist and log.* file
> > Read the Amanda-Catalog file
> > * perl/*, server-src/*, common-src/conffile.c: Major changes
> > * installcheck/*: Fix tests
> > * Amanda-Catalog: New documentation document
> >
> > Jan 2018 is after Amanda 3.5 was released so this is still
> > work-in-progress, and I don't know how "mature" this transition got
> > before Jean-Louis left
>
> Digging a bit in the new Catalog code
>
> The style makes for difficult reading. Clearly what docs there are
> show that work was being done on SQLite. The DB schema is not readily
> apparent.
>
> It seems to me that Amanda is heavily dependent upon its logs to
> coordinate and keep it running. It would be very nice to have a solid
> set of docs outlining the log syntax for each log and describing the
> various roles of each log in the overall scheme of Amanda's operation.
>
> Without some sort of understanding of the larger picture of how things
> operate, its going to be mighty difficult picking up and moving
> forward with development.
>
+1 for understatement of the month.

> I have been involved in development (thousands of lines of code) and
> project management of another large OSS package written primarily in
> Perl. I am certainly willing to contribute to Amanda, but the
> "activation energy" (aka learning curve) appears way too high at the
> moment.
>
> I think when Dustin left, the end of true open development in Amanda
> left with him.
>
> My $0.02 worth.
>
And it will take more than the famous 2 cents in order for anybody else 
to think about picking up the reins. BTDT, too many times already on 
older stuff. The toughest part is getting into the last guys head and 
figuring out what he had in mind.
 
> Kind regards,
> Chris



Copyright 2018 by Maurice E. Heskett
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page

Re: Configuration confusion

Hi Charles,

> I think the simplest solution to your problem is to take Gene's advice
> and run the three backups in sequence, using a simple shell script.
> That is what I do. All these issues go away.

I don't think I understand your advice, via Gene.

To refresh your memory, I have three configs for three clients which 
effectively backup each to it's own vtape pool:
c0.tclc.org --> nas0.tclc.org
c1.tclc.org --> nas1.tclc.org
c2.tclc.org --> nas2.tclc.org

I am working with a one week backup cycle -- seven backups, during which each 
of the three clients must complete a level 0 backup. 

If I run three backups, serial or otherwise, then do they know about each 
other? Meaning, is AMANDA smart enough to know not to run more than one level 0 
dump per night? The problem is that level 0 backups take several hours and if I 
run multiple then I will still be completing last nights backup when everybody 
comes in the next morning. That would be embarrassing. "Sorry, I didn't 
complete my work last night, so you can't continue yours."

It seems like I should be able to combine my several configs into a single 
config so AMANDA will actually know the full scope of the problem and schedule 
accordingly, but I don't understand enough of the configuration discipline to 
describe multiple DLE --> vtape mappings to run in a single AMANDA execution. I 
think this comes from "disklist" having a list, but amanda.conf having only one 
tapedev. Even if there were latitude to define multiple tapedev, I don't see 
syntax that would map a set of DLEs to each tapedev. For example, can I specify 
the "disklist" in the definition of the tapedev? Unfortunately "man tapedev" 
does not produce anything. Is there, for example a BNF definition of syntax for 
tapedev anywhere?

Thanks for the help, Charles.
-- 
Chris. 

V:916.974.0424 
F:916.974.0428

Re: Does anyone know how to make an amadmin $config estimate work for new dle's?

On Thursday 15 November 2018 07:36:37 Austin S. Hemmelgarn wrote:

> On 2018-11-15 06:16, Gene Heskett wrote:
> > I ask because after last nights run it showed one huge and 3 teeny
> > level 0's for the 4 new dle's.  So I just re-adjusted the locations
> > of some categories and broke the big one up into 2 pieces.
> > "./[A-P]*" and ./[Q-Z]*", so the next run will have 5 new dle's.
> >
> > But an estimate does not show the new names that results in. I've
> > even took the estimate assignment calcsize back out of the global
> > dumptype, which ack the manpage, forces the estimates to be derived
> > from a dummy run of tar, didn't help.
> >
> > Clues? Having this info from an estimate query might take a couple
> > hours, but it sure would be helpfull when redesigning ones dle's.I'm
> > fairly certain you can't, because it specifically shows server-side
>
> estimates, which have no data to work from if there has never been a
> dump run for the DLE.

Even if you told it to user tar for the estimate phase? That has enough 
legs to be called a bug. IMO anyway.

Copyright 2018 by Maurice E. Heskett
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page

Re: Does anyone know how to make an amadmin $config estimate work for new dle's?

On 2018-11-15 11:21, Chris Nighswonger wrote:
On Thu, Nov 15, 2018 at 7:40 AM Austin S. Hemmelgarn 
mailto:ahferro...@gmail.com>> wrote:

On 2018-11-15 06:16, Gene Heskett wrote:
 > I ask because after last nights run it showed one huge and 3
teeny level
 > 0's for the 4 new dle's.  So I just re-adjusted the locations of some
 > categories and broke the big one up into 2 pieces. "./[A-P]*"
 > and ./[Q-Z]*", so the next run will have 5 new dle's.
 >
 > But an estimate does not show the new names that results in. I've
even
 > took the estimate assignment calcsize back out of the global
dumptype,
 > which ack the manpage, forces the estimates to be derived from a
dummy
 > run of tar, didn't help.
 >
 > Clues? Having this info from an estimate query might take a
couple hours,
 > but it sure would be helpfull when redesigning ones dle's.

I'm fairly certain you can't, because it specifically shows server-side
estimates, which have no data to work from if there has never been a
dump run for the DLE.

What would be the downside to having the amanda client execute ' du -s' 
or some such on the DLE and return the results when amcheck and friends 
realize there is no reliable size estimate? This would seem to be a much 
more accurate estimate than a non-existent server estimate.

My guess is that it's intentionally limited to server estimates to avoid 
putting load on the client systems.  Both calcsize and client estimates 
require reading a nontrivial amount of data on the client side, and 
client estimates also involve a nontrivial amount of processing.

That said, it would be nice to be able to explicitly run any of the 
three types of estimate.

Re: Monitor and Manage

Hi Ned,

> on my linux system, I have, in amanda.conf
> 
> mailto "ned.danie...@duke.edu"
> 
> and amdump runs from cron. when amdump completes, I get an email with a
> list of all the DLEs that were dumped, including the level, the size,
> and the time taken.

I also have that line, but I was getting no report. Turned out to be a 
permissions problem preventing PostFix from running. "# postfix -c /etc/postfix 
start" revealed the problem, which was easily fixed, and now I get reports!

Thanks for the help, Ned.

Chris.

Re: Does anyone know how to make an amadmin $config estimate work for new dle's?

2018-11-15 Thread Chris Nighswonger

On Thu, Nov 15, 2018 at 7:40 AM Austin S. Hemmelgarn 
wrote:

> On 2018-11-15 06:16, Gene Heskett wrote:
> > I ask because after last nights run it showed one huge and 3 teeny level
> > 0's for the 4 new dle's.  So I just re-adjusted the locations of some
> > categories and broke the big one up into 2 pieces. "./[A-P]*"
> > and ./[Q-Z]*", so the next run will have 5 new dle's.
> >
> > But an estimate does not show the new names that results in. I've even
> > took the estimate assignment calcsize back out of the global dumptype,
> > which ack the manpage, forces the estimates to be derived from a dummy
> > run of tar, didn't help.
> >
> > Clues? Having this info from an estimate query might take a couple hours,
> > but it sure would be helpfull when redesigning ones dle's.



> I'm fairly certain you can't, because it specifically shows server-side
> estimates, which have no data to work from if there has never been a
> dump run for the DLE.
>

What would be the downside to having the amanda client execute ' du -s' or
some such on the DLE and return the results when amcheck and friends
realize there is no reliable size estimate? This would seem to be a much
more accurate estimate than a non-existent server estimate.

Chris

Re: Amanda's DB

2018-11-15 Thread Chris Nighswonger

On Wed, Nov 14, 2018 at 7:08 PM Nathan Stratton Treadway 
wrote:

> On Wed, Nov 14, 2018 at 15:35:50 -0500, Chris Nighswonger wrote:
> > Can anyone point me to "good" documentation that describes Amanda's DB?
> > Things like schema, field descriptions and/or uses, etc.
>
>
[snip]

>
> > I notice that the later versions of ZWC use SQL.
>
> I saw that the Amanda source repository master branch includes the
> following commit:
>
> commit ea4dd29d265b44a914e794d5150e53bce9d9435d
> Author: Jean-Louis Martineau 
> Date:   Thu Jan 4 12:52:28 2018 +
>
> Use a database (SQLite, MySQL or ProstgreSQL) to replace the tapelist
> and log.* file
> Read the Amanda-Catalog file
> * perl/*, server-src/*, common-src/conffile.c: Major changes
> * installcheck/*: Fix tests
> * Amanda-Catalog: New documentation document
>
> Jan 2018 is after Amanda 3.5 was released so this is still
> work-in-progress, and I don't know how "mature" this transition got
> before Jean-Louis left
>

Digging a bit in the new Catalog code

The style makes for difficult reading. Clearly what docs there are show
that work was being done on SQLite. The DB schema is not readily apparent.

It seems to me that Amanda is heavily dependent upon its logs to coordinate
and keep it running. It would be very nice to have a solid set of docs
outlining the log syntax for each log and describing the various roles of
each log in the overall scheme of Amanda's operation.

Without some sort of understanding of the larger picture of how things
operate, its going to be mighty difficult picking up and moving forward
with development.

I have been involved in development (thousands of lines of code) and
project management of another large OSS package written primarily in Perl.
I am certainly willing to contribute to Amanda, but the "activation energy"
(aka learning curve) appears way too high at the moment.

I think when Dustin left, the end of true open development in Amanda left
with him.

My $0.02 worth.

Kind regards,
Chris

Re: Does anyone know how to make an amadmin $config estimate work for new dle's?


On 2018-11-15 06:16, Gene Heskett wrote:

I ask because after last nights run it showed one huge and 3 teeny level
0's for the 4 new dle's.  So I just re-adjusted the locations of some
categories and broke the big one up into 2 pieces. "./[A-P]*"
and ./[Q-Z]*", so the next run will have 5 new dle's.

But an estimate does not show the new names that results in. I've even
took the estimate assignment calcsize back out of the global dumptype,
which ack the manpage, forces the estimates to be derived from a dummy
run of tar, didn't help.

Clues? Having this info from an estimate query might take a couple hours,
but it sure would be helpfull when redesigning ones dle's.I'm fairly certain you can't, because it specifically shows server-side 
estimates, which have no data to work from if there has never been a 
dump run for the DLE.

Does anyone know how to make an amadmin $config estimate work for new dle's?