Re: Only increasing incrementals

[Austin S. Hemmelgarn]

On 2018-12-12 19:11, Debra S Baddorf wrote:

Oh, that’s right — Chris DID tell us what he was trying to do,  last week:
(in case it helps with any further answers)


OK, given that what you want appears to be frequent snapshots, Amanda is 
almost certainly _not_ the correct tool for this job for three reasons:


* It can not do atomic snapshots of the filesystem state, so either you 
need to freeze all write I/O to the DLE, or you won't have coherent copy 
of the state of the DLE from when you ran the backup.  This doesn't 
matter most of the time for regular backup usage, because you just run 
Amanda during off-hours when nobody's doing anything and the system is 
idle, but for this it's going to be a potential issue.


* It takes a _lot_ of system resources to do a backup with Amanda.  This 
is mitigated by your proposed approach of doing constantly increasing 
incremental levels, but even for that Amanda has to call `stat()` on 
_everything_ in the DLE.


* It's not trivial (as you have found out) to get this type of thing to 
work reliably.


I would suggest looking at the following alternative approaches:

* Use ZFS, BTRFS, or another filesystem that supports native 
snapshotting, and just do snapshots regularly.  This is likely to be 
your best approach.  In some cases, depending on the platform and 
filesystem, you may not even need to do anything (for example, NILFS2 on 
Linux has implicit snapshots built in because it's a log structured 
filesystem).


* Store all the data on a NAS device that can do snapshots (for example, 
something running FreeNAS), and have it do regular snapshots.  This 
largely reduces to the above, just indirected over the network.


* Use a filesystem that supports automatic native file versioning.  The 
classic example is Files-11 from OpenVMS.  Other options for this 
include GitFS [1], CopyFS [2], Plan 9's Fossil filesystem


* Store all the data on a NAS device that does automatic native file 
versioning.


* If all else fails, you can technically do this with Amanda by using 
`amadmin force` to force the level 0 dump, and `amadmin force-bump` for 
each backup _after the second_ (the first backup after a level 0 will 
always be a level 1) to get the increasing incrementals.


[1] https://www.presslabs.com/code/gitfs/
[2] https://boklm.eu/copyfs/




On Dec 7, 2018, at 12:04 PM, Chris Miller  wrote:

Hi Folks,

I'm about to start a project during which I want to be able to:
• Request a backup at any moment and have that backup be either an incremental 
backup (Level N+1), meaning everything that has changed since the last backup (Level N), 
or a differential backup, meaning everything that has changed since the last full backup 
(Level 1). The second provision, "differential backup" is pretty straight 
forward, but I have no idea how to configure a constantly increasing dump level.
• The first backup of the day, meaning the first backup after midnight, 
will be a full filesystem backup.

Discussion on point 1:
The provision is for capturing changes that occur during a given period of time, and no 
so much for "backup" per se, so AMANDA may not be the best tool, but it is what 
I have, so I'm tying to make it fit. I know how to request a backup, so that's not my 
problem, but I don't know how to force a given level. In particular, I don't know how to 
force a Level N+1 backup. I could replace the Level N+1 requirement with a forced Level 
1, run my experiment, and force a level 2, and this would meet my requirement of 
capturing all the changes during a particular interval. But, again, that requires forcing 
AMANDA to take direction about backup levels and I don't know how to do that.

Before anybody reminds me that this is why god invented git, I would like to add, that 
the scope of git is typically only known parts of the project, and I want to capture log 
files and other files that are sometimes created in temporary locations with temporary 
names, which are not known a priori and therefore can't be "managed" with git.

Discussion on point 2:
The "first backup of the day", will run as a cron job, but it must be a level 
0, full filesystem backup so no work for the day is lost. It is more forcing AMANDA to 
take direction, I don't know exactly how to do this.

I don't think I like the idea of forcing AMANDA (#MeToo) to do things, but I'm 
not above payment in kind. (-:

Thanks for the help,

Re: Only increasing incrementals

[Gene Heskett]

On Wednesday 12 December 2018 17:10:08 Jon LaBadie wrote:

> On Wed, Dec 12, 2018 at 02:18:57PM +0700, Olivier wrote:
> > Nathan Stratton Treadway  writes:
> > > On Thu, Nov 22, 2018 at 11:18:25 +0700, Olivier wrote:
> > >> Hello,
> > >>
> > >> UI am wondering if there is a way to define a DLE that would all
> > >> incrementals but only with increasing level:
> > >>
> > >> - full (0)
> > >> - incremental 1
> > >> - incremental 2
> > >> - incremental 3
> > >> - etc.
> > >>
> > >> But never: 0, 1, 1, 1, 2
> > >>
> > >> Each back-up level must be above the previous one or be a full
> > >> back-up.
>
> Have you tried:
>
>   bumpdays0
>   bumpmult0
>   bumppercent 0
>   bumpsize0
>
> I don't know if it will work, or maybe bork somehow.
>
> Jon

ISTR I did something like that, back up  the log a year or so, and 
amcheck had a hungry bawling calf.  Anytime you do something odd, let 
amcheck see if it has an opinion. It does try to make you miss your foot 
when shooting at it.

Copyright 2018 by Maurice E. Heskett
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: Only increasing incrementals

[Debra S Baddorf]

Oh, that’s right — Chris DID tell us what he was trying to do,  last week:
(in case it helps with any further answers)

> On Dec 7, 2018, at 12:04 PM, Chris Miller  wrote:
> 
> Hi Folks,
> 
> I'm about to start a project during which I want to be able to:
>   • Request a backup at any moment and have that backup be either an 
> incremental backup (Level N+1), meaning everything that has changed since the 
> last backup (Level N), or a differential backup, meaning everything that has 
> changed since the last full backup (Level 1). The second provision, 
> "differential backup" is pretty straight forward, but I have no idea how to 
> configure a constantly increasing dump level.
>   • The first backup of the day, meaning the first backup after midnight, 
> will be a full filesystem backup.
> 
> Discussion on point 1:
> The provision is for capturing changes that occur during a given period of 
> time, and no so much for "backup" per se, so AMANDA may not be the best tool, 
> but it is what I have, so I'm tying to make it fit. I know how to request a 
> backup, so that's not my problem, but I don't know how to force a given 
> level. In particular, I don't know how to force a Level N+1 backup. I could 
> replace the Level N+1 requirement with a forced Level 1, run my experiment, 
> and force a level 2, and this would meet my requirement of capturing all the 
> changes during a particular interval. But, again, that requires forcing 
> AMANDA to take direction about backup levels and I don't know how to do that.
> 
> Before anybody reminds me that this is why god invented git, I would like to 
> add, that the scope of git is typically only known parts of the project, and 
> I want to capture log files and other files that are sometimes created in 
> temporary locations with temporary names, which are not known a priori and 
> therefore can't be "managed" with git.
> 
> Discussion on point 2:
> The "first backup of the day", will run as a cron job, but it must be a level 
> 0, full filesystem backup so no work for the day is lost. It is more forcing 
> AMANDA to take direction, I don't know exactly how to do this.
> 
> I don't think I like the idea of forcing AMANDA (#MeToo) to do things, but 
> I'm not above payment in kind. (-:
> 
> Thanks for the help,
> --
> Chris.
> 
> V:916.974.0424
> F:916.974.0428

Re: Only increasing incrementals

[Jon LaBadie]

On Wed, Dec 12, 2018 at 02:18:57PM +0700, Olivier wrote:
> Nathan Stratton Treadway  writes:
> 
> > On Thu, Nov 22, 2018 at 11:18:25 +0700, Olivier wrote:
> >> Hello,
> >> 
> >> UI am wondering if there is a way to define a DLE that would all
> >> incrementals but only with increasing level:
> >> 
> >> - full (0)
> >> - incremental 1
> >> - incremental 2
> >> - incremental 3
> >> - etc.
> >> 
> >> But never: 0, 1, 1, 1, 2
> >> 
> >> Each back-up level must be above the previous one or be a full back-up.
> >

Have you tried:

  bumpdays0
  bumpmult0
  bumppercent 0
  bumpsize0

I don't know if it will work, or maybe bork somehow.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Only increasing incrementals

[Debra S Baddorf]

> On Dec 12, 2018, at 6:09 AM, Austin S. Hemmelgarn  
> wrote:
> 
> On 2018-12-12 02:18, Olivier wrote:
>> Nathan Stratton Treadway  writes:
>>> On Thu, Nov 22, 2018 at 11:18:25 +0700, Olivier wrote:
 Hello,
 
 UI am wondering if there is a way to define a DLE that would all
 incrementals but only with increasing level:
 
 - full (0)
 - incremental 1
 - incremental 2
 - incremental 3
 - etc.
 
 But never: 0, 1, 1, 1, 2
 
 Each back-up level must be above the previous one or be a full back-up.
>>> 
>>> I am not sure what you are trying to accomplish,
>> I am trying to backup something that can only have incremental with
>> increasing levels: it cannot do two level 1 in a row, levels must be 1,
>> then 2, then 3, etc. (think some successive snapshots).
>> According to amanda.conf(5) man page:
>>bumpdays int
>>Default: 2 days. To insure redundancy in the dumps, Amanda keeps
>>filesystems at the same incremental level for at least bumpdays
>>days, even if the other bump threshold criteria are met.
>> I want to absolutely cancel that feature, each incremental must have a
>> level creater than the previous dump and an incremental level can not be
>> bumped (only level 0 can be bumped).
> OK, I"m actually curious what your exact reasoning for requiring this is, 
> because I'm seeing exactly zero circumstances where this makes sense at all, 
> and can think of multiple ways it's a bad thing (for example, losing your 
> level one incremental makes all of your backups for that cycle useless).


Ditto.  I’d like to know the problem that you are trying to solve.

 Also,  as far as I know, amanda does something to every DLE, every time you 
run amdump.  
It won’t skip the DLE just because no changes have occurred.
You might go up to really odd levels of incremental, if you have even a 
week-long cycle.   Level 0, 1, 2, 3, 4, 5, 6.
Some might even be empty,  if nothing has changed.  But it’ll have to create 
them anyway.
Unless you tell it your "dumpcycle”   is only 4  or 3 days long.  Or some such.
And then you are forced to have all of those tapes (vtapes)  available for any 
backup.

You could possibly do  “amadmin config  force-bump”   every night,  in your 
cronjob.

Just random thoughts, since we don’t understand the actual problem.

Deb Baddorf

Re: Only increasing incrementals

[Austin S. Hemmelgarn]

On 2018-12-12 02:18, Olivier wrote:

Nathan Stratton Treadway  writes:


On Thu, Nov 22, 2018 at 11:18:25 +0700, Olivier wrote:

Hello,

UI am wondering if there is a way to define a DLE that would all
incrementals but only with increasing level:

- full (0)
- incremental 1
- incremental 2
- incremental 3
- etc.

But never: 0, 1, 1, 1, 2

Each back-up level must be above the previous one or be a full back-up.


I am not sure what you are trying to accomplish,


I am trying to backup something that can only have incremental with
increasing levels: it cannot do two level 1 in a row, levels must be 1,
then 2, then 3, etc. (think some successive snapshots).

According to amanda.conf(5) man page:

bumpdays int
Default: 2 days. To insure redundancy in the dumps, Amanda keeps
filesystems at the same incremental level for at least bumpdays
days, even if the other bump threshold criteria are met.

I want to absolutely cancel that feature, each incremental must have a
level creater than the previous dump and an incremental level can not be
bumped (only level 0 can be bumped).
OK, I"m actually curious what your exact reasoning for requiring this 
is, because I'm seeing exactly zero circumstances where this makes sense 
at all, and can think of multiple ways it's a bad thing (for example, 
losing your level one incremental makes all of your backups for that 
cycle useless).

Re: Only increasing incrementals

[Olivier]

Nathan Stratton Treadway  writes:

> On Thu, Nov 22, 2018 at 11:18:25 +0700, Olivier wrote:
>> Hello,
>> 
>> UI am wondering if there is a way to define a DLE that would all
>> incrementals but only with increasing level:
>> 
>> - full (0)
>> - incremental 1
>> - incremental 2
>> - incremental 3
>> - etc.
>> 
>> But never: 0, 1, 1, 1, 2
>> 
>> Each back-up level must be above the previous one or be a full back-up.
>
> I am not sure what you are trying to accomplish,

I am trying to backup something that can only have incremental with
increasing levels: it cannot do two level 1 in a row, levels must be 1,
then 2, then 3, etc. (think some successive snapshots).

According to amanda.conf(5) man page:

   bumpdays int
   Default: 2 days. To insure redundancy in the dumps, Amanda keeps
   filesystems at the same incremental level for at least bumpdays
   days, even if the other bump threshold criteria are met.

I want to absolutely cancel that feature, each incremental must have a
level creater than the previous dump and an incremental level can not be
bumped (only level 0 can be bumped).

Thainks,

Olivier

Re: Only increasing incrementals

[Jon LaBadie]

On Thu, Nov 22, 2018 at 11:18:25AM +0700, Olivier wrote:
> Hello,
> 
> UI am wondering if there is a way to define a DLE that would all
> incrementals but only with increasing level:
> 
> - full (0)
> - incremental 1
> - incremental 2
> - incremental 3
> - etc.
> 
> But never: 0, 1, 1, 1, 2
> 
> Each back-up level must be above the previous one or be a full back-up.

You can use "amadmin  force [host] [disk]"
to force a level 0 dump.

You can use "amadmin  force-bump [host] [disk]"
to force a bump to the next level.

Run them from crontab to automate the process.

Jon
> 
> Best regards,
> 
> Olivier
> -- 
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: Only increasing incrementals

[Nathan Stratton Treadway]

On Thu, Nov 22, 2018 at 11:18:25 +0700, Olivier wrote:
> Hello,
> 
> UI am wondering if there is a way to define a DLE that would all
> incrementals but only with increasing level:
> 
> - full (0)
> - incremental 1
> - incremental 2
> - incremental 3
> - etc.
> 
> But never: 0, 1, 1, 1, 2
> 
> Each back-up level must be above the previous one or be a full back-up.

I am not sure what you are trying to accomplish, but in general I don't
believe you can ever force Amanda to do a level N+1 dump if level N is
already empty.  (This makes sense, in that there's no reason to require
the operator to pull N+1 tapes for recovery if N tapes will give you the
same result) So if your DLE doesn't change every day (or only has
very small changes), I don't believe you'll be able to strictly enforce
the rule as described.

But to see the knobs you can tune to get closer to what you want, see
the the four bump options (bumpdays, bumpmult, bumppercent, bumpsize)
in the amandna.conf(5) man page (in both the global and DUMPTYPE
sections).

Also, to see the thresholds that Amanda calculates based on those
parameters you can use the the "amadmin  bumpsize" command (though
I guess that command only looks at the global bump options, not any
dumptype-specific ones).


Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: Only increasing incrementals

[Gene Heskett]

On Wednesday 21 November 2018 23:18:25 Olivier wrote:

> Hello,
>
> UI am wondering if there is a way to define a DLE that would all
> incrementals but only with increasing level:
>
> - full (0)
> - incremental 1
> - incremental 2
> - incremental 3
> - etc.
>
> But never: 0, 1, 1, 1, 2

You do this in the amanda.conf, there is a var that sets the number of 
days before a level can be promoted.  And its global.  Recommended at 2  
so if one is duff, you still have a duplicate provided nothing in that 
dle was modified between the runs. For busy dle's 1 might be a disaster 
that might find a place to happen eventually. I leave mine at 2.

That said, there are 4 more vars there that control the level increment, 
and they may hold a level for additional days. I had to make mine more 
lenient to get the promotions I thought should happen.  


> Each back-up level must be above the previous one or be a full
> back-up.
>
> Best regards,
>
> Olivier



Copyright 2018 by Maurice E. Heskett
-- 
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page