Re: VTape configuration

[Jon LaBadie]

On Wed, Jan 03, 2018 at 06:22:41PM -0500, Winston Sorfleet wrote:
> On 2018-01-03 05:46 PM, Chris Miller wrote:
> > Hi Jean-Louis,
> >
> > A vtape do not reserve the space, it only use the space of the
> > dump you put on it.
> > The vtape size should be the maximum size of any days or larger,
> > they could be 120GB or 2TG, the result will be the same.
> > Some vtape will use 5GB, some will use 120GB.
> >
> > Thanks very much. That is quite helpful.
> >
> > Is it possible to re-use a vtape for several successive backups? If
> > the vtape were twice the size of a level 0 backup, then the level 1
> > backups would be appended, but the next level 0 would be too large and
> > would trigger a new tape.
> >
> > Additionally, How do I configure larger cycles, meaning, I will use
> > 150GB each week, and 2TB can hold 13 weeks. How do I tell Amanda that
> > after 13 weeks, she should start over by deleting the oldest vtape?
> 
> That's exactly what it will do, unless you explicitly tell Amanda that a
> tape is out of commission.  So, just like your traditional robot, it
> will cycle.
> 

One parameter you specify is how many tapes (minimum) you have in
rotation.  Amanda will not reuse any tape until it has used that
many tapes.

I presently have 240 vtapes of 100GB each spread over 6 hard disks.
I have specified my "tapecycle" as 140, not 240.  All 240 tapes are
used in sequence and then things rotate back to the beginning.  But
the lower number is my protection against one of the disks failing
and amanda not having anywhere to put new backups.

> HOWEVER, note that Amanda will not write to a tape that has been
> recently written, whether it's 5 GB or 150 GB.  It does /not/ append to
> tapes (barring some painful and unnecessary manual configuration, that
> is).  So, you're better off using small vtapes.  Because if it only
> writes 5 GB to 150 GB in a 2 TB NAS, that remaining 145 GB is
> unavailable to Amanda until that vtape comes up for overwriting.
> 
Perhaps I misunderstand Winston's comment here.  But if you write 5GB
to a "150GB" vtape, the "remaining 145GB" are available for other vtapes.
My 240 x 100GB vtapes (nominally 24TB) are on 20TB of disk.  None of the
six disks are over 80% full.

jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: VTape configuration

[Jon LaBadie]

On Wed, Jan 03, 2018 at 01:08:37PM -0800, Chris Miller wrote:
> Hi Winston, 
> 
> | I highly suggest a read of this FAQ: [
> | 
> http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F
> | ] ; particularly the section about Amanda's planning strategy. If you 
> "insist"
> | on constraining Amanda to one-volume-per-backup, you are basically going
> | against the strategy; without that capability, I don't think that Amanda's
> | overhead gives you anything you can't do with tar and a cron job.
> 
> I understand how Amanda wants to try to "smooth" the mix of backup
> levels and filesystem sizes so that backup costs about the same amount
> of time and storage each cycle, and that is a very worthwhile goal,
> so I don't want to impede that. I also understand that tape discipline
> is already built-in to Amanda at a fundamental level, so I don't want
> to mess with that, either. So, I seek advice.
> 
> Suppose "amanda.example.com" is backing-up "client.example.com" to
> "NAS0.example.com". My level 0 backups are typically 120GB and my level 1
> backups are typically 5GB, and I have 2TB on NAS0. That's 120+6*5 = 
> 150GB/week,
> meaning I have sufficient room for thirteen weeks of backup. This seems to me
> like it might be a pretty common scenario and that there might be example
> configs floating around that would size the vtapes for optimal use. Is there 
> one? 

I'm not sure you do understand amanda's approach Chris.  Amanda does not
try to smooth things across a dump cycle, but across each run within a
dump cycle.  If your dumpcycle is 7 days with daily amdump runs, amanda's
goal would be 5GB incrementals plus 120GB/7 (about 18GB) for a daily
dump of about 23GB.

That ideal might be achievable if you had about 14 DLE averaging about
5GB each.  Then amanda could decide which 1,2,3, or 4 DLEs to do level 0
dumps each run.  But in real life you have some small and some large
DLEs and amanda can't meet that ideal.  Assuming you have at least a
few DLEs in those 120GB, some days might have a 60GB level 0 DLE to
dump, other days maybe a 5 and a 10 GB, and still others maybe none
and you will only have incrementals.

In such a scenario I would try to size my vtapes so the largest (60GB
in my example) would fit comfortably in about 3 tapes including a set
of incrementals.  So something like 25GB/vtape and tell amanda it can
use up to 3 tapes per amdump run.  The day that large DLE gets a full
dump, 3 tapes will be used.  Most other days probably only 1 tape will
be needed.

> I have some questions: 

Again, based on the questions below, you don't yet get the amanda scheme.
There will not be A (as in "a single") level 0, each DLE will get its
own schedule of level 0's.

Amanda is nothing if not flexible.  You can force it to do things in a
"non-amanda" way.  My comments after each question are about how you
might achieve the goal, but then why use amanda?  .
> 
> 1. Can I make my vtapes all 150GB, and then instruct Amanda to put
> one cycle (one level 0, and six level 1) on one vtape, meaning re-use a
> vtape multiple time in a backup cycle? I like this approach quite a bit,
> if it is possible. It "packages" one level 0 with all the attendant
> level 1 differentials and eliminates my strongest reservations about
> vtapes -- namely that I don't know where anything is. 

Amanda will not write a new amdump to the end of an existing tape containing
a previous dump.

I've not seen mention of a "holding disk" in this thread.  If you are using
one (strongly recommended) and it is sufficiently large, you could run each
amdump with the option to not write to tape.  The dumps will then collect
on the holding disk.  Then one day a week have your cronjob also do an
amflush which will move all the dumps, full and incr., to one or more vtapes.

You could even do your traditional all level 0's the same day if you
want.  Specify a dumptype of incremental only and on the day you choose
specify a forced full dump.

> 2. Failing that, should I make my vtapes 120GB, so I can fit a
> level 0 backup on one vtape, but then will Amanda truncate level 1 backups,
> so that the vtapes storage requirement is NOT 120G, but more like 5GB? 

You could make a 100 120GB or 150GB vtapes.  They are just directories
and take no space.  Their size is the maximum amount of data amanda will
write to a single vtape.  So if only 10GB get written to each tape it
those 100 tapes will only take 1/2 of your 2TB space on NAS0.

Amanda will not truncate any dumps.  Again assuming you are using a
holding disk, any dumps that don't fit the available vtape(s) will
remain on the holding disk until flushed to a vtape.  This can happen
automatically on the next amdump run.

> 3. Alternatively, I could make my vtapes all 5GB and then Amanda will
> have to span fourteen vtapes for the level 0? This might be optimal use of
> storage, but it scares me with added complexity. I won't know where anything
> is, meaning 

Re: VTape configuration

[Debra S Baddorf]

> On Jan 3, 2018, at 5:22 PM, Winston Sorfleet  wrote:
> 
> On 2018-01-03 05:46 PM, Chris Miller wrote:
>> Hi Jean-Louis,
>> A vtape do not reserve the space, it only use the space of the dump you put 
>> on it.
>> The vtape size should be the maximum size of any days or larger, they could 
>> be 120GB or 2TG, the result will be the same.
>> Some vtape will use 5GB, some will use 120GB.
>> Thanks very much. That is quite helpful.
>> 
>> Is it possible to re-use a vtape for several successive backups? If the 
>> vtape were twice the size of a level 0 backup, then the level 1 backups 
>> would be appended, but the next level 0 would be too large and would trigger 
>> a new tape.
>> 
>> Additionally, How do I configure larger cycles, meaning, I will use 150GB 
>> each week, and 2TB can hold 13 weeks. How do I tell Amanda that after 13 
>> weeks, she should start over by deleting the oldest vtape?
> 
> That's exactly what it will do, unless you explicitly tell Amanda that a tape 
> is out of commission.  So, just like your traditional robot, it will cycle.
> 
> HOWEVER, note that Amanda will not write to a tape that has been recently 
> written, whether it's 5 GB or 150 GB.  It does not append to tapes (barring 
> some painful and unnecessary manual configuration, that is).  So, you're 
> better off using small vtapes.  Because if it only writes 5 GB to 150 GB in a 
> 2 TB NAS, that remaining 145 GB is unavailable to Amanda until that vtape 
> comes up for overwriting.

Except that we were just assured that it only uses 5G if that’s how much data 
you have.  The 145G  isn’t unavailable.   You just “oversubscribe” the disk, 
since you know most of the backups will only be 5G.

Still doing tapes myself, but I’m pretty sure I understood that much.
Deb Baddorf
Fermilab
> 
> 
>> 
>> Thanks for the help,
>> -- 
>> Chris.
>> 
>> V:916.974.0424
>> F:916.974.0428
> 
> 
> 

Re: VTape configuration

[Winston Sorfleet]

On 2018-01-03 05:46 PM, Chris Miller wrote:
> Hi Jean-Louis,
>
> A vtape do not reserve the space, it only use the space of the
> dump you put on it.
> The vtape size should be the maximum size of any days or larger,
> they could be 120GB or 2TG, the result will be the same.
> Some vtape will use 5GB, some will use 120GB.
>
> Thanks very much. That is quite helpful.
>
> Is it possible to re-use a vtape for several successive backups? If
> the vtape were twice the size of a level 0 backup, then the level 1
> backups would be appended, but the next level 0 would be too large and
> would trigger a new tape.
>
> Additionally, How do I configure larger cycles, meaning, I will use
> 150GB each week, and 2TB can hold 13 weeks. How do I tell Amanda that
> after 13 weeks, she should start over by deleting the oldest vtape?

That's exactly what it will do, unless you explicitly tell Amanda that a
tape is out of commission.  So, just like your traditional robot, it
will cycle.

HOWEVER, note that Amanda will not write to a tape that has been
recently written, whether it's 5 GB or 150 GB.  It does /not/ append to
tapes (barring some painful and unnecessary manual configuration, that
is).  So, you're better off using small vtapes.  Because if it only
writes 5 GB to 150 GB in a 2 TB NAS, that remaining 145 GB is
unavailable to Amanda until that vtape comes up for overwriting.


>
> Thanks for the help,
> -- 
> Chris.
>
> V:916.974.0424
> F:916.974.0428

Re: VTape configuration

[Chris Miller]

Hi Jean-Louis, 

| A vtape do not reserve the space, it only use the space of the dump you put on
| it.
| The vtape size should be the maximum size of any days or larger, they could be
| 120GB or 2TG, the result will be the same.
| Some vtape will use 5GB, some will use 120GB.

Thanks very much. That is quite helpful. 

Is it possible to re-use a vtape for several successive backups? If the vtape 
were twice the size of a level 0 backup, then the level 1 backups would be 
appended, but the next level 0 would be too large and would trigger a new tape. 

Additionally, How do I configure larger cycles, meaning, I will use 150GB each 
week, and 2TB can hold 13 weeks. How do I tell Amanda that after 13 weeks, she 
should start over by deleting the oldest vtape? 

Thanks for the help, 
-- 
Chris. 

V:916.974.0424 
F:916.974.0428 

Re: VTape configuration

[Jean-Louis Martineau]

A vtape do not reserve the space, it only use the space of the dump you 
put on it.
The vtape size should be the maximum size of any days or larger, they 
could be 120GB or 2TG, the result will be the same.

Some vtape will use 5GB, some will use 120GB.

Jean-Louis

On 03/01/18 04:08 PM, Chris Miller wrote:

Hi Winston,

For the longest time I did traditional backups (fulls and
incrementals, via tar).  If that model still fits your needs, you
can stay with that.

Once I began backing up a small cluster of machines, Amanda's
paradigm began to show its value.  However, it relies upon volume
management, and it requires an acceptance that "her" algorithms
allow a more optimal distribution of backups based on both
availability of backup media (virtual or otherwise) and the amount
of individual and collective changes on the client machines - as
well as certain parameters I set such as the maximum interval I am
willing to accept between full backups.  The benefit is that with
the amount of space I've allocated to vtapes, I get the maximum
amount of change data on backup.  It isn't overprovisioning; it's
about optimization.  It's also proven itself in restores, where
instead of having to restore a full directory and then every L1
and L2 delta, I can simply tell Amanda to restore
file-version-as-of-specific-date.

I highly suggest a read of this FAQ:

http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F

;
particularly the section about Amanda's planning strategy.  If you
"insist" on constraining Amanda to one-volume-per-backup, you are
basically going against the strategy; without that capability, I
don't think that Amanda's overhead gives you anything you can't do
with tar and a cron job.


I understand how Amanda wants to try to "smooth" the mix of backup 
levels and filesystem sizes so that backup costs about the same amount 
of time and storage each cycle, and that is a very worthwhile goal, so 
I don't want to impede that. I also understand that tape discipline is 
already built-in to Amanda at a fundamental level, so I don't want to 
mess with that, either. So, I seek advice.


Suppose "amanda.example.com 
" 
is backing-up "client.example.com 
" 
to "NAS0.example.com 
". 
My level 0 backups are typically 120GB and my level 1 backups are 
typically 5GB, and I have 2TB on NAS0. That's 120+6*5 = 150GB/week, 
meaning I have sufficient room for thirteen weeks of backup.  This 
seems to me like it might be a pretty common scenario and that there 
might be example configs floating around that would size the vtapes 
for optimal use. Is there one?


I have some questions:

 1. Can I make my vtapes all 150GB, and then instruct Amanda to put
one cycle (one level 0, and six level 1) on one vtape, meaning
re-use a vtape multiple time in a backup cycle? I like this
approach quite a bit, if it is possible. It "packages" one level 0
with all the attendant level 1 differentials and eliminates my
strongest reservations about vtapes -- namely that I don't know
where anything is.
 2. Failing that, should I make my vtapes 120GB, so I can fit a level
0 backup on one vtape, but then will Amanda truncate level 1
backups, so that the vtapes storage requirement is NOT 120G, but
more like 5GB?
 3. Alternatively, I could make my vtapes all 5GB and then Amanda will
have to span fourteen vtapes for the level 0? This might be
optimal use of storage, but it scares me with added complexity. I
won't know where anything is, meaning I will have to have Amanda
tools to unpack a backup, and in the case of a disaster, that may
be really inconvenient.

I sure would like to have option 1, if I could...
--
Chris.

V:916.974.0424
F:916.974.0428

This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

Re: VTape configuration

[Stefan G. Weichinger]

Am 2018-01-03 um 22:08 schrieb Chris Miller:


I have some questions:

 1. Can I make my vtapes all 150GB, and then instruct Amanda to put one
cycle (one level 0, and six level 1) on one vtape, meaning re-use a
vtape multiple time in a backup cycle? I like this approach quite a
bit, if it is possible. It "packages" one level 0 with all the
attendant level 1 differentials and eliminates my strongest
reservations about vtapes -- namely that I don't know where anything is.
 2. Failing that, should I make my vtapes 120GB, so I can fit a level 0
backup on one vtape, but then will Amanda truncate level 1 backups,
so that the vtapes storage requirement is NOT 120G, but more like 5GB?
 3. Alternatively, I could make my vtapes all 5GB and then Amanda will
have to span fourteen vtapes for the level 0? This might be optimal
use of storage, but it scares me with added complexity. I won't know
where anything is, meaning I will have to have Amanda tools to
unpack a backup, and in the case of a disaster, that may be really
inconvenient.

I sure would like to have option 1, if I could...


I would say, go with option 2 if you want to avoid complexity.
The vtapes won't all use up the full defined size if the actual daily 
dump needs less space. So why not specify that a bit larger than the 
expected lev0/FULL-run and see how that works out?


AFAIK option 1 is not possible with current Amanda-strategies (at least 
not easily or without other complexities involved). But maybe I am wrong 
here ...

Re: VTape configuration

[Chris Miller]

Hi Winston, 

| For the longest time I did traditional backups (fulls and incrementals, via
| tar). If that model still fits your needs, you can stay with that.

| Once I began backing up a small cluster of machines, Amanda's paradigm began 
to
| show its value. However, it relies upon volume management, and it requires an
| acceptance that "her" algorithms allow a more optimal distribution of backups
| based on both availability of backup media (virtual or otherwise) and the
| amount of individual and collective changes on the client machines - as well 
as
| certain parameters I set such as the maximum interval I am willing to accept
| between full backups. The benefit is that with the amount of space I've
| allocated to vtapes, I get the maximum amount of change data on backup. It
| isn't overprovisioning; it's about optimization. It's also proven itself in
| restores, where instead of having to restore a full directory and then every 
L1
| and L2 delta, I can simply tell Amanda to restore
| file-version-as-of-specific-date.

| I highly suggest a read of this FAQ: [
| 
http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F
| |
| 
http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F
| ] ; particularly the section about Amanda's planning strategy. If you "insist"
| on constraining Amanda to one-volume-per-backup, you are basically going
| against the strategy; without that capability, I don't think that Amanda's
| overhead gives you anything you can't do with tar and a cron job.

I understand how Amanda wants to try to "smooth" the mix of backup levels and 
filesystem sizes so that backup costs about the same amount of time and storage 
each cycle, and that is a very worthwhile goal, so I don't want to impede that. 
I also understand that tape discipline is already built-in to Amanda at a 
fundamental level, so I don't want to mess with that, either. So, I seek 
advice. 

Suppose "amanda.example.com" is backing-up "client.example.com" to 
"NAS0.example.com". My level 0 backups are typically 120GB and my level 1 
backups are typically 5GB, and I have 2TB on NAS0. That's 120+6*5 = 150GB/week, 
meaning I have sufficient room for thirteen weeks of backup. This seems to me 
like it might be a pretty common scenario and that there might be example 
configs floating around that would size the vtapes for optimal use. Is there 
one? 

I have some questions: 

1. Can I make my vtapes all 150GB, and then instruct Amanda to put one 
cycle (one level 0, and six level 1) on one vtape, meaning re-use a vtape 
multiple time in a backup cycle? I like this approach quite a bit, if it is 
possible. It "packages" one level 0 with all the attendant level 1 
differentials and eliminates my strongest reservations about vtapes -- namely 
that I don't know where anything is. 
2. Failing that, should I make my vtapes 120GB, so I can fit a level 0 
backup on one vtape, but then will Amanda truncate level 1 backups, so that the 
vtapes storage requirement is NOT 120G, but more like 5GB? 
3. Alternatively, I could make my vtapes all 5GB and then Amanda will have 
to span fourteen vtapes for the level 0? This might be optimal use of storage, 
but it scares me with added complexity. I won't know where anything is, meaning 
I will have to have Amanda tools to unpack a backup, and in the case of a 
disaster, that may be really inconvenient. 

I sure would like to have option 1, if I could... 
-- 
Chris. 

V:916.974.0424 
F:916.974.0428