Re: Backup Strategy

[Gene Heskett]

On Monday 13 January 2014 12:56:52 Andrius D. Ilgunas did opine:

> Thanks a brazillion all!
> 
> There's a lot of good info here that I'll need to take some time to
> digest. Amanda sure is a big sandwich.
> 
A good way to put it in the vernacular, yes it is.

> While I'm processing it all, might someone point me to the syntax of
> what might be called a 'compound DLE' ?
> 
> Instead of something like:
> 
> 
> localhost /bin simple-gnutar-local
> 
> 
> I've seen some entries with the following format:
> 
> 
> host entry_name device_or_directory {
>   dumptype
>   include "regex"
>   exclude [append] "regex"
> }  somenumber someword
> 
> 
> Though I haven't deciphered what somenumber and someword mean.  It
> appears that somenumber might be a priority and someword might be the
> authentication method?
>
To use a piece of my disklist:
host directory  dumptype spindle#   access
coyote  /usr/pixcoyote-tar  1   local

where coyote is the alias for this machine, contained in /etc/hosts file

directory is that dir on that host

dumptype profile is the name of the stanza to use, contained in
/usr/local/etc/amanda/Daily/amanda.conf.

Spindle# (somenumber) is used to prevent amanda from hammering the same 
physical drive with more than one backup operation at a time as the seek 
times would combine and slow the overall operation much more than the 
serialization this enforces does.

The last (someword) is an indicator to amanda as to whether the host/dir is 
local to this machine, or if leo, it has to go out on the network to find 
this particular host/dir.

coyote  /usr/sbin   comp-coyote-tar 1   local
coyote  /usr/share  comp-coyote-tar 1   local
coyote  /usr/srccomp-coyote-tar 1   local
coyote  /usr/games  comp-coyote-tar 1   local

This next entry (on both machines below, is mainly to pickup the LCNC 
config and nc_files directories in the /home dir of those 2 machines.
IOW, backing up the machine configuration files, and any of my gcode
I've written so I don't have to reinvent those wheels again.

shop/home   shop-tar-comp   1   le0

The next 2 entries pickup the machine(host) local amanda database kept 
on each machine.

shop/usr/lib/amanda shop-tar-comp   1   le0
shop/var/lib/amanda shop-tar-comp   1   le0

And last, pickup the locally installed stuff.

shop/usr/local  shop-tar-comp   1   le0
lathe   /home   shop-tar-comp   1   le0
lathe   /usr/local  shop-tar-comp   1   le0
lathe   /usr/lib/amanda shop-tar-comp   1   le0
lathe   /var/lib/amanda shop-tar-comp   1   le0

You will not find those dumptype profiles in your amanda.conf as they are 
combination profiles since you can use the existing ones in the 
example/amanda.conf by going to the bottom of that set of defines in your 
amanda.conf, and add your own, including one of the supplied ones by the 
use of an include "string" line in your own.

This is kind of verbose but shows how shop-tar-comp was derived.

define dumptype global {
comment "Global definitions"
# This is quite useful for setting global parameters, so you don't have
# to type them everywhere.  All dumptype definitions in this sample file
# do include these definitions, either directly or indirectly.
# There's nothing special about the name `global'; if you create any
# dumptype that does not contain the word `global' or the name of any
# other dumptype that contains it, these definitions won't apply.
# Note that these definitions may be overridden in other
# dumptypes, if the redefinitions appear *after* the `global'
# dumptype name.
# You may want to use this for globally enabling or disabling
# indexing, recording, etc.  Some examples:
index yes
program "GNUTAR"
auth "bsdtcp"
#application "/usr/local/libexec/amanda/application/amgtar"
#   program "APPLICATION"
# record no
##   tape_splitsize 1G
##   split_diskbuffer   "/dumps"
#   fallback_splitsize  64m

exclude list "/GenesAmandaHelper-0.61/excludes"
}

define dumptype shop-tar-comp {
global
compress client best
}

> Is there a document somewhere that describes this syntax and options
> available?

Man pages are there to clarify, but much of it is in the example tree 
off the root of the tarball if building from a tarball.

And we don't generally advise using a distro's amanda packaging because
the limits of the packaging system muck seriously with amanda's security
profiles.  Specifically, neither rpm nor dpkg can apply the needed suid
settings at install to make amanda run like the tarball build will.

To automate as much of that for me as possible, I carved up a bash script
several years ago, called gh.cf that does all that. First I become root
with a sudo -i, unpack the tarbal

Re: Backup Strategy

[Jon LaBadie]

On Mon, Jan 13, 2014 at 08:18:22AM -0800, Andrius D. Ilgunas wrote:
> Thanks a brazillion all!
> 
> There's a lot of good info here that I'll need to take some time to digest.
>  Amanda sure is a big sandwich.
> 
> While I'm processing it all, might someone point me to the syntax of what
> might be called a 'compound DLE' ?
> 
> Instead of something like:
> 
> 
> localhost /bin simple-gnutar-local
> 
> 
> I've seen some entries with the following format:
> 
> 
> host entry_name device_or_directory {
>   dumptype
>   include "regex"
>   exclude [append] "regex"
> }  somenumber someword
> 
> 
> Though I haven't deciphered what somenumber and someword mean.  It appears
> that somenumber might be a priority and someword might be the
> authentication method?

Somenumber and someword could also be applied to the end
of your first sample DLE.  See first, the relevant manpages.

The simple-gnutar-local and the {...} are the dumptype
for that DLE.  Dumptypes are defined in amanda.conf and
can be given a name (ex simple-gnutar-local).  If the
dumptype has not been named in a previous definition,
it can be defined in the braces.

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (609) 477-8330 (C)

Re: Backup Strategy

[Jean-Francois Malouin]

* Andrius D. Ilgunas  [20140113 11:19]:
> Thanks a brazillion all!
> 
> There's a lot of good info here that I'll need to take some time to digest.
>  Amanda sure is a big sandwich.
> 
> While I'm processing it all, might someone point me to the syntax of what
> might be called a 'compound DLE' ?

The manpages for disklist and amanda.conf will tell you that.
amanda.conf is a mammoth so you might want to first peruse the FAQ and
HOWTOs at the zmanda wiki, http://wiki.zmanda.com

> 
> Instead of something like:
> 
> 
> localhost /bin simple-gnutar-local
> 
> 
> I've seen some entries with the following format:
> 
> 
> host entry_name device_or_directory {
>   dumptype
>   include "regex"
>   exclude [append] "regex"
> }  somenumber someword
> 
> 
> Though I haven't deciphered what somenumber and someword mean.  It appears
> that somenumber might be a priority and someword might be the
> authentication method?

somenumber is a spindle (from the manpage)

   spindle
   Default: -1. A number used to balance backup load on a host. Amanda
   will not run multiple backups at the same time on the same spindle,
   unless the spindle number is -1, which means there is no spindle
   restriction.

someword is a network interface.
In the real word, I've never used it and I'm not sure if it really
does anything. The section in amanda.conf that references it says:
" At the moment, this is of little use:...

Those in the know, correct me if I'm wrong!

jf

> 
> Is there a document somewhere that describes this syntax and options
> available?
> 
> Again, thanks so much for everybody's responses!  I think I'm off to a good
> start now.
> 
> 
> 
> 
> --
> Andrius
> 
> 
> On Mon, Jan 13, 2014 at 2:57 AM, Toomas Aas wrote:
> 
> > On Sun, 12 Jan 2014 Gene Heskett wrote:
> >
> >
> >> If separate configs, which I can't personally find an overpowering reason
> >> for, you would need,  most likely, two separate tape "libraries" each
> >> containing its own drive(s), or 2 separate big hard drives.
> >>
> >
> > Actually two separate configs do not require separate tape libraries or
> > drives, just tapes labelled with different "labelstr".
> >
> > By this I don't mean to imply that two separate configs are the best
> > solution for OP.
> >
> > --
> > Toomas Aas
> >
> >

-- 
<° >< Jean-François Malouin   
Systems/Network Manager | McConnell Brain Imaging Centre | MNI
3801 University St, Suite WB219, Montreal, QC, H3A 2B4, Canada

Re: Backup Strategy

[Andrius D. Ilgunas]

Thanks a brazillion all!

There's a lot of good info here that I'll need to take some time to digest.
 Amanda sure is a big sandwich.

While I'm processing it all, might someone point me to the syntax of what
might be called a 'compound DLE' ?

Instead of something like:


localhost /bin simple-gnutar-local


I've seen some entries with the following format:


host entry_name device_or_directory {
  dumptype
  include "regex"
  exclude [append] "regex"
}  somenumber someword


Though I haven't deciphered what somenumber and someword mean.  It appears
that somenumber might be a priority and someword might be the
authentication method?

Is there a document somewhere that describes this syntax and options
available?

Again, thanks so much for everybody's responses!  I think I'm off to a good
start now.




--
Andrius


On Mon, Jan 13, 2014 at 2:57 AM, Toomas Aas wrote:

> On Sun, 12 Jan 2014 Gene Heskett wrote:
>
>
>> If separate configs, which I can't personally find an overpowering reason
>> for, you would need,  most likely, two separate tape "libraries" each
>> containing its own drive(s), or 2 separate big hard drives.
>>
>
> Actually two separate configs do not require separate tape libraries or
> drives, just tapes labelled with different "labelstr".
>
> By this I don't mean to imply that two separate configs are the best
> solution for OP.
>
> --
> Toomas Aas
>
>

Re: Backup Strategy

[Toomas Aas]

On Sun, 12 Jan 2014 Gene Heskett wrote:



If separate configs, which I can't personally find an overpowering reason
for, you would need,  most likely, two separate tape "libraries" each
containing its own drive(s), or 2 separate big hard drives.


Actually two separate configs do not require separate tape libraries  
or drives, just tapes labelled with different "labelstr".


By this I don't mean to imply that two separate configs are the best  
solution for OP.


--
Toomas Aas

Re: Backup Strategy

[Jon LaBadie]

On Sun, Jan 12, 2014 at 12:47:56PM -0800, Andrius D. Ilgunas wrote:
> 
> On Sun, Jan 12, 2014 at 10:17 AM, Jon LaBadie  wrote:
> 
> > Again, you are thinking there will be a weekly full dump.  Unless that
> > is a separate config that does only full dumps, and you run it once
> > each dumpcycle.  Sounds like a lot of duplication of effort.
> >
> 
> You're right, and I see now how my understanding was a bit off the mark.  I
> wasn't intending to run a separate config just to ensure full dumps on
> Sundays, but I was under a misconception. Thanks for correcting that!
> 

Note, a separate archive config has its place.  One client had a
regulatory requirement to maintain "permanant data backups".  He
used a separate config and the tapes were couriered to a secure
data vault.

I used to have an archive config.  One use I had for it was before
major changes to a system.  I could manually run amdump for those
file system or disks that were going to be affected.

BTW, the traditional style of backups is "possible" with Amanda.
One way is to set the config with a strategy of incremental only.
Then in your in your crontab, for the day you want full backups
you use amdump options to override the normal strategy and to
force full dumps.

She's quite flexible.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (609) 477-8330 (C)

Re: Backup Strategy

[Gene Heskett]

On Sunday 12 January 2014 15:56:32 Andrius D. Ilgunas did opine:

> --
> Andrius
> 
> 
> On Sun, Jan 12, 2014 at 1:21 AM, Gene Heskett  wrote:
> 
> so when 500Gig and up hard drives became available, I converted to
> vtapes
> 
> > on a hard drive, which has turned out to be, dollar for dollar, about
> > 100x more dependable,
> 
> Awesome, that answers another question I had - "Why use tapes these
> days?"
> 
> > Linux has excellent drive health drive monitoring utilities, so the 3
> > drive failures I have had since converting about 8 or 9 years ago,
> > has always given me sufficient warning that no data has been lost,
> > not a byte.
> 
> Yes it does. Though I'm running a mostly FreeBSD shop and I just
> recently started using zfs primarily because of its mirroring/data
> correction capabilities, which I find very valuable in these days of
> >1TB HDDs.
> 
> Speaking of which, I see that amanda has a amzfs-sendrecv function,
> which I presume is a wrapper around getting zfs snapshots stored
> somewhere.  But I'm not yet clear on how it (zfs/sendrecv) "fits" in
> amanda's backup scheme.  Would it simply take snapshots and save them
> on a (v)tape?  And when it comes time to recover, I would guess that
> amrecover would rebuild the fs by restoring snapshot deltas?  Sounds
> like some more reading in this facet is needed.
> 
> > Then about 12 years back I realized that a bare metal recovery to a
> > newly formatted operating drive would always be a 1 days loss because
> > the amanda data would also be lost, what was on the tapes was a day
> > old, I wrote some wrapper scripts that make a tarball each of the
> > instant configuration directory, and a tarball of the data directory
> > amanda keeps and appends them both to the tape just used, so that
> > when doing the full recovery, it started from the last known good
> > state.
> 
> Good one!  I'll keep that in mind when building the backup system.
> 
>  > Unfortunately, even with the binlog enabled, some of the queries
>  > don't
>  > 
> > > get recorded (long story),
> > 
> > And one that needs attended to in case forensics are required.
> 
> Totally.  It's just one of those problems inherited from the previous
> DBA/SA
> 
> > > Another part of my backup plan is going to have two configs.  One
> > > for the database instance ('hypercritical data') that will be
> > > backed up insanely often, and another config for merely really
> > > important data (svn repos, wikis, etc) which will have a less
> > > taxing schedule.
> > 
> > Separate configs need separate storage, you cannot mix tapes if you
> > want to keep your sanity.  Let each amanda config keep its own
> > database and its own named tapes.  Its really quite good at it.
> 
> I'm not quite clear on your response.  Creating a separate storage
> volume isn't a problem since I'm using vtapes.  Do you mean that it
> would be easier to have one config to rule them all?  Where "them" are
> the different priority files and "them" are different clients backing
> up to the backup server.

If separate configs, which I can't personally find an overpowering reason 
for, you would need,  most likely, two separate tape "libraries" each 
containing its own drive(s), or 2 separate big hard drives.  With the tapes 
labeled specifically for each use so that if somebody puts the wrong tape 
in the libraries changer, it would refuse to use it, thereby safe keeping 
the data.  I think that same idea, where you have large data dumps, could 
easily be extended to HD's.

Now, here is a thought, and I have no clue about zfs,  I have never used 
it, but Tiger Direct has a 3 drive quick change cage that holds 3 3.5" 
drives in the same rack space as 2 half height 5.25 drives.  No other 
special hardware, you just power down, open the retainer door in front of 
the drive, it slides out about 1/2", you finish pulling it out and slide 
another drive into the slot, close the door & power up.  Sells for about a 
70 dollar bill, and surprise, the commodity 1 Tb seagate also sells for 
about a 70 dollar bill. $280 gives your 3Tb of quickly removable storage.  
I am sure you could find a way to make that simplify things.

A warning since linux has had a propensity to move drives around from sda 
to sdd or wherever, I think depending on the timing of the drive ready 
discovery process, that you will NOT want to mount drives as 
/dev/sd(abcdef) or as UUID=big number statements but as a LABEL=name drive 
is labeled as type of mount statements in the /etc/fstab file.  That way 
you could have a dozen drives labeled as DailyOS##, and another dozen 
labeled as DailyDB## and with some advice from Jean-Louis, figure out a way 
to have a years worth of 30 day backups no farther away than a 
compartmented file drawer.  It would take all that much effort to come up 
with a divider design that could hold those 2 dozen drives in one desk 
drawer, or better yet, a slide out on HD drawer sliders tray in a fireproof 
safe.

Another thought about those d

Re: Backup Strategy

[Andrius D. Ilgunas]

--
Andrius


On Sun, Jan 12, 2014 at 1:10 PM, Gerrit A. Smit wrote:

>
> Andrius D. Ilgunas wrote on 12-01-14 21:42:
>
>
>
>  I'm not quite clear on your response.  Creating a separate storage
> volume isn't a problem since I'm using vtapes.  Do you mean that it would
> be easier to have one config to rule them all?  Where "them" are the
> different priority files and "them" are different clients backing up to the
> backup server.
>
>  Or would it make more sense to have separate configs, e.g.
>  CriticalDatabasesOnHost1_config, ImportantFilesOnHost1_config,
> ImportantFilesOnHost2_config, MiscFilesOnHost3_config, etc ?
>
> No, its about an Amanda-config which runs daily and does things the way
> Amanda is designed to do them (all levels).
>
> And a separate config which runs e.g. weekly, where the config forces
> Amanda not to think but just make level 0 dumps only on separate (level 0
> dumps) media.
>
>
Hm.  That confused me even more.  I'll keep reading/experimenting until it
makes sense :)


>  Greetings from
>
> --
> Gerrit A. Smit
>

Re: Backup Strategy

[Andrius D. Ilgunas]

Hey again Jon,

On Sun, Jan 12, 2014 at 10:17 AM, Jon LaBadie  wrote:

> > Heck, I even learned about the Tower of Hanoi scheme!  But one piece of
> > advice stuck with me, and that is "when restoring data, you don't have
> time
> > to try to remember/decipher which files are on which tapes". In other
> > words, backup / on a regular schedule, even if it'll cost you some disk
> > space/timeit'll be worth it when it comes time to recover.  I
> suppose I
> > can make things more complex when I get really bored. ;)
>
> Assuming you have your config keep an index, amrecover is an interactive
> tool
> that does this for you.  With a date set it can determine which 'tapes'
> contain the file as of that date.
>

Aha!.  After I get a system working properly, it's likely that I'll go back
to reconfigure it armed with a bit more experience and understanding to
make use of these more advanced schemes.  For example, I now know that
amrecover can do this for me, assuming I have my config keep an index :)



> >
> > As I'm working out the backup plan, it's starting to seem that I'll want
> to
> > have a dumpcycle of 7 days, and then keep those weekly full dumps
> > elsewhere.
>
> Again, you are thinking there will be a weekly full dump.  Unless that
> is a separate config that does only full dumps, and you run it once
> each dumpcycle.  Sounds like a lot of duplication of effort.
>

You're right, and I see now how my understanding was a bit off the mark.  I
wasn't intending to run a separate config just to ensure full dumps on
Sundays, but I was under a misconception. Thanks for correcting that!



> There has been regular talk about adding a feature to extract a set
> of full dumps.  I think amvault is a command that may implement this
> feature.  I'm unsure of its status.
>

I'll be sure to take a gander at that.

>
> Jon
> --
> Jon H. LaBadie j...@jgcomp.com
>  11226 South Shore Rd.  (703) 787-0688 (H)
>  Reston, VA  20190  (609) 477-8330 (C)
>

Re: Backup Strategy

[Andrius D. Ilgunas]

--
Andrius


On Sun, Jan 12, 2014 at 1:21 AM, Gene Heskett  wrote:

so when 500Gig and up hard drives became available, I converted to vtapes
> on a hard drive, which has turned out to be, dollar for dollar, about 100x
> more dependable,


Awesome, that answers another question I had - "Why use tapes these days?"


> Linux has excellent drive health drive monitoring utilities, so the 3 drive
> failures I have had since converting about 8 or 9 years ago, has always
> given me sufficient warning that no data has been lost, not a byte.
>

Yes it does. Though I'm running a mostly FreeBSD shop and I just recently
started using zfs primarily because of its mirroring/data correction
capabilities, which I find very valuable in these days of >1TB HDDs.

Speaking of which, I see that amanda has a amzfs-sendrecv function, which I
presume is a wrapper around getting zfs snapshots stored somewhere.  But
I'm not yet clear on how it (zfs/sendrecv) "fits" in amanda's backup
scheme.  Would it simply take snapshots and save them on a (v)tape?  And
when it comes time to recover, I would guess that amrecover would rebuild
the fs by restoring snapshot deltas?  Sounds like some more reading in this
facet is needed.


> Then about 12 years back I realized that a bare metal recovery to a newly
> formatted operating drive would always be a 1 days loss because the amanda
> data would also be lost, what was on the tapes was a day old, I wrote some
> wrapper scripts that make a tarball each of the instant configuration
> directory, and a tarball of the data directory amanda keeps and appends
> them both to the tape just used, so that when doing the full recovery, it
> started from the last known good state.


Good one!  I'll keep that in mind when building the backup system.

 > Unfortunately, even with the binlog enabled, some of the queries don't

> > get recorded (long story),
>
> And one that needs attended to in case forensics are required.
>
>
Totally.  It's just one of those problems inherited from the previous DBA/SA


>  >
> > Another part of my backup plan is going to have two configs.  One for
> > the database instance ('hypercritical data') that will be backed up
> > insanely often, and another config for merely really important data
> > (svn repos, wikis, etc) which will have a less taxing schedule.
>
> Separate configs need separate storage, you cannot mix tapes if you want to
> keep your sanity.  Let each amanda config keep its own database and its own
> named tapes.  Its really quite good at it.
>
>
I'm not quite clear on your response.  Creating a separate storage volume
isn't a problem since I'm using vtapes.  Do you mean that it would be
easier to have one config to rule them all?  Where "them" are the different
priority files and "them" are different clients backing up to the backup
server.

Or would it make more sense to have separate configs, e.g.
 CriticalDatabasesOnHost1_config, ImportantFilesOnHost1_config,
ImportantFilesOnHost2_config, MiscFilesOnHost3_config, etc ?



> >
> Thats my $0.02, but adjust for inflation since 1934.  Yeah, I'm an older
> user. ;-)
>
>
Thanks so much for your input Gene! And, that just goes to show how much
value can be found in $0.02 from an older user. :) Your 2 pennies are worth
$4.94 !!



> Cheers, Gene
> --
> "There are four boxes to be used in defense of liberty:
>  soap, ballot, jury, and ammo. Please use in that order."
> -Ed Howdershelt (Author)
> Genes Web page 
>
> "I'd love to go out with you, but my favorite commercial is on TV."
> A pen in the hand of this president is far more
> dangerous than 200 million guns in the hands of
>  law-abiding citizens.
>

Re: Backup Strategy

[Jon LaBadie]

On Sat, Jan 11, 2014 at 10:33:26PM -0800, Andrius D. Ilgunas wrote:
> Excellent discussion Jon!  Thank you so very much for your thoughts!!
> 
> I was aware of Amanda's philosophy of "I want one full dump every dumpcycle
> which is n days" and then to let Amanda find the optimal combination of
> full/incrementals, and that's one of the reasons that Amanda was so
> attractive, other than it uses native compression tools (BIG plus for me).
> 
> Heck, I even learned about the Tower of Hanoi scheme!  But one piece of
> advice stuck with me, and that is "when restoring data, you don't have time
> to try to remember/decipher which files are on which tapes". In other
> words, backup / on a regular schedule, even if it'll cost you some disk
> space/timeit'll be worth it when it comes time to recover.  I suppose I
> can make things more complex when I get really bored. ;)

Assuming you have your config keep an index, amrecover is an interactive tool
that does this for you.  With a date set it can determine which 'tapes'
contain the file as of that date.
> 
> As I'm working out the backup plan, it's starting to seem that I'll want to
> have a dumpcycle of 7 days, and then keep those weekly full dumps
> elsewhere.

Again, you are thinking there will be a weekly full dump.  Unless that
is a separate config that does only full dumps, and you run it once
each dumpcycle.  Sounds like a lot of duplication of effort.

There has been regular talk about adding a feature to extract a set
of full dumps.  I think amvault is a command that may implement this
feature.  I'm unsure of its status.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (609) 477-8330 (C)

Re: Backup Strategy

[Gene Heskett]

t; seems like we'll be keeping many full dumps.
> 
> 
> Another part of my backup plan is going to have two configs.  One for
> the database instance ('hypercritical data') that will be backed up
> insanely often, and another config for merely really important data
> (svn repos, wikis, etc) which will have a less taxing schedule.

Separate configs need separate storage, you cannot mix tapes if you want to 
keep your sanity.  Let each amanda config keep its own database and its own 
named tapes.  Its really quite good at it.
 
> Now I suppose that I could combine these two schedules into one config,
> but it seems that it would make more sense from an administrative view
> to have two (more?) configs.
> 
> Do you/anyone else have any thoughts on that?
> 
Thats my $0.02, but adjust for inflation since 1934.  Yeah, I'm an older 
user. ;-)
 
> 
> --
> Andrius
> 
> On Sat, Jan 11, 2014 at 9:10 PM, Jon LaBadie  wrote:
> > On Sat, Jan 11, 2014 at 12:31:52PM -0800, Andrius D. Ilgunas wrote:
> > > Hmm...Jon's question got me to a'thnkin' about something that I
> > > haven't
> > 
> > yet
> > 
> > > read about.
> > > 
> > > So let's say the dump cycle is 30 days and I want to keep a copy of
> > > the previous month's dump before the tape1 gets overwritten so that
> > > I can
> > 
> > keep
> > 
> > > a record of the state of the database from two months ago.
> > > 
> > > If I understand the terminology, this would be an archive?
> > 
> > Andrius,
> > 
> > You might want to read further about the way Amanda differs from most
> > backup systems.  Typically they act as you are describing, a full dump
> > of everything once in a while and incrementals in-between.  So you get
> > a monster dump followed by a bunch of small dumps.
> > 
> > Amanda tries to level out the size of the dumps.  You define objects
> > to be backed up, called DiskList Entries (DLEs) in Amanda, and how
> > you want them backed up (they need not all be the same).  Amanda then
> > spreads the full back ups of the DLEs across the whole dump cycle. 
> > At the beginning of each dump run Amanda spends time determining how
> > big a dump it expects from each DLE as several levels, at least two,
> > level 0 full and level 1 incremental.  Then it determines how best to
> > achieve balanced dumping while still keeping to your defined
> > parameters.  For any specific DLE it may continue to do a level 1
> > incremental, switch it to a smaller level 2 incremental, or promote
> > it to an early level 0 full dump.
> > 
> > > So a strategy might be: on the 30th day, do a full dump, and copy it
> > > to 'somewhere safe', whether on S3, or a DVD.That seems like a
> > > pretty straight-forward concept, but might this idea be improved
> > > upon somehow?
> > 
> > A strong recommendation is to never have less than two full
> > dumpcycles. I suspect most Amanda installations maintain well over 2
> > dumpcycles.  If you are reusing your storage media, and you only have
> > one level 0, should it be overwritten by mistake or something bad
> > happen to it, you only have less useful incrementals.
> > 
> > Jon
> > --
> > Jon H. LaBadie j...@jgcomp.com
> > 
> >  11226 South Shore Rd.  (703) 787-0688 (H)
> >  Reston, VA  20190  (609) 477-8330 (C)


Cheers, Gene
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

"I'd love to go out with you, but my favorite commercial is on TV."
A pen in the hand of this president is far more
dangerous than 200 million guns in the hands of
 law-abiding citizens.

Re: Backup Strategy

[Andrius D. Ilgunas]

Excellent discussion Jon!  Thank you so very much for your thoughts!!

I was aware of Amanda's philosophy of "I want one full dump every dumpcycle
which is n days" and then to let Amanda find the optimal combination of
full/incrementals, and that's one of the reasons that Amanda was so
attractive, other than it uses native compression tools (BIG plus for me).

Heck, I even learned about the Tower of Hanoi scheme!  But one piece of
advice stuck with me, and that is "when restoring data, you don't have time
to try to remember/decipher which files are on which tapes". In other
words, backup / on a regular schedule, even if it'll cost you some disk
space/timeit'll be worth it when it comes time to recover.  I suppose I
can make things more complex when I get really bored. ;)

As I'm working out the backup plan, it's starting to seem that I'll want to
have a dumpcycle of 7 days, and then keep those weekly full dumps
elsewhere.  At this time, we don't have a massive quantity of critical
data, and we do have enough backup storage to keep 52 full dumps/year.  One
the reasons that I'm going to keep so many is because of the way one of our
database has been (poorly) designed. There have been questions asked such
as:

"On which day did this record change?"

Unfortunately, even with the binlog enabled, some of the queries don't get
recorded (long story), so at least for the database config, it seems like
we'll be keeping many full dumps.


Another part of my backup plan is going to have two configs.  One for the
database instance ('hypercritical data') that will be backed up insanely
often, and another config for merely really important data (svn repos,
wikis, etc) which will have a less taxing schedule.

Now I suppose that I could combine these two schedules into one config, but
it seems that it would make more sense from an administrative view to have
two (more?) configs.

Do you/anyone else have any thoughts on that?



--
Andrius


On Sat, Jan 11, 2014 at 9:10 PM, Jon LaBadie  wrote:

> On Sat, Jan 11, 2014 at 12:31:52PM -0800, Andrius D. Ilgunas wrote:
> > Hmm...Jon's question got me to a'thnkin' about something that I haven't
> yet
> > read about.
> >
> > So let's say the dump cycle is 30 days and I want to keep a copy of the
> > previous month's dump before the tape1 gets overwritten so that I can
> keep
> > a record of the state of the database from two months ago.
> >
> > If I understand the terminology, this would be an archive?
>
> Andrius,
>
> You might want to read further about the way Amanda differs from most
> backup systems.  Typically they act as you are describing, a full dump
> of everything once in a while and incrementals in-between.  So you get
> a monster dump followed by a bunch of small dumps.
>
> Amanda tries to level out the size of the dumps.  You define objects to
> be backed up, called DiskList Entries (DLEs) in Amanda, and how you want
> them backed up (they need not all be the same).  Amanda then spreads the
> full back ups of the DLEs across the whole dump cycle.  At the beginning
> of each dump run Amanda spends time determining how big a dump it expects
> from each DLE as several levels, at least two, level 0 full and level 1
> incremental.  Then it determines how best to achieve balanced dumping
> while still keeping to your defined parameters.  For any specific DLE it
> may continue to do a level 1 incremental, switch it to a smaller level 2
> incremental, or promote it to an early level 0 full dump.
> >
> > So a strategy might be: on the 30th day, do a full dump, and copy it to
> > 'somewhere safe', whether on S3, or a DVD.That seems like a pretty
> > straight-forward concept, but might this idea be improved upon somehow?
>
> A strong recommendation is to never have less than two full dumpcycles.
> I suspect most Amanda installations maintain well over 2 dumpcycles.  If
> you are reusing your storage media, and you only have one level 0,
> should it be overwritten by mistake or something bad happen to it, you
> only have less useful incrementals.
>
> Jon
> --
> Jon H. LaBadie j...@jgcomp.com
>  11226 South Shore Rd.  (703) 787-0688 (H)
>  Reston, VA  20190  (609) 477-8330 (C)
>

Re: Backup Strategy

[Jon LaBadie]

On Sat, Jan 11, 2014 at 12:31:52PM -0800, Andrius D. Ilgunas wrote:
> Hmm...Jon's question got me to a'thnkin' about something that I haven't yet
> read about.
> 
> So let's say the dump cycle is 30 days and I want to keep a copy of the
> previous month's dump before the tape1 gets overwritten so that I can keep
> a record of the state of the database from two months ago.
> 
> If I understand the terminology, this would be an archive?

Andrius,

You might want to read further about the way Amanda differs from most
backup systems.  Typically they act as you are describing, a full dump
of everything once in a while and incrementals in-between.  So you get
a monster dump followed by a bunch of small dumps.

Amanda tries to level out the size of the dumps.  You define objects to
be backed up, called DiskList Entries (DLEs) in Amanda, and how you want
them backed up (they need not all be the same).  Amanda then spreads the
full back ups of the DLEs across the whole dump cycle.  At the beginning
of each dump run Amanda spends time determining how big a dump it expects
from each DLE as several levels, at least two, level 0 full and level 1
incremental.  Then it determines how best to achieve balanced dumping
while still keeping to your defined parameters.  For any specific DLE it
may continue to do a level 1 incremental, switch it to a smaller level 2
incremental, or promote it to an early level 0 full dump.
> 
> So a strategy might be: on the 30th day, do a full dump, and copy it to
> 'somewhere safe', whether on S3, or a DVD.That seems like a pretty
> straight-forward concept, but might this idea be improved upon somehow?

A strong recommendation is to never have less than two full dumpcycles.
I suspect most Amanda installations maintain well over 2 dumpcycles.  If
you are reusing your storage media, and you only have one level 0,
should it be overwritten by mistake or something bad happen to it, you
only have less useful incrementals.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (609) 477-8330 (C)

Re: Backup Strategy

[Charles Curley]

On Sat, 11 Jan 2014 12:05:58 -0800
"Andrius D. Ilgunas"  wrote:

> I would expect them to be network-type speeds on a T1 even as the
> buckets are mounted via FUSE.  I don't have numbers, but I can say
> that a copy of one of my databases to /mnt/s3/bucket1 of around 250MB
> will timeout approximately 30% of the time.  But I'm sure that
> copying over incremental changes, either via amanda or rsync would be
> successful.

I'm not so sure about using FUSE. Check my thinking here, folks.

If you rsync between two computers, then rsync runs on both machines.
The two instances calculate hashes for chunks of the data, and send
the hashes back and forth. Based on the hashes, the originating instance
then sends only the changed chunks to the other machine. So you don't
have to carry the entire data set across the network in order to back
up the entire data set.

If you mount via FUSE, Samba, NFS or whatever, both instances of rsync
runs locally. So the "remote" rsync reads the entire archive over the
network in order to calculate its hashes. That's a huge overhead.

I have 69 virtual tapes, and typically use one, rarely two, tapes a
day. If I archive daily, I only have to move a few percent of my "tape
library" to the archive. Inverting that, if I archived via a network
mount, I would increase my overhead by hundreds of percent.

Can you have a remote virtual machine and rsync to it? Preferably over
SSH.

Of course, your data set may be small enough that you don't care.
That's fine, too.


-- 

The right of the people to be secure in their persons, houses, papers,
and effects, against unreasonable searches and seizures, shall not be
violated, and no Warrants shall issue, but upon probable cause,
supported by Oath or affirmation, and particularly describing the
place to be searched, and the persons or things to be seized.
-- U.S. Const. Amendment IV

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB

Re: Backup Strategy

[Andrius D. Ilgunas]

Hmm...Jon's question got me to a'thnkin' about something that I haven't yet
read about.

So let's say the dump cycle is 30 days and I want to keep a copy of the
previous month's dump before the tape1 gets overwritten so that I can keep
a record of the state of the database from two months ago.

If I understand the terminology, this would be an archive?

So a strategy might be: on the 30th day, do a full dump, and copy it to
'somewhere safe', whether on S3, or a DVD.That seems like a pretty
straight-forward concept, but might this idea be improved upon somehow?

I just finished reading O'Reilly's Backup and Restore (a very disappointing
book) that only brushed over some strategies, and Google seems to show me
only the same few pages on how to setup a config with only a topical
discussion about scheduling.

Anyway, thanks for your responses and ideas!


--
Andrius


On Sat, Jan 11, 2014 at 12:05 PM, Andrius D. Ilgunas wrote:

> Hey Jon,
>
> I would expect them to be network-type speeds on a T1 even as the buckets
> are mounted via FUSE.  I don't have numbers, but I can say that a copy of
> one of my databases to /mnt/s3/bucket1 of around 250MB will timeout
> approximately 30% of the time.  But I'm sure that copying over incremental
> changes, either via amanda or rsync would be successful.
>
> At this time, my holding disk (two 2TB drives - zfs mirrored) is
> sufficient to contain two or three full dumps of / of both clients that I'm
> interested in backing up, and probably will be for another year or so - the
> data growth is expected to be around 7% annually.
>
> And now that I brought that up, I haven't yet considered any kind of
> long-term archiving. One step at a time though.
>
>
>
> --
> Andrius
>
>
> On Sat, Jan 11, 2014 at 11:38 AM, Jon LaBadie  wrote:
>
>> On Sat, Jan 11, 2014 at 10:29:27AM -0800, Andrius D. Ilgunas wrote:
>> > Thanks a brazillion Charles!!
>> >
>> > I'll probably setup my systems likewise, but I wonder if anyone has any
>> > other opinions on this.
>> >
>> > --
>> > Andrius
>>
>> What kind of write speed to S3 can you expect?
>> Can your backups to holding disk exceed that by a sufficient
>> amount that holding disk capacity might be a problem?
>>
>> Jon
>> >
>> >
>> > On Sat, Jan 11, 2014 at 10:24 AM, Charles Curley <
>> > charlescur...@charlescurley.com> wrote:
>> >
>> > > On Sat, 11 Jan 2014 09:44:07 -0800
>> > > "Andrius D. Ilgunas"  wrote:
>> > >
>> > > > We're setting up amanda on our servers, and the primary backup is
>> > > > going to be on a dedicated disk/virtual tapes.  One of the offsite
>> > > > locations is going to be a bucket on Amazon's S3.
>> > > >
>> > > > Now I see that amanda has the capability of writing to multiple
>> > > > volumes in parallel, but I'm wondering if it would be more efficient
>> > > > to have amada write the backup to the localdisk first and then use
>> > > > rsync to mirror that backup onto the S3 bucket.
>> > > >
>> > > > Or would it be a better idea to have amanda manage both
>> > > > "tapechangers"?
>> > > >
>> > > > Any discussion would be most welcome!
>> > >
>> > > What I do is similar to your first proposal. I back up to virtual
>> > > tapes, then rsync to external hard drive for off site backup.
>> > >
>> > >
>> > >
>> http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html
>> > >
>> > > You probably want to save the metadata so that restores from offsite
>> > > will be easier.
>> > >
>> > >
>> > >
>> http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html
>> > >
>> > > --
>> > >
>> > > The right of the people to be secure in their persons, houses, papers,
>> > > and effects, against unreasonable searches and seizures, shall not be
>> > > violated, and no Warrants shall issue, but upon probable cause,
>> > > supported by Oath or affirmation, and particularly describing the
>> > > place to be searched, and the persons or things to be seized.
>> > > -- U.S. Const. Amendment IV
>> > >
>> > > Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
>> > >
>> >>> End of included message <<<
>>
>> --
>> Jon H. LaBadie j...@jgcomp.com
>>  11226 South Shore Rd.  (703) 787-0688 (H)
>>  Reston, VA  20190  (609) 477-8330 (C)
>>
>
>

Re: Backup Strategy

[Andrius D. Ilgunas]

Hey Jon,

I would expect them to be network-type speeds on a T1 even as the buckets
are mounted via FUSE.  I don't have numbers, but I can say that a copy of
one of my databases to /mnt/s3/bucket1 of around 250MB will timeout
approximately 30% of the time.  But I'm sure that copying over incremental
changes, either via amanda or rsync would be successful.

At this time, my holding disk (two 2TB drives - zfs mirrored) is sufficient
to contain two or three full dumps of / of both clients that I'm interested
in backing up, and probably will be for another year or so - the data
growth is expected to be around 7% annually.

And now that I brought that up, I haven't yet considered any kind of
long-term archiving. One step at a time though.



--
Andrius


On Sat, Jan 11, 2014 at 11:38 AM, Jon LaBadie  wrote:

> On Sat, Jan 11, 2014 at 10:29:27AM -0800, Andrius D. Ilgunas wrote:
> > Thanks a brazillion Charles!!
> >
> > I'll probably setup my systems likewise, but I wonder if anyone has any
> > other opinions on this.
> >
> > --
> > Andrius
>
> What kind of write speed to S3 can you expect?
> Can your backups to holding disk exceed that by a sufficient
> amount that holding disk capacity might be a problem?
>
> Jon
> >
> >
> > On Sat, Jan 11, 2014 at 10:24 AM, Charles Curley <
> > charlescur...@charlescurley.com> wrote:
> >
> > > On Sat, 11 Jan 2014 09:44:07 -0800
> > > "Andrius D. Ilgunas"  wrote:
> > >
> > > > We're setting up amanda on our servers, and the primary backup is
> > > > going to be on a dedicated disk/virtual tapes.  One of the offsite
> > > > locations is going to be a bucket on Amazon's S3.
> > > >
> > > > Now I see that amanda has the capability of writing to multiple
> > > > volumes in parallel, but I'm wondering if it would be more efficient
> > > > to have amada write the backup to the localdisk first and then use
> > > > rsync to mirror that backup onto the S3 bucket.
> > > >
> > > > Or would it be a better idea to have amanda manage both
> > > > "tapechangers"?
> > > >
> > > > Any discussion would be most welcome!
> > >
> > > What I do is similar to your first proposal. I back up to virtual
> > > tapes, then rsync to external hard drive for off site backup.
> > >
> > >
> > >
> http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html
> > >
> > > You probably want to save the metadata so that restores from offsite
> > > will be easier.
> > >
> > >
> > >
> http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html
> > >
> > > --
> > >
> > > The right of the people to be secure in their persons, houses, papers,
> > > and effects, against unreasonable searches and seizures, shall not be
> > > violated, and no Warrants shall issue, but upon probable cause,
> > > supported by Oath or affirmation, and particularly describing the
> > > place to be searched, and the persons or things to be seized.
> > > -- U.S. Const. Amendment IV
> > >
> > > Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
> > >
> >>> End of included message <<<
>
> --
> Jon H. LaBadie j...@jgcomp.com
>  11226 South Shore Rd.  (703) 787-0688 (H)
>  Reston, VA  20190  (609) 477-8330 (C)
>

Re: Backup Strategy

[Jon LaBadie]

On Sat, Jan 11, 2014 at 10:29:27AM -0800, Andrius D. Ilgunas wrote:
> Thanks a brazillion Charles!!
> 
> I'll probably setup my systems likewise, but I wonder if anyone has any
> other opinions on this.
> 
> --
> Andrius

What kind of write speed to S3 can you expect?
Can your backups to holding disk exceed that by a sufficient
amount that holding disk capacity might be a problem?

Jon
> 
> 
> On Sat, Jan 11, 2014 at 10:24 AM, Charles Curley <
> charlescur...@charlescurley.com> wrote:
> 
> > On Sat, 11 Jan 2014 09:44:07 -0800
> > "Andrius D. Ilgunas"  wrote:
> >
> > > We're setting up amanda on our servers, and the primary backup is
> > > going to be on a dedicated disk/virtual tapes.  One of the offsite
> > > locations is going to be a bucket on Amazon's S3.
> > >
> > > Now I see that amanda has the capability of writing to multiple
> > > volumes in parallel, but I'm wondering if it would be more efficient
> > > to have amada write the backup to the localdisk first and then use
> > > rsync to mirror that backup onto the S3 bucket.
> > >
> > > Or would it be a better idea to have amanda manage both
> > > "tapechangers"?
> > >
> > > Any discussion would be most welcome!
> >
> > What I do is similar to your first proposal. I back up to virtual
> > tapes, then rsync to external hard drive for off site backup.
> >
> >
> > http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html
> >
> > You probably want to save the metadata so that restores from offsite
> > will be easier.
> >
> >
> > http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html
> >
> > --
> >
> > The right of the people to be secure in their persons, houses, papers,
> > and effects, against unreasonable searches and seizures, shall not be
> > violated, and no Warrants shall issue, but upon probable cause,
> > supported by Oath or affirmation, and particularly describing the
> > place to be searched, and the persons or things to be seized.
> > -- U.S. Const. Amendment IV
> >
> > Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
> >
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (609) 477-8330 (C)

Re: Backup Strategy

[Andrius D. Ilgunas]

Thanks a brazillion Charles!!

I'll probably setup my systems likewise, but I wonder if anyone has any
other opinions on this.

--
Andrius


On Sat, Jan 11, 2014 at 10:24 AM, Charles Curley <
charlescur...@charlescurley.com> wrote:

> On Sat, 11 Jan 2014 09:44:07 -0800
> "Andrius D. Ilgunas"  wrote:
>
> > We're setting up amanda on our servers, and the primary backup is
> > going to be on a dedicated disk/virtual tapes.  One of the offsite
> > locations is going to be a bucket on Amazon's S3.
> >
> > Now I see that amanda has the capability of writing to multiple
> > volumes in parallel, but I'm wondering if it would be more efficient
> > to have amada write the backup to the localdisk first and then use
> > rsync to mirror that backup onto the S3 bucket.
> >
> > Or would it be a better idea to have amanda manage both
> > "tapechangers"?
> >
> > Any discussion would be most welcome!
>
> What I do is similar to your first proposal. I back up to virtual
> tapes, then rsync to external hard drive for off site backup.
>
>
> http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html
>
> You probably want to save the metadata so that restores from offsite
> will be easier.
>
>
> http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html
>
> --
>
> The right of the people to be secure in their persons, houses, papers,
> and effects, against unreasonable searches and seizures, shall not be
> violated, and no Warrants shall issue, but upon probable cause,
> supported by Oath or affirmation, and particularly describing the
> place to be searched, and the persons or things to be seized.
> -- U.S. Const. Amendment IV
>
> Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
>

Re: Backup Strategy

[Charles Curley]

On Sat, 11 Jan 2014 09:44:07 -0800
"Andrius D. Ilgunas"  wrote:

> We're setting up amanda on our servers, and the primary backup is
> going to be on a dedicated disk/virtual tapes.  One of the offsite
> locations is going to be a bucket on Amazon's S3.
> 
> Now I see that amanda has the capability of writing to multiple
> volumes in parallel, but I'm wondering if it would be more efficient
> to have amada write the backup to the localdisk first and then use
> rsync to mirror that backup onto the S3 bucket.
> 
> Or would it be a better idea to have amanda manage both
> "tapechangers"?
> 
> Any discussion would be most welcome!

What I do is similar to your first proposal. I back up to virtual
tapes, then rsync to external hard drive for off site backup.

http://charlescurley.com/blog/articles/off_site_backups_for_amanda/index.html

You probably want to save the metadata so that restores from offsite
will be easier.

http://charlescurley.com/blog/articles/backing_up_amanda_metadata/index.html

-- 

The right of the people to be secure in their persons, houses, papers,
and effects, against unreasonable searches and seizures, shall not be
violated, and no Warrants shall issue, but upon probable cause,
supported by Oath or affirmation, and particularly describing the
place to be searched, and the persons or things to be seized.
-- U.S. Const. Amendment IV

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB

Backup Strategy

[Andrius D. Ilgunas]

Hey All!

We're setting up amanda on our servers, and the primary backup is going to
be on a dedicated disk/virtual tapes.  One of the offsite locations is
going to be a bucket on Amazon's S3.

Now I see that amanda has the capability of writing to multiple volumes in
parallel, but I'm wondering if it would be more efficient to have amada
write the backup to the localdisk first and then use rsync to mirror that
backup onto the S3 bucket.

Or would it be a better idea to have amanda manage both "tapechangers"?

Any discussion would be most welcome!


--
Andrius

Re: Does 'force bump' roughly equate to 'strategy incronly'?

[Jean-Louis Martineau]

There is a planner bug with force-bump when a dle is overdue.
The attached patch should fix it.

As a workaround, you can set a large dumpcycle (eg. 1).

force-bump will prevent full, but it will also bump the level, this 
might not be what you want.


Setting a large dumpcycle and a small maxpromoteday should do 
incremental without bumping the level.


Jean-Louis

On 12/22/2011 07:20 AM, Bryan Hodgson wrote:

On Wed, Dec 21, 2011 at 03:42:32PM -0500, Bryan Hodgson wrote:

On Wed, Dec 21, 2011 at 01:11:00PM -0500, Jon LaBadie wrote:

On Wed, Dec 21, 2011 at 10:01:36AM -0500, Bryan Hodgson wrote:

One day next week I want to prevent any level 0 dumps, and run only
incrementals for that one night.

It's not obvious to me from the docs that 'amadmin force bump' will
actually prevent amanda from concluding that it's time for a level
0.  Would 'force bump' serve my purpose?  (My guess = no.)

There is more than one runcycle in our tape cycle, and we won't be
over-writing the most recent level 0 for any dump.


I looked at the man page and I agree the question is not addressed:
"if a level 0 were due, would force-bump cause an incremental instead?"

Jon

Okay, so it's not just me.  Thanks.

124 file systems, 2 are due today, all others are due for level 0 in
4 to 6 days.  And, of course, amanda usually moves some forward
early.

Okay, so I switched dumpcycle to 3 and all DLEs to force-bump for
tonight's run.  amadmin due now thinks there are some level 0's due
today.  We'll see what happens.

Bryan


It didn't work.  No obvious reason.  planner results look okay, but
amdump failed as soon as planner finished with all DLEs showing
'RESULTS MISSING' via amreport.  Ran unforce-bump on DLEs, reset
dumpcycle to its regular value, and amdump runs successfully.

Not a problem, will steer a different course.

Bryan


diff --git a/server-src/planner.c b/server-src/planner.c
index 4e70e24..5b61c97 100644
--- a/server-src/planner.c
+++ b/server-src/planner.c
@@ -3051,7 +3051,8 @@ static int promote_hills(void)
 }
 
 for(dp = schedq.head; dp != NULL; dp = dp->next) {
-   days = est(dp)->next_level0;   /* This is > 0 by definition */
+   days = est(dp)->next_level0;
+   if (days < 0) days = 0;
if(daysskip_full && dp->strategy != DS_NOFULL &&
   dp->strategy != DS_INCRONLY) {
sp[days].disks++;

Re: Does 'force bump' roughly equate to 'strategy incronly'?

[Bryan Hodgson]

On Wed, Dec 21, 2011 at 03:42:32PM -0500, Bryan Hodgson wrote:
> On Wed, Dec 21, 2011 at 01:11:00PM -0500, Jon LaBadie wrote:
> > On Wed, Dec 21, 2011 at 10:01:36AM -0500, Bryan Hodgson wrote:
> > > 
> > > One day next week I want to prevent any level 0 dumps, and run only
> > > incrementals for that one night.
> > > 
> > > It's not obvious to me from the docs that 'amadmin force bump' will
> > > actually prevent amanda from concluding that it's time for a level
> > > 0.  Would 'force bump' serve my purpose?  (My guess = no.)
> > > 
> > > There is more than one runcycle in our tape cycle, and we won't be
> > > over-writing the most recent level 0 for any dump.
> > > 
> > I looked at the man page and I agree the question is not addressed:
> > "if a level 0 were due, would force-bump cause an incremental instead?"
> > 
> > Jon
> 
> Okay, so it's not just me.  Thanks.
> 
> 124 file systems, 2 are due today, all others are due for level 0 in
> 4 to 6 days.  And, of course, amanda usually moves some forward
> early.
> 
> Okay, so I switched dumpcycle to 3 and all DLEs to force-bump for
> tonight's run.  amadmin due now thinks there are some level 0's due
> today.  We'll see what happens.
> 
> Bryan
> 

It didn't work.  No obvious reason.  planner results look okay, but
amdump failed as soon as planner finished with all DLEs showing
'RESULTS MISSING' via amreport.  Ran unforce-bump on DLEs, reset
dumpcycle to its regular value, and amdump runs successfully.

Not a problem, will steer a different course.

Bryan

Re: Does 'force bump' roughly equate to 'strategy incronly'?

[Bryan Hodgson]

On Wed, Dec 21, 2011 at 01:11:00PM -0500, Jon LaBadie wrote:
> On Wed, Dec 21, 2011 at 10:01:36AM -0500, Bryan Hodgson wrote:
> > 
> > One day next week I want to prevent any level 0 dumps, and run only
> > incrementals for that one night.
> > 
> > It's not obvious to me from the docs that 'amadmin force bump' will
> > actually prevent amanda from concluding that it's time for a level
> > 0.  Would 'force bump' serve my purpose?  (My guess = no.)
> > 
> > There is more than one runcycle in our tape cycle, and we won't be
> > over-writing the most recent level 0 for any dump.
> > 
> I looked at the man page and I agree the question is not addressed:
> "if a level 0 were due, would force-bump cause an incremental instead?"
> 
> Jon

Okay, so it's not just me.  Thanks.

124 file systems, 2 are due today, all others are due for level 0 in
4 to 6 days.  And, of course, amanda usually moves some forward
early.

Okay, so I switched dumpcycle to 3 and all DLEs to force-bump for
tonight's run.  amadmin due now thinks there are some level 0's due
today.  We'll see what happens.

Bryan

Re: Does 'force bump' roughly equate to 'strategy incronly'?

[Jon LaBadie]

On Wed, Dec 21, 2011 at 10:01:36AM -0500, Bryan Hodgson wrote:
> 
> One day next week I want to prevent any level 0 dumps, and run only
> incrementals for that one night.
> 
> It's not obvious to me from the docs that 'amadmin force bump' will
> actually prevent amanda from concluding that it's time for a level
> 0.  Would 'force bump' serve my purpose?  (My guess = no.)
> 
> There is more than one runcycle in our tape cycle, and we won't be
> over-writing the most recent level 0 for any dump.
> 
I looked at the man page and I agree the question is not addressed:
"if a level 0 were due, would force-bump cause an incremental instead?"

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (609) 477-8330 (C)

Does 'force bump' roughly equate to 'strategy incronly'?

[Bryan Hodgson]

One day next week I want to prevent any level 0 dumps, and run only
incrementals for that one night.

It's not obvious to me from the docs that 'amadmin force bump' will
actually prevent amanda from concluding that it's time for a level
0.  Would 'force bump' serve my purpose?  (My guess = no.)

There is more than one runcycle in our tape cycle, and we won't be
over-writing the most recent level 0 for any dump.

I recognize that changing strategy to 'incronly' for all dumptypes in
amanda.conf should have the desired effect, but would prefer to do
this through amadmin.

TIA.

Bryan

Re: backup strategy

[Chris Hoogendyk]

Stefan Zörner wrote:

Hi,

we are about to build up a backup system onto a network drive. We do 
not need long term full backup, we just want to be able to access data 
maximal a month old.
We thought of doing one full backup and then just doing cumulative 
backups. We are planning on using it over a longer period and since 
after a year the data can differ a lot from the original full backup 
each cumulative backup can get quite large. A possibility of course is 
to do a full backup now and then.


My question is, that instead of working with cumulative backup I want 
to use incremental ones. Now normally at the end of a cycle the oldest 
incremental backup is overwritten and the whole backup becomes 
worthless. Is there a method were the oldest increments is included 
into the full backup itself after a cycle has been reached?

Example:
- one full backup
- 10 increments
before starting increment 11. Renew full backup by including oldest 
incremental backup.


Thanks in advance for any help or pointers. 


You should take a look at this:

  
http://wiki.zmanda.com/index.php/FAQ:How_are_backup_levels_defined_and_how_does_Amanda_use_them%3F


before launching off into backup strategy ideas based on prior knowledge 
from other kinds of backup software. After you have read through that a 
couple of times, you might have different questions.



--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Rory Campbell-Lange]

On 17/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
> Cyrille Bollu wrote:
> >Chris Hoogendyk wrote:
> >> I wouldn't put the holding disks in raid.

> http://blogs.zdnet.com/Ou/?p=484.
> 
> While this guy is looking at things like database servers and
> exchange, we ought to be able to interpret this for Amanda. A couple
> of points to note for Amanda: Amanda will use all the holding disk
> drives while doing parallel backups and storing output on the
> holding disks. When it is writing to tape, it is constrained by the
> sequential nature of the tape, and will only be doing one DLE at a
> time from those that it has completed on the holding disks. Also,
> Amanda's access is heavily sequential, although it may have multiple
> parallel processes hitting the drives.

A slow RAID1 off two 7200 RPM SATA disks on a BBU-backed LSI hardware
raid controller can do about 62031 KiB/s write and 86399 KiB/s read.
Those sorts of numbers improve steadily the number of spindles you add
to a RAID collection and the higher the RAID number and (in the case of
writing) if cacheing is enabled.

On 16/08/09, Rory Campbell-Lange (r...@campbell-lange.net) wrote:
> On 14/08/09, Frank Smith (fsm...@hoovers.com) wrote:
> > Chris Hoogendyk wrote:
> With reference to Chris Hoogendyk's email "clarification on
> parallelism", I am very curious to learn if Amanda "...still require[s]
> a DLE to be completed to holding disk before it will send any of it to
> tape..." In our case this is a particularly important question as,
> although we can add in more AoE storage for a DLE, this will only run at
> the speeds above. Do we need a 1TB SAS disk array too?

>From the discussion here it seems preferable to have a DLE on two major
counts. One is that compression can happen prior to writing to tape,
which could result in shoe-shining, and another is that Amanda will be
clearer about the amount of data it will be trying to write to a tape,
in other words it will do a better fit of data to tape.

The most important question I now have to ask is:

How fast can a SAS-based LTO4 drive write to tape?

Regards
Rory


-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Chris Hoogendyk]

Cyrille Bollu wrote:

Chris Hoogendyk wrote:
> I wouldn't put the holding disks in raid.

Hu hu... Interesting... I have a 4 disks RAID-0 holding disk, and it 
isn't fast... I always wondered if I should use seperated (non-RAID) 
drives...


Here is an extremely interesting article that everyone should take a 
look at. Rather than just giving theoretical comparisons that you see in 
places like the Wikipedia raid article (which is nevertheless an 
excellent reference), this article is a case study analysis with both 
real and test environments throwing data at raid and instrumenting it -- 
http://blogs.zdnet.com/Ou/?p=484.


While this guy is looking at things like database servers and exchange, 
we ought to be able to interpret this for Amanda. A couple of points to 
note for Amanda: Amanda will use all the holding disk drives while doing 
parallel backups and storing output on the holding disks. When it is 
writing to tape, it is constrained by the sequential nature of the tape, 
and will only be doing one DLE at a time from those that it has 
completed on the holding disks. Also, Amanda's access is heavily 
sequential, although it may have multiple parallel processes hitting the 
drives.



--
---

Chris Hoogendyk

-
O__  Systems Administrator
c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst



---

Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Alan Hodgson]

> On Monday 17 August 2009, Cyrille Bollu wrote:
> >> I wouldn't put the holding disks in raid.
> >
> >Hu hu... Interesting... I have a 4 disks RAID-0 holding disk, and it
> > isn't fast... I always wondered if I should use seperated (non-RAID)
> > drives...

To drive an LTO-4 your holding disk needs to read somewhat over 100MB/sec 
sequentially, which requires at least 2 drives striped, but should be easy 
enough with any modern raid controller or software raid. 

Complicating this, it may also need to write at similar speed, 
simultaneously, which introduces a random access element and ups the demand 
considerably. I would think you would probably want at least 4 big SATA 
drives striped together to reliably feed an LTO-4 drive at full speed. 
Conveniently, this could also give you over 5TB of very cheap holding disk 
space.

Also, unless you're backing up exclusively large files over a fast SAN link 
(faster than Gig-E), I doubt you could get anywhere close to full tape 
performance without a holding disk.


-- 
"... a serious depression seems improbable; [we expect] recovery of business
next spring, with further improvement in the fall." Harvard Economic
Society, November 10, 1929

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Gene Heskett]

On Monday 17 August 2009, Cyrille Bollu wrote:
>> I wouldn't put the holding disks in raid.
>
>Hu hu... Interesting... I have a 4 disks RAID-0 holding disk, and it isn't
>fast... I always wondered if I should use seperated (non-RAID) drives...

Its been my observation that software raids are slower. Not tremendously so 
though.  When Jim put together a raid-5 with 4 drives several years ago, the 
drives were about 70meg/sec drives, and the overall was just a hair over 
50meg/sec.  I know he has rebuilt it with bigger & faster drives 2 or 3 times 
since, along with more iron in the cpu, so I don't know its current speed.  
Being retired means being out of the loop. :(

>Cyrille Bollu
>Responsable systèmes
>Fedasil - ICT
>tel: +32.2.213.43.49
>gsm: +32.478.23.08.15


-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
The NRA is offering FREE Associate memberships to anyone who wants them.


A prig is a fellow who is always making you a present of his opinions.
-- George Eliot

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Cyrille Bollu]

> 
> I wouldn't put the holding disks in raid.

Hu hu... Interesting... I have a 4 disks RAID-0 holding disk, and it isn't 
fast... I always wondered if I should use seperated (non-RAID) drives... 

Cyrille Bollu
Responsable systèmes
Fedasil - ICT
tel: +32.2.213.43.49
gsm: +32.478.23.08.15

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Chris Hoogendyk]

Rory Campbell-Lange wrote:

On 14/08/09, Frank Smith (fsm...@hoovers.com) wrote:
  

Chris Hoogendyk wrote:

Amanda will do the compression for you. You define it in the dumptype in 
amanda.conf. If you have a holding disk, then it will compress the data 
as it goes onto the holding disk. If you don't have a holding disk, then 
you might have issues with being able to stream a backup to tape, 
compressing it on the fly. Even with a really fast cpu, I don't know if 
you can maintain the throughput to drive LTO4 at a good speed.
  

You might want to consider configuring for client compression.  Not
only will that give you more CPU for feeding your tape, it also
minimizes network bandwidth. As usual, YMMV, it all depends on where
the bottlenecks are in your environment.



In our case the server _is_ the only client, with up to 30TB of direct
attached storage, with the storage running at between 80MB/s and 120MB/s
access speeds (Bytes rather than bytes).

I don't know if this is fast enough to deal with a SAS connected LTO4
drive, particularly if it is doing software compression along the way.

With reference to Chris Hoogendyk's email "clarification on
parallelism", I am very curious to learn if Amanda "...still require[s]
a DLE to be completed to holding disk before it will send any of it to
tape..." In our case this is a particularly important question as,
although we can add in more AoE storage for a DLE, this will only run at
the speeds above. Do we need a 1TB SAS disk array too?


You will get the best performance if you can do that. If the disk that 
is being copied to tape can give the speed the tape needs, that's going 
to do a better job of keeping things moving.


You have a couple of options.

You can go without a holding disk, and then each DLE will be streamed 
sequentially to tape. This will stretch out your backups. It will also 
mean that any compression you do in software will be done in line with 
that sequential stream. Your system may not be able to keep that all 
flying fast enough for the tape, and you may end up with shoe shining 
and very low speeds. You can certainly try it and see what happens. If 
(when?) that fails, you could try using hardware compression on the tape 
drive. The backups will still be sequential, one DLE streaming to tape 
at a time, and if your drives can't keep up, it will be slower than you 
might like. But, at least you are not dealing with network backups.


The option I would try, budget allowing, would be to add a couple of SAS 
drives to be used as holding disks. Then break up your DLEs so that each 
DLE is substantially smaller than the holding disks. Then Amanda can run 
them in parallel, compress them on the holding disk, and then stream 
completed, compressed DLEs from the holding disk to the tape.


I wouldn't put the holding disks in raid.


--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Rory Campbell-Lange]

On 14/08/09, Frank Smith (fsm...@hoovers.com) wrote:
> Chris Hoogendyk wrote:
> > Amanda will do the compression for you. You define it in the dumptype in 
> > amanda.conf. If you have a holding disk, then it will compress the data 
> > as it goes onto the holding disk. If you don't have a holding disk, then 
> > you might have issues with being able to stream a backup to tape, 
> > compressing it on the fly. Even with a really fast cpu, I don't know if 
> > you can maintain the throughput to drive LTO4 at a good speed.
> 
> You might want to consider configuring for client compression.  Not
> only will that give you more CPU for feeding your tape, it also
> minimizes network bandwidth. As usual, YMMV, it all depends on where
> the bottlenecks are in your environment.

In our case the server _is_ the only client, with up to 30TB of direct
attached storage, with the storage running at between 80MB/s and 120MB/s
access speeds (Bytes rather than bytes).

I don't know if this is fast enough to deal with a SAS connected LTO4
drive, particularly if it is doing software compression along the way.

With reference to Chris Hoogendyk's email "clarification on
parallelism", I am very curious to learn if Amanda "...still require[s]
a DLE to be completed to holding disk before it will send any of it to
tape..." In our case this is a particularly important question as,
although we can add in more AoE storage for a DLE, this will only run at
the speeds above. Do we need a 1TB SAS disk array too?

Rory

-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Frank Smith]

Chris Hoogendyk wrote:
> 
> Rory Campbell-Lange wrote:
>> Hi Chris
>>
>> On 13/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
>>   
>> 
>>> Typically, we set up Amanda with holding disk space.
>>> 
>> 
>>
>> If all the storage is locally attached (actually, AoE drives storage
>> units connected over Ethernet), I am hoping to avoid the disk space if I
>> can write to tape fast enough. I'd like to avoid paying for up to 15TB
>> of fast holding disk space if I can avoid it.
> 
> So, one way would be to logically divide the storage into smaller DLE's. 
> A DLE (Disk List Entry -- http://wiki.zmanda.com/man/disklist.5.html) 
> for Amanda can be a mount point or directory. Obviously, I don't know 
> how your storage is organized; but, if you can define your DLE's as 
> separate directories on the storage device, each one of which is much 
> smaller, then you could use a smaller holding disk and still benefit 
> from Amanda's parallelism. In one of the other departments here, the 
> sysadmin has successfully divided a large array this way and is driving 
> LTO4 near top speed.
> 
>>> Compression can be done either on the client, on the server, or on
>>> the tape drive. Obviously, if you use software compression, you want
>>> to turn off the tape drive compression. I use server side
>>> compression, because I have a dedicated Amanda server that can
>>> handle it. By not using the tape drive compression, Amanda has more
>>> complete information on data size and tape usage for its planning.
>>> If your server is more constrained than your clients, you could use
>>> client compression. This is specified in your dumptypes in your
>>> amanda.conf.
>>> 
>> I don't have any clients, so this is an interesting observation. I'll be
>> trying to do sofware compression then I think. The Unix backup book
>> (google for "amanda software compression") suggests that compression can
>> be used on a "per-image basis"; presumably I can pass the backup data
>> stream through gzip or bzip2 on the way to a tape?
> 
> Amanda will do the compression for you. You define it in the dumptype in 
> amanda.conf. If you have a holding disk, then it will compress the data 
> as it goes onto the holding disk. If you don't have a holding disk, then 
> you might have issues with being able to stream a backup to tape, 
> compressing it on the fly. Even with a really fast cpu, I don't know if 
> you can maintain the throughput to drive LTO4 at a good speed.

You might want to consider configuring for client compression.  Not
only will that give you more CPU for feeding your tape, it also
minimizes network bandwidth. As usual, YMMV, it all depends on where
the bottlenecks are in your environment.

Frank


-- 
Frank Smith  fsm...@hoovers.com
Sr. Systems Administrator   Voice: 512-374-4673
Hoover's Online   Fax: 512-374-4501

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Chris Hoogendyk]

Rory Campbell-Lange wrote:

Hi Chris

On 13/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
  


Typically, we set up Amanda with holding disk space.




If all the storage is locally attached (actually, AoE drives storage
units connected over Ethernet), I am hoping to avoid the disk space if I
can write to tape fast enough. I'd like to avoid paying for up to 15TB
of fast holding disk space if I can avoid it.


So, one way would be to logically divide the storage into smaller DLE's. 
A DLE (Disk List Entry -- http://wiki.zmanda.com/man/disklist.5.html) 
for Amanda can be a mount point or directory. Obviously, I don't know 
how your storage is organized; but, if you can define your DLE's as 
separate directories on the storage device, each one of which is much 
smaller, then you could use a smaller holding disk and still benefit 
from Amanda's parallelism. In one of the other departments here, the 
sysadmin has successfully divided a large array this way and is driving 
LTO4 near top speed.



Compression can be done either on the client, on the server, or on
the tape drive. Obviously, if you use software compression, you want
to turn off the tape drive compression. I use server side
compression, because I have a dedicated Amanda server that can
handle it. By not using the tape drive compression, Amanda has more
complete information on data size and tape usage for its planning.
If your server is more constrained than your clients, you could use
client compression. This is specified in your dumptypes in your
amanda.conf.



I don't have any clients, so this is an interesting observation. I'll be
trying to do sofware compression then I think. The Unix backup book
(google for "amanda software compression") suggests that compression can
be used on a "per-image basis"; presumably I can pass the backup data
stream through gzip or bzip2 on the way to a tape?


Amanda will do the compression for you. You define it in the dumptype in 
amanda.conf. If you have a holding disk, then it will compress the data 
as it goes onto the holding disk. If you don't have a holding disk, then 
you might have issues with being able to stream a backup to tape, 
compressing it on the fly. Even with a really fast cpu, I don't know if 
you can maintain the throughput to drive LTO4 at a good speed.



--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Chris Hoogendyk]

Cyrille Bollu wrote:

Here's my (very) small personnal experience:

A few years ago, when I tried it, I couldn't enable server-side 
software compression while bypassing the holding disk with my IBM 
ULTIUM LTO-3 drive: Tape speed was sinking to about 5MB/s.


My backup server was a Dell PowerEdge 2850 with 4 Intel Xeon 3GHz and 
8MB RAM using RHEL-4.0 and amanda-2.4.4p3-1.


Maybe did I do something wrong at that time (I just had 1 try). Beware 
though. 


That actually makes perfect sense.

By not using a holding disk, you are disabling Amanda's ability to run 
multiple things in parallel. The tape device now controls everything. 
That is to say, you cannot do a backup without streaming it to the tape. 
So, you cannot do more than one at a time. Furthermore, as that one 
backup gets done, the compression has to be done as it is being streamed 
to the tape. So all the processes from reading a remote disk, to 
transferring it over the network, to compressing it, to writing it to 
the tape are all tied together in a single pipe. Any slowdown along that 
pipe will affect everything else. When the tape doesn't get what it 
needs to keep going, it will stop and then have to start up again and 
reposition, and then you get shoe shining.


When you use a holding disk, Amanda can stream multiple backups to the 
holding disk simultaneously. It can compress them there when it has them 
and do it in parallel with other processes. Once it has something ready 
to go to tape, it can dd it straight from the disk to the tape as an 
independent process in parallel with the other things that are going on. 
That final step out to the tape is no longer constrained by any of the 
other steps along the way. Now all you have to worry about is tuning 
various pieces of hardware and software to get the throughput you want.



--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Cyrille Bollu]

Hi,

Here's my (very) small personnal experience:

A few years ago, when I tried it, I couldn't enable server-side software 
compression while bypassing the holding disk with my IBM ULTIUM LTO-3 
drive: Tape speed was sinking to about 5MB/s.

My backup server was a Dell PowerEdge 2850 with 4 Intel Xeon 3GHz and 8MB 
RAM using RHEL-4.0 and amanda-2.4.4p3-1.

Maybe did I do something wrong at that time (I just had 1 try). Beware 
though.

Cyrille

owner-amanda-us...@amanda.org wrote on 14/08/2009 15:57:45:

> Hi Chris
> 
> On 13/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
> > ... the solution is akin to the Japanese monks caring for Bonzai
> 
> I liked this idea about tape archives -- constant pruning and
> maintenance. Difficult to sell though.
> 
> > As for your specific questions:
> > 
> > You should be able to do LVM snapshots. I use fssnap on Solaris 9 and
> > 10, and scanning through, here are just a couple of references I find
> > to people using LVM snapshots with Amanda:
> 
> > With the latest releases of Amanda, there is a new API that could make
> > it even easier to implement.
> 
> Great; thanks for the pointers.
> 
> > Typically, we set up Amanda with holding disk space.
> 
> 
> If all the storage is locally attached (actually, AoE drives storage
> units connected over Ethernet), I am hoping to avoid the disk space if I
> can write to tape fast enough. I'd like to avoid paying for up to 15TB
> of fast holding disk space if I can avoid it.
> 
> > Compression can be done either on the client, on the server, or on
> > the tape drive. Obviously, if you use software compression, you want
> > to turn off the tape drive compression. I use server side
> > compression, because I have a dedicated Amanda server that can
> > handle it. By not using the tape drive compression, Amanda has more
> > complete information on data size and tape usage for its planning.
> > If your server is more constrained than your clients, you could use
> > client compression. This is specified in your dumptypes in your
> > amanda.conf.
> 
> I don't have any clients, so this is an interesting observation. I'll be
> trying to do sofware compression then I think. The Unix backup book
> (google for "amanda software compression") suggests that compression can
> be used on a "per-image basis"; presumably I can pass the backup data
> stream through gzip or bzip2 on the way to a tape?
> 
> > Deduplication is not available with Amanda. However, some people
> > stage different kinds of tools and use Amanda for the final staging
> > and management of tapes and archives. So, in some situations,
> > BackupPC could be used to do deduplication from, say, desktop
> > clients to a server archive which is then backed up by Amanda. That
> > could start complicating your 12 year recovery scenario and what
> > happens when software is not available or doesn't run.
> 
> Great -- thanks for the details.
> 
> > Amanda uses the term "index" rather than "catalog" -- see
> > http://wiki.zmanda.com/index.php/Amanda_Index.
> > 
> > Note that if you are putting tapes into a long term archive with no
> > intent of recycling them in subsequent backups, you can use amadmin
> > to mark them as no-reuse. I periodically (typically at the end of
> > semesters) do a force full, mark the tapes as no-reuse, and then
> > pull them out of my tapecycle and put them in storage.
> 
> Very useful again, thanks.
> 
> Regards
> Rory
> -- 
> Rory Campbell-Lange
> Director
> r...@campbell-lange.net
> 
> Campbell-Lange Workshop
> www.campbell-lange.net
> 0207 6311 555
> 3 Tottenham Street London W1T 2AF
> Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Rory Campbell-Lange]

Hi Chris

On 13/08/09, Chris Hoogendyk (hoogen...@bio.umass.edu) wrote:
> ... the solution is akin to the Japanese monks caring for Bonzai

I liked this idea about tape archives -- constant pruning and
maintenance. Difficult to sell though.

> As for your specific questions:
> 
> You should be able to do LVM snapshots. I use fssnap on Solaris 9 and
> 10, and scanning through, here are just a couple of references I find
> to people using LVM snapshots with Amanda:

> With the latest releases of Amanda, there is a new API that could make
> it even easier to implement.

Great; thanks for the pointers.

> Typically, we set up Amanda with holding disk space.


If all the storage is locally attached (actually, AoE drives storage
units connected over Ethernet), I am hoping to avoid the disk space if I
can write to tape fast enough. I'd like to avoid paying for up to 15TB
of fast holding disk space if I can avoid it.

> Compression can be done either on the client, on the server, or on
> the tape drive. Obviously, if you use software compression, you want
> to turn off the tape drive compression. I use server side
> compression, because I have a dedicated Amanda server that can
> handle it. By not using the tape drive compression, Amanda has more
> complete information on data size and tape usage for its planning.
> If your server is more constrained than your clients, you could use
> client compression. This is specified in your dumptypes in your
> amanda.conf.

I don't have any clients, so this is an interesting observation. I'll be
trying to do sofware compression then I think. The Unix backup book
(google for "amanda software compression") suggests that compression can
be used on a "per-image basis"; presumably I can pass the backup data
stream through gzip or bzip2 on the way to a tape?

> Deduplication is not available with Amanda. However, some people
> stage different kinds of tools and use Amanda for the final staging
> and management of tapes and archives. So, in some situations,
> BackupPC could be used to do deduplication from, say, desktop
> clients to a server archive which is then backed up by Amanda. That
> could start complicating your 12 year recovery scenario and what
> happens when software is not available or doesn't run.

Great -- thanks for the details.

> Amanda uses the term "index" rather than "catalog" -- see
> http://wiki.zmanda.com/index.php/Amanda_Index.
> 
> Note that if you are putting tapes into a long term archive with no
> intent of recycling them in subsequent backups, you can use amadmin
> to mark them as no-reuse. I periodically (typically at the end of
> semesters) do a force full, mark the tapes as no-reuse, and then
> pull them out of my tapecycle and put them in storage.

Very useful again, thanks.

Regards
Rory
-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Chris Hoogendyk]

Rory Campbell-Lange wrote:

On 13/08/09, Charles Curley (charlescur...@charlescurley.com) wrote:
  

On Thu, 13 Aug 2009 01:08:03 -0400
Jon LaBadie  wrote:


On Wed, Aug 12, 2009 at 06:17:17PM -0400, rorycl wrote:
  
So maybe you should provide a complete OS distribution, including the

backup software.  Like a customized version of one of the "live CD"
releases of Linux.  But wait, will that distribution's included
device drivers work on the devices that will exist in 12 years?  Will
that era's computers still have CD drives.  Will they be bootable?
  

Oh, folks 12 years hence ought to be able to dig out 12 year old
computers to run their 12 year old distributions on. 



Many thanks for this note, Charles, and to the other notes Chris,
Charles and Jon about their commentary about using Amanda to provide a
long-term archive format. The points about being able to use standard
Unix tools to retrieve information is well made, as is the point that
the current machines and architectures (and CDs!) may not be around in
12 years' time. Thanks very much for those observations. 


I'd like to return the other part of my question if I may:

  

The backup tape format is to be LT04 and we have a second-hand Dell
PowerVault 124T 16 tape autoloader to work with currently. Backup from
a pool may be taken off a Linux LVM (or hopefully soon a BTRFS)
snapshot ensuring that the source data does not change during the
backup process. We have the possibility of pre-preparing backup or
compressed images if this is advisable.



I'd be grateful to learn specifically if the approach I have set out
seems feasible. Also:
 
- is the snapshot volume or secondary holding pool advisable?
- is compression / deduplication possible?   
- after scanning through the wiki I can't see any references to what
  I think of as a backup job "catalogue". How does one know what
  files were part of a particular backup job?   


Thanks for any further advice.


One further comment on the nature of long term archives (and then on to 
your specific questions):


I used to work in the Systems Office of the University Library. I 
handled backups there, and had close contact with a group of librarians 
who were into digital content, archives and special collections. Among 
other things, we kicked around ideas about how to archive digital 
collections, the life expectancies and failure rates of various types of 
CDs (generally terrible in reality), etc. When librarians talk about 
archives, they don't just talk decades. They expect things to last a 
hundred years and more. In that light, they have concluded that the 
solution is akin to the Japanese monks caring for Bonzai. There are 
records of Bonzai trees that have been cared for for hundreds of years. 
So, think of sysadmins as monks caring for data. The archive librarians 
solution is raid6 with hot spares and mirrored to another location. The 
sysadmins maintain and update hardware and software and transfer data 
when necessary. Although I haven't kept up with that area, they were 
developing a cooperative distributed archive software as an open source. 
The idea was that different libraries join the cooperative, run the 
software, and they end up with multiple copies of their digital 
collections distributed geographically among other libraries. If your 
library burns down, you rebuild, set up the software, and bring your 
collection back. Sort of a cloud library, if you will.


So, as technology changes, you need to be frequently reviewing the state 
of your archives, keeping an eye on compatibility bottlenecks and 
transferring data to newer media when it becomes necessary. I have a 
faculty member who ran his own backups on AIT2 for years. His drives are 
fairly old now. I periodically urge him to read them back in to a disk 
archive and allow me to put them on AIT5. He's too busy. Ah, well. It's 
his data.


--

As for your specific questions:

You should be able to do LVM snapshots. I use fssnap on Solaris 9 and 
10, and scanning through, here are just a couple of references I find to 
people using LVM snapshots with Amanda:

http://wiki.zmanda.com/index.php/FAQ:Which_backup_program_for_filesystems_is_better%3F
http://archives.zmanda.com/amanda-archives/viewtopic.php?t=2711&sid=f1535cf0b0782bf2b99aebc033e91c9c
http://archives.zmanda.com/amanda-archives/viewtopic.php?p=9823&sid=8e54f6a0b4ab2cd58bd02e048c299844

In the past that sort of thing had always been done with a wrapper 
script (described toward the end of 
http://wiki.zmanda.com/index.php/Backup_client). Paul Bijens refers to a 
script that he uses in one of the above links. With the latest releases 
of Amanda, there is a new API that could make it even easier to implement.


Typically, we set up Amanda with holding disk space.
See the section of the sample amanda.conf partway down regarding holding 
disks -- http:

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Rory Campbell-Lange]

On 13/08/09, Charles Curley (charlescur...@charlescurley.com) wrote:
> On Thu, 13 Aug 2009 01:08:03 -0400
> Jon LaBadie  wrote:
> > On Wed, Aug 12, 2009 at 06:17:17PM -0400, rorycl wrote:

> > So maybe you should provide a complete OS distribution, including the
> > backup software.  Like a customized version of one of the "live CD"
> > releases of Linux.  But wait, will that distribution's included
> > device drivers work on the devices that will exist in 12 years?  Will
> > that era's computers still have CD drives.  Will they be bootable?
> 
> Oh, folks 12 years hence ought to be able to dig out 12 year old
> computers to run their 12 year old distributions on. 

Many thanks for this note, Charles, and to the other notes Chris,
Charles and Jon about their commentary about using Amanda to provide a
long-term archive format. The points about being able to use standard
Unix tools to retrieve information is well made, as is the point that
the current machines and architectures (and CDs!) may not be around in
12 years' time. Thanks very much for those observations. 

I'd like to return the other part of my question if I may:

> The backup tape format is to be LT04 and we have a second-hand Dell
> PowerVault 124T 16 tape autoloader to work with currently. Backup from
> a pool may be taken off a Linux LVM (or hopefully soon a BTRFS)
> snapshot ensuring that the source data does not change during the
> backup process. We have the possibility of pre-preparing backup or
> compressed images if this is advisable.

I'd be grateful to learn specifically if the approach I have set out
seems feasible. Also:
 
- is the snapshot volume or secondary holding pool advisable?
- is compression / deduplication possible?   
- after scanning through the wiki I can't see any references to what
  I think of as a backup job "catalogue". How does one know what
  files were part of a particular backup job?   

Thanks for any further advice.

Rory

-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Charles Curley]

On Thu, 13 Aug 2009 01:08:03 -0400
Jon LaBadie  wrote:

> On Wed, Aug 12, 2009 at 06:17:17PM -0400, rorycl wrote:
> > 

> So maybe you should provide a complete OS distribution, including the
> backup software.  Like a customized version of one of the "live CD"
> releases of Linux.  But wait, will that distribution's included
> device drivers work on the devices that will exist in 12 years?  Will
> that era's computers still have CD drives.  Will they be bootable?

Oh, folks 12 years hence ought to be able to dig out 12 year old
computers to run their 12 year old distributions on. I suspect the
bottle neck will be finding drives to read LT04 tapes; maybe keep one or
two spares around?

Or put two or three complete machines aside and keep them for the
purpose.


-- 

Charles Curley  /"\ASCII Ribbon Campaign
Looking for fine software   \ /Respect for open standards
and/or writing?  X No HTML/RTF in email
http://www.charlescurley.com/ \No M$ Word docs in email

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB

Re: Post subject: [Amanda-users] Advice needed on Linux backup strategy to LTO

[Rory Campbell-Lange]

Apologies that this email doesn't have the correct threading ID. I
posted through Backup Central originally and I can't get hold of the
original mails on the Amanda users list since subscribing properly to it
-- the advertised ftp archives don't seem to exist any more.

Many thanks to Chris, Charles and Jon about their commentary about using
Amanda to provide a long-term archive format. The points about being
able to use standard Unix tools to retrieve information is well made, as
is the point that the current machines and architectures may not be
around in 12 years' time. Thanks very much for those points.

I'd like to return the other part of my question if I may:

> The backup tape format is to be LT04 and we have a second-hand Dell
> PowerVault 124T 16 tape autoloader to work with currently. Backup from
> a pool may be taken off a Linux LVM (or hopefully soon a BTRFS)
> snapshot ensuring that the source data does not change during the
> backup process. We have the possibility of pre-preparing backup or
> compressed images if this is advisable. 

I'd be grateful to learn specifically if the approach I have set out
seems feasible. Also:

- is the snapshot volume or secondary holding pool advisable?
- is compression / deduplication possible?
- after scanning through the wiki I can't see any references to what
  I think of as a backup job "catalogue". How does one know what
  files were part of a particular backup job?

Thanks for any further advice.

Rory


-- 
Rory Campbell-Lange
Director
r...@campbell-lange.net

Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Jon LaBadie]

On Wed, Aug 12, 2009 at 06:17:17PM -0400, rorycl wrote:
> 
> I'm going to cross-post this text on the Amanda and Bacula lists.
> Apologies in advance if you see this twice.
> 
> Our company is about to provide centralised backups for several pools of
> backup data of between 1 and 15TB in size. Each pool changes daily but
> backups to tape will only occur once a month for each pool.
> 
> The backup tape format is to be LT04 and we have a second-hand Dell
> PowerVault 124T 16 tape autoloader to work with currently. Backup from a
> pool may be taken off a Linux LVM (or hopefully soon a BTRFS) snapshot
> ensuring that the source data does not change during the backup process.
> We have the possibility of pre-preparing backup or compressed images if
> this is advisable.
> 
> An important aspect of the system is that the tapes should be readable
> for 12 years, by other parties if necessary. From this point of view we
> like the idea of providing a CD with each tape set of the software
> needed to extract the contents, together with a listing of the enclosed
> files in a UTF8 text file. We will be required to audit each backup set
> by successfully extracting files from tape.

Others have mentioned that even without amanda software, amanda backups
are recoverable with standard unix/linux tools.

I question whether the concept of providing the software is reasonable.
Programs are compiled for a particular environment, versions of libraries,
devices, operating system, etc.  Amanda software, compiled for today's
systems, is unlikely to be able to run on systems a dozen years from now.
What systems were around in 1997?  Many instances of them still running?  

So maybe you should provide a complete OS distribution, including the backup
software.  Like a customized version of one of the "live CD" releases of
Linux.  But wait, will that distribution's included device drivers work on
the devices that will exist in 12 years?  Will that era's computers still
have CD drives.  Will they be bootable?

This requirement may take some additional thought.

jl
-- 
Jon H. LaBadie  j...@jgcomp.com
 JG Computing
 12027 Creekbend Drive  (703) 787-0884
 Reston, VA  20194  (703) 787-0922 (fax)

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Charles Curley]

On Wed, 12 Aug 2009 18:17:17 -0400
rorycl  wrote:


> An important aspect of the system is that the tapes should be readable
> for 12 years, by other parties if necessary. From this point of view
> we like the idea of providing a CD with each tape set of the software
> needed to extract the contents, together with a listing of the
> enclosed files in a UTF8 text file. We will be required to audit each
> backup set by successfully extracting files from tape.

I assume you already have verified that your tapes will last that long
before print-through makes them unreadable.

Another thought is to provide a CD/DVD of a suitable distribution of
Linux with Amanda, mt, etc. That way you don't have compatibility
problems with the current version of Amanda and some future
distribution of Linux. Install and go, and at least then you should be
able to read the tapes.

-- 

Charles Curley  /"\ASCII Ribbon Campaign
Looking for fine software   \ /Respect for open standards
and/or writing?  X No HTML/RTF in email
http://www.charlescurley.com/ \No M$ Word docs in email

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB

Re: [Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[Chris Hoogendyk]

rorycl wrote:

An important aspect of the system is that the tapes should be readable
for 12 years, by other parties if necessary. From this point of view we
like the idea of providing a CD with each tape set of the software
needed to extract the contents, together with a listing of the enclosed
files in a UTF8 text file. We will be required to audit each backup set
by successfully extracting files from tape.


Just taking up that one point for the moment -- Amanda is not just open 
source and open format, but the tape format is based on standard 
UNIX/Linux tools. If you pull off the first file of the tape, it 
actually tells you how to read the tape. You don't need Amanda or any 
Amanda tools to read it. Just standard UNIX/Linux tools that come with 
every distribution, such as dd, gnutar, and gzip.


That said, it is easier to read and recover using the Amanda tools, 
because they will give you an index, allow you to specify what it is you 
want to recover, tell you which tapes you need, and get it for you. But, 
in the event that the tape lands in the hands of a UNIX/Linux admin who 
has never heard of Amanda, but who needs to recover the data, it can be 
done. And those tools are more likely to be available in stable or 
compatible forms in 12 years. It just happens that 12 years is about the 
lifecycle of a particular version of Solaris. That is, from the first 
introduction of Solaris X to its final EOL and drop of all support is 
about 12 years. I think Linux turns over faster than that, but the basic 
tools are typically compatible between versions.


If you want, you can use amreport to generate a report on the contents 
of a backup. Since you won't need a CD of software (and won't need to 
worry about whether it will run, whether the right libraries will be 
available, etc.), you might decide that a printout provided with each 
tape might be easier. Sysadmin looks at printout and immediately sees 
what's on the tape and, Oh, gee, it's that easy to read the tape. That 
avoids the difficulty of a CD not being stable or readable. The tapes 
are typically going to outlive a CD.



--
---

Chris Hoogendyk

-
  O__   Systems Administrator
 c/ /'_ --- Biology & Geology Departments
(*) \(*) -- 140 Morrill Science Center
~~ - University of Massachusetts, Amherst 




--- 


Erdös 4

[Amanda-users] Advice needed on Linux backup strategy to LTO-4 tape

[rorycl]

I'm going to cross-post this text on the Amanda and Bacula lists.
Apologies in advance if you see this twice.

Our company is about to provide centralised backups for several pools of
backup data of between 1 and 15TB in size. Each pool changes daily but
backups to tape will only occur once a month for each pool.

The backup tape format is to be LT04 and we have a second-hand Dell
PowerVault 124T 16 tape autoloader to work with currently. Backup from a
pool may be taken off a Linux LVM (or hopefully soon a BTRFS) snapshot
ensuring that the source data does not change during the backup process.
We have the possibility of pre-preparing backup or compressed images if
this is advisable.

An important aspect of the system is that the tapes should be readable
for 12 years, by other parties if necessary. From this point of view we
like the idea of providing a CD with each tape set of the software
needed to extract the contents, together with a listing of the enclosed
files in a UTF8 text file. We will be required to audit each backup set
by successfully extracting files from tape.

We are very familiar with working on the command-line in Linux,
Postgresql and Python.

As we have not run backup to tape on Linux before I would be very
grateful to receive advice on what approach members of this list would
take to meeting the above requirements.

Many thanks,
Rory

+--
|This was sent by r...@campbell-lange.net via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--

Re: How to configure Amanda for our backup strategy?

[Jon LaBadie]

On Mon, Oct 01, 2007 at 08:50:59AM -0500, [EMAIL PROTECTED] wrote:
> I'm setting up Amanda for the first time and would like to configure
> it for our existing backup strategy.  We want to do complete backups
> nightly during weekdays, i.e. Monday-Friday night.  Once a week, let's
> say Wednesday, we want to take one tape to an off-site vault and bring
> another tape back, meaning we'd be taking a tape out of the normal
> rotation and bringing another one back into the rotation.  We'd want
> to have a minimum of 4 tapes in rotation (3 here and 1 off-site),
> possibly a few more, but not a huge number of tapes.  All will be
> complete backups, no incrementals.
> 
> What parameters would work best in Amanda for doing this with a
> minimal need for manual "adjustments" during the week?  Is it possible
> to just take a tape out of the rotation for a week and give Amanda a
> different tape than it expects, or does a person have to tell Amanda
> they've done this each time they do it?
> 
> Thanks for any help or tips anyone might have!

Set your dumpcycle and runspercycle to 1.  That will do full dumps
each run.  Set your tapecycle to 1 or 2.  If set to 1, amanda will
use any available tape.  If set to 2 it will not reuse the most
recently used tape.  But all the others will be usable.  If the
tape it would prefer to use is off-site, it will use one of the
other available tapes.  You may get some messages about not being
able to use the tape it wants.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

How to configure Amanda for our backup strategy?

[DavidG]

I'm setting up Amanda for the first time and would like to configure
it for our existing backup strategy.  We want to do complete backups
nightly during weekdays, i.e. Monday-Friday night.  Once a week, let's
say Wednesday, we want to take one tape to an off-site vault and bring
another tape back, meaning we'd be taking a tape out of the normal
rotation and bringing another one back into the rotation.  We'd want
to have a minimum of 4 tapes in rotation (3 here and 1 off-site),
possibly a few more, but not a huge number of tapes.  All will be
complete backups, no incrementals.

What parameters would work best in Amanda for doing this with a
minimal need for manual "adjustments" during the week?  Is it possible
to just take a tape out of the rotation for a week and give Amanda a
different tape than it expects, or does a person have to tell Amanda
they've done this each time they do it?

Thanks for any help or tips anyone might have!

David

Advice for setup and backup strategy needed

[Markus Koppenberger]

Hi,

I'm quite new to amanda but as far as I have read about it and after
some first tries, it seems that it could do the job for our needs, but I
would need some advice for a backup strategy and for how to setup up amanda

I will describe you our current situation and our needs as well as my
thoughts of how to do it.

Every help is very apreciated!


We are planing to change from hard-disk backup (mirroring) to use amanda
to backup our RAID storage(s) with a total of 9 TB available on a 8x
LTO-2 (200GB) tape changer.

Currently about 4.8 TB of total storage space on three partitions is used:
  - 3.0T on a 3.3T RAID,
  - 1.4T on a 1.9T RAID and
  - 0.4T on another 1.9TB RAID.

Most of the stored data will not change over time, but the total amount
of data will (slowly) grow.

So what we would need is an initial full backup followed by regular
(maybe daily) incremental backups (which will be quite small)

Of course it is possible to split up the backup into smaller chunks
(single directories, not 3.3 T at once) but most of these smaller chunks
still would be greater then the 200G tapes we have and therefor I would
use tape-spanning.

Most of these directories have the following structure:

- directory1
   +- bigdata (> 200 GB, but does not change [often])
   +- otherdata (smaller [<< 200 GB], but changes more often)
- directory2
   +- bigdata
   +- otherdata
- ...


This could maybe used to define different backup-set:
- set(s): bigdata:
contains the 'bigdata' subdirectory of one (ore more,
depending on total size) directories be run maybe manually when
new files are added.
- set(s): otherdata:
contains the 'otherdata' sub-directories of various directories and
which will be run on daily basis with maybe monthly or two-weekly
full backup

Is this possible and does it make sense to split up the directories on
sub-directory basis into two (well actually more) backup sets ('big' and
'other')?

And if this would be the right way to do it, how would I configure the
different sets?

Maybe you could give me a general hint on how to use tape-spanning,
especially how to configure the 'runtapes' parameter.
Am I right that I have to configure it that 'runtapes' * 'size-of-tape'
= 'total_size_to_backup'? So, that when I have to backup let's say 1TB I
would need a minimum of runtapes=5 that the whole (full) backup fits on
the tapes ( 5 x 200GB)?

Thanks in advance,
koppi

-- 
Grupo de Tecnologia Musical (MTG)
Universidad Pompeu Fabra (UPF), Barcelona, Espana
http://www.iua.upf.es/mtg/



signature.asc
Description: OpenPGP digital signature

Re: no incremental backup in strategy standard

[Thomas Ginestet]

Ello list,

With the help of Jim Summers, i added "maxpromoteday" in the dumptype 
definition and give it the value of 4.
This ended up my problem for all my directories backed up except for 3 
of them.


It gives me:
WARNING planner disk pdc:/etc, estimate of level 1 failed.
WARNING planner disk pdc:/home/organismes, estimate of level 1 failed.
WARNING planner disk pdc:/home, estimate of level 1 failed.

(the /home and /home/organismes are separated because of an exclude list).

The next level 0 was due in 6 days and so I was expected theses 3 DLE  
would not be promoted to level 0.


Here is an extract of the amstatus:

pdc://S0/C$/amanda  10m finished (20:54:38)
pdc://S0/PMSSYBEL   1   10m finished (20:58:49)
pdc://S0/WEF10m finished (20:54:41)
pdc:/etc00m finished (20:56:35)
pdc:/home   058648m finished (8:07:16)
pdc:/home/organismes061356m finished (2:49:55)
www-dev:/etc   10m finished (20:53:21)
www-dev:/var/lib/postgresql15m finished (20:57:41)
www-dev:/var/lib/zope2.7   10m finished (20:54:21)


Any other similar experience about that ?


Thomas


Thomas Ginestet a écrit :

Hi list,

I've got a problem with my dump cycle. In my dumpcycle of 7 days (with 
5 runs per cycle) and a strategy standard, I've got 5 full backups 
instead of one full and 4 incremental backups. Here is my amanda.conf:



org "test"  # your organization name for reports
dumpuser "backup"   # the user to run dumps under

inparallel 8# maximum dumpers that will run in parallel
displayunit "m"
maxdumps 2
etimeout 2500
dtimeout 5000
netusage  10240 # maximum net bandwidth for Amanda, in KB per sec

# a filesystem is due for a full backup once every  days
dumpcycle 7 # the number of days in the normal dump cycle
runspercycle 5
runtapes 2
tapecycle 20 tapes  # the number of tapes in rotation

amrecover_do_fsf yes   amrecover_check_label yes

bumpsize 10 MB  # minimum savings (threshold) to bump 
level 1 -> 2

bumppercent  0
bumpdays 1  # minimum days at each level
bumpmult 4  # threshold = bumpsize * (level-1)**bumpmult

tpchanger "chg-zd-mtx"
changerdev "/dev/sg5"
tapedev "/dev/nst0"
changerfile "/etc/amanda/Tests/loader"
tapetype LTO2   # what kind of tape it is (see tapetypes below)
labelstr "P552LJ[0-9][0-9]$"# label constraint regex: all tapes 
must match

label_new_tapes "P552LJ%%"

autoflush on

infofile "/var/lib/amanda/Tests/curinfo"# database filename
logfile  "/var/log/amanda/Tests/log"# log filename

indexdir "/var/lib/amanda/Tests/index"

define tapetype LTO2 {
   comment "LTO-2"
   length 199680 mbytes
   filemark 0 kbytes
   speed 30810 kps
}

define dumptype test-dump {
   comment "dump of our hosts"
   program "GNUTAR"
   tape_splitsize 1 gb
   priority high
   strategy standard
   index yes
   compress client best
   fallback_splitsize 100 mb
}

holdingdisk test-tmp {
   directory "/var/lib/amanda/dumps"
   use -2 gb
}

Is there something wrong ?
Any help would be appreciate :)

Thomas

no incremental backup in strategy standard

[Thomas Ginestet]

Hi list,

I've got a problem with my dump cycle. In my dumpcycle of 7 days (with 5 
runs per cycle) and a strategy standard, I've got 5 full backups instead 
of one full and 4 incremental backups. 
Here is my amanda.conf:



org "test"  # your organization name for reports
dumpuser "backup"   # the user to run dumps under

inparallel 8# maximum dumpers that will run in parallel
displayunit "m"
maxdumps 2
etimeout 2500
dtimeout 5000
netusage  10240 # maximum net bandwidth for Amanda, in KB per sec

# a filesystem is due for a full backup once every  days
dumpcycle 7 # the number of days in the normal dump cycle
runspercycle 5
runtapes 2
tapecycle 20 tapes  # the number of tapes in rotation

amrecover_do_fsf yes   
amrecover_check_label yes


bumpsize 10 MB  # minimum savings (threshold) to bump 
level 1 -> 2

bumppercent  0
bumpdays 1  # minimum days at each level
bumpmult 4  # threshold = bumpsize * (level-1)**bumpmult

tpchanger "chg-zd-mtx"
changerdev "/dev/sg5"
tapedev "/dev/nst0"
changerfile "/etc/amanda/Tests/loader"
tapetype LTO2   # what kind of tape it is (see tapetypes below)
labelstr "P552LJ[0-9][0-9]$"# label constraint regex: all tapes must 
match

label_new_tapes "P552LJ%%"

autoflush on

infofile "/var/lib/amanda/Tests/curinfo"# database filename
logfile  "/var/log/amanda/Tests/log"# log filename

indexdir "/var/lib/amanda/Tests/index"

define tapetype LTO2 {
   comment "LTO-2"
   length 199680 mbytes
   filemark 0 kbytes
   speed 30810 kps
}

define dumptype test-dump {
   comment "dump of our hosts"
   program "GNUTAR"
   tape_splitsize 1 gb
   priority high
   strategy standard
   index yes
   compress client best
   fallback_splitsize 100 mb
}

holdingdisk test-tmp {
   directory "/var/lib/amanda/dumps"
   use -2 gb
}

Is there something wrong ?
Any help would be appreciate :)

Thomas

"strategy incronly" and "skip-full" again: According to manual page, both are buggy. Have bugs been fixed?

[Toralf Lund]

Regarding my recent post regarding "strategy incronly" and "skip-full", 
I just found out more about the problems I've had in the past with this 
simply by checking the amanda.conf manual page it says:


   skip-full /boolean/ 


   Default: no. If true and planner has scheduled a full backup,
   these disks will be skipped, and full backups should be run
   off-line on these days. It was reported that Amanda only
   schedules level 1 incrementals in this configuration; this is
   probably a bug.

and also

   strategy {standard|nofull|noinc|skip|incronly}

   [ ... ]
   incronly : Only do incremental dumps. amadmin force should be
   used to tell Amanda that a full dump has been performed
   off-line, so that it resets to level 1. It is similar to
   skip-full, but with incronly full dumps may be scheduled
   manually. Unfortunately, it appears that Amanda will perform
   full backups with this configuration, which is probably a bug.

So, the problem with setting up an incremental-only config is that with 
"skip-full", incremental level will never be increased above 1, while 
with "strategy incronly", it is updated correctly, but the level 0s 
(scheduled via "amadmin force") won't be skipped correctly. Or at least, 
that's the way it's been in the past. Does anyone know if anything has 
been done lately to address these issues?


- Toralf

Re: HD backup strategy ?

[Mike Delaney]

On Wed, Aug 17, 2005 at 03:03:59PM +0300, Toomas Aas wrote:
> Jon LaBadie wrote:
> 
> >Haven't seen anyone on the list mention using it, but Iomega
> >introduced some interesting hardware last year.  I think they
> >call it "Rev", basically a small, removalble hard drive
> >cartridge.  Think high capacity, tiny zip drive as it has
> >35GB native capacity and a builtin compressor.  Along with
> >their single slot device, they also introduce an autoloader
> >that can hold 10 cartridges.
> 
> Funny, I just stumbled upon this on another mailing list yesterday. 
> Looks like IOMega continues to live up to its reputation:
> 
> http://lists.freebsd.org/pipermail/freebsd-questions/2005-July/094114.html

Or someone didn't look hard enough.  While most systems device drivers
do detect REV drives as CD-ROMs, that doesn't prevent one from writing
to it.  And while the filesystem format isn't ISO-9660, it's not some
Iomega proprietary thing either: it's UDF.

Basically, from the system's perspective REV drives look and act like
high capacity DVD-RAM units.  I've been using one with SuSE Linux for
several months now.  

Re: HD backup strategy ?

[Jon LaBadie]

On Wed, Aug 17, 2005 at 03:03:59PM +0300, Toomas Aas wrote:
> Jon LaBadie wrote:
> 
> >Haven't seen anyone on the list mention using it, but Iomega
> >introduced some interesting hardware last year.  I think they
> >call it "Rev", basically a small, removalble hard drive
> >cartridge.  Think high capacity, tiny zip drive as it has
> >35GB native capacity and a builtin compressor.  Along with
> >their single slot device, they also introduce an autoloader
> >that can hold 10 cartridges.
> 
> Funny, I just stumbled upon this on another mailing list yesterday. 
> Looks like IOMega continues to live up to its reputation:
> 
> http://lists.freebsd.org/pipermail/freebsd-questions/2005-July/094114.html

A peek at the IOMygosh website confirms the articles comment that
it is windows only!  Time for me to stop even thinking of it as a
potential amanda backup device.

jl
-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: HD backup strategy ?

[Toomas Aas]

Jon LaBadie wrote:


Haven't seen anyone on the list mention using it, but Iomega
introduced some interesting hardware last year.  I think they
call it "Rev", basically a small, removalble hard drive
cartridge.  Think high capacity, tiny zip drive as it has
35GB native capacity and a builtin compressor.  Along with
their single slot device, they also introduce an autoloader
that can hold 10 cartridges.


Funny, I just stumbled upon this on another mailing list yesterday. 
Looks like IOMega continues to live up to its reputation:


http://lists.freebsd.org/pipermail/freebsd-questions/2005-July/094114.html

--
Toomas Aas 
|arvutivõrgu peaspetsialist | head specialist on computer networks|
|Tartu Linnakantselei   | Tartu City Office   |
- +372 736 1274

Re: HD backup strategy ?

[Jon LaBadie]

On Tue, Aug 16, 2005 at 01:02:33PM -0400, Joshua Baker-LePain wrote:
> On Tue, 16 Aug 2005 at 12:13pm, Jon LaBadie wrote
> 
> > On Fri, Aug 12, 2005 at 09:40:59AM -0400, FM wrote:
> > > Hello everybody,
> > > 
> > > My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
> > > thing it's time to switch for HD backups.
> > > 
> > > Does some of you using HD backup ? if so what is you stategy ?
> > > How you take backup off-site ?
> > > 
> > > What kind of hardware are you using ? SCSI or SATA RAID ?
> > 
> > Haven't seen anyone on the list mention using it, but Iomega
> > introduced some interesting hardware last year.  I think they
> > call it "Rev", basically a small, removalble hard drive
> > cartridge.  Think high capacity, tiny zip drive as it has
> > 35GB native capacity and a builtin compressor.  Along with
> > their single slot device, they also introduce an autoloader
> > that can hold 10 cartridges.
> > 
> > If the capacity is reasonable for your needs you might have
> > your HD backup and off-site capability combined.
> 
> Why does the thought of trusting my backups to anything made by Iomega 
> scare the living [EMAIL PROTECTED] out of me?
> 

That is why I made sure to mention the manufacturer.
But it is a neat concept.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: HD backup strategy ?

[Tobias Bluhm]

Joshua Baker-LePain <[EMAIL PROTECTED]> 08/16/2005 01:02:33 PM:

> On Tue, 16 Aug 2005 at 12:13pm, Jon LaBadie wrote
> 
> > On Fri, Aug 12, 2005 at 09:40:59AM -0400, FM wrote:
> > > Hello everybody,
> > > 
> > > My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
> > > thing it's time to switch for HD backups.
> > > 
> > > Does some of you using HD backup ? if so what is you stategy ?
> > > How you take backup off-site ?
> > > 
> > > What kind of hardware are you using ? SCSI or SATA RAID ?
> > 
> > Haven't seen anyone on the list mention using it, but Iomega
> > introduced some interesting hardware last year.  I think they
> > call it "Rev", basically a small, removalble hard drive
> > cartridge.  Think high capacity, tiny zip drive as it has
> > 35GB native capacity and a builtin compressor.  Along with
> > their single slot device, they also introduce an autoloader
> > that can hold 10 cartridges.
> > 
> > If the capacity is reasonable for your needs you might have
> > your HD backup and off-site capability combined.
> 
> Why does the thought of trusting my backups to anything made by Iomega 
> scare the living [EMAIL PROTECTED] out of me?


Hehe - you're not thinking of the i'll-eat-your-disk-then-die zip-100, are 
you?


-
toby bluhm
philips medical systems, cleveland ohio
[EMAIL PROTECTED]
440-483-5323

Re: HD backup strategy ?

[Joshua Baker-LePain]

On Tue, 16 Aug 2005 at 12:13pm, Jon LaBadie wrote

> On Fri, Aug 12, 2005 at 09:40:59AM -0400, FM wrote:
> > Hello everybody,
> > 
> > My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
> > thing it's time to switch for HD backups.
> > 
> > Does some of you using HD backup ? if so what is you stategy ?
> > How you take backup off-site ?
> > 
> > What kind of hardware are you using ? SCSI or SATA RAID ?
> 
> Haven't seen anyone on the list mention using it, but Iomega
> introduced some interesting hardware last year.  I think they
> call it "Rev", basically a small, removalble hard drive
> cartridge.  Think high capacity, tiny zip drive as it has
> 35GB native capacity and a builtin compressor.  Along with
> their single slot device, they also introduce an autoloader
> that can hold 10 cartridges.
> 
> If the capacity is reasonable for your needs you might have
> your HD backup and off-site capability combined.

Why does the thought of trusting my backups to anything made by Iomega 
scare the living [EMAIL PROTECTED] out of me?

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: HD backup strategy ?

[Jon LaBadie]

On Fri, Aug 12, 2005 at 09:40:59AM -0400, FM wrote:
> Hello everybody,
> 
> My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
> thing it's time to switch for HD backups.
> 
> Does some of you using HD backup ? if so what is you stategy ?
> How you take backup off-site ?
> 
> What kind of hardware are you using ? SCSI or SATA RAID ?

Haven't seen anyone on the list mention using it, but Iomega
introduced some interesting hardware last year.  I think they
call it "Rev", basically a small, removalble hard drive
cartridge.  Think high capacity, tiny zip drive as it has
35GB native capacity and a builtin compressor.  Along with
their single slot device, they also introduce an autoloader
that can hold 10 cartridges.

If the capacity is reasonable for your needs you might have
your HD backup and off-site capability combined.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: HD backup strategy ?

[Bernhard Ott]

FM wrote:
> Hello everybody,
> 
> My ibm 35810 died and I will cost use 2 CA$ to replace it. So I
> thing it's time to switch for HD backups.
> 
> Does some of you using HD backup ? if so what is you stategy ?
> How you take backup off-site ?
> 
> What kind of hardware are you using ? SCSI or SATA RAID ?
> 
> Thanks !
> 
> 
Have a look at backuppc: http://backuppc.sourceforge.net/
The archiving can't be compared to amanda, but still it's quite
powerfull ...
I'm using amanda for off-site backups and backuppc (SATA Raid) to give
users permanent access to their data .

Regards, Bernhard

Re: HD backup strategy ?

[Gene Heskett]

On Friday 12 August 2005 09:40, FM wrote:
>Hello everybody,
>
>My ibm 35810 died and I will cost use 2 CA$ to replace it. So I
>thing it's time to switch for HD backups.
>
>Does some of you using HD backup ? if so what is you stategy ?
>How you take backup off-site ?
>
>What kind of hardware are you using ? SCSI or SATA RAID ?
>
>Thanks !

Here, a home site, I'm, using a 200GB atapi drive, with 180GB of it
partitioned for amanda.  Its worked flawlessly for everythng for
about a year. It is not removeable other than by a powerdown & ata133
cable disconnection.  I'd not have an offsite storage place available
anyway.  I'm backing up the working parts of 2 machines & may add a
third if it ever gets to stay on 24/7 like the first 2 do.

Candid opinion:  HD's are far more friendly than tapes when it comes
to recovery operations, and hundreds of times faster for recovery.  I
did not find that tapes (but then I was using DDS2 (spit) tapes too)
were in fact all that dependable, with something getting a tummy ache
at weekly intervals.  I've had no such tummy aches with the HD setup.

BUT:  Quality tapes would have the option of removing for permanent
storage, unlike using a non-removable HD.  If I really thought my
stuff was *that* precious, I'd burn a stack of dvd's once  month. 
But, I'd have to justify the cost of the dvd+(or-)r's as an ongoing
expense, something my SS would have to support as I'm just your
typical old fart, half retired, tv chief engineer whose played with
computers since 1977 & refuses to act my age of 70. 

Trying to stay in shape, and needing to get it done, I've moved about
4 yards of dirt with a shovel and wheelborrow in the last 3 days
with the temps in the high 80's.   And thats burning about 2 lbs a day
off me that I can well afford to lose, and keeping the sugar in check
too.  But when I'm done, it'll be mowable. Nothings flat in West
Virginia unless you "make it so"...

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: HD backup strategy ?

[Toomas Aas]

FM wrote:


Does some of you using HD backup ? if so what is you stategy ?
How you take backup off-site ?


At one site, I'm using two external FireWire HDDs (Maxtor 5000DV), 5 
'virtual tapes' on each. One of the HDDs is off-site, every Monday it's 
brought in and the HDD that was used during previus week is then 
disconnected and taken off-site.





---
... How can you tell when you've run out of invisible ink?

HD backup strategy ?

[FM]

Hello everybody,

My ibm 35810 died and I will cost use 2 CA$ to replace it. So I 
thing it's time to switch for HD backups.


Does some of you using HD backup ? if so what is you stategy ?
How you take backup off-site ?

What kind of hardware are you using ? SCSI or SATA RAID ?

Thanks !

Re: What tapes can I send off-site ? How do I implement this backup strategy ?

[L Sydow]

How can I manage to do full backups of every server each FRIDAY ? I
know I should create a new config specifying always full and an
infinite tapecylce and run it on fridays, and run the "regular"
schedule on the other days.
Instead of having cron run amdump directly, you could have it run a 
script like:

set -- `date`
case $1 in
Fri)
su backup /local/sbin/amdump Full
;;
*)
su backup /local/sbin/amdump Daily
;;
esac

Re: What tapes can I send off-site ? How do I implement this backup strategy ?

[Matt Hyclak]

On Tue, May 10, 2005 at 03:45:57PM -0400, Guy Dallaire enlightened us:
> We use a nice GUI program to do that, but the product is outdated and
> buying a new licence is too costly. Amanda seems good, and we use more
> and more open source software here so we decided to try it out.
> 
> Our current method of backup is the traditional incrementals Mon, Tue,
> Wed, Thu, and Friday we do a Full backup. No backups on week end.
> 
> The tapes are stored in a locked cabinet outside the server room.
> After 3 "business days" on site, the tapes are sent to a remote
> location.  This permits to have the tape handy if we have to restore a
> file that was modified and deleted in the last 2 days.
> 
> Monthly tapes (Full backup) are kept forever for archiving/financial
> regulation  pupose.
> 
> I'm having a hard time figuring out how to do something similar with amanda.
> 

You'll need to keep "dumpcycle" worth of tapes on site. If you set dumpcycle
to 1 week, and runspercycle to 5 (M-F), then you'll need to keep the latest
5 tapes to be able to do those quick restores. What a lot of people will do
with enough tapes is have 4 "sets" of tapes. In your case you might have
something like the following:

* Tape1, Tape2, Tape3, Tape4, Tape5 - "current" set that backups are performed
  on

* Tape6, Tape7, Tape8, Tape9, Tape10 - "most recent" set from last week,
  stored in your locked cabinet

* Tape11, Tape12, Tape13, Tape14, Tape15 - set stored offsite

* Tape16, Tape17, Tape18, Tape19, Tape20 - in transit back onsite to become
  next weeks "current"

Depending on how accessible your offsite location is, you can probably get
away with 3 "sets"

> I'm not familiar with backup levels. It's a concept I always had a
> difficulty to grasp.  If I send the tapes older than 3 days to the
> remote site, will the tapes of the last three days ALWAYS contain the
> file modified in the last 3 days ?
>

Backup levels work like this:

Level 0 contains every file
Level 1 contains every file that has changed since the last Level 0
Level 2 contains every file that has changed since the last Level 1
Level n contains every file that has changed since the last Level n-1

For a simple example, let's consider some directory "foo". "foo" contains the
files "a", "b", "c", and "d".

* Monday - Level 0 - contains a,b,c and d
- User edits file b
* Tuesday - Level 1 - contains b
- User edits file c
* Wednesday - Level 1 - contains b and c
- User edits file a
* Thursday - Level 2 - contains a

So the idea is that if you need to restore an entire directory, you use the
latest-dated of each level, in this case Monday, Wednesday and Thursday's
tapes.

Amanda uses the bump* variables to decide when to increase to the next
level. Above it seems that the level was bumped arbitrarily from 1 to 2, but
what happened for example was that b and c were both extremely large files,
and a was very small. It wouldn't make much sense to use tape for those
large files when the only change is the small "a" file, so it bumps the
level up.

> How can I manage to do full backups of every server each FRIDAY ? I
> know I should create a new config specifying always full and an
> infinite tapecylce and run it on fridays, and run the "regular"
> schedule on the other days.
>

It is generally not recommended to do so. Why do you feel it necessary to
only do full dumps on Friday? Let amanda even out the schedule and you will
find good usage accross tapes, and fairly consistent backup runtimes.

> One last question: Is there an easy way with amanda to do an "ad hoc"
> backup ? I mean, say I have to backup a group of directories on a
> server, do I have to create a new config, a new disklist, etc... ?
> There must be an easier way ?
> 

The host would have to be in the disklist, but you can do individual
backups. amdump takes optional host and disk parameters, so if a machine was
offline during the backup, the next morning you can bring it up and run
something like:

amdump CONFIGNAME missinghost /backupdir

> I'm reading the documentation, but there is a lot of doc, and not
> enough samples and use-cases IMHO.
> 

I hope my examples helped.

Matt

-- 
Matt Hyclak
Department of Mathematics 
Department of Social Work
Ohio University
(740) 593-1263


pgpuEy3ZZ3Uuh.pgp
Description: PGP signature

What tapes can I send off-site ? How do I implement this backup strategy ?

[Guy Dallaire]

We use a nice GUI program to do that, but the product is outdated and
buying a new licence is too costly. Amanda seems good, and we use more
and more open source software here so we decided to try it out.

Our current method of backup is the traditional incrementals Mon, Tue,
Wed, Thu, and Friday we do a Full backup. No backups on week end.

The tapes are stored in a locked cabinet outside the server room.
After 3 "business days" on site, the tapes are sent to a remote
location.  This permits to have the tape handy if we have to restore a
file that was modified and deleted in the last 2 days.

Monthly tapes (Full backup) are kept forever for archiving/financial
regulation  pupose.

I'm having a hard time figuring out how to do something similar with amanda.

I'm not familiar with backup levels. It's a concept I always had a
difficulty to grasp.  If I send the tapes older than 3 days to the
remote site, will the tapes of the last three days ALWAYS contain the
file modified in the last 3 days ?

How can I manage to do full backups of every server each FRIDAY ? I
know I should create a new config specifying always full and an
infinite tapecylce and run it on fridays, and run the "regular"
schedule on the other days.

One last question: Is there an easy way with amanda to do an "ad hoc"
backup ? I mean, say I have to backup a group of directories on a
server, do I have to create a new config, a new disklist, etc... ?
There must be an easier way ?

I'm reading the documentation, but there is a lot of doc, and not
enough samples and use-cases IMHO.

Thanks

Re: Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Chuck Amadi]

Great

I use the following for a So called Friday-Tape-Question.

dumpcycle 5
runspercycle 5
tapecycle 25 
maxdumps 1

Cheers

On Wed, 2005-04-13 at 15:33 +0200, Stefan G. Weichinger wrote:
> Hi, Chuck,
> 
> on Mittwoch, 13. April 2005 at 12:30 you wrote to amanda-users:
> 
> CA> Has anyone example of amanda implement the infinite
> CA> grandfather-father-son strategy.
> 
> This is what we call the friday-tape-question:
> 
> http://www.amanda.org/docs/topten.html#id2519662
> 
-- 
Unix/ Linux Systems Administrator

The Surgical Material Testing Laboratory (SMTL), 
Princess of Wales Hospital 
Coity Road 
Bridgend, 
United Kingdom, CF31 1RQ.

Tel: +44 1656 752820 
Fax: +44 1656 752830

Re: Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Paul Bijnens]

Chuck Amadi wrote:
Hi agian
I now got it.
almost :-)
My Friday tapes i.e weeklySets their amanda.conf will have level 0 
and thus I just pull out the last Friday of the month for archiving.
and run amlabel to create the pulled tape.

dumpcycle 5 # 4/5 Friday tapes last tape to be pulled.
runspercycle 5 # 4 tapes for mon to thurs in rotation
tapecycle 14 # one tape for errors and one for cleaning before being  
overwritten.
maxdumps 1
You probably want:
dumpcycle 0 # just to force a full dump allways
runspercycle 1  # just a number
tapecycle 14# how many tapes you have in this config
I think you're mixing up the meaning of maxdumps with something else:
this has little to do with the scheduling of backups, only with
concurrency during a run.

My Mon to Thursday dailysets will have their unqiue amanda.conf file
dumpcycle 5
runspercycle 4 # 4 tapes for mon to thurs.
tapecycle 6 # tape for errors and one for cleaning
dumpcycle 1 week
runspercycle 4
tapecycle  6

Im getting there.
closer and closer indeed.
--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***

Re: Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Chuck Amadi]

Hi agian

I now got it.

My Friday tapes i.e weeklySets their amanda.conf will have level 0 
and thus I just pull out the last Friday of the month for archiving.
and run amlabel to create the pulled tape.

dumpcycle 5 # 4/5 Friday tapes last tape to be pulled.
runspercycle 5 # 4 tapes for mon to thurs in rotation
tapecycle 14 # one tape for errors and one for cleaning before being
overwritten.

maxdumps 1

My Mon to Thursday dailysets will have their unqiue amanda.conf file

dumpcycle 5
runspercycle 4 # 4 tapes for mon to thurs.
tapecycle 6 # tape for errors and one for cleaning

Im getting there.

Cheers
Chuck

.On Wed, 2005-04-13 at 11:40 +0100, Martin Hepworth wrote:
> Chuck
> 
> Amanda does it's own cycling of full and incremental backups, tying to 
> get an even backup size/time over the tape sets.
> 
> If you want to do 'permanent archive' sets (we do) then you need a 
> separate config  that only does level 0 backups and has the tapes marked 
> as don't reuse.
> 
> Have a read of the documentation and esp how the planner handles 
> incrementals/full backups.
> 
> --
> Martin Hepworth
> Snr Systems Administrator
> Solid State Logic
> Tel: +44 (0)1865 842300
> 
> 
> Chuck Amadi wrote:
> > Has anyone example of amanda implement the infinite
> > grandfather-father-son strategy.
> > 
> > Cheers
> > 
> > Chuck
> 
> **
> 
> This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they
> are addressed. If you have received this email in error please notify
> the system manager.
> 
> This footnote confirms that this email message has been swept
> for the presence of computer viruses and is believed to be clean. 
> 
> **
> 
-- 
Unix/ Linux Systems Administrator

The Surgical Material Testing Laboratory (SMTL), 
Princess of Wales Hospital 
Coity Road 
Bridgend, 
United Kingdom, CF31 1RQ.

Tel: +44 1656 752820 
Fax: +44 1656 752830

Re: Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Jon LaBadie]

On Wed, Apr 13, 2005 at 12:22:49PM +0100, Chuck Amadi wrote:
> Hi Yes I have but before I run my first test I would like confirmation
> that my amanda.conf handle incremental monday to thursday and a full
> backup on a friday and I will pull out the last friday of that month.
> 
> dumpcycle 1 week # Incremental weekday mon-thurs dump only 
> runpercycle 5 # Friday Full dump and thus I will pull out for
> archives(12 tapes).
> 
> tapecycle 25 # Number of tapes in circulation to be overwritten.
> # NB 3 tapes for error/ampflush and one tape for cleaning purposes.
> 
> Am I on the right track.
> 
> Chuck
>
> On Wed, 2005-04-13 at 11:40 +0100, Martin Hepworth wrote:
> > Chuck
> > 
> > Amanda does it's own cycling of full and incremental backups, tying to 
> > get an even backup size/time over the tape sets.
> > 
> > If you want to do 'permanent archive' sets (we do) then you need a 
> > separate config  that only does level 0 backups and has the tapes marked 
> > as don't reuse.
> > 
> > Have a read of the documentation and esp how the planner handles 
> > incrementals/full backups.
> > 
> > 
> > Chuck Amadi wrote:
> > > Has anyone example of amanda implement the infinite
> > > grandfather-father-son strategy.


You may have read, but not understood.

Normally amanda MIXES full dumps and incrementals on each amdump run.
Its design tries to avoid the monster dump on friday by balancing each
days' dump with a mix.

Many (most?) hate amanda's scheme ... until they try it.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Stefan G. Weichinger]

Hi, Chuck,

on Mittwoch, 13. April 2005 at 12:30 you wrote to amanda-users:

CA> Has anyone example of amanda implement the infinite
CA> grandfather-father-son strategy.

This is what we call the friday-tape-question:

http://www.amanda.org/docs/topten.html#id2519662

-- 
best regards,
Stefan

Stefan G. Weichinger
mailto:[EMAIL PROTECTED]

Re: Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Martin Hepworth]

Chuck
err ... no
amanda's planner will decide when to do the full dumps, based on how 
big/how long the dumps take.

if you want a definite archive set you need a separate config that just 
has a zero dumpcyle..

maxdumps 1
dumpcycle 0
runspercycle 1
tapecycle 2000 tapes## I run this once per month so it give me lots 
of tapes...

--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
Chuck Amadi wrote:
Hi Yes I have but before I run my first test I would like confirmation
that my amanda.conf handle incremental monday to thursday and a full
backup on a friday and I will pull out the last friday of that month.
dumpcycle 1 week # Incremental weekday mon-thurs dump only 
runpercycle 5 # Friday Full dump and thus I will pull out for
archives(12 tapes).

tapecycle 25 # Number of tapes in circulation to be overwritten.
# NB 3 tapes for error/ampflush and one tape for cleaning purposes.
Am I on the right track.
Cheers
Chuck
On Wed, 2005-04-13 at 11:40 +0100, Martin Hepworth wrote:
Chuck
Amanda does it's own cycling of full and incremental backups, tying to 
get an even backup size/time over the tape sets.

If you want to do 'permanent archive' sets (we do) then you need a 
separate config  that only does level 0 backups and has the tapes marked 
as don't reuse.

Have a read of the documentation and esp how the planner handles 
incrementals/full backups.

--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
Chuck Amadi wrote:
Has anyone example of amanda implement the infinite
grandfather-father-son strategy.
Cheers
Chuck
**
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.   
**
**
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.   
**

Re: Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Paul Bijnens]

Chuck Amadi wrote:
Am I on the right track.
No, you're not.

On Wed, 2005-04-13 at 11:40 +0100, Martin Hepworth wrote:
Have a read of the documentation and esp how the planner handles 
incrementals/full backups.
Have you done that?
And do you have a reason why not to follow the amanda-planner?
--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***

Re: Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Chuck Amadi]

Hi Yes I have but before I run my first test I would like confirmation
that my amanda.conf handle incremental monday to thursday and a full
backup on a friday and I will pull out the last friday of that month.

dumpcycle 1 week # Incremental weekday mon-thurs dump only 
runpercycle 5 # Friday Full dump and thus I will pull out for
archives(12 tapes).

tapecycle 25 # Number of tapes in circulation to be overwritten.
# NB 3 tapes for error/ampflush and one tape for cleaning purposes.

Am I on the right track.

Cheers

Chuck
On Wed, 2005-04-13 at 11:40 +0100, Martin Hepworth wrote:
> Chuck
> 
> Amanda does it's own cycling of full and incremental backups, tying to 
> get an even backup size/time over the tape sets.
> 
> If you want to do 'permanent archive' sets (we do) then you need a 
> separate config  that only does level 0 backups and has the tapes marked 
> as don't reuse.
> 
> Have a read of the documentation and esp how the planner handles 
> incrementals/full backups.
> 
> --
> Martin Hepworth
> Snr Systems Administrator
> Solid State Logic
> Tel: +44 (0)1865 842300
> 
> 
> Chuck Amadi wrote:
> > Has anyone example of amanda implement the infinite
> > grandfather-father-son strategy.
> > 
> > Cheers
> > 
> > Chuck
> 
> **
> 
> This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they
> are addressed. If you have received this email in error please notify
> the system manager.
> 
> This footnote confirms that this email message has been swept
> for the presence of computer viruses and is believed to be clean. 
> 
> **
> 
-- 
Unix/ Linux Systems Administrator

The Surgical Material Testing Laboratory (SMTL), 
Princess of Wales Hospital 
Coity Road 
Bridgend, 
United Kingdom, CF31 1RQ.

Tel: +44 1656 752820 
Fax: +44 1656 752830

Re: Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Martin Hepworth]

Chuck
Amanda does it's own cycling of full and incremental backups, tying to 
get an even backup size/time over the tape sets.

If you want to do 'permanent archive' sets (we do) then you need a 
separate config  that only does level 0 backups and has the tapes marked 
as don't reuse.

Have a read of the documentation and esp how the planner handles 
incrementals/full backups.

--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
Chuck Amadi wrote:
Has anyone example of amanda implement the infinite
grandfather-father-son strategy.
Cheers
Chuck
**
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.   
**

Example of amanda implement the infinite grandfather-father-son strategy dumpcycle.

[Chuck Amadi]

Has anyone example of amanda implement the infinite
grandfather-father-son strategy.

Cheers

Chuck
-- 
Unix/ Linux Systems Administrator

The Surgical Material Testing Laboratory (SMTL), 
Princess of Wales Hospital 
Coity Road 
Bridgend, 
United Kingdom, CF31 1RQ.

Tel: +44 1656 752820 
Fax: +44 1656 752830

Re: Determining appropriate backup strategy

[Joshua Baker-LePain]

On Thu, 31 Mar 2005 at 5:24pm, Vicki Stanfield wrote

> I think what I am really talking about is the dumptype. I have the ones 
> I listed defined. I think I might be confusing them with priority. I 
> suppose the dumptype could only mean level of compression. But then I 
> need more information about how Amanda decides what to back up when. 
> I'll look back to the amanda.org site.

Every DLE is backed up every night.  The level depends not on the dumptype 
(unless you've specified always-full or something like that) but on 
dumpcycle and runspercycle.

dumpcycle=max amount of time before full (level 0) backups
runspercycle=number of 'amdump' runs per dumpcycle

You'll also need to coordinate these with tapecycle, which is the number 
of tapes you'll let amanda use.  Generally you'll want 
tapecycle>=1.5*runspercycle for error protection.  Note also that all of 
the above assumes runtapes=1.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: Determining appropriate backup strategy

[Jon LaBadie]

On Thu, Mar 31, 2005 at 03:47:51PM -0500, Vicki Stanfield wrote:
> I am using amanda to back up my servers. I have the following situation:
> 
> There is about 42G of data to be backed up on one particular machine. 
> This data is mostly static data in a directory structure which creates a 
> new directory whenever 1000 files are in the current directory and 
> numbers the new directory incrementally.  It looks like this:
> 
> /project/newarchive/0 contains files 0-999
> /project/newarchive/1 contains files 1000-1999
> /project/newarchive/1 contains files 2000-2999
> etc.
> 
> The previous directories are necessary for the program to run 
> successfully but will remain static once the next directory is created. 
> I need a backup solution that is appropriate for this. What is 
> appropriate for this situation?

Wouldn't the standard solution be appropriate here?

I presume you do want an occassional full backup of even the static data.
For the incrementals, only things which change (I guess files added) are
backed up.  As long as the files haven't changed since the last higher
level dump (lower numerically) then those files will not appear in the
incrementals.

You may want to play with bumpdays and bumpmultiplier to ensure you get
several dump levels between level 0 dumps.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: Determining appropriate backup strategy

[Jon LaBadie]

On Thu, Mar 31, 2005 at 05:24:43PM -0500, Vicki Stanfield wrote:
> Jon LaBadie wrote:
> 
> >On Thu, Mar 31, 2005 at 03:47:51PM -0500, Vicki Stanfield wrote:
> > 
> >
> >>P.S. I think I asked before but didn't get an answer. Is there a source 
> >>for an explanation of the different priorities of backups. I have the 
> >>following defined from an inherited amanda.conf file:
> >>
> >>always-full (obvious)
> >>comp-low-tar
> >>comp-med-tar
> >>comp-high-tar
> >>low-tar
> >>med-tar
> >>high-tar
> >>comp-low-dump
> >>comp-med-dump
> >>comp-high-dump
> >>low-dump
> >>med-dump
> >>high-dump
> >>
> >>The explanations in the amanda.conf are insufficient. I need to know 
> >>what decisions amanda makes when determining how often to back up each 
> >>dumptype.
> >>   
> >>
> >
> >A good additional place to read is the man pages.
> >Under priority this appears:
> >
> >priority "string"
> > Default:  medium.  When there is no tape to  write  to,
> > Amanda will do incremental backups in priority order to
> > the  holding  disk.   The  priority  may  be   high(2),
> > medium(1), low(0) or a number of your choice.
> >
> >So the difference appears important only when saving ONLY to holding disk.
> >Then you can define which come earlier or later incase the holding disk
> >fills.
> >
> > 
> >
> I think what I am really talking about is the dumptype. I have the ones 
> I listed defined. I think I might be confusing them with priority. I 
> suppose the dumptype could only mean level of compression. But then I 
> need more information about how Amanda decides what to back up when. 
> I'll look back to the amanda.org site.
> 

Set one of your DLE's to the dumptype.  Maybe even a dummy DLE.
Then run amadmin ... disklist ... command to see the relevant
parameters.  Change the dumptype and rerun amadmin to compare.
A sample run:

  # amadmin DS1 disklist butch /var
  line 10:
host butch:
interface default
disk /var:
program "GNUTAR"
exclude list "/usr/local/etc/amanda/exclude.gtar"
priority 0
dumpcycle 7
maxdumps 8
maxpromoteday 1
strategy STANDARD
compress CLIENT FAST
comprate 0.50 0.50
auth BSD
kencrypt NO
holdingdisk YES
record YES
index YES
skip-incr NO
skip-full NO

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: Determining appropriate backup strategy

[Vicki Stanfield]

Jon LaBadie wrote:
On Thu, Mar 31, 2005 at 03:47:51PM -0500, Vicki Stanfield wrote:
 

P.S. I think I asked before but didn't get an answer. Is there a source 
for an explanation of the different priorities of backups. I have the 
following defined from an inherited amanda.conf file:

always-full (obvious)
comp-low-tar
comp-med-tar
comp-high-tar
low-tar
med-tar
high-tar
comp-low-dump
comp-med-dump
comp-high-dump
low-dump
med-dump
high-dump
The explanations in the amanda.conf are insufficient. I need to know 
what decisions amanda makes when determining how often to back up each 
dumptype.
   

A good additional place to read is the man pages.
Under priority this appears:
priority "string"
 Default:  medium.  When there is no tape to  write  to,
 Amanda will do incremental backups in priority order to
 the  holding  disk.   The  priority  may  be   high(2),
 medium(1), low(0) or a number of your choice.
So the difference appears important only when saving ONLY to holding disk.
Then you can define which come earlier or later incase the holding disk
fills.
 

I think what I am really talking about is the dumptype. I have the ones 
I listed defined. I think I might be confusing them with priority. I 
suppose the dumptype could only mean level of compression. But then I 
need more information about how Amanda decides what to back up when. 
I'll look back to the amanda.org site.

Vicki

Re: Determining appropriate backup strategy

[Jon LaBadie]

On Thu, Mar 31, 2005 at 03:47:51PM -0500, Vicki Stanfield wrote:
> 
> P.S. I think I asked before but didn't get an answer. Is there a source 
> for an explanation of the different priorities of backups. I have the 
> following defined from an inherited amanda.conf file:
> 
> always-full (obvious)
> comp-low-tar
> comp-med-tar
> comp-high-tar
> low-tar
> med-tar
> high-tar
> comp-low-dump
> comp-med-dump
> comp-high-dump
> low-dump
> med-dump
> high-dump
> 
> The explanations in the amanda.conf are insufficient. I need to know 
> what decisions amanda makes when determining how often to back up each 
> dumptype.

A good additional place to read is the man pages.
Under priority this appears:

 priority "string"
  Default:  medium.  When there is no tape to  write  to,
  Amanda will do incremental backups in priority order to
  the  holding  disk.   The  priority  may  be   high(2),
  medium(1), low(0) or a number of your choice.

So the difference appears important only when saving ONLY to holding disk.
Then you can define which come earlier or later incase the holding disk
fills.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Determining appropriate backup strategy

[Vicki Stanfield]

I am using amanda to back up my servers. I have the following situation:
There is about 42G of data to be backed up on one particular machine. 
This data is mostly static data in a directory structure which creates a 
new directory whenever 1000 files are in the current directory and 
numbers the new directory incrementally.  It looks like this:

/project/newarchive/0 contains files 0-999
/project/newarchive/1 contains files 1000-1999
/project/newarchive/1 contains files 2000-2999
etc.
The previous directories are necessary for the program to run 
successfully but will remain static once the next directory is created. 
I need a backup solution that is appropriate for this. What is 
appropriate for this situation?

Vicki
P.S. I think I asked before but didn't get an answer. Is there a source 
for an explanation of the different priorities of backups. I have the 
following defined from an inherited amanda.conf file:

always-full (obvious)
comp-low-tar
comp-med-tar
comp-high-tar
low-tar
med-tar
high-tar
comp-low-dump
comp-med-dump
comp-high-dump
low-dump
med-dump
high-dump
The explanations in the amanda.conf are insufficient. I need to know 
what decisions amanda makes when determining how often to back up each 
dumptype.

Re: amanda tape use strategy

[Chris Loken]

Paul & Jon - thanks for the tip about using taperalgo. I wasn't aware of 
this parameter. Will try it out but I'm not sure how well it will work 
for me - I'm only dumping from one client, there are a lot of 
filesystems that are large compared to the tape size and there isn't 
enough space to keep all the dumps on the holding disk (HD is size of 
one tape, total space to be dumped is several tapes worth). I've been 
using maxdumps=1 so turning that up might help (but I'm afraid of the 
load it will place on the client) as well as playing with dumporder. 
Increasing the HD size might also help :-)

I was kind of hoping that the dumper would be smart enough to dump in an 
optimal order (the 40GB space, then one of the 55's, then the other 40GB 
and then the remaining 55GB in my example from below). Presumably amanda 
has all the relevant info since it does a planning stage first.  But 
while this may be simple in the case of one client and no compression it 
could be tricky with multiple clients, differing bandwidths, compression 
 etc.

According to Paul, I can't avoid the fact that amanda will attempt to 
write a file to tape even when it knows there isn't enough space (if 
compression is off this should be very clear). A closer look at the man 
page confirms this. Oh well.

Thanks for the responses!
Chris
Paul Bijnens wrote:
Chris Loken wrote:
Apologies if this has come up before. Couldn't find anything relevant. 
I'm talking about level 0 dumps (not incrementals). Using ait-3 tapes, 
GNUTAR (not linux "dump"), a tape library, 104GB holding disk  and 
setting runtapes greater than 1 (I need to dump several tapes-worth of 
data). NO software or hardware compression. amanda-2.4.4p1-0.3E

Amanda works great for me. These are efficiency questions - not 
critical problems.

a) When backing-up multiple disks from a single client, amanda seems 
to dump and then write them to tape in order of increasing size. This 
can be inefficient.  e.g. if you have filesystems that are 40GB, 40GB, 
55GB and 55GB and write to 100GB tapes, amanda will use 3 tapes when 
the backup would fit on 2 tapes.

b) amanda doesn't seem to know that it will run out of space on a 
tape. E.g. if I have two 65GB filesystems to dump to a 100GB tape, 
amanda will write the first and then start writing the second one and 
keep going until it hits the end of tape. It then puts in the next 
tape and rewrites which is great but a lot of time was wasted writing 
a file which wasn't going to fit on the first tape. I'm running 
amtapetype for myself right now but have been using a tape entry from 
the amanda faq that sets "length 108890 mbytes" (but there are two 
different ones for the same drive and tape; Sony SDX-700C and AIT-3).

So, my questions are 1) is what I've described above true (seems to be 
for me) and 2) is there anything I can do (easily) to change this 
behaviour?

Use the parameter "taperalgo largestfit".
But beware of some interaction with dumporder, inparallel and the
taperalgo:
http://marc.theaimsgroup.com/?l=amanda-users&m=106277015010167&w=2
About nr2: you can't change it, but it hurts less when the largestfit
algoritm is applied.

Re: amanda tape use strategy

[Paul Bijnens]

Frank Smith wrote:
How difficult would it be to implement a 'best fit' strategy?
The number of possibilities is extremely large, and
because the tapecapacity is only an approximation, not worth
the cost.  My AIT-1 tapes hit end of tape between 33 and 34 Gbyte.
And rarely there is one which suddenly hits EOT at 31.8 too.
One that would collect all the dumps to holdingdisk and then
fit them onto the tapes, the functional equivalent of running
amdump without a tape and then running amflush.  I realize
that it would require holdingdisk size to be greater than the
total size of the backup, but it would allow for the most
efficient use of tape.  
The "largestfit" performs very well on my setup:  3 of my 4
AIT-1 tapes are filled to nearly 100% on my archive run.
Last week one tape had even 100.6%.
  Also, does Amanda always try to write another dump image
to tape until it hits EOT, even if the size of the dump is
greater than the amount of remaining tape?
For people that use hardware compression, amanda does not know
what remains on the tape.  And even in my config, I had to
specify the tapelength a little pessimistic, because not all
tapes have equal capacity.  The capacity of a tape is also
affected by the status of the write head and tape quality:
the drive reads with one head what it just wrote, and, if it
concludes that the written part is not good enough, the tapedrive
writes the same block again, up to a number of times (much faster
than stopping, rewinding and rewriting); this uses tapecapacity
too.  That's why my tapelengths are less than the real length.
An improvement could be that, when nothing fits, amanda chooses
the smallest, which has the best chance of fitting on the uncertain
remaining inches (which in my case can be 1 Gbyte!).

--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***

Re: amanda tape use strategy

[Jon LaBadie]

On Tue, Dec 07, 2004 at 03:36:56PM -0600, Frank Smith wrote:
> --On Tuesday, December 07, 2004 14:13:43 -0500 Jon LaBadie <[EMAIL 
> PROTECTED]> wrote:
> > 
> > Note, this does not affect the order of dumping.  Other parameters
> > do that.  It only affects which, of the already completed dumps,
> > amanda will choose to tape next.  It will not wait around for
> > one that fits.
> 
> How difficult would it be to implement a 'best fit' strategy?
> One that would collect all the dumps to holdingdisk and then
> fit them onto the tapes, the functional equivalent of running
> amdump without a tape and then running amflush.  I realize
> that it would require holdingdisk size to be greater than the
> total size of the backup, but it would allow for the most
> efficient use of tape.  

I seem to recall that amanda had some way to say "only dump
this DLE at such and such a time".  If dumper can have delays,
maybe taper can too.  Then maybe you could say don't start
taping until ...  That way a number of DLEs would have been
completed and the largest fit taperalgo would have several
to choose from.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: amanda tape use strategy

[Frank Smith]

--On Tuesday, December 07, 2004 14:13:43 -0500 Jon LaBadie <[EMAIL PROTECTED]> 
wrote:
> 
> Note, this does not affect the order of dumping.  Other parameters
> do that.  It only affects which, of the already completed dumps,
> amanda will choose to tape next.  It will not wait around for
> one that fits.

How difficult would it be to implement a 'best fit' strategy?
One that would collect all the dumps to holdingdisk and then
fit them onto the tapes, the functional equivalent of running
amdump without a tape and then running amflush.  I realize
that it would require holdingdisk size to be greater than the
total size of the backup, but it would allow for the most
efficient use of tape.  
  Also, does Amanda always try to write another dump image
to tape until it hits EOT, even if the size of the dump is
greater than the amount of remaining tape?

Frank

-- 
Frank Smith  [EMAIL PROTECTED]
Sr. Systems Administrator   Voice: 512-374-4673
Hoover's Online   Fax: 512-374-4501

Re: amanda tape use strategy

[Jon LaBadie]

On Tue, Dec 07, 2004 at 12:35:29PM -0500, Chris Loken wrote:
> 
> Apologies if this has come up before. Couldn't find anything relevant. 
> I'm talking about level 0 dumps (not incrementals). Using ait-3 tapes, 
> GNUTAR (not linux "dump"), a tape library, 104GB holding disk  and 
> setting runtapes greater than 1 (I need to dump several tapes-worth of 
> data). NO software or hardware compression. amanda-2.4.4p1-0.3E
> 
> Amanda works great for me. These are efficiency questions - not critical 
> problems.
> 
> a) When backing-up multiple disks from a single client, amanda seems to 
> dump and then write them to tape in order of increasing size. This can 
> be inefficient.  e.g. if you have filesystems that are 40GB, 40GB, 55GB 
> and 55GB and write to 100GB tapes, amanda will use 3 tapes when the 
> backup would fit on 2 tapes.
> 
> b) amanda doesn't seem to know that it will run out of space on a tape. 
> E.g. if I have two 65GB filesystems to dump to a 100GB tape, amanda will 
> write the first and then start writing the second one and keep going 
> until it hits the end of tape. It then puts in the next tape and 
> rewrites which is great but a lot of time was wasted writing a file 
> which wasn't going to fit on the first tape. I'm running amtapetype for 
> myself right now but have been using a tape entry from the amanda faq 
> that sets "length 108890 mbytes" (but there are two different ones for 
> the same drive and tape; Sony SDX-700C and AIT-3).
> 
> So, my questions are 1) is what I've described above true (seems to be 
> for me) and 2) is there anything I can do (easily) to change this behaviour?
> 

Checking the setting of your "taperalgo" parameter is in order.
A suitable setting for what you want might be "largest dump
in the holding disk that should fit the remaining tape space".
Probably named something like "largestfit", but DHMTI. :)

Note, this does not affect the order of dumping.  Other parameters
do that.  It only affects which, of the already completed dumps,
amanda will choose to tape next.  It will not wait around for
one that fits.
-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: amanda tape use strategy

[Paul Bijnens]

Chris Loken wrote:
Apologies if this has come up before. Couldn't find anything relevant. 
I'm talking about level 0 dumps (not incrementals). Using ait-3 tapes, 
GNUTAR (not linux "dump"), a tape library, 104GB holding disk  and 
setting runtapes greater than 1 (I need to dump several tapes-worth of 
data). NO software or hardware compression. amanda-2.4.4p1-0.3E

Amanda works great for me. These are efficiency questions - not critical 
problems.

a) When backing-up multiple disks from a single client, amanda seems to 
dump and then write them to tape in order of increasing size. This can 
be inefficient.  e.g. if you have filesystems that are 40GB, 40GB, 55GB 
and 55GB and write to 100GB tapes, amanda will use 3 tapes when the 
backup would fit on 2 tapes.

b) amanda doesn't seem to know that it will run out of space on a tape. 
E.g. if I have two 65GB filesystems to dump to a 100GB tape, amanda will 
write the first and then start writing the second one and keep going 
until it hits the end of tape. It then puts in the next tape and 
rewrites which is great but a lot of time was wasted writing a file 
which wasn't going to fit on the first tape. I'm running amtapetype for 
myself right now but have been using a tape entry from the amanda faq 
that sets "length 108890 mbytes" (but there are two different ones for 
the same drive and tape; Sony SDX-700C and AIT-3).

So, my questions are 1) is what I've described above true (seems to be 
for me) and 2) is there anything I can do (easily) to change this 
behaviour?
Use the parameter "taperalgo largestfit".
But beware of some interaction with dumporder, inparallel and the
taperalgo:
http://marc.theaimsgroup.com/?l=amanda-users&m=106277015010167&w=2
About nr2: you can't change it, but it hurts less when the largestfit
algoritm is applied.
--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***

amanda tape use strategy

[Chris Loken]

Apologies if this has come up before. Couldn't find anything relevant. 
I'm talking about level 0 dumps (not incrementals). Using ait-3 tapes, 
GNUTAR (not linux "dump"), a tape library, 104GB holding disk  and 
setting runtapes greater than 1 (I need to dump several tapes-worth of 
data). NO software or hardware compression. amanda-2.4.4p1-0.3E

Amanda works great for me. These are efficiency questions - not critical 
problems.

a) When backing-up multiple disks from a single client, amanda seems to 
dump and then write them to tape in order of increasing size. This can 
be inefficient.  e.g. if you have filesystems that are 40GB, 40GB, 55GB 
and 55GB and write to 100GB tapes, amanda will use 3 tapes when the 
backup would fit on 2 tapes.

b) amanda doesn't seem to know that it will run out of space on a tape. 
E.g. if I have two 65GB filesystems to dump to a 100GB tape, amanda will 
write the first and then start writing the second one and keep going 
until it hits the end of tape. It then puts in the next tape and 
rewrites which is great but a lot of time was wasted writing a file 
which wasn't going to fit on the first tape. I'm running amtapetype for 
myself right now but have been using a tape entry from the amanda faq 
that sets "length 108890 mbytes" (but there are two different ones for 
the same drive and tape; Sony SDX-700C and AIT-3).

So, my questions are 1) is what I've described above true (seems to be 
for me) and 2) is there anything I can do (easily) to change this behaviour?

   Thanks,
 Chris

Re: Laptop Backup Strategy

[Jonathan Dill]

I'm beginning to investigate using external USB 2.0 / Firewire drives to 
do backups of some systems.  The idea is that we could keep 2-3 spare 
PCs around, then if your computer is toast, we just ship it out for 
repairs, plug your external drive into one of the spare PCs, and rebuild 
the system from the external drive.

--jonathan

Re: Laptop Backup Strategy

[Geert Uytterhoeven]

On Wed, 27 Oct 2004, Richard Karnesky wrote:
> 3. Force users to backup to a server

If you have sufficient spare diskspace, let the user rsync from time to time to
this space. Then let Amanda backup this space, together with the other machines
that are always available.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [EMAIL PROTECTED]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Laptop Backup Strategy

[Richard Karnesky]

I'd like to add several laptops to my current configuration.  I was 
wondering what solutions people used for "transient" hosts.  Ideally, the 
main backup configuration (which handles workstations) would run in the 
evening.  The downside of this is that laptops are VERY rarely (if ever) on 
the network at this time.  So some of the possible solutions:

1. Force users to occasionally leave their laptops on the network over 
night.  Probably not going to happen.

2. Setup a second config, to be executed for only transient hosts during 
the work day.
Is there a way to keep this on a holding disk & trick amanda into dumping 
it with the first config, so that I only use a single set of tapes?  We 
have tapes that are somewhat large & somewhat expensive & it would be 
overly wastefull to use 2 tapes/day when 1 could be sufficient.  It would 
also be a shame to maintain two indexes & figure out which to restore from.

3. Force users to backup to a server
What do Windows & OS X users use for this?  BackupPC?  A tar or some 
proprietary backup format saved to a samba share?

I was wondering which of these solutions (or others that I didn't think of 
and couldn't find in the list archives) people use & if there is any 
additional advice that could be useful.

Thanks,
Richard Karnesky

Re: Bare-metal restore strategy

[Paul Bijnens]

John Bossert wrote:
Are there "best practices" for a bare-metal restore with amanda?  In the 
past, if I needed to recover a machine, I'd get the tapes I needed (that 
I'd produced with "dump" with the machine quiesced); boot the machine 
from CD (or tape - yes, I'm that old...); newfs the disk partitions as 
necessary; and restore the partitions with "restore"
Perfect.
An obvious deficiency in my current ruleset, for example, is the 
presumption that gtar would be available on my boot CD, which is not the 
case (at least for Solaris9.)
When my amanda server used to be a solaris machine, I made the
backups of the system partitions of the amanda-server with ufsdump.
I had (still have actually) a separate partition with all optional
software, like amanda and gnutar.  That filesystem is mirrored
(with rsync nightly) on, another machine.
The rest of the network uses read-only nfs mounts from those two
machines.
The amanda server used to be one of those machines, and that
partition was back(ed?)up(ed?) with ufsdump too. The partition
on the mirror machine was backup up with gnutar.
In case of a amanda-server crash, I could restore the needed
partitions using ufsrestore.  In case another machine crashed,
it is sufficient to nfs-mount one of the two software partitions,
after booting from a CD.
I never had to do a bare-metal restore on Solaris (yet).
I once experimented with booting from a Solaris CD, activating
network, and nfs-mount the software.  The test worked.  I'm sure
I found something about this on the WWW.  Can't find it back right now.
(!! Things to do next week, and print it out too :-) !! )
On Linux, you can always boot from e.g. a Knoppix CD, and have all
tools available.  A bare metal recovery worked out fine.  (One hour
to diagnose that the disk was indeed irrecoverably damaged + one hour
to install spare disk, boot from CD, partition, format, amrestore
over the network, and final reboot.)
I'm confident that I can recover one or more individual files with my 
current backup, but I'd like to have more of a cookbook process for 
disaster planning.

Any example amanda.conf's (and while I'm asking,) disaster recovery 
checklists/cookbooks would be much appreciated.  I'm looking to put 
together something that a third-shift operator could follow...

Thanks in advance...

--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***

Bare-metal restore strategy

[John Bossert]

Just getting started with Amanda and am successfully backing up 
partitions on both my amanda server and a separate client with gtar - so 
far so good.

Are there "best practices" for a bare-metal restore with amanda?  In the 
past, if I needed to recover a machine, I'd get the tapes I needed (that 
I'd produced with "dump" with the machine quiesced); boot the machine 
from CD (or tape - yes, I'm that old...); newfs the disk partitions as 
necessary; and restore the partitions with "restore"

An obvious deficiency in my current ruleset, for example, is the 
presumption that gtar would be available on my boot CD, which is not the 
case (at least for Solaris9.)

I'm confident that I can recover one or more individual files with my 
current backup, but I'd like to have more of a cookbook process for 
disaster planning.

Any example amanda.conf's (and while I'm asking,) disaster recovery 
checklists/cookbooks would be much appreciated.  I'm looking to put 
together something that a third-shift operator could follow...

Thanks in advance...

Re: Amanda backup strategy thoughts, full and incremental backups to tape once a week.

[pll+amanda]

In a message dated: Tue, 06 Apr 2004 12:12:05 BST
"William Hargrove" said:

>I was thinking I could do the incremental backups to a hard disk area on the tape
>server each night and then run Amanda once a week to archive the full backups plus
>the HDD incremental ones.

I did this once, way back when all I had was a single DLT drive and 
no autoloader.  I would just leave the drive empty all week, and on 
Friday mornings I would pop that week's tape in then amflush 
everything to tape.  When that was done, I would place the weekend 
tape in the drive and the full dumps on Saturday night would go 
direct to tape, or, if I forgot, end up on the holding disk for 
amflushing Monday morning.

This worked quite well IMO for the small environment I had at the 
time.

HTH,
-- 
Seeya,
Paul

GPG Key fingerprint = 1660 FECC 5D21 D286 F853  E808 BB07 9239 53F1 28EE

 If you're not having fun, you're not doing it right!

Re: Amanda backup strategy thoughts, full and incremental backups to tape once a week.

[Jonathan Dill]

Here's a strategy that I implemented about a month ago that is working 
pretty well so far:

1. run amdump every night to large RAID w/o tape, mix of full and incr
2. run script to separate images to amanda-incr and amanda-full
3. when amanda-full exceeds size of tape, run amflush
4. when RAID approaches capacity, flush some incrs to tape
Since it's a RAID, there is less chance it will go to crap than a 
regular disk partition, but it is still possible.  And if it does, I 
still have the full dumps on tape to fall back on.

To give you some gauge of capacities, I'm backing up about 300 GB of 
disks, have ~600 GB RAID5 (on a Snap 4500) and am using 100/200 GB LTO 
(Ultrium-1) tapes.  So far, it's mostly web servers, but I haven't moved 
everything to the new backup cycle yet.  I have flushed 2 tapes worth of 
full dumps, and accumulated about 167 GB of incrementals that are still 
on the RAID.

The goal and idea is that > 90% of the time, I will only need to load 
one tape to do a restore, and the rest will come out of holding files.  
Even better, there is a good chance that the tape that I need will be 
the last one that I flushed, so I won't have to change the tape.

/snap/amanda-hd is the directory that amanda.conf looks for, which is 
actually a symbolic link to /snap/amanda-full.  Therefore, I just move 
the incremental dumps from amanda-full to amanda-incr, both actually on 
the same filesystem (a simple directory rename in the filesystem inode 
table).  If I need to flush incrementals, I just change the link 
amanda-hd -> amanda-incr and run the flush (this could be done more 
elegantly with a patch, but this is so easy, so why bother?)  In the 
cron job, I make sure amanda-hd -> amanda-full before amdump runs.

I'm using NFSv3 to XFS filesystem through a dedicated 100baseTX FD 
connection, so I'm using a chunksize of ~600 GB, and that makes the 
rename script very simple since I don't have to worry about any backup 
getting split into chunks.  AFAIK NFSv3 and XFS both support 64-bit 
addressing, so I shouldn't run into any filesize limits.

--jonathan

Re: Amanda backup strategy thoughts, full and incremental backups to tape once a week.

[Jon LaBadie]

On Tue, Apr 06, 2004 at 12:12:05PM +0100, William Hargrove wrote:
> Hello All,
> 
> I would like to ask a general Amanda backup 'strategy' question here, given
> by backup 'goals' below. I have a configured and working Amanda set-up but I
> want to try and fit a backup policy that I believe is best suited to our
> site around Amanda.
> 
> I have an 8-tape (DLT 35/70gb each) auto changer library linked to Amanda
> via mtx, this works well.
> 
> I want to do a full backup of various file systems on a remote host once a
> week and then, ideally, incremental backups once a day to that same tape. At
> the end of a week's backup, the tape should switch. One week's backup is far
> less than a single tape's capacity.
> 
> I would like to be able to hold 8 weeks worth of backups on tape, that is
> one tape a week and then reuse the oldest tape.
> 
> I'm not sure this is possible with Amanda.. as I believe I can't append the
> daily incremental backups onto the same tape as the full backups. Is it the
> case that if I want to run Amanda every day that I would also have to use a
> tape every day, thus meaning the tapes in the library are all used after a
> week. I'm fixed in the number of tapes I can use and can't add/replace them,
> so they would get reused after a week and wouldn't provide much of a 'backup
> horizon'.
> 
> If this isn't possible does anyone have some suggestions? I was thinking I
> could do the incremental backups to a hard disk area on the tape server each
> night and then run Amanda once a week to archive the full backups plus the
> HDD incremental ones.
> 


Where is that "Top Ten Questions" document when you need it?

Amanda will not reuse a tape for a new run of amdump, fulls or
incrementals, until it cycles through "tapecycle" number of
tapes.  So you are correct, amanda will NOT append a second
amdump to a tape.  Amanda only overwrites tapes!

However, an amflush can overwrite a tape with multiple dumps
from the holding disk.  Thus collect as many amdumps as you
want with the tape offline.  Then amflush when you want to
tape them.

-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Amanda backup strategy thoughts, full and incremental backups to tape once a week.

[William Hargrove]

Hello All,

I would like to ask a general Amanda backup 'strategy' question here, given
by backup 'goals' below. I have a configured and working Amanda set-up but I
want to try and fit a backup policy that I believe is best suited to our
site around Amanda.

I have an 8-tape (DLT 35/70gb each) auto changer library linked to Amanda
via mtx, this works well.

I want to do a full backup of various file systems on a remote host once a
week and then, ideally, incremental backups once a day to that same tape. At
the end of a week's backup, the tape should switch. One week's backup is far
less than a single tape's capacity.

I would like to be able to hold 8 weeks worth of backups on tape, that is
one tape a week and then reuse the oldest tape.

I'm not sure this is possible with Amanda.. as I believe I can't append the
daily incremental backups onto the same tape as the full backups. Is it the
case that if I want to run Amanda every day that I would also have to use a
tape every day, thus meaning the tapes in the library are all used after a
week. I'm fixed in the number of tapes I can use and can't add/replace them,
so they would get reused after a week and wouldn't provide much of a 'backup
horizon'.

If this isn't possible does anyone have some suggestions? I was thinking I
could do the incremental backups to a hard disk area on the tape server each
night and then run Amanda once a week to archive the full backups plus the
HDD incremental ones.

Many thanks in advance for everyone's thoughts.

William.

Re: strategy

[Paul Bijnens]

Bruce Fletcher wrote:
On Tuesday, June 3, 2003, at 12:57  AM, Paul Bijnens wrote:
The global dumpcycle can be overruled for some specific dumptypes.
So that you can have e.g. a global dumpcycle of 7 days, but a few
important DLE's can specify a dumpcycle 0 (= full backups allways).
Runspercycle and tapecycle can only be global.
I assume there is nothing special about a DLE dumpcycle of 0, is there?
I.e. a DLE dumpcycle of 3 won't somehow conflict with a (global)
runspercycle of 5?
Nothing special at all.


And how does one cope as data storage requirements start to increase?
My understanding is that you can do the following:
- if necessary, label a few more tapes and increase the tapecycle
- increase the global and DLE dumpcycles as required
Yes.


There are no other steps involved?  If I have that right, it just
seems too easy.
Not everything has to be complicated :-)



--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***