Re: to compress or not to compress ???

[Michael D. Schleif]

Also sprach Paul Bijnens (Fri 04 Jul 02003 at 10:51:47AM +0200):
> 

> In docs/RESTORE, you can find all the commands you need to use
> an amanda tape without amanda software.

That is an excellent document!  I'm sorry that I bothered the list
before finding and reading it . . .

-- 
Best Regards,

mds
mds resource
877.596.8237
-
Dare to fix things before they break . . .
-
Our capacity for understanding is inversely proportional to how much
we think we know.  The more I know, the more I know I don't know . . .
--


pgp0.pgp
Description: PGP signature

Re: to compress or not to compress ???

[Paul Bijnens]

Michael D. Schleif wrote:
Also sprach Joshua Baker-LePain (Thu 03 Jul 02003 at 05:41:32PM -0400):
On Thu, 3 Jul 2003 at 4:20pm, Michael D. Schleif wrote
Also, using only amrestore, is it possible to get at individual
files/directories, or is it only a matter of restoring the entire
dump/tarball?
Yes, depending.  If you use dump, you can usually pipe amrestore to 
'restore -i', the interactive restore, which will let you pick and choose.  
If you use tar, you can do a 'tar t' to get a table of contents, and then 
'tar x myfile ./mydir/myfile2' to get  particular files.


This is where I'm getting lost:

# sudo mt -f /dev/nst0 rewind

# sudo tar tvf /dev/nst0
tar: /dev/nst0: Cannot read: Cannot allocate memory
tar: At beginning of tape, quitting now
tar: Error is not recoverable: exiting now
What do you think?

First you have to realize that on a tape you can put any
sequence of bytes.  To read and write those bytes from/to
tape, it's also important that you do that with a suitable
blocksize. In theory you could read the tape with "cat", but
cat does not have a way to specify the read blocksize.  That's
why we use the "dd" command.
If you read a tape, you have to do it with a blocksize equal
or greater than the blocksize used when written.
Amanda writes your tapes with a 32k blocksize (you can
change the default blocksize in recent versions -- know your
config!)
A file on tape is any sequence of bytes.  You can put many
files on tape sequentially, each separated with a filemarker.
After the last file is an end of data marker.
The format of an amanda is as follows:
The first file is the amanda header. Then follow the different
dump images.
The first block (32k in de the default setup) contains in simple
ascii the description of what follows/
The header looks like:
  AMANDA: TAPESTART DATE 20030702 TAPE Test-01
  ^L
  [EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED] nulls to fill up the rest of 
the blocksize
The next files are each contain a header of 1 block, that contains
the description of the next bytes:
 AMANDA: FILE 20030702 sunny /space lev 0 comp N program 
/usr/local/bin/amgtar
 To restore, position tape at start of file and run:
dd if= bs=32k skip=1 | /usr/local/bin/amgtar -f... -
 ^L
 [EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@ascii nulls to fill up the rest of the block

As you can see, amanda explains even what command you can use to
read it:  it contains a tar file, and to read it skip over the 32k
header and feed those bytes to tar.
What you should be doing was:
  mt -f /dev/nst0 rewind
  mt -f /dev/nst0 fsf 1 # skip over the tape label
  dd if=/dev/nst0 bs=32k skip=1 | tar -tvf -
If you had used software compression, add the "z" flag to gnutar.

In docs/RESTORE, you can find all the commands you need to use
an amanda tape without amanda software.
--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***

Re: to compress or not to compress ???

[Paul Bijnens]

Michael D. Schleif wrote:
I am curious as to a procedure of manually -- without amanda -- viewing
and restoring from amanda-written tapes . . .
There is a good explanation of all the possible restore scenario's
in docs/RESTORE.
--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***

Re: to compress or not to compress ???

[Paul Bijnens]

Michael D. Schleif wrote:
Also, what is the best way to turn off compression?
...
Will this persist across power cycles?  Will previously hardware
compressed tapes turn hardware compression back on?
See also:

 http://groups.yahoo.com/group/amanda-users/message/44453
 http://groups.yahoo.com/group/amanda-users/message/43705
(I have this strange feeling of deja-vu :-) )

--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***

Re: to compress or not to compress ???

[Gene Heskett]

On Thursday 03 July 2003 18:03, bao wrote:
>Joshua Baker-LePain wrote:
>>On Thu, 3 Jul 2003 at 4:20pm, Michael D. Schleif wrote
>>
>>>Am I right that amrecover is useless *without* an index?
>>
>>Yep.
>
>No argument about it.
>But, I don't keep index files along with the backup image on tape.
>Before running amrecover,
>I would run a script to extract and recreate the index files and use
>them to run amrecover.
>Are there any drawbacks to that scheme ???

Yes.  If the indices do make it into the backup tape (I've had it 
miss-fire here and have taken other means to assure I have an up2date 
backup of them on the same tape here), they will still be a day old 
compared to the tapes actual contents.

>>>Also, using only amrestore, is it possible to get at individual
>>>files/directories, or is it only a matter of restoring the entire
>>>dump/tarball?
>>
>>Yes, depending.  If you use dump, you can usually pipe amrestore to
>>'restore -i', the interactive restore, which will let you pick and
>> choose. If you use tar, you can do a 'tar t' to get a table of
>> contents, and then 'tar x myfile ./mydir/myfile2' to get 
>> particular files.

-- 
Cheers, Gene
AMD [EMAIL PROTECTED] 320M
[EMAIL PROTECTED]  512M
99.26% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2003 by Maurice Eugene Heskett, all rights reserved.

Re: to compress or not to compress ???

[Gene Heskett]

On Thursday 03 July 2003 15:10, Michael D. Schleif wrote:
>Also sprach Gene Heskett (Thu 03 Jul 02003 at 02:51:39PM -0400):
>> On Thursday 03 July 2003 11:42, Michael D. Schleif wrote:
>> >Yes, I am learning -- at the expense of many questions ;>
>> >
>> >First, a brief overview:
>> >
>> >I have five (5) Linux servers, totaling ~50 Gb used diskspace,
>> > divided roughly even across all five.
>> >
>> >I have several DAT tape drives, the largest of which is an HP
>> > DDS-3. I have twelve (12) DDS-3 tapes, and twenty (20) DDS-2
>> > tapes, as well as several cleaning tapes.
>> >
>> >So far, I have configured:
>> >
>> >dumpcycle 7
>> >runspercycle  7
>> >runtapes  1
>>
>> You left out tapecycle, which is the number of tapes in the
>> rotation pool, in this case it should be not less than 15.
>
>Wouldn't that be eight (8)?
No, not if you don't want to be overwriting the only full you have with one that might 
just fail, leaving you with no full to anchor the recovery to.
>
>   runspercycle * runtapes + 1
>
Most of us consider that to be 2*runspercycle*runtapes+1

>
>
>> One thing to be aware of is that a tape, once written in the
>> compressed mode, remembers that, and will overwrite your choices
>> unless you go to a rather detailed method of removing the
>> compressed flags.
>
>How do I do this?
>
>These tape drives have all used compression, and many of these tapes
>have been used once or twice.  So, it looks like I will *not* use
>hardware compression, and I want to reap all of the benefits of that
>strategy.
>
Clip this script and adjust to suit.
--
#!/bin/sh
if [ `whoami` != 'root' ]; then
echo
echo "! Warning !!"
echo "fixcomp needs to be run by the user root,"
echo "else the perms on some commands will be denied."
echo
exit 1
fi

# blatently stolen from the bash manual
NUMBERS="0 1 2 3" # Increase or decrease to match the number of slots
# in your magazine, mine holds 4 DDS2 tapes
for number in $NUMBERS
do
echo "amtape DailySet1 slot "$number
su amanda -c 'amtape DailySet1 slot '$number
mt -f /dev/nst0 rewind
dd if=/dev/st0 of=./scratch bs=64K count=1
ls -l ./scratch # see how big the label is
mt -f /dev/nst0 compression off
mt -f /dev/nst0 defcompression -1
# write it back, with fixed 32k block in case the read
# above was short
dd if=./scratch of=/dev/nst0 bs=32K conv=sync
mt -f /dev/nst0 tell
# This seems to get rid of compressed headers
echo "forcing buffer flush with an 4+ meg write to tape # "$number
dd if=/dev/zero bs=32K count=130 of=/dev/st0
echo "Now reading the label"
# display it on the console to verify its undamaged
dd if=/dev/st0 bs=32K
mt -f /dev/st0 status
done
exit 0

This is probably not the only way to do it, but it seems to 
work here just fine.  This does however, assume that you 
have located the jumper or dipswitch that sets the drives default
compression status and set it to the off position.

>Also, what is the best way to turn off compression?
>
># sudo mt-gnu -f /dev/nst0 datcompression
>Compression on.
>Compression capable.
>Decompression capable.
>
># sudo mt-gnu -f /dev/nst0 datcompression 0
>Compression off.
>Compression capable.
>Decompression capable.
>
># sudo mt-gnu -f /dev/nst0 datcompression
>Compression off.
>Compression capable.
>Decompression capable.
>
>Will this persist across power cycles?

No, on powerup the drive will revert to the dipswitch settings.

>  Will previously hardware
>compressed tapes turn hardware compression back on?

Yes, as soon as the drive recognizes the tape.  The above 
script should fix that.  In my tests here, if the extra 
4 meg write isn't done, then the tape will be uncompressed, 
but the header still will be.  The extra write to force the
buffer flush seems to fix that.

-- 
Cheers, Gene
AMD [EMAIL PROTECTED] 320M
[EMAIL PROTECTED]  512M
99.26% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2003 by Maurice Eugene Heskett, all rights reserved.

Re: to compress or not to compress ???

[Michael D. Schleif]

Also sprach Joshua Baker-LePain (Thu 03 Jul 02003 at 05:41:32PM -0400):
> On Thu, 3 Jul 2003 at 4:20pm, Michael D. Schleif wrote
> 
> > Am I right that amrecover is useless *without* an index?
> 
> Yep.
> 
> > Also, using only amrestore, is it possible to get at individual
> > files/directories, or is it only a matter of restoring the entire
> > dump/tarball?
> 
> Yes, depending.  If you use dump, you can usually pipe amrestore to 
> 'restore -i', the interactive restore, which will let you pick and choose.  
> If you use tar, you can do a 'tar t' to get a table of contents, and then 
> 'tar x myfile ./mydir/myfile2' to get  particular files.

This is where I'm getting lost:

# sudo mt -f /dev/nst0 rewind

# sudo tar tvf /dev/nst0
tar: /dev/nst0: Cannot read: Cannot allocate memory
tar: At beginning of tape, quitting now
tar: Error is not recoverable: exiting now

What do you think?

-- 
Best Regards,

mds
mds resource
877.596.8237
-
Dare to fix things before they break . . .
-
Our capacity for understanding is inversely proportional to how much
we think we know.  The more I know, the more I know I don't know . . .
--


pgp0.pgp
Description: PGP signature

Re: to compress or not to compress ???

[bao]

Joshua Baker-LePain wrote:

On Thu, 3 Jul 2003 at 4:20pm, Michael D. Schleif wrote

 

Am I right that amrecover is useless *without* an index?
   

Yep.
 

No argument about it.
But, I don't keep index files along with the backup image on tape. 
Before running amrecover,
I would run a script to extract and recreate the index files and use 
them to run amrecover.
Are there any drawbacks to that scheme ???

 

Also, using only amrestore, is it possible to get at individual
files/directories, or is it only a matter of restoring the entire
dump/tarball?
   

Yes, depending.  If you use dump, you can usually pipe amrestore to 
'restore -i', the interactive restore, which will let you pick and choose.  
If you use tar, you can do a 'tar t' to get a table of contents, and then 
'tar x myfile ./mydir/myfile2' to get  particular files.

 

Re: to compress or not to compress ???

[Joshua Baker-LePain]

On Thu, 3 Jul 2003 at 4:20pm, Michael D. Schleif wrote

> Am I right that amrecover is useless *without* an index?

Yep.

> Also, using only amrestore, is it possible to get at individual
> files/directories, or is it only a matter of restoring the entire
> dump/tarball?

Yes, depending.  If you use dump, you can usually pipe amrestore to 
'restore -i', the interactive restore, which will let you pick and choose.  
If you use tar, you can do a 'tar t' to get a table of contents, and then 
'tar x myfile ./mydir/myfile2' to get  particular files.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: to compress or not to compress ???

[Michael D. Schleif]

Also sprach George Kelbley (Thu 03 Jul 02003 at 02:41:12PM -0600):
> restoring w/o amrestore can be done with dd, gzip, tar, and so on, way
> messy compared to amrecover (or amrestore) but its possible.  That's one
> of the plus's to amanda.
> 
> Kurt Yoder wrote:
> > 
> > Assuming you're running Linux, all you need is some form of Linux
> > rescue disk. I've got a bunch of Debian installer cd's lying around
> > and have used them for similar purposes before. Probably most Linux
> > installer cd's can be used like this, and I'm quite certain
> > something like Knoppix would include all the utilities anyone would
> > need. Who knows, might even work to restore tapes made with
> > non-Linux amanda servers.
> > 
> > BTW, if restoring without an amanda server, doesn't one also need
> > amrestore? The other day I tried restoring an amanda image, and had
> > to run it through amrestore before I could untar it.
> > 
> > Eric Siegerman said:
> > > On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote:
> > >> Also, with hardware compression, I know I can restore the tape
> > >> without having
> > >> to worry about finding the right libraries and programs to do the
> > >> restore.
> > >
> > > True.  But one can work around that by backing up / uncompressed,
> > > and making sure it contains a (possibly statically linked) copy
> > > of gzip.  Hmmm, something to add to my to-do list :-(

Am I right that amrecover is useless *without* an index?

Also, using only amrestore, is it possible to get at individual
files/directories, or is it only a matter of restoring the entire
dump/tarball?

I am curious as to a procedure of manually -- without amanda -- viewing
and restoring from amanda-written tapes . . .

-- 
Best Regards,

mds
mds resource
877.596.8237
-
Dare to fix things before they break . . .
-
Our capacity for understanding is inversely proportional to how much
we think we know.  The more I know, the more I know I don't know . . .
--


pgp0.pgp
Description: PGP signature

Re: to compress or not to compress ???

[Kurt Yoder]

How do you recover an amanda dump file without amrestore? I had dd'd
one to disk and tried to untar it but got the message "not a tar
file". Once I ran it through amrestore I was able to untar it
though.

George Kelbley said:
> restoring w/o amrestore can be done with dd, gzip, tar, and so on,
> way
> messy compared to amrecover (or amrestore) but its possible.  That's
> one
> of the plus's to amanda.
>
> Kurt Yoder wrote:
>>
>> Assuming you're running Linux, all you need is some form of Linux
>> rescue disk. I've got a bunch of Debian installer cd's lying
>> around
>> and have used them for similar purposes before. Probably most
>> Linux
>> installer cd's can be used like this, and I'm quite certain
>> something like Knoppix would include all the utilities anyone
>> would
>> need. Who knows, might even work to restore tapes made with
>> non-Linux amanda servers.
>>
>> BTW, if restoring without an amanda server, doesn't one also need
>> amrestore? The other day I tried restoring an amanda image, and
>> had
>> to run it through amrestore before I could untar it.
>>
>> Eric Siegerman said:
>> > On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote:
>> >> Also, with hardware compression, I know I can restore the tape
>> >> without having
>> >> to worry about finding the right libraries and programs to do
>> the
>> >> restore.
>> >
>> > True.  But one can work around that by backing up /
>> uncompressed,
>> > and making sure it contains a (possibly statically linked) copy
>> > of gzip.  Hmmm, something to add to my to-do list :-(
>>
>> --
>> Kurt Yoder
>> Sport & Health network administrator
>
> --
> George KelbleySystem Support Group
> Computer Science Department   University of New Mexico
> 505-277-6502  Fax: 505-277-6927
>


-- 
Kurt Yoder
Sport & Health network administrator

Re: to compress or not to compress ???

[Joshua Baker-LePain]

On Thu, 3 Jul 2003 at 4:34pm, Kurt Yoder wrote

> Assuming you're running Linux, all you need is some form of Linux
> rescue disk. I've got a bunch of Debian installer cd's lying around
> and have used them for similar purposes before. Probably most Linux
> installer cd's can be used like this, and I'm quite certain
> something like Knoppix would include all the utilities anyone would
> need. Who knows, might even work to restore tapes made with
> non-Linux amanda servers.
> 
> BTW, if restoring without an amanda server, doesn't one also need
> amrestore? The other day I tried restoring an amanda image, and had
> to run it through amrestore before I could untar it.

You can do with mt, dd, (gzip), and tar/restore.  You just need to make 
sure you strip off the 32KB amanda header on the front of each tape 
"file".

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: to compress or not to compress ???

[George Kelbley]

restoring w/o amrestore can be done with dd, gzip, tar, and so on, way
messy compared to amrecover (or amrestore) but its possible.  That's one
of the plus's to amanda.

Kurt Yoder wrote:
> 
> Assuming you're running Linux, all you need is some form of Linux
> rescue disk. I've got a bunch of Debian installer cd's lying around
> and have used them for similar purposes before. Probably most Linux
> installer cd's can be used like this, and I'm quite certain
> something like Knoppix would include all the utilities anyone would
> need. Who knows, might even work to restore tapes made with
> non-Linux amanda servers.
> 
> BTW, if restoring without an amanda server, doesn't one also need
> amrestore? The other day I tried restoring an amanda image, and had
> to run it through amrestore before I could untar it.
> 
> Eric Siegerman said:
> > On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote:
> >> Also, with hardware compression, I know I can restore the tape
> >> without having
> >> to worry about finding the right libraries and programs to do the
> >> restore.
> >
> > True.  But one can work around that by backing up / uncompressed,
> > and making sure it contains a (possibly statically linked) copy
> > of gzip.  Hmmm, something to add to my to-do list :-(
> 
> --
> Kurt Yoder
> Sport & Health network administrator

-- 
George Kelbley  System Support Group
Computer Science Department University of New Mexico
505-277-6502Fax: 505-277-6927

Re: to compress or not to compress ???

[Kurt Yoder]

Michael D. Schleif said:
> Also, what is the best way to turn off compression?
>
> # sudo mt-gnu -f /dev/nst0 datcompression
> Compression on.
> Compression capable.
> Decompression capable.
>
> # sudo mt-gnu -f /dev/nst0 datcompression 0
> Compression off.
> Compression capable.
> Decompression capable.
>
> # sudo mt-gnu -f /dev/nst0 datcompression
> Compression off.
> Compression capable.
> Decompression capable.
>
> Will this persist across power cycles?  Will previously hardware
> compressed tapes turn hardware compression back on?

mt datcompression 0 will work fine for you. Just make sure you run
that command every time you boot. Put it in one of your boot
scripts. I'm using debian, and made a file called "nocompress" which
contains:

#!/bin/sh
#turn off tape drive hardware compression
/bin/mt -f /dev/nst0 datcompression 0

This file resides in /etc/rc.boot and is mode -rwxr-xr-x (755).

-- 
Kurt Yoder
Sport & Health network administrator

Re: to compress or not to compress ???

[Kurt Yoder]

Assuming you're running Linux, all you need is some form of Linux
rescue disk. I've got a bunch of Debian installer cd's lying around
and have used them for similar purposes before. Probably most Linux
installer cd's can be used like this, and I'm quite certain
something like Knoppix would include all the utilities anyone would
need. Who knows, might even work to restore tapes made with
non-Linux amanda servers.

BTW, if restoring without an amanda server, doesn't one also need
amrestore? The other day I tried restoring an amanda image, and had
to run it through amrestore before I could untar it.

Eric Siegerman said:
> On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote:
>> Also, with hardware compression, I know I can restore the tape
>> without having
>> to worry about finding the right libraries and programs to do the
>> restore.
>
> True.  But one can work around that by backing up / uncompressed,
> and making sure it contains a (possibly statically linked) copy
> of gzip.  Hmmm, something to add to my to-do list :-(


-- 
Kurt Yoder
Sport & Health network administrator

Re: to compress or not to compress ???

[Anthony A. D. Talltree]

>True.  But one can work around that by backing up / uncompressed,
>and making sure it contains a (possibly statically linked) copy
>of gzip.

... or just write a couple of copies of a CD with gzip and whatever else
you might need.

Re: to compress or not to compress ???

[Michael D. Schleif]

Also sprach Gene Heskett (Thu 03 Jul 02003 at 02:51:39PM -0400):
> On Thursday 03 July 2003 11:42, Michael D. Schleif wrote:
> >Yes, I am learning -- at the expense of many questions ;>
> >
> >First, a brief overview:
> >
> >I have five (5) Linux servers, totaling ~50 Gb used diskspace,
> > divided roughly even across all five.
> >
> >I have several DAT tape drives, the largest of which is an HP DDS-3.
> >  I have twelve (12) DDS-3 tapes, and twenty (20) DDS-2 tapes, as
> > well as several cleaning tapes.
> >
> >So far, I have configured:
> >
> >dumpcycle 7
> >runspercycle  7
> >runtapes  1
> >
> You left out tapecycle, which is the number of tapes in the rotation 
> pool, in this case it should be not less than 15.

Wouldn't that be eight (8)?

runspercycle * runtapes + 1


> One thing to be aware of is that a tape, once written in the 
> compressed mode, remembers that, and will overwrite your choices 
> unless you go to a rather detailed method of removing the compressed 
> flags.

How do I do this?

These tape drives have all used compression, and many of these tapes
have been used once or twice.  So, it looks like I will *not* use
hardware compression, and I want to reap all of the benefits of that
strategy.

Also, what is the best way to turn off compression?

# sudo mt-gnu -f /dev/nst0 datcompression
Compression on.
Compression capable.
Decompression capable.

# sudo mt-gnu -f /dev/nst0 datcompression 0
Compression off.
Compression capable.
Decompression capable.

# sudo mt-gnu -f /dev/nst0 datcompression
Compression off.
Compression capable.
Decompression capable.

Will this persist across power cycles?  Will previously hardware
compressed tapes turn hardware compression back on?

What do you think?

-- 
Best Regards,

mds
mds resource
877.596.8237
-
Dare to fix things before they break . . .
-
Our capacity for understanding is inversely proportional to how much
we think we know.  The more I know, the more I know I don't know . . .
--


pgp0.pgp
Description: PGP signature

Re: to compress or not to compress ???

[Gene Heskett]

On Thursday 03 July 2003 11:42, Michael D. Schleif wrote:
>Yes, I am learning -- at the expense of many questions ;>
>
>First, a brief overview:
>
>I have five (5) Linux servers, totaling ~50 Gb used diskspace,
> divided roughly even across all five.
>
>I have several DAT tape drives, the largest of which is an HP DDS-3.
>  I have twelve (12) DDS-3 tapes, and twenty (20) DDS-2 tapes, as
> well as several cleaning tapes.
>
>So far, I have configured:
>
>dumpcycle 7
>runspercycle  7
>runtapes  1
>
You left out tapecycle, which is the number of tapes in the rotation 
pool, in this case it should be not less than 15.

>I am studying _Using Amanda_ here:
>
>   
>
>I am confused about two (2) things:
>
>[1] Should I use hardware compression?

Not if you can help it, for the reasons I'll develop.
>
>There seem to be several schools of thought here.  I want to know
> how Amanda works with hardware compression?

Amanda can use hardware compression, but since the hardware compressor 
hides the true tape capacity from amanda, you must cheat on the 
tapetype size entry, often by significant amounts.

>  What are the
> advantages of using software compression?

Amanda can know quite well how much a tape can hold because it counts 
bytes of compressed data fed to the drive.  The tradeoff of course is 
cpu horsepower to do the compression.  In a client-sever world, the 
compression can be offloaded to the client, and several clients can 
be doing their compression in parallel, so its not as big a concern 
as it first appears.

>  What are the
> disadvantages of using *both* hardware and software compression?
>
With hardware smunching, amanda has no idea how much data has actually 
been written to the tape.  Sparse stuff can compress to maybe 1/2.6 
of its original size, but amanda doesn't have any way of knowing 
that.  OTOH, feed a bunch of tar.gz's, and .bz2's to that hardware 
compressor and it will get a tummy ache and make the output data 
stream as much as 15% bigger than the input was.
>
>[2] What are the optimal dumptypes for my situation?
>
>Yes, I have already struggled with and overcome dump vs. GNUTAR.  My
>first mistake was using comp-root and comp-user on localhost.
>
>Yes, I understand that Amanda can facilitate planning and scheduling
> full vs. incremental backups.
>
>However, I am concerned about developing a simple recovery strategy.
>  I am currently having problems with amrecover; but, I think that
> is due to short vs. FQDN usage -- so, I'll save that for another
> time.

Just make sure that your tar is at least 1.13-19, and prefereably 
1.13-25.  Indexes are fubared in earlier versions.

>I am running exclusively Debian woody on all systems.  I believe
> that I have a good working filesystem design.  I am on a fast
> network.
>
>Yes, I come from a traditional system administrator's backup
> mindset, and I do not want that to undermine Amanda's design.
>
>
>What do you think?

One thing to be aware of is that a tape, once written in the 
compressed mode, remembers that, and will overwrite your choices 
unless you go to a rather detailed method of removing the compressed 
flags.

-- 
Cheers, Gene
AMD [EMAIL PROTECTED] 320M
[EMAIL PROTECTED]  512M
99.26% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2003 by Maurice Eugene Heskett, all rights reserved.

Re: to compress or not to compress ???

[Super-User]

On Thu, Jul 03, 2003 at 02:18:26PM -0400, Eric Siegerman wrote:
> 
> - Better compression, probably.  Hardware compression is
>   typically some variant of LZ, isn't it?  I don't know how
>   gzip -1 (the default "compress-fast") compares with that, but
>   gzip -9 (the default "compress-best") does a lot better.
> 
>   Ok, here's one quickie far-from-representative test.  Sorted in
>   order of decreasing size, a largish, mostly-text file, and its
>   compression by compress, and by the several grades of gzip.
>   SizeCPU File
>   --- 
>   5560320   0 amanda-2.4.4.tar
>   2096458   0.88  amanda-2.4.4.tar.Z
>   1496904   0.68  amanda-2.4.4.tar.gz1
>   1227454   1.28  amanda-2.4.4.tar.gz6
>   1220934   2.01  amanda-2.4.4.tar.gz9

My tests have always shown similar results.
I wish we could do comprss-default (no -level option),
which is the same as -6, and get nearly the same compression
as -9 but with far less cpu.


-- 
Jon H. LaBadie  [EMAIL PROTECTED]
 JG Computing
 4455 Province Line Road(609) 252-0159
 Princeton, NJ  08540-4322  (609) 683-7220 (fax)

Re: to compress or not to compress ???

[Eric Siegerman]

On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote:
> Also, with hardware compression, I know I can restore the tape without having
> to worry about finding the right libraries and programs to do the restore.

True.  But one can work around that by backing up / uncompressed,
and making sure it contains a (possibly statically linked) copy
of gzip.  Hmmm, something to add to my to-do list :-(

--

|  | /\
|-_|/  >   Eric Siegerman, Toronto, Ont.[EMAIL PROTECTED]
|  |  /
When I came back around from the dark side, there in front of me would
be the landing area where the crew was, and the Earth, all in the view
of my window. I couldn't help but think that there in front of me was
all of humanity, except me.
- Michael Collins, Apollo 11 Command Module Pilot

Re: to compress or not to compress ???

[Eric Siegerman]

On Thu, Jul 03, 2003 at 11:57:22AM -0400, Kurt Yoder wrote:
> I prefer software compression personally:
> 
> -Amanda can make a more accurate estimate of how much tape is
> needed.  So if you know your tape is 20 GB, and your
> software-compressed dump files total 21 GB, you know they won't all
> fit. With hardware compression you just have to guess-timate

Conversely, if you have a DLE full of something that compresses
down to 20% of its original size, Amanda will know that there's
more room on the tape for other stuff.  With hardware
compression, Amanda won't know that that DLE compresses better
than your other ones.

- Better compression, probably.  Hardware compression is
  typically some variant of LZ, isn't it?  I don't know how
  gzip -1 (the default "compress-fast") compares with that, but
  gzip -9 (the default "compress-best") does a lot better.

  Ok, here's one quickie far-from-representative test.  Sorted in
  order of decreasing size, a largish, mostly-text file, and its
  compression by compress, and by the several grades of gzip.
  Size  CPU File
  ---   
  5560320   0   amanda-2.4.4.tar
  2096458   0.88amanda-2.4.4.tar.Z
  1496904   0.68amanda-2.4.4.tar.gz1
  1227454   1.28amanda-2.4.4.tar.gz6
  1220934   2.01amanda-2.4.4.tar.gz9

  This time, even gzip -1 beat LZ.  I don't know whether that's
  typical.

--

|  | /\
|-_|/  >   Eric Siegerman, Toronto, Ont.[EMAIL PROTECTED]
|  |  /
When I came back around from the dark side, there in front of me would
be the landing area where the crew was, and the Earth, all in the view
of my window. I couldn't help but think that there in front of me was
all of humanity, except me.
- Michael Collins, Apollo 11 Command Module Pilot

Re: to compress or not to compress ???

[Paul Bijnens]

Michael D. Schleif wrote:
[1] Should I use hardware compression?

There seem to be several schools of thought here.  I want to know how
Amanda works with hardware compression?  What are the advantages of
using software compression?  What are the disadvantages of using *both*
hardware and software compression?
See Kurt's answer.  May I add that using both hardware compression
and software compression at the same time  actually lowers the
capacity of your tapedrive (a DAT-tape, you said).
If you blindly impose the compression algorithm on an already
compressed data will enlarge that data.  Now much depends on the
algorithm.  DAT-tapes use something from the "compress" family.
Try:
 compress somefile
 cat somefile.Z | compress -v | wc -c
Gzip does much better, by implementing a way to indicate
that next few bytes are not compressed.
 cat somefile.Z | gzip -v | wc -c
The only drive I know without that problem is an LTO1 drive.
see:
  http://groups.yahoo.com/group/amanda-users/message/44455

[2] What are the optimal dumptypes for my situation?
I would go for gnutar, compress client fast, index, record, and
apropriate priority (user-data = higher, system = lower).
--
Paul Bijnens, XplanationTel  +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax  +32 16 397.512
http://www.xplanation.com/  email:  [EMAIL PROTECTED]
***
* I think I've got the hang of it now:  exit, ^D, ^C, ^\, ^Z, ^Q, F6, *
* quit,  ZZ, :q, :q!,  M-Z, ^X^C,  logoff, logout, close, bye,  /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt,  abort,  hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e,  kill -1 $$,  shutdown, *
* kill -9 1,  Alt-F4,  Ctrl-Alt-Del,  AltGr-NumLock,  Stop-A,  ...*
* ...  "Are you sure?"  ...   YES   ...   Phew ...   I'm out  *
***

RE: to compress or not to compress ???

[Joshua Baker-LePain]

On Thu, 3 Jul 2003 at 12:17pm, Ean Kingston wrote

> I Have the opposite viewpoint. I prefer hardware compression. It allows 
> me to offload processing required to the tape drive (instead of my 
> computers) Since some of my systems (including the backup server itself) 
> can be slow, this actually speeds things up for me. Also, with hardware 
> compression, I know I can restore the tape without having to worry about 
> finding the right libraries and programs to do the restore. Also (AFAIK) 
> you can't do remote compression with samba which I use for about half of 
> my backups.
> 
It really depends on your data.  I have filesystems which are completely 
uncompressable in software (and expand in hardware), and filesystems which 
compress to <20% of their original size.  Using software compressions lets 
me pick and choose which DLEs get compressed or not.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

RE: to compress or not to compress ???

[Ean Kingston]

I Have the opposite viewpoint. I prefer hardware compression. It allows me to offload 
processing required to the tape drive (instead of my computers) Since some of my 
systems (including the backup server itself) can be slow, this actually speeds things 
up for me. Also, with hardware compression, I know I can restore the tape without 
having to worry about finding the right libraries and programs to do the restore. Also 
(AFAIK) you can't do remote compression with samba which I use for about half of my 
backups.

On the down side, I need more holding disk and I have to guess at my tape capacity.

As for both, don't do it. That is a huge waste of cpu and tape.  The first compression 
algorithm will work fine. The second one will waste space with its headers and such 
since it probably won't be able to get the data any smaller than the software 
compression program did.

If you don't believe me, try to zip a gzip file. See if it gets any smaller.

> -Original Message-
> From: Kurt Yoder [mailto:[EMAIL PROTECTED]
> Sent: Thursday, July 03, 2003 11:57 AM
> To: Michael D. Schleif
> Cc: amanda mailing list
> Subject: Re: to compress or not to compress ???
> 
> 
> 
> Michael D. Schleif said:
> > [1] Should I use hardware compression?
> >
> > There seem to be several schools of thought here.  I want to know
> > how
> > Amanda works with hardware compression?  What are the advantages of
> > using software compression?  What are the disadvantages of using
> > *both*
> > hardware and software compression?
> 
> I prefer software compression personally:
> 
> -Amanda can make a more accurate estimate of how much tape is
> needed.  So if you know your tape is 20 GB, and your
> software-compressed dump files total 21 GB, you know they won't all
> fit. With hardware compression you just have to guess-timate
> 
> -Less bandwidth consumed if you do your compression on the client
> side (eg, before it comes to the tape server)
> 
> -Less disk space used on your holding disk
> 
> -If you back up to disks instead of tapes, hardware compression is
> not even an option
> 
> 
> The only drawback to software compression that I can see is the
> greater amount of cpu power consumed. For me, this is not really a
> problem, since my backups all happen in the wee hours when no-one is
> on my systems anyway.
> 
> -- 
> Kurt Yoder
> Sport & Health network administrator
> 
> 

Re: to compress or not to compress ???

[Kurt Yoder]

Michael D. Schleif said:
> [1] Should I use hardware compression?
>
> There seem to be several schools of thought here.  I want to know
> how
> Amanda works with hardware compression?  What are the advantages of
> using software compression?  What are the disadvantages of using
> *both*
> hardware and software compression?

I prefer software compression personally:

-Amanda can make a more accurate estimate of how much tape is
needed.  So if you know your tape is 20 GB, and your
software-compressed dump files total 21 GB, you know they won't all
fit. With hardware compression you just have to guess-timate

-Less bandwidth consumed if you do your compression on the client
side (eg, before it comes to the tape server)

-Less disk space used on your holding disk

-If you back up to disks instead of tapes, hardware compression is
not even an option


The only drawback to software compression that I can see is the
greater amount of cpu power consumed. For me, this is not really a
problem, since my backups all happen in the wee hours when no-one is
on my systems anyway.

-- 
Kurt Yoder
Sport & Health network administrator

to compress or not to compress ???

[Michael D. Schleif]

Yes, I am learning -- at the expense of many questions ;>

First, a brief overview:

I have five (5) Linux servers, totaling ~50 Gb used diskspace, divided
roughly even across all five.

I have several DAT tape drives, the largest of which is an HP DDS-3.  I
have twelve (12) DDS-3 tapes, and twenty (20) DDS-2 tapes, as well as
several cleaning tapes.

So far, I have configured:

dumpcycle 7
runspercycle  7
runtapes  1

I am studying _Using Amanda_ here:



I am confused about two (2) things:

[1] Should I use hardware compression?

There seem to be several schools of thought here.  I want to know how
Amanda works with hardware compression?  What are the advantages of
using software compression?  What are the disadvantages of using *both*
hardware and software compression?


[2] What are the optimal dumptypes for my situation?

Yes, I have already struggled with and overcome dump vs. GNUTAR.  My
first mistake was using comp-root and comp-user on localhost.

Yes, I understand that Amanda can facilitate planning and scheduling full
vs. incremental backups.

However, I am concerned about developing a simple recovery strategy.  I
am currently having problems with amrecover; but, I think that is due to
short vs. FQDN usage -- so, I'll save that for another time.

I am running exclusively Debian woody on all systems.  I believe that I
have a good working filesystem design.  I am on a fast network.

Yes, I come from a traditional system administrator's backup mindset,
and I do not want that to undermine Amanda's design.


What do you think?

-- 
Best Regards,

mds
mds resource
877.596.8237
-
Dare to fix things before they break . . .
-
Our capacity for understanding is inversely proportional to how much
we think we know.  The more I know, the more I know I don't know . . .
--


pgp0.pgp
Description: PGP signature