On Wednesday 07 July 2010, McGraw, Robert P wrote:
>> -----Original Message-----
>> From: owner-amanda-us...@amanda.org [mailto:owner-amanda-
>> us...@amanda.org] On Behalf Of Gene Heskett
>> Sent: Wednesday, July 07, 2010 11:33 AM
>> To: amanda-users@amanda.org
>> Subject: Re: Tape Usage Question
>>
>> On Wednesday 07 July 2010, McGraw, Robert P wrote:
>> >> -----Original Message-----
>> >> From: djmit...@gmail.com [mailto:djmit...@gmail.com] On Behalf Of
>> >> Dustin J. Mitchell
>> >> Sent: Wednesday, July 07, 2010 1:14 AM
>> >> To: McGraw, Robert P
>> >> Cc: amanda-users@amanda.org
>> >> Subject: Re: Tape Usage Question
>> >>
>> >>
>> >> On Tue, Jul 6, 2010 at 11:56 PM, McGraw, Robert P
>>
>> <rmcg...@purdue.edu>
>>
>> >> wrote:
>> >> > In my case there were several dump image that could have been
>>
>> written
>>
>> >> to the tape. Amanda did not seem to pick the largest dump image that
>> >> would fit on the tape.
>> >>
>> >> Well, bear in mind that the tape drive does not give any indication
>>
>> of
>>
>> >> space left -- all Amanda has to go on is dead reckoning based on the
>> >> parameters in the tapetype and what it has written so far. So if
>> >> Amanda chose the 37GB dump, it probably expected at least 37GB
>> >> remained on tape at that point, although you knew better. Are your
>> >> tapetype parameters correct?
>> >>
>> >> Dustin
>> >>
>> >> --
>> >> Open Source Storage Engineer
>> >> http://www.zmanda.com
>> >
>> >[McGraw, Robert P]
>> >
>> >Dustin,
>> >
>> >I use an LTO-2 drive which is supposed to give 200GB uncompressed and
>> > 400GB compressed. I realize that the 400GB is the absolute max and
>>
>> would
>>
>> > never be obtainable. So I make tapetype equal to a pretend 350GB
>>
>> which
>>
>> > seems to be a good number. So to amanda it thinks that the tape size
>>
>> is
>>
>> > 350GB and bases it calculations on this number.
>> >
>> >define tapetype LTO2-HWC {
>> > comment "LTO-2-Hardware Compression on."
>> > blocksize 1024 kbytes
>> > length 350000 mbytes #350G compressed
>> > filemark 0 kbytes
>> > speed 27315 kps #27 Mb/s
>> >}
>>
>> Humm "Hardware compression on". So amanda has not even a SWAG clue as
>> to
>> what the tape will hold. If you are also doing software compression,
>> using
>> such as gzip, please bear in mind that gzip can do a much better job of
>> it
>> in many cases than the hardware compressors can, AND that there is a
>> high
>> probability that a gzip'd file, already having been compressed, will
>> actually grow, sometimes quite a bit, in that hardware compressor.
>>
>> This is saying that it is generally bad kharma to use both. The choice
>> is
>> tipped very quickly and positively to the gzip method, with the
>> hardware
>> compressor turned off by the fact that amanda now knows exactly how
>> many
>> bytes have been sent to the tape as it tracks the size of the gzip'd
>> output
>> file. And the best-fit works as expected.
>>
>> Be aware that turning off the hardware compressor can be a frustrating
>> experience because the state of the compressor is usually stored on the
>> tape, in a data block at the beginning that is over-written each time.
>> The
>> problem with that is that if it is enabled, then regardless of your
>> commands
>> to disable it, it will be re-enabled from this hidden header each time
>> the
>> tape in inserted and re-read by the drive so the drive can configure
>> itself
>> to that exact tape.
>>
>> And amanda re-does this tape recognition sequence in checking the label
>> on
>> the tape, forcing the hardware compressor on if that flag bit is set.
>>
>> I only know of one foolproof method to turn it off in that case and it
>> doesn't use amanda to do it:
>>
>> 1. Insert the tape, make sure it is rewound. The drive will generally
>> do
>> this in recognizing the tape.
>>
>> 2. Using dd, read out the first block containing the label and save it
>> to
>> disk.
>>
>> 3. Rewind the tape with mt.
>>
>> 4. issue the command to disable the hardware compression, probably
>> using mt.
>>
>> 5. re-write that label block file using dd. This will re-write that
>> hidden
>> block too, turning the hardware compression off for this tape, and it
>> also
>> effectively trashes the rest of the tape, rendering it unreadable.
>>
>> So it must be treated as a new tape in every way but the use count if
>> you
>> are tracking that. From this point on, the hardware compression should
>> be
>> off, and you can use amanda's housekeeping utilities to re-label the
>> tapes,
>> and that should be done so amanda knows it is losing its database as
>> you
>> amrmtape, and amlabel relabel the tapes.
>>
>> I have been known to run a dump 5 or 6 times in a row in the same day
>> so
>> amanda will rebuild the database, and the normal backup runs won't
>> waste
>> several days doing a level 0 on the whole system, rather than a few
>> each
>> night because one tape doesn't have enough room for a level 0 on every
>> DLE.
>> Its forcing the issue, but amanda seems to settle into its routine a
>> lot
>> faster if I do this.
>>
>> >The problem is not the size of the physical tape, the problem is that
>> > amanda did not pick the largestfit dump image based what amanda knew
>>
>> to
>>
>> > be how much was left on the tape based on tapetype length size of
>>
>> 350GB.
>>
>> >1) amanda knows the tape size based on tapetype information which in
>>
>> my
>>
>> > case is 350GB.
>>
>> Reset this to 200GB, the true size of the tape.
>>
>> >2) amanda knows the amount of data the is written to tape. You can see
>> > that in the amreports and amstatus information.
>> >
>> >3) amanda knows how much disk spaced is left of the 350GB and from
>>
>> this
>>
>> > number amanda should get the largestfit dump image that is less than
>>
>> the
>>
>> > disk space left. In my case amanda picked a dump image that was
>>
>> bigger
>>
>> > that the disk space left. So as the templar knight said in the
>>
>> Indiana
>>
>> > Jones "The Last Crusade" amanda "did not pick well".
>> >
>> >Oh well I do not want to beat a dead horse.
>>
>> And the above advice should revive the horse. ;-)
>>
>> >Thanks
>> >
>> >Robert
>>
>> --
>> Cheers, Gene
>> "There are four boxes to be used in defense of liberty:
>> soap, ballot, jury, and ammo. Please use in that order."
>> -Ed Howdershelt (Author)
>> The nation that controls magnetism controls the universe.
>> -- Chester Gould/Dick Tracy
>
>[McGraw, Robert P]
>
>Gene,
>
>Thank for the information.
>
>I only use hardware compression, no software compression. When I tried
> software compression it was taking way to long.
>
>Thanks again
>
>Robert
>
That is one of the tradeoffs, but you get better far compression ratios with
some dle's that way, in fact I enable it for all new dle's, and only disable
it it for that dle if the output size is more then 75% of input size, but
typically the output is in the 10% to 30% range.
Me, I threw iron at it about 2 years ago & built a new box. ;-) Phenom 9550
at 2.1Ghz, 4Gb of ram, so it is no longer an all night project for me. But
then my average backup, after compressing that which will compress, is
probably under 30Gb. Someone with 300Gb might want to toss a little more
iron at it to keep it from bothering the morning shift.
I know when it starts, 00:30 am, but it would be nice (hint hint Dustin),
if, when it sends the report to the printer, if it also logged the total
elapsed time. Generally under an hour here, using vtapes, plus another 3 or
4 minutes for my wrapper scripts to finish up.
--
Cheers, Gene
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
* JHM wonders what Joey did to earn "I'd just like to say, for the record,
that Joey rules."
-- Seen on #Debian
