Re: Do you use this library

2011-08-25 Thread Brian Cuttler

As long as we are talking about HW vs SW compression, I should
say that we've installed pigz, a parallelized version of gzip,
on some of our systems with good result. If you have a system
that will support it and SW compression is running long you might
want to test it out.


On Wed, Aug 24, 2011 at 05:19:44PM -0400, gene heskett wrote:
 On Wednesday, August 24, 2011 05:13:46 PM McGraw, Robert P did opine:
 
  Gene,
  
  Thanks for your information.
  
  I tried the gzip way but it took too long on my backup server.
  
 My iron here is beginning to took toward collecting SS one day, but it was 
 good when I built it from boxes of parts about 4 years ago.  Quad core 
 Phenom running at 2.1Ghz, 4Gb of DDR2 ram.  Today that isn't much to brag 
 about.  A typical run here grabs 2 boxes for 36 DLE's, and is done in a 
 long hour most nights, backing up a bit less than 30Gb most runs.  To 
 vtapes on a 1Tb drive.
 
  With my old LTO2 drive I took 90% of the compressed value and used that
  as the length. With   device-property LEOM true and part_size
  40GB, I was getting close to filling my tapes to 100%.
  
  If I ever get a faster backup server I will retry the gizp way again.
  
  Thanks again,
  
 No problem Robert, but I did need to emphasize the 'price' of the trade 
 offs.  ;-)
 
  Robert
  
   -Original Message-
   From: owner-amanda-us...@amanda.org
   [mailto:owner-amanda-us...@amanda.org] On Behalf Of gene heskett
   Sent: Wednesday, August 24, 2011 10:43 AM
   To: amanda-users@amanda.org
   Subject: Re: Do you use this library
   
   On Wednesday, August 24, 2011 10:17:33 AM McGraw, Robert P did opine:
JF,


Recently you sent me your tapetype for an lto4 drive. I have a
question about the length parameter.

An lto-4 tape has a non-compress length of ~800GB and in compress
about double. I know these are max values.

In you answer below you have length set to ~772GB. If I am going to
run in compress mode so should the length value be double or
~1.4TB.
   
   You cannot double compress in normal practice.  Feeding a highly
   compressed
   file to the lightning fast compression methods used in tape drives
   will often result in the file growing larger by many percentage
   points.
   
   Gzip can generally beat the tape drives at their own game, albeit at
   the expense of cpu time to do it right.
   
   There are also pretty valid arguments against using the drives
   compression as it isolates the backup program, preventing it,
   regardless of the name of
   the program in question, from keeping track of how much of the tape
   has been used since the best the programs can do is count the bytes
   sent down the cable to the drive.  If the drive is massaging the
   data, shrinking or expending it, then the program has no real clue
   when the tape is full till the drive reports EOT.
   
   Because of these considerations, it is pretty universal here that the
   recommendation is to turn off the drives compressor and let software
   methods do the data smunching.  Then amanda, or any another other
   program that does track the tape capacity, will then know how much of
   the tape has been used.
   
   This can be very difficult to actually do for those tape formats, most
   of them, that record the compressor's state in a hidden header file
   we never see.  DDS is one such format where you will have to write a
   script that:
   
   dd reads the first block of the tape out to a file, then
   mtx re-winds the tape, then
   mtx turns off the drives compressor, then
   dd re-writes the first block of the tape with the saved file
   
   All without ejecting the tape or going through a tape recognition
   cycle as most drives do for a freshly inserted tape, and which would
   turn the compression back on despite your wishes.  Only a similar
   operation will actually turn it of for most drives today if it has
   ever been turned on and
   the tape written to.
   
   You can of course ignore this advice Robert.
   
Thanks

Robert
   
   [...]
   
   Cheers, gene
   --
   
   There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order.
   
   -Ed Howdershelt (Author)
   
   He's dead, Jim.
 
 
 Cheers, gene
 -- 
 There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order.
 -Ed Howdershelt (Author)
 
 The covers of this book are too far apart.
   -- Book review by Ambrose Bierce.
---
   Brian R Cuttler brian.cutt...@wadsworth.org
   Computer Systems Support(v) 518 486-1697
   Wadsworth Center(f) 518 473-6384
   NYS Department of HealthHelp Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure.  It
is intended only

Re: Do you use this library

2011-08-25 Thread gene heskett
On Thursday, August 25, 2011 09:17:04 AM Brian Cuttler did opine:

 As long as we are talking about HW vs SW compression, I should
 say that we've installed pigz, a parallelized version of gzip,
 on some of our systems with good result. If you have a system
 that will support it and SW compression is running long you might
 want to test it out.
 
That could be handy if it is easily configured to be used, which I have not 
investigated. Making amanda use its parallelism might be a challenging 
project however.

Another compressor also comes to mind where much of the data being 
compressed is in a common format.  I have a 220 megabyte file here which 
contains the src trees of every linux kernel released from 1.0 to about 
2.6.37, which was done by and can be unpacked if the disk space is 
available, by lrzip, the lr meaning long range as it finds and removes 
the redundancies in a tree of data like that archive has.  I would suspect 
that speed is not one of its strong points, but you have to be impressed by 
that compression ratio, as that is probably more than 15Gb of data 
unpacked.

This was done by (IIRC) Con Kovilas who also maintains the bfs (brain fuck 
scheduler) patches for the linux kernel, and which I have been running here 
for several months.  It brings a whole new feel to the users desktop 
experience, unless I see the cpu usages all fired up when amanda is running 
in the gkrellm panel, I am not aware that amanda has fired up, which 
usually results in me being virtually locked out of the system by amanda's 
running, instead remaining quite usable in spite of a session of gzip -best 
showing 98% cpu on one or another of the 4 cores.  The difference between 
the usual cfs scheduler and the bfs version is quite impressive to me.

No, I am not building these kernels, I get them from the PCLos 2011 repo, 
which is what I am running on this machine.

Cheers, gene
-- 
There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order.
-Ed Howdershelt (Author)
My web page: http://204.111.67.138:85/gene
Maryann's Law:
You can always find what you're not looking for.


Re: Do you use this library

2011-08-25 Thread Charles Curley
On Thu, 25 Aug 2011 09:45:38 -0400
gene heskett ghesk...@wdtv.com wrote:

 That could be handy if it is easily configured to be used, which I
 have not investigated.

http://wiki.zmanda.com/index.php/How_To:Use_pigz_to_speed_up_compression

 Making amanda use its parallelism might be a
 challenging project however.

Other emails on this list indicate that using pigz does speed up amanda
runs.

Just for the halibut, I did a bit of testing on an otherwise idle
desktop with two AMD cores. I created a tarball using my /var
directory. Then:

root@dzur:/# time pigz var.test.tar 

real6m1.411s
user9m5.290s
sys 0m44.550s
root@dzur:/# ll var.test.tar.gz 
-rw-r--r-- 1 root root 6786436232 2011-08-25 10:07 var.test.tar.gz
root@dzur:/# time gzip -d var.test.tar.gz 

real4m16.611s
user1m43.080s
sys 0m38.410s
root@dzur:/# ll var.test.tar 
-rw-r--r-- 1 root root 9458923520 2011-08-25 10:07 var.test.tar
root@dzur:/# tar -tf var.test.tar
# output omitted; no error messages
root@dzur:/# time gzip var.test.tar 

real9m45.992s
user8m23.070s
sys 0m32.970s
root@dzur:/# ll var.test.tar.gz 
-rw-r--r-- 1 root root 6785153296 2011-08-25 10:07 var.test.tar.gz
root@dzur:/# gunzip var.test.tar.gz 
root@dzur:/# ll var.test.tar 
-rw-r--r-- 1 root root 9458923520 2011-08-25 10:07 var.test.tar
root@dzur:/# pigz --version
pigz 2.1.6
root@dzur:/#


Note that gzip took noticeably longer but produced a marginally smaller
output file (possibly due to different default compression values; I
did not investigate). Testing the output file as above produced no error
message. The uncompressed tarball retained its size.

I used gzip -d to test rather than pigz -d in order to verify
compatibility.

While we're at it, I am glad I never did implement bzip2 compression
for Amanada:

root@dzur:/# time bzip2 var.test.tar 

real157m54.710s
user156m41.360s
sys 0m35.550s
root@dzur:/# ll var.test.tar.bz2 
-rw-r--r-- 1 root root 6780647698 2011-08-25 10:07 var.test.tar.bz2
root@dzur:/# 


-- 

Charles Curley  /\ASCII Ribbon Campaign
Looking for fine software   \ /Respect for open standards
and/or writing?  X No HTML/RTF in email
http://www.charlescurley.com/ \No M$ Word docs in email

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB


Re: Do you use this library

2011-08-25 Thread Brian Cuttler


All we did at my site was to essentially replace the gzip binary.

Its wrong and its cheating, but that is what we did.


On Thu, Aug 25, 2011 at 02:21:13PM -0600, Charles Curley wrote:
 On Thu, 25 Aug 2011 09:45:38 -0400
 gene heskett ghesk...@wdtv.com wrote:
 
  That could be handy if it is easily configured to be used, which I
  have not investigated.
 
 http://wiki.zmanda.com/index.php/How_To:Use_pigz_to_speed_up_compression
 
  Making amanda use its parallelism might be a
  challenging project however.
 
 Other emails on this list indicate that using pigz does speed up amanda
 runs.
 
 Just for the halibut, I did a bit of testing on an otherwise idle
 desktop with two AMD cores. I created a tarball using my /var
 directory. Then:
 
 root@dzur:/# time pigz var.test.tar 
 
 real  6m1.411s
 user  9m5.290s
 sys   0m44.550s
 root@dzur:/# ll var.test.tar.gz 
 -rw-r--r-- 1 root root 6786436232 2011-08-25 10:07 var.test.tar.gz
 root@dzur:/# time gzip -d var.test.tar.gz 
 
 real  4m16.611s
 user  1m43.080s
 sys   0m38.410s
 root@dzur:/# ll var.test.tar 
 -rw-r--r-- 1 root root 9458923520 2011-08-25 10:07 var.test.tar
 root@dzur:/# tar -tf var.test.tar
 # output omitted; no error messages
 root@dzur:/# time gzip var.test.tar 
 
 real  9m45.992s
 user  8m23.070s
 sys   0m32.970s
 root@dzur:/# ll var.test.tar.gz 
 -rw-r--r-- 1 root root 6785153296 2011-08-25 10:07 var.test.tar.gz
 root@dzur:/# gunzip var.test.tar.gz 
 root@dzur:/# ll var.test.tar 
 -rw-r--r-- 1 root root 9458923520 2011-08-25 10:07 var.test.tar
 root@dzur:/# pigz --version
 pigz 2.1.6
 root@dzur:/#
 
 
 Note that gzip took noticeably longer but produced a marginally smaller
 output file (possibly due to different default compression values; I
 did not investigate). Testing the output file as above produced no error
 message. The uncompressed tarball retained its size.
 
 I used gzip -d to test rather than pigz -d in order to verify
 compatibility.
 
 While we're at it, I am glad I never did implement bzip2 compression
 for Amanada:
 
 root@dzur:/# time bzip2 var.test.tar 
 
 real  157m54.710s
 user  156m41.360s
 sys   0m35.550s
 root@dzur:/# ll var.test.tar.bz2 
 -rw-r--r-- 1 root root 6780647698 2011-08-25 10:07 var.test.tar.bz2
 root@dzur:/# 
 
 
 -- 
 
 Charles Curley  /\ASCII Ribbon Campaign
 Looking for fine software   \ /Respect for open standards
 and/or writing?  X No HTML/RTF in email
 http://www.charlescurley.com/ \No M$ Word docs in email
 
 Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
---
   Brian R Cuttler brian.cutt...@wadsworth.org
   Computer Systems Support(v) 518 486-1697
   Wadsworth Center(f) 518 473-6384
   NYS Department of HealthHelp Desk 518 473-0773



IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure.  It
is intended only for the addressee.  If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments.  Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.




Re: Do you use this library

2011-08-25 Thread Charles Curley
On Thu, 25 Aug 2011 09:45:38 -0400
gene heskett ghesk...@wdtv.com wrote:

 Making amanda use its parallelism might be a challenging 
 project however.

Actually, not challenging at all. According to the man page,
http://www.zlib.net/pigz/pigz.pdf:

  Pigz compresses using threads to make use of multiple processors and
  cores. The input is broken up into 128 KB chunks with each compressed
  in parallel. The individual check value for each chunk is also
  calculated in parallel. The compressed data is written in order to
  the output, and a combined check value is calculated from the
  individual check values.

  The compressed data format generated is in the gzip, zlib, or
  single-entry zip format using the deflate compression method. The
  compression produces partial raw deflate streams which are
  concatenated by a single write thread and wrapped with the
  appropriate header and trailer, where the trailer contains the
  combined check value.

  ...

  The number of compress threads is set by default to the number of
  online processors, which can be changed using the -p option.


-- 

Charles Curley  /\ASCII Ribbon Campaign
Looking for fine software   \ /Respect for open standards
and/or writing?  X No HTML/RTF in email
http://www.charlescurley.com/ \No M$ Word docs in email

Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB


Re: Do you use this library

2011-08-25 Thread gene heskett
On Thursday, August 25, 2011 07:20:20 PM Charles Curley did opine:

 On Thu, 25 Aug 2011 09:45:38 -0400
 
 gene heskett ghesk...@wdtv.com wrote:
  That could be handy if it is easily configured to be used, which I
  have not investigated.
 
 http://wiki.zmanda.com/index.php/How_To:Use_pigz_to_speed_up_compression
 
  Making amanda use its parallelism might be a
  challenging project however.
 
 Other emails on this list indicate that using pigz does speed up amanda
 runs.
 
 Just for the halibut, I did a bit of testing on an otherwise idle
 desktop with two AMD cores. I created a tarball using my /var
 directory. Then:
 
 root@dzur:/# time pigz var.test.tar
 
 real  6m1.411s
 user  9m5.290s
 sys   0m44.550s
 root@dzur:/# ll var.test.tar.gz
 -rw-r--r-- 1 root root 6786436232 2011-08-25 10:07 var.test.tar.gz
 root@dzur:/# time gzip -d var.test.tar.gz
 
 real  4m16.611s
 user  1m43.080s
 sys   0m38.410s
 root@dzur:/# ll var.test.tar
 -rw-r--r-- 1 root root 9458923520 2011-08-25 10:07 var.test.tar
 root@dzur:/# tar -tf var.test.tar
 # output omitted; no error messages
 root@dzur:/# time gzip var.test.tar
 
 real  9m45.992s
 user  8m23.070s
 sys   0m32.970s
 root@dzur:/# ll var.test.tar.gz
 -rw-r--r-- 1 root root 6785153296 2011-08-25 10:07 var.test.tar.gz
 root@dzur:/# gunzip var.test.tar.gz
 root@dzur:/# ll var.test.tar
 -rw-r--r-- 1 root root 9458923520 2011-08-25 10:07 var.test.tar
 root@dzur:/# pigz --version
 pigz 2.1.6
 root@dzur:/#
 
 
 Note that gzip took noticeably longer but produced a marginally smaller
 output file (possibly due to different default compression values; I
 did not investigate). Testing the output file as above produced no error
 message. The uncompressed tarball retained its size.
 
 I used gzip -d to test rather than pigz -d in order to verify
 compatibility.
 
 While we're at it, I am glad I never did implement bzip2 compression
 for Amanada:
 
 root@dzur:/# time bzip2 var.test.tar
 
 real  157m54.710s
 user  156m41.360s
 sys   0m35.550s
 root@dzur:/# ll var.test.tar.bz2
 -rw-r--r-- 1 root root 6780647698 2011-08-25 10:07 var.test.tar.bz2
 root@dzur:/#

This latter is not a bit surprising.  And its been known to make mistakes 
in the decompress without reporting an error.  I once unpacked a kernel 
tree but could not get it to build, not once but 3 times and each time the 
tree was missing about a megabytes worth of a driver subdir.  By then I was 
getting schmardter from the error messages  knew which subdir was missing.

Prepared to download it again, I blew that tree away and unpacked the .bz2 
again.  On the 4th try, that subdir finally showed up, and it then built as 
usual.  From that time on, I have never pulled the .bz2 unless the .gz was 
not available.

Cheers, gene
-- 
There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order.
-Ed Howdershelt (Author)
My web page: http://204.111.67.138:85/gene
When you die, you lose a very important part of your life.
-- Brooke Shields


Re: Do you use this library

2011-08-24 Thread gene heskett
On Wednesday, August 24, 2011 10:17:33 AM McGraw, Robert P did opine:

 JF,
 
 
 Recently you sent me your tapetype for an lto4 drive. I have a question
 about the length parameter.
 
 An lto-4 tape has a non-compress length of ~800GB and in compress about
 double. I know these are max values.
 
 In you answer below you have length set to ~772GB. If I am going to run
 in compress mode so should the length value be double or ~1.4TB.

You cannot double compress in normal practice.  Feeding a highly compressed 
file to the lightning fast compression methods used in tape drives will 
often result in the file growing larger by many percentage points.

Gzip can generally beat the tape drives at their own game, albeit at the 
expense of cpu time to do it right.

There are also pretty valid arguments against using the drives compression 
as it isolates the backup program, preventing it, regardless of the name of 
the program in question, from keeping track of how much of the tape has 
been used since the best the programs can do is count the bytes sent down 
the cable to the drive.  If the drive is massaging the data, shrinking or 
expending it, then the program has no real clue when the tape is full till 
the drive reports EOT.

Because of these considerations, it is pretty universal here that the 
recommendation is to turn off the drives compressor and let software 
methods do the data smunching.  Then amanda, or any another other program 
that does track the tape capacity, will then know how much of the tape has 
been used.

This can be very difficult to actually do for those tape formats, most of 
them, that record the compressor's state in a hidden header file we never 
see.  DDS is one such format where you will have to write a script that:

dd reads the first block of the tape out to a file, then 
mtx re-winds the tape, then 
mtx turns off the drives compressor, then 
dd re-writes the first block of the tape with the saved file

All without ejecting the tape or going through a tape recognition cycle as 
most drives do for a freshly inserted tape, and which would turn the 
compression back on despite your wishes.  Only a similar operation will 
actually turn it of for most drives today if it has ever been turned on and 
the tape written to.

You can of course ignore this advice Robert.

 Thanks
 
 Robert
[...]

Cheers, gene
-- 
There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order.
-Ed Howdershelt (Author)

He's dead, Jim.


RE: Do you use this library

2011-08-24 Thread McGraw, Robert P
Gene,

Thanks for your information.

I tried the gzip way but it took too long on my backup server.

With my old LTO2 drive I took 90% of the compressed value and used that as the 
length. With   device-property LEOM true and part_size 40GB, I was 
getting close to filling my tapes to 100%.

If I ever get a faster backup server I will retry the gizp way again.

Thanks again,

Robert




 -Original Message-
 From: owner-amanda-us...@amanda.org [mailto:owner-amanda-us...@amanda.org]
 On Behalf Of gene heskett
 Sent: Wednesday, August 24, 2011 10:43 AM
 To: amanda-users@amanda.org
 Subject: Re: Do you use this library
 
 
 On Wednesday, August 24, 2011 10:17:33 AM McGraw, Robert P did opine:
 
  JF,
 
 
  Recently you sent me your tapetype for an lto4 drive. I have a question
  about the length parameter.
 
  An lto-4 tape has a non-compress length of ~800GB and in compress about
  double. I know these are max values.
 
  In you answer below you have length set to ~772GB. If I am going to run
  in compress mode so should the length value be double or ~1.4TB.
 
 You cannot double compress in normal practice.  Feeding a highly
 compressed
 file to the lightning fast compression methods used in tape drives will
 often result in the file growing larger by many percentage points.
 
 Gzip can generally beat the tape drives at their own game, albeit at the
 expense of cpu time to do it right.
 
 There are also pretty valid arguments against using the drives compression
 as it isolates the backup program, preventing it, regardless of the name
 of
 the program in question, from keeping track of how much of the tape has
 been used since the best the programs can do is count the bytes sent down
 the cable to the drive.  If the drive is massaging the data, shrinking or
 expending it, then the program has no real clue when the tape is full till
 the drive reports EOT.
 
 Because of these considerations, it is pretty universal here that the
 recommendation is to turn off the drives compressor and let software
 methods do the data smunching.  Then amanda, or any another other program
 that does track the tape capacity, will then know how much of the tape has
 been used.
 
 This can be very difficult to actually do for those tape formats, most of
 them, that record the compressor's state in a hidden header file we never
 see.  DDS is one such format where you will have to write a script that:
 
 dd reads the first block of the tape out to a file, then
 mtx re-winds the tape, then
 mtx turns off the drives compressor, then
 dd re-writes the first block of the tape with the saved file
 
 All without ejecting the tape or going through a tape recognition cycle as
 most drives do for a freshly inserted tape, and which would turn the
 compression back on despite your wishes.  Only a similar operation will
 actually turn it of for most drives today if it has ever been turned on
 and
 the tape written to.
 
 You can of course ignore this advice Robert.
 
  Thanks
 
  Robert
 [...]
 
 Cheers, gene
 --
 There are four boxes to be used in defense of liberty:
  soap, ballot, jury, and ammo. Please use in that order.
 -Ed Howdershelt (Author)
 
 He's dead, Jim.



Re: Do you use this library

2011-08-24 Thread Jean-Francois Malouin
* McGraw, Robert P rmcg...@purdue.edu [20110824 11:46]:
 Gene,
 
 Thanks for your information.
 
 I tried the gzip way but it took too long on my backup server.
 
 With my old LTO2 drive I took 90% of the compressed value and used that as 
 the length. With   device-property LEOM true and part_size 40GB, I was 
 getting close to filling my tapes to 100%.
 
 If I ever get a faster backup server I will retry the gizp way again.

I've always used hardware compression with LTO3/4/5 since these
drives will detect if the data stream is compressible or not so I dont
bother disabling it. 
But the amtapetype output I sent you was with compression disabled though.

jf

 
 Thanks again,
 
 Robert
 
 
 
 
  -Original Message-
  From: owner-amanda-us...@amanda.org [mailto:owner-amanda-us...@amanda.org]
  On Behalf Of gene heskett
  Sent: Wednesday, August 24, 2011 10:43 AM
  To: amanda-users@amanda.org
  Subject: Re: Do you use this library
  
  
  On Wednesday, August 24, 2011 10:17:33 AM McGraw, Robert P did opine:
  
   JF,
  
  
   Recently you sent me your tapetype for an lto4 drive. I have a question
   about the length parameter.
  
   An lto-4 tape has a non-compress length of ~800GB and in compress about
   double. I know these are max values.
  
   In you answer below you have length set to ~772GB. If I am going to run
   in compress mode so should the length value be double or ~1.4TB.
  
  You cannot double compress in normal practice.  Feeding a highly
  compressed
  file to the lightning fast compression methods used in tape drives will
  often result in the file growing larger by many percentage points.
  
  Gzip can generally beat the tape drives at their own game, albeit at the
  expense of cpu time to do it right.
  
  There are also pretty valid arguments against using the drives compression
  as it isolates the backup program, preventing it, regardless of the name
  of
  the program in question, from keeping track of how much of the tape has
  been used since the best the programs can do is count the bytes sent down
  the cable to the drive.  If the drive is massaging the data, shrinking or
  expending it, then the program has no real clue when the tape is full till
  the drive reports EOT.
  
  Because of these considerations, it is pretty universal here that the
  recommendation is to turn off the drives compressor and let software
  methods do the data smunching.  Then amanda, or any another other program
  that does track the tape capacity, will then know how much of the tape has
  been used.
  
  This can be very difficult to actually do for those tape formats, most of
  them, that record the compressor's state in a hidden header file we never
  see.  DDS is one such format where you will have to write a script that:
  
  dd reads the first block of the tape out to a file, then
  mtx re-winds the tape, then
  mtx turns off the drives compressor, then
  dd re-writes the first block of the tape with the saved file
  
  All without ejecting the tape or going through a tape recognition cycle as
  most drives do for a freshly inserted tape, and which would turn the
  compression back on despite your wishes.  Only a similar operation will
  actually turn it of for most drives today if it has ever been turned on
  and
  the tape written to.
  
  You can of course ignore this advice Robert.
  
   Thanks
  
   Robert
  [...]
  
  Cheers, gene
  --
  There are four boxes to be used in defense of liberty:
   soap, ballot, jury, and ammo. Please use in that order.
  -Ed Howdershelt (Author)
  
  He's dead, Jim.


Re: Do you use this library

2011-08-24 Thread gene heskett
On Wednesday, August 24, 2011 05:13:46 PM McGraw, Robert P did opine:

 Gene,
 
 Thanks for your information.
 
 I tried the gzip way but it took too long on my backup server.
 
My iron here is beginning to took toward collecting SS one day, but it was 
good when I built it from boxes of parts about 4 years ago.  Quad core 
Phenom running at 2.1Ghz, 4Gb of DDR2 ram.  Today that isn't much to brag 
about.  A typical run here grabs 2 boxes for 36 DLE's, and is done in a 
long hour most nights, backing up a bit less than 30Gb most runs.  To 
vtapes on a 1Tb drive.

 With my old LTO2 drive I took 90% of the compressed value and used that
 as the length. With   device-property LEOM true and part_size
 40GB, I was getting close to filling my tapes to 100%.
 
 If I ever get a faster backup server I will retry the gizp way again.
 
 Thanks again,
 
No problem Robert, but I did need to emphasize the 'price' of the trade 
offs.  ;-)

 Robert
 
  -Original Message-
  From: owner-amanda-us...@amanda.org
  [mailto:owner-amanda-us...@amanda.org] On Behalf Of gene heskett
  Sent: Wednesday, August 24, 2011 10:43 AM
  To: amanda-users@amanda.org
  Subject: Re: Do you use this library
  
  On Wednesday, August 24, 2011 10:17:33 AM McGraw, Robert P did opine:
   JF,
   
   
   Recently you sent me your tapetype for an lto4 drive. I have a
   question about the length parameter.
   
   An lto-4 tape has a non-compress length of ~800GB and in compress
   about double. I know these are max values.
   
   In you answer below you have length set to ~772GB. If I am going to
   run in compress mode so should the length value be double or
   ~1.4TB.
  
  You cannot double compress in normal practice.  Feeding a highly
  compressed
  file to the lightning fast compression methods used in tape drives
  will often result in the file growing larger by many percentage
  points.
  
  Gzip can generally beat the tape drives at their own game, albeit at
  the expense of cpu time to do it right.
  
  There are also pretty valid arguments against using the drives
  compression as it isolates the backup program, preventing it,
  regardless of the name of
  the program in question, from keeping track of how much of the tape
  has been used since the best the programs can do is count the bytes
  sent down the cable to the drive.  If the drive is massaging the
  data, shrinking or expending it, then the program has no real clue
  when the tape is full till the drive reports EOT.
  
  Because of these considerations, it is pretty universal here that the
  recommendation is to turn off the drives compressor and let software
  methods do the data smunching.  Then amanda, or any another other
  program that does track the tape capacity, will then know how much of
  the tape has been used.
  
  This can be very difficult to actually do for those tape formats, most
  of them, that record the compressor's state in a hidden header file
  we never see.  DDS is one such format where you will have to write a
  script that:
  
  dd reads the first block of the tape out to a file, then
  mtx re-winds the tape, then
  mtx turns off the drives compressor, then
  dd re-writes the first block of the tape with the saved file
  
  All without ejecting the tape or going through a tape recognition
  cycle as most drives do for a freshly inserted tape, and which would
  turn the compression back on despite your wishes.  Only a similar
  operation will actually turn it of for most drives today if it has
  ever been turned on and
  the tape written to.
  
  You can of course ignore this advice Robert.
  
   Thanks
   
   Robert
  
  [...]
  
  Cheers, gene
  --
  
  There are four boxes to be used in defense of liberty:
   soap, ballot, jury, and ammo. Please use in that order.
  
  -Ed Howdershelt (Author)
  
  He's dead, Jim.


Cheers, gene
-- 
There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order.
-Ed Howdershelt (Author)

The covers of this book are too far apart.
-- Book review by Ambrose Bierce.


RE: Do you use this library

2011-08-23 Thread McGraw, Robert P
JF,


Recently you sent me your tapetype for an lto4 drive. I have a question about 
the length parameter. 

An lto-4 tape has a non-compress length of ~800GB and in compress about double. 
I know these are max values.

In you answer below you have length set to ~772GB. If I am going to run in 
compress mode so should the length value be double or ~1.4TB.

Thanks

Robert



 -Original Message-
 From: Jean-Francois Malouin [mailto:Jean-
 francois.malo...@bic.mni.mcgill.ca]
 Sent: Thursday, July 21, 2011 3:33 PM
 To: McGraw, Robert P
 Subject: Re: Do you use this library
 
 
 * McGraw, Robert P rmcg...@purdue.edu [20110721 14:56]:
 
  We just purchased a Oracle/Sun StorageTek SL24, SCSI, with HP LTO4 FH
 tape drive.
 
  Does anyone use this model unit. If so could you send me your tapetype
 for the LTO4 drive.
 
 Got many of those.
 
 Here's my tapetype for these drives:
 
 define tapetype tape-lto4 {
 length 772096 mbytes
 filemark 0 kbytes
 speed 102216 kps
 blocksize 2048 kbytes
 }
 
 
 hth,
 jf
 
 
  _
  Robert P. McGraw, Jr.
  Manager, Computer SystemEMAIL: rmcg...@purdue.edu
  Purdue UniversityROOM: MATH-807
  Department of Mathematics   PHONE: (765) 494-6055
  150 N. University Street
  West Lafayette, IN 47907-2067
 
 
 
 --
 °  Jean-François Malouin  McConnell Brain Imaging Centre
 Systems/Network Administrator   Montréal Neurological Institute
 3801 Rue University, Suite WB219  Montréal, Québec, H3A 2B4
 Phone: 514-398-8924   Fax: 514-398-8948



Do you use this library

2011-07-21 Thread McGraw, Robert P

We just purchased a Oracle/Sun StorageTek SL24, SCSI, with HP LTO4 FH tape 
drive.

Does anyone use this model unit. If so could you send me your tapetype for the 
LTO4 drive.

_
Robert P. McGraw, Jr.
Manager, Computer SystemEMAIL: rmcg...@purdue.edu
Purdue UniversityROOM: MATH-807
Department of Mathematics   PHONE: (765) 494-6055
150 N. University Street  
West Lafayette, IN 47907-2067