Re: Do you use this library
On Wednesday, August 24, 2011 10:17:33 AM McGraw, Robert P did opine: JF, Recently you sent me your tapetype for an lto4 drive. I have a question about the length parameter. An lto-4 tape has a non-compress length of ~800GB and in compress about double. I know these are max values. In you answer below you have length set to ~772GB. If I am going to run in compress mode so should the length value be double or ~1.4TB. You cannot double compress in normal practice. Feeding a highly compressed file to the lightning fast compression methods used in tape drives will often result in the file growing larger by many percentage points. Gzip can generally beat the tape drives at their own game, albeit at the expense of cpu time to do it right. There are also pretty valid arguments against using the drives compression as it isolates the backup program, preventing it, regardless of the name of the program in question, from keeping track of how much of the tape has been used since the best the programs can do is count the bytes sent down the cable to the drive. If the drive is massaging the data, shrinking or expending it, then the program has no real clue when the tape is full till the drive reports EOT. Because of these considerations, it is pretty universal here that the recommendation is to turn off the drives compressor and let software methods do the data smunching. Then amanda, or any another other program that does track the tape capacity, will then know how much of the tape has been used. This can be very difficult to actually do for those tape formats, most of them, that record the compressor's state in a hidden header file we never see. DDS is one such format where you will have to write a script that: dd reads the first block of the tape out to a file, then mtx re-winds the tape, then mtx turns off the drives compressor, then dd re-writes the first block of the tape with the saved file All without ejecting the tape or going through a tape recognition cycle as most drives do for a freshly inserted tape, and which would turn the compression back on despite your wishes. Only a similar operation will actually turn it of for most drives today if it has ever been turned on and the tape written to. You can of course ignore this advice Robert. Thanks Robert [...] Cheers, gene -- There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) He's dead, Jim.
RE: Do you use this library
Gene, Thanks for your information. I tried the gzip way but it took too long on my backup server. With my old LTO2 drive I took 90% of the compressed value and used that as the length. With device-property LEOM true and part_size 40GB, I was getting close to filling my tapes to 100%. If I ever get a faster backup server I will retry the gizp way again. Thanks again, Robert -Original Message- From: owner-amanda-us...@amanda.org [mailto:owner-amanda-us...@amanda.org] On Behalf Of gene heskett Sent: Wednesday, August 24, 2011 10:43 AM To: amanda-users@amanda.org Subject: Re: Do you use this library On Wednesday, August 24, 2011 10:17:33 AM McGraw, Robert P did opine: JF, Recently you sent me your tapetype for an lto4 drive. I have a question about the length parameter. An lto-4 tape has a non-compress length of ~800GB and in compress about double. I know these are max values. In you answer below you have length set to ~772GB. If I am going to run in compress mode so should the length value be double or ~1.4TB. You cannot double compress in normal practice. Feeding a highly compressed file to the lightning fast compression methods used in tape drives will often result in the file growing larger by many percentage points. Gzip can generally beat the tape drives at their own game, albeit at the expense of cpu time to do it right. There are also pretty valid arguments against using the drives compression as it isolates the backup program, preventing it, regardless of the name of the program in question, from keeping track of how much of the tape has been used since the best the programs can do is count the bytes sent down the cable to the drive. If the drive is massaging the data, shrinking or expending it, then the program has no real clue when the tape is full till the drive reports EOT. Because of these considerations, it is pretty universal here that the recommendation is to turn off the drives compressor and let software methods do the data smunching. Then amanda, or any another other program that does track the tape capacity, will then know how much of the tape has been used. This can be very difficult to actually do for those tape formats, most of them, that record the compressor's state in a hidden header file we never see. DDS is one such format where you will have to write a script that: dd reads the first block of the tape out to a file, then mtx re-winds the tape, then mtx turns off the drives compressor, then dd re-writes the first block of the tape with the saved file All without ejecting the tape or going through a tape recognition cycle as most drives do for a freshly inserted tape, and which would turn the compression back on despite your wishes. Only a similar operation will actually turn it of for most drives today if it has ever been turned on and the tape written to. You can of course ignore this advice Robert. Thanks Robert [...] Cheers, gene -- There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) He's dead, Jim.
Re: Do you use this library
* McGraw, Robert P rmcg...@purdue.edu [20110824 11:46]: Gene, Thanks for your information. I tried the gzip way but it took too long on my backup server. With my old LTO2 drive I took 90% of the compressed value and used that as the length. With device-property LEOM true and part_size 40GB, I was getting close to filling my tapes to 100%. If I ever get a faster backup server I will retry the gizp way again. I've always used hardware compression with LTO3/4/5 since these drives will detect if the data stream is compressible or not so I dont bother disabling it. But the amtapetype output I sent you was with compression disabled though. jf Thanks again, Robert -Original Message- From: owner-amanda-us...@amanda.org [mailto:owner-amanda-us...@amanda.org] On Behalf Of gene heskett Sent: Wednesday, August 24, 2011 10:43 AM To: amanda-users@amanda.org Subject: Re: Do you use this library On Wednesday, August 24, 2011 10:17:33 AM McGraw, Robert P did opine: JF, Recently you sent me your tapetype for an lto4 drive. I have a question about the length parameter. An lto-4 tape has a non-compress length of ~800GB and in compress about double. I know these are max values. In you answer below you have length set to ~772GB. If I am going to run in compress mode so should the length value be double or ~1.4TB. You cannot double compress in normal practice. Feeding a highly compressed file to the lightning fast compression methods used in tape drives will often result in the file growing larger by many percentage points. Gzip can generally beat the tape drives at their own game, albeit at the expense of cpu time to do it right. There are also pretty valid arguments against using the drives compression as it isolates the backup program, preventing it, regardless of the name of the program in question, from keeping track of how much of the tape has been used since the best the programs can do is count the bytes sent down the cable to the drive. If the drive is massaging the data, shrinking or expending it, then the program has no real clue when the tape is full till the drive reports EOT. Because of these considerations, it is pretty universal here that the recommendation is to turn off the drives compressor and let software methods do the data smunching. Then amanda, or any another other program that does track the tape capacity, will then know how much of the tape has been used. This can be very difficult to actually do for those tape formats, most of them, that record the compressor's state in a hidden header file we never see. DDS is one such format where you will have to write a script that: dd reads the first block of the tape out to a file, then mtx re-winds the tape, then mtx turns off the drives compressor, then dd re-writes the first block of the tape with the saved file All without ejecting the tape or going through a tape recognition cycle as most drives do for a freshly inserted tape, and which would turn the compression back on despite your wishes. Only a similar operation will actually turn it of for most drives today if it has ever been turned on and the tape written to. You can of course ignore this advice Robert. Thanks Robert [...] Cheers, gene -- There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) He's dead, Jim.
Re: Do you use this library
On Wednesday, August 24, 2011 05:13:46 PM McGraw, Robert P did opine: Gene, Thanks for your information. I tried the gzip way but it took too long on my backup server. My iron here is beginning to took toward collecting SS one day, but it was good when I built it from boxes of parts about 4 years ago. Quad core Phenom running at 2.1Ghz, 4Gb of DDR2 ram. Today that isn't much to brag about. A typical run here grabs 2 boxes for 36 DLE's, and is done in a long hour most nights, backing up a bit less than 30Gb most runs. To vtapes on a 1Tb drive. With my old LTO2 drive I took 90% of the compressed value and used that as the length. With device-property LEOM true and part_size 40GB, I was getting close to filling my tapes to 100%. If I ever get a faster backup server I will retry the gizp way again. Thanks again, No problem Robert, but I did need to emphasize the 'price' of the trade offs. ;-) Robert -Original Message- From: owner-amanda-us...@amanda.org [mailto:owner-amanda-us...@amanda.org] On Behalf Of gene heskett Sent: Wednesday, August 24, 2011 10:43 AM To: amanda-users@amanda.org Subject: Re: Do you use this library On Wednesday, August 24, 2011 10:17:33 AM McGraw, Robert P did opine: JF, Recently you sent me your tapetype for an lto4 drive. I have a question about the length parameter. An lto-4 tape has a non-compress length of ~800GB and in compress about double. I know these are max values. In you answer below you have length set to ~772GB. If I am going to run in compress mode so should the length value be double or ~1.4TB. You cannot double compress in normal practice. Feeding a highly compressed file to the lightning fast compression methods used in tape drives will often result in the file growing larger by many percentage points. Gzip can generally beat the tape drives at their own game, albeit at the expense of cpu time to do it right. There are also pretty valid arguments against using the drives compression as it isolates the backup program, preventing it, regardless of the name of the program in question, from keeping track of how much of the tape has been used since the best the programs can do is count the bytes sent down the cable to the drive. If the drive is massaging the data, shrinking or expending it, then the program has no real clue when the tape is full till the drive reports EOT. Because of these considerations, it is pretty universal here that the recommendation is to turn off the drives compressor and let software methods do the data smunching. Then amanda, or any another other program that does track the tape capacity, will then know how much of the tape has been used. This can be very difficult to actually do for those tape formats, most of them, that record the compressor's state in a hidden header file we never see. DDS is one such format where you will have to write a script that: dd reads the first block of the tape out to a file, then mtx re-winds the tape, then mtx turns off the drives compressor, then dd re-writes the first block of the tape with the saved file All without ejecting the tape or going through a tape recognition cycle as most drives do for a freshly inserted tape, and which would turn the compression back on despite your wishes. Only a similar operation will actually turn it of for most drives today if it has ever been turned on and the tape written to. You can of course ignore this advice Robert. Thanks Robert [...] Cheers, gene -- There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) He's dead, Jim. Cheers, gene -- There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) The covers of this book are too far apart. -- Book review by Ambrose Bierce.
