Re: Backups to tape consistently under 60% tape capacity
On Thursday 23 October 2014 01:28:01 Tom Robinson did opine
And Gene did reply:
Now I have to work out why my tape is reporting as smaller! amtapetype
reports my tape is only half as big for the same block size...(was
1483868160 is now 743424512). :-/
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 67415370.8307692 bytes/sec
Wrote fixed (compressible) data at 273874944 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 761266700288 bytes at 57975 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (7612661760 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 743424512 kbytes
filemark 987 kbytes
speed 57975 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
On 23/10/14 03:19, Debra S Baddorf wrote:
Re-running amtapetype might be a very good idea. It might point to
where the problem isn’t, at least!
Do double check your cables. People have found problems in
cables which look like
reduced throughput. “Mpath” — I don’t know what it is, but could
it have changed with your OS upgrade?
Wouldn’t hurt to check that the tape driver setting haven’t
changed with the OS work …..
but otherwise, it sounds good.
Deb
On Oct 21, 2014, at 7:43 PM, Tom Robinson tom.robin...@motec.com.au
wrote:
Hmm, I did check my tape driver settings. When I set the tape
library up, I spent a long time getting the driver settings right
(on OmniOS) and took copious notes on the settings. My queries
reveal that I'm not using compression (which is what I wanted as
the vtapes are already compressed). LTO5 native is 1.5T; compressed
is 3T.
All my tapes are 'newish' (about one year old). The tape unit is the
same age.
For months I was consistently getting over 110% (highest 117%), then
capacity dropped once to 86% and then consistently to about 56%
(lowest 42%). Is there a block size issue (2x56=112)?
All the weekly dumps are local so network shouldn't be an issue. The
tape unit is using redundant SAS connectors. Maybe it's a mpath
thing?
Should I re-run amtapetype to see what it thinks the 'new' tape
capacity is after upgrading the OS?
On 22/10/14 10:47, Debra S Baddorf wrote:
Yeah, it sure does look like it ought to fit!
Could the tapes be dirty and not holding as much any more??? I
have no idea if that’s even possible. But it kinds seems like your
tapes are saying “I don’t want that much data”.Compression
issues?
Your tapes were previously getting 117% capacity, and now are
only doing 86%. Is that the general summary?
Unless somebody else can suggest some network (cable to tape
drive?) or system problems which might make the tapes appear
smaller than before? Compression issues? Read the original
message at the bottom of this email for the original problem
complaint.
Deb
On Oct 21, 2014, at 6:20 PM, Tom Robinson
tom.robin...@motec.com.au wrote:
Hi Debra,
A brilliant motivational speech. Thanks. Well worth the read. In
homage, I strongly suggest anyone who hasn't read it to go and do
that now. Here it is again for those whose mouse wheel is
dysfunctional:
http://www.appleseeds.org/Big-Rocks_Covey.htm
I will try your suggestions but want to make clear that the
virtual tapes you see are the product of a daily run which is
disk only. The weekly run puts all those daily dumps onto tape
which then leaves the building. So I have both virtual and real
tapes. The issues I'm having are in the weekly run (the dump to
real tape of a set of virtual tapes).
The tapes can be viewed as a bunch of big/little rocks. The total
amount of data, however they are stacked, should still fit on a
single LTO5 tape (amtapetype told me: length 1483868160 kbytes):
$ pwd
/data/backup/amanda/vtapes/daily
$ du -shc *
512 data
1.0Kinfo
119Gslot1
144Gslot2
115Gslot3
101Gslot4
80G slot5
157Gslot6
189Gslot7
117Gslot8
1019G total
Plus:
4.2G/
212M/export
212M/export/home
212M/export/home/tom
So, it looks like I do still have some big rocks to put in first
but on the surface of it, it looks like it should all fit in
anyway (Did I sum that up wrongly? Looks less than my tape
length.).
Thanks,
Tom
(BTW, your last email is not so much a diatribe as a oratory or
allocution).
On 22/10/14 03:31, Debra S Baddorf wrote:
Since nobody else is chiming in, I’ll have another go.
I don’t think there IS a dry-run of the taping process, since so
much depends on the timing of
Re: Backups to tape consistently under 60% tape capacity
On Oct 23, 2014, at 9:34 AM, Gene Heskett ghesk...@wdtv.com wrote:
On Thursday 23 October 2014 01:28:01 Tom Robinson did opine
And Gene did reply:
Now I have to work out why my tape is reporting as smaller! amtapetype
reports my tape is only half as big for the same block size...(was
1483868160 is now 743424512). :-/
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 67415370.8307692 bytes/sec
Wrote fixed (compressible) data at 273874944 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 761266700288 bytes at 57975 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (7612661760 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 743424512 kbytes
filemark 987 kbytes
speed 57975 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
On 23/10/14 03:19, Debra S Baddorf wrote:
Re-running amtapetype might be a very good idea. It might point to
where the problem isn’t, at least!
Do double check your cables. People have found problems in
cables which look like
reduced throughput. “Mpath” — I don’t know what it is, but could
it have changed with your OS upgrade?
Wouldn’t hurt to check that the tape driver setting haven’t
changed with the OS work …..
but otherwise, it sounds good.
Deb
On Oct 21, 2014, at 7:43 PM, Tom Robinson tom.robin...@motec.com.au
wrote:
Hmm, I did check my tape driver settings. When I set the tape
library up, I spent a long time getting the driver settings right
(on OmniOS) and took copious notes on the settings. My queries
reveal that I'm not using compression (which is what I wanted as
the vtapes are already compressed). LTO5 native is 1.5T; compressed
is 3T.
All my tapes are 'newish' (about one year old). The tape unit is the
same age.
For months I was consistently getting over 110% (highest 117%), then
capacity dropped once to 86% and then consistently to about 56%
(lowest 42%). Is there a block size issue (2x56=112)?
All the weekly dumps are local so network shouldn't be an issue. The
tape unit is using redundant SAS connectors. Maybe it's a mpath
thing?
Should I re-run amtapetype to see what it thinks the 'new' tape
capacity is after upgrading the OS?
On 22/10/14 10:47, Debra S Baddorf wrote:
Yeah, it sure does look like it ought to fit!
Could the tapes be dirty and not holding as much any more??? I
have no idea if that’s even possible. But it kinds seems like your
tapes are saying “I don’t want that much data”.Compression
issues?
Your tapes were previously getting 117% capacity, and now are
only doing 86%. Is that the general summary?
Unless somebody else can suggest some network (cable to tape
drive?) or system problems which might make the tapes appear
smaller than before? Compression issues? Read the original
message at the bottom of this email for the original problem
complaint.
Deb
On Oct 21, 2014, at 6:20 PM, Tom Robinson
tom.robin...@motec.com.au wrote:
Hi Debra,
A brilliant motivational speech. Thanks. Well worth the read. In
homage, I strongly suggest anyone who hasn't read it to go and do
that now. Here it is again for those whose mouse wheel is
dysfunctional:
http://www.appleseeds.org/Big-Rocks_Covey.htm
I will try your suggestions but want to make clear that the
virtual tapes you see are the product of a daily run which is
disk only. The weekly run puts all those daily dumps onto tape
which then leaves the building. So I have both virtual and real
tapes. The issues I'm having are in the weekly run (the dump to
real tape of a set of virtual tapes).
The tapes can be viewed as a bunch of big/little rocks. The total
amount of data, however they are stacked, should still fit on a
single LTO5 tape (amtapetype told me: length 1483868160 kbytes):
$ pwd
/data/backup/amanda/vtapes/daily
$ du -shc *
512 data
1.0Kinfo
119Gslot1
144Gslot2
115Gslot3
101Gslot4
80G slot5
157Gslot6
189Gslot7
117Gslot8
1019G total
Plus:
4.2G/
212M/export
212M/export/home
212M/export/home/tom
So, it looks like I do still have some big rocks to put in first
but on the surface of it, it looks like it should all fit in
anyway (Did I sum that up wrongly? Looks less than my tape
length.).
Thanks,
Tom
(BTW, your last email is not so much a diatribe as a oratory or
allocution).
On 22/10/14 03:31, Debra S Baddorf wrote:
Since nobody else is chiming in, I’ll have another go.
I don’t think there IS a dry-run of the taping process, since so
much depends on the timing of when a DLE is finished and ready
to go to tape, and
Re: Backups to tape consistently under 60% tape capacity
On Thursday 23 October 2014 12:41:59 Debra S Baddorf did opine And Gene did reply: On Oct 23, 2014, at 9:34 AM, Gene Heskett ghesk...@wdtv.com wrote: [huge snip] If you are feeding the tape device compressed files, and the drives compressor is enabled too, this will quite often cause file expansions on the tape itself. The drives compressor, because it is intended to handle the compression on the fly, is generally not sophisticated enough to do any further compression and will add to the datasize, expanding what actually goes down the cable to the drives heads. For that reason, and because it also tends to cripple amanda's estimation ability, most of us long term amanda users turned off the drives compressors at least a decade ago. Most drives record that compressor on/off status in a short file ahead of the tape content you can see, which makes it difficult to turn the compression off, so a procedure like this must be done to every tape in the house. 1. Rewind the tape. 2. Read out to a scratch file, the first 32 kilobytes of the tape, this is the amanda header. 3. Rewind the tape again. 4. Use mt to turn off the compression. 5. Immediately write that scratch file back to the tape so that this hidden flag on the tape is turned off. If you eject the tape and then reload it after turning off the compression without doing the immediate rewrite, the drive will scan that beginning of the tape to establish what kind of a tape its dealing with, and that will re-enable the compression flag you just turned off. So do not do anything but rewind it, turn off the compression, and re-write the amanda identification header, all in one swell foop. Cheers, Gene Heskett If you do this relabeling just before you re-use each tape, then you can also (rather than reading and re-writing the label) do amrmtape config tape-label do the above steps 3 and 4 (rewind and turn of compression) write the same label anew with amlabel -f ….. Not that this is conceptually any easier, really. Deb When I was forced to do that, I was screwing with a travan drive tapes at the time, circa 1999 or so. I tried that, but amlabel was moving the drive before the write, probably to see what label it was over-writing and that was restoring the flag somehow. So I effectively went to the head of the line made the drive an offer it could not refuse. 15 years later its possible it would now work, but someone with a tape drive needs to check it and see if it will now work, as I don't know haven't used any tape in 6 or 7 years. All vdisk on a hard drive, and currently on the 2nd terrabyte drive in all those years. Considerably faster backups, and very fast restores since the hard drive really is random access. This is such a PITA when dealing with real tapes, that what I would like to see is amlabel reaching into the amanda.conf, and reading the DRIVE_COMPRESSION label, and applying that unconditionally as it labels or re-labels a tape. Set it once and forget it IOW. And at my age, 80, I am not about to go back to the tapes drives I can afford (DDS-2's), those were crap. Cheers Deb, Gene Heskett -- There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) Genes Web page http://geneslinuxbox.net:6309/gene US V Castleman, SCOTUS, Mar 2014 is grounds for Impeaching SCOTUS
Re: Backups to tape consistently under 60% tape capacity
On Thu, Oct 23, 2014 at 10:34:38AM -0400, Gene Heskett wrote: On Thursday 23 October 2014 01:28:01 Tom Robinson did opine ... If you are feeding the tape device compressed files, and the drives compressor is enabled too, this will quite often cause file expansions on the tape itself. The drives compressor, because it is intended to handle the compression on the fly, is generally not sophisticated enough to do any further compression and will add to the datasize, expanding what actually goes down the cable to the drives heads. Tom is using an LTO drive (-5 I think). Most modern tape drives, including all LTO's do not exhibit the bad behavior of the DDS drives with their run-length encoding scheme. IIRC, they have enough cpu smarts and memory to first collect the data in memory, try to compress it to another another memory buffer, and if it is enlarged the block is saved uncompressed. Note, instead of a flag at the start of the tape indicating compressed or uncompressed, there is a flag for each tape block. jl -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (609) 477-8330 (C)
Re: Backups to tape consistently under 60% tape capacity
On Thu, Oct 23, 2014 at 04:28:01PM +1100, Tom Robinson wrote:
Now I have to work out why my tape is reporting as smaller! amtapetype
reports my tape is only half
as big for the same block size...(was 1483868160 is now 743424512). :-/
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 67415370.8307692 bytes/sec
Wrote fixed (compressible) data at 273874944 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 761266700288 bytes at 57975 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (7612661760 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 743424512 kbytes
filemark 987 kbytes
speed 57975 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
Note it is not only reporting the lower size, but dumps
are experiencing it as well.
IIRC, you are using an LTO-5. My peek at the web says
that format can record at up to 280Mbps. You are now
only seeing 58Mbps. Is that a big change from your
previous runs?
Feeding a drive too slowly, i.e. below its ability to
stream continuously, can reduce the apparent capacity
of the tape.
If this is the case you may have to find ways to
increase the data flow rate to your drive.
Jon
--
Jon H. LaBadie j...@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (609) 477-8330 (C)
Re: Backups to tape consistently under 60% tape capacity
On 24/10/14 07:48, Jon LaBadie wrote:
On Thu, Oct 23, 2014 at 10:34:38AM -0400, Gene Heskett wrote:
On Thursday 23 October 2014 01:28:01 Tom Robinson did opine
...
If you are feeding the tape device compressed files, and the drives
compressor is enabled too, this will quite often cause file expansions on
the tape itself. The drives compressor, because it is intended to handle
the compression on the fly, is generally not sophisticated enough to do
any further compression and will add to the datasize, expanding what
actually goes down the cable to the drives heads.
Tom is using an LTO drive (-5 I think). Most modern tape
drives, including all LTO's do not exhibit the bad behavior
of the DDS drives with their run-length encoding scheme.
IIRC, they have enough cpu smarts and memory to first
collect the data in memory, try to compress it to another
another memory buffer, and if it is enlarged the block
is saved uncompressed.
Note, instead of a flag at the start of the tape indicating
compressed or uncompressed, there is a flag for each tape
block.
jl
Thanks for all the feedback.
Jon is correct, I'm using an LTO5 drive.
Note that, about a year ago I posed this question (and received no response):
Tape Compression: Is it on or off
(https://www.mail-archive.com/amanda-users%40amanda.org/msg47097.html)
If you read that post, there is a riddle to be answered by checking the tape
driver flags to
determine if compression is on or off. I set the driver flags and used a
'non-compression' device
node, as per the man page (/dev/rmt/0bn or BSD style, no-compression,
no-rewind). All seemed well
until recently.
For reference, below are my amtapetype outputs from back then and from
yesterday. Notably,
compression always reports as 'enabled'. Also, I'm dumping the same, compressed
DLEs as before. I'm
not sure compression is the factor here.
Regards,
Tom
Back then (2014-10-14) amtapetype showed:
$ amtapetype -f -t ULT3580-TD5 weekly /dev/rmt/0bn
Checking for FSF_AFTER_FILEMARK requirement
device-property FSF_AFTER_FILEMARK false
Applying heuristic check for compression.
Wrote random (uncompressible) data at 73561559.3650794 bytes/sec
Wrote fixed (compressible) data at 193099093.33 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 1515988615168 bytes at 73464 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (15159885824 bytes) to determine filemark.
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 1480457632 kbytes
filemark 0 kbytes
speed 73464 kps
blocksize 32 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
today (2014-10-23) I get:
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 67415370.8307692 bytes/sec
Wrote fixed (compressible) data at 273874944 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 761266700288 bytes at 57975 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (7612661760 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 743424512 kbytes
filemark 987 kbytes
speed 57975 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
signature.asc
Description: OpenPGP digital signature
Re: Backups to tape consistently under 60% tape capacity
On Oct 23, 2014, at 3:48 PM, Jon LaBadie j...@jgcomp.com wrote: On Thu, Oct 23, 2014 at 10:34:38AM -0400, Gene Heskett wrote: On Thursday 23 October 2014 01:28:01 Tom Robinson did opine ... If you are feeding the tape device compressed files, and the drives compressor is enabled too, this will quite often cause file expansions on the tape itself. The drives compressor, because it is intended to handle the compression on the fly, is generally not sophisticated enough to do any further compression and will add to the datasize, expanding what actually goes down the cable to the drives heads. Tom is using an LTO drive (-5 I think). Most modern tape drives, including all LTO's do not exhibit the bad behavior of the DDS drives with their run-length encoding scheme. IIRC, they have enough cpu smarts and memory to first collect the data in memory, try to compress it to another another memory buffer, and if it is enlarged the block is saved uncompressed. Note, instead of a flag at the start of the tape indicating compressed or uncompressed, there is a flag for each tape block. jl -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (609) 477-8330 (C) Oooo — so with LTOx tapes one can forget that “turn off hardware compression by rewriting start of tapes” bit? I’ve just moved to LTO2 tapes, and will be going to LTO5 tapes as soon as my new hardware is setup. This would be good to know! Deb Baddorf Fermilab
Re: Backups to tape consistently under 60% tape capacity
On 24/10/14 07:59, Jon LaBadie wrote:
On Thu, Oct 23, 2014 at 04:28:01PM +1100, Tom Robinson wrote:
Now I have to work out why my tape is reporting as smaller! amtapetype
reports my tape is only half
as big for the same block size...(was 1483868160 is now 743424512). :-/
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 67415370.8307692 bytes/sec
Wrote fixed (compressible) data at 273874944 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 761266700288 bytes at 57975 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (7612661760 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 743424512 kbytes
filemark 987 kbytes
speed 57975 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
Note it is not only reporting the lower size, but dumps
are experiencing it as well.
IIRC, you are using an LTO-5. My peek at the web says
that format can record at up to 280Mbps. You are now
only seeing 58Mbps. Is that a big change from your
previous runs?
Feeding a drive too slowly, i.e. below its ability to
stream continuously, can reduce the apparent capacity
of the tape.
If this is the case you may have to find ways to
increase the data flow rate to your drive.
Jon
Thanks Jon.
I was getting 'speed 73464 kps', previously. So, yes, there is some performance
degradation. I never
saw anything close to 280Mbps in my tests with amtapetype. Is there another way
to accurately test
tape write performance?
Regards,
Tom
signature.asc
Description: OpenPGP digital signature
Re: Backups to tape consistently under 60% tape capacity
* Jon LaBadie j...@jgcomp.com [20141023 16:59]:
On Thu, Oct 23, 2014 at 04:28:01PM +1100, Tom Robinson wrote:
Now I have to work out why my tape is reporting as smaller! amtapetype
reports my tape is only half
as big for the same block size...(was 1483868160 is now 743424512). :-/
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 67415370.8307692 bytes/sec
Wrote fixed (compressible) data at 273874944 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 761266700288 bytes at 57975 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (7612661760 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 743424512 kbytes
filemark 987 kbytes
speed 57975 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
Note it is not only reporting the lower size, but dumps
are experiencing it as well.
IIRC, you are using an LTO-5. My peek at the web says
that format can record at up to 280Mbps. You are now
only seeing 58Mbps. Is that a big change from your
previous runs?
Feeding a drive too slowly, i.e. below its ability to
stream continuously, can reduce the apparent capacity
of the tape.
If this is the case you may have to find ways to
increase the data flow rate to your drive.
Jon
--
Jon H. LaBadie j...@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (609) 477-8330 (C)
Stepping in, I miss the earlier comments so maybe this is not
appropriate or OT, in which case just toss me in the dust bin.
Your length and speed are way off.
This is my tapetype for a HP Ultrium LTO-5
define tapetype tape-lto5 {
comment Created by amtapetype; compression disabled
length 1480900608 kbytes
filemark 3413 kbytes
speed 107063 kps
blocksize 2048 kbytes
part-size 100gb
part-cache-max-size 100gb
part-cache-type disk
part-cache-dir /holddisk
}
In this case, amtapetype disappointedly reported only ~100MBs (native
speed per specs is 140MBs) but in my local setup I frequently see
values up to 300MBs with 'averaged xfer rate' around the specs value,
eg, see the attached munin graph of my holdding disk performance from
last night run, the green line is data from the holdding disk to the
tape. LTO-5 will stream from 40MBs (I think) to 150Mbs, lower than
that you're shoe-shinning.
If the data xfer from the holdding disk to the drive can sustain the max
xfer data rate to the drive (140MBs) this suggest you have a dud or
experiencing other hardware issues. I would test the hardware directly
without amanda in the way using native OS tool, dd or whatever you
fancy.
cheers,
jf
--
° Jean-François Malouin
Systems/Network Manager | McConnell Brain Imaging Centre | MNI
3801 University St, Suite WB219, Montreal, QC, H3A 2B4, Canada
Re: Backups to tape consistently under 60% tape capacity
On 24/10/14 08:30, Jean-Francois Malouin wrote:
* Jon LaBadie j...@jgcomp.com [20141023 16:59]:
On Thu, Oct 23, 2014 at 04:28:01PM +1100, Tom Robinson wrote:
Now I have to work out why my tape is reporting as smaller! amtapetype
reports my tape is only half
as big for the same block size...(was 1483868160 is now 743424512). :-/
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 67415370.8307692 bytes/sec
Wrote fixed (compressible) data at 273874944 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 761266700288 bytes at 57975 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (7612661760 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 743424512 kbytes
filemark 987 kbytes
speed 57975 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
Note it is not only reporting the lower size, but dumps
are experiencing it as well.
IIRC, you are using an LTO-5. My peek at the web says
that format can record at up to 280Mbps. You are now
only seeing 58Mbps. Is that a big change from your
previous runs?
Feeding a drive too slowly, i.e. below its ability to
stream continuously, can reduce the apparent capacity
of the tape.
If this is the case you may have to find ways to
increase the data flow rate to your drive.
Jon
--
Jon H. LaBadie j...@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (609) 477-8330 (C)
Stepping in, I miss the earlier comments so maybe this is not
appropriate or OT, in which case just toss me in the dust bin.
Your length and speed are way off.
This is my tapetype for a HP Ultrium LTO-5
define tapetype tape-lto5 {
comment Created by amtapetype; compression disabled
length 1480900608 kbytes
filemark 3413 kbytes
speed 107063 kps
blocksize 2048 kbytes
part-size 100gb
part-cache-max-size 100gb
part-cache-type disk
part-cache-dir /holddisk
}
In this case, amtapetype disappointedly reported only ~100MBs (native
speed per specs is 140MBs) but in my local setup I frequently see
values up to 300MBs with 'averaged xfer rate' around the specs value,
eg, see the attached munin graph of my holdding disk performance from
last night run, the green line is data from the holdding disk to the
tape. LTO-5 will stream from 40MBs (I think) to 150Mbs, lower than
that you're shoe-shinning.
If the data xfer from the holdding disk to the drive can sustain the max
xfer data rate to the drive (140MBs) this suggest you have a dud or
experiencing other hardware issues. I would test the hardware directly
without amanda in the way using native OS tool, dd or whatever you
fancy.
Quite possibly the tape driver is at fault. I'm using a generic SCSI Tape
driver (st) together with
the SCSI Generic dirver (sgen) for the tape library. I tried installing the
IBMtape driver, but
failed to get it to work on OmniOS.
signature.asc
Description: OpenPGP digital signature
Re: Backups to tape consistently under 60% tape capacity
On Fri, Oct 24, 2014 at 08:23:55AM +1100, Tom Robinson wrote:
Thanks for all the feedback.
Jon is correct, I'm using an LTO5 drive.
Note that, about a year ago I posed this question (and received no response):
Tape Compression: Is it on or off
(https://www.mail-archive.com/amanda-users%40amanda.org/msg47097.html)
If you read that post, there is a riddle to be answered by checking the tape
driver flags to
determine if compression is on or off. I set the driver flags and used a
'non-compression' device
node, as per the man page (/dev/rmt/0bn or BSD style, no-compression,
no-rewind). All seemed well
until recently.
For reference, below are my amtapetype outputs from back then and from
yesterday. Notably,
compression always reports as 'enabled'. Also, I'm dumping the same,
compressed DLEs as before. I'm
not sure compression is the factor here.
Regards,
Tom
As reported on the manpage for amtapetype, the compression enabled
test can be fooled by several factors including very fast drives.
I think the test assumes that tape writing speed is the limiting
factor. Thus uncompressible data approximates the actual write
speed and if more compressible data can be written in the same
time, compression must be enabled.
My LTO experience is limited, but I wonder about block size.
You do not specify a tape block size in your amtapetype runs
(-b size) and thus are defaulting to 32KB. Is this also
true in your amdump runs, i.e. do you specify a blocksize in
your tapetype definition?
It is possible you can get better performance using a larger
tape block size. Check it out with a couple of amtapetype runs.
As to the performance, it seems lower than it should be and
lower than it used to be. What has changed? Have you added
more devices to the controller that the LTO-5 drive uses? Is
it still using the same controller? I recall some reports of
amanda users dedicating a high-performance controller to their
LTO drives. Otherwise they couldn't feed the drive as fast as
it could write.
Back then (2014-10-14) amtapetype showed:
$ amtapetype -f -t ULT3580-TD5 weekly /dev/rmt/0bn
Checking for FSF_AFTER_FILEMARK requirement
device-property FSF_AFTER_FILEMARK false
Applying heuristic check for compression.
Wrote random (uncompressible) data at 73561559.3650794 bytes/sec
Wrote fixed (compressible) data at 193099093.33 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 1515988615168 bytes at 73464 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (15159885824 bytes) to determine filemark.
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 1480457632 kbytes
filemark 0 kbytes
speed 73464 kps
blocksize 32 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
today (2014-10-23) I get:
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 67415370.8307692 bytes/sec
Wrote fixed (compressible) data at 273874944 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 761266700288 bytes at 57975 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (7612661760 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 743424512 kbytes
filemark 987 kbytes
speed 57975 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
End of included message
--
Jon H. LaBadie j...@jgcomp.com
11226 South Shore Rd. (703) 787-0688 (H)
Reston, VA 20190 (609) 477-8330 (C)
Re: Backups to tape consistently under 60% tape capacity
Tom Robinson
IT Manager/System Administrator
MoTeC Pty Ltd
121 Merrindale Drive
Croydon South
3136 Victoria
Australia
T: +61 3 9761 5050
F: +61 3 9761 5051
E: tom.robin...@motec.com.au
On 24/10/14 09:23, Jon LaBadie wrote:
On Fri, Oct 24, 2014 at 08:23:55AM +1100, Tom Robinson wrote:
Thanks for all the feedback.
Jon is correct, I'm using an LTO5 drive.
Note that, about a year ago I posed this question (and received no response):
Tape Compression: Is it on or off
(https://www.mail-archive.com/amanda-users%40amanda.org/msg47097.html)
If you read that post, there is a riddle to be answered by checking the tape
driver flags to
determine if compression is on or off. I set the driver flags and used a
'non-compression' device
node, as per the man page (/dev/rmt/0bn or BSD style, no-compression,
no-rewind). All seemed well
until recently.
For reference, below are my amtapetype outputs from back then and from
yesterday. Notably,
compression always reports as 'enabled'. Also, I'm dumping the same,
compressed DLEs as before. I'm
not sure compression is the factor here.
Regards,
Tom
As reported on the manpage for amtapetype, the compression enabled
test can be fooled by several factors including very fast drives.
I think the test assumes that tape writing speed is the limiting
factor. Thus uncompressible data approximates the actual write
speed and if more compressible data can be written in the same
time, compression must be enabled.
My LTO experience is limited, but I wonder about block size.
You do not specify a tape block size in your amtapetype runs
(-b size) and thus are defaulting to 32KB. Is this also
true in your amdump runs, i.e. do you specify a blocksize in
your tapetype definition?
It is possible you can get better performance using a larger
tape block size. Check it out with a couple of amtapetype runs.
As to the performance, it seems lower than it should be and
lower than it used to be. What has changed? Have you added
more devices to the controller that the LTO-5 drive uses? Is
it still using the same controller? I recall some reports of
amanda users dedicating a high-performance controller to their
LTO drives. Otherwise they couldn't feed the drive as fast as
it could write.
Sorry, I've copied the wrong output. Last year I did do the amtapetype test
specifying block size
512 kbytes. Here's what I got last year:
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 85721088 bytes/sec
Wrote fixed (compressible) data at 295261525.33 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 1519480995840 bytes at 85837 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (15194390528 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 1483868160 kbytes
filemark 868 kbytes
speed 85837 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
The recent test with amtapetype is also specifying block size of 512 kbytes:
Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 67415370.8307692 bytes/sec
Wrote fixed (compressible) data at 273874944 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 761266700288 bytes at 57975 kb/sec
Got LEOM indication, so drive and kernel together support LEOM
Writing smaller files (7612661760 bytes) to determine filemark.
device-property FSF_AFTER_FILEMARK false
define tapetype ULT3580-TD5 {
comment Created by amtapetype; compression enabled
length 743424512 kbytes
filemark 987 kbytes
speed 57975 kps
blocksize 512 kbytes
}
# for this drive and kernel, LEOM is supported; add
# device-property LEOM TRUE
# for this device.
As to what has changed, about 25 days ago (September 31, 2014) we upgraded the
OS (prompted by
issues unrelated to backup) but the tape performance was already suffering
degradation before that
(starting in August with degradation from 117% (2014-08-10) to 86% tape
capacity the following week,
then steadily declining weekly by week to 57.6%, 56.6%, 42.9%, 54.4%, etc.,
remaining at about 56%,
...).
Some speed tests with dd:
$ dd if=/dev/zero of=/data/backup/amanda/file1 bs=1024k count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 1.34568 s, 1.6 GB/s
$ time dd if=/data/backup/amanda/file1 of=/dev/rmt/0b bs=512k count=4000
4000+0 records in
4000+0 records out
2097152000 bytes (2.1 GB) copied, 13.4122 s, 156 MB/s
real0m13.427s
user0m0.015s
sys 0m1.519s
$ time dd if=/data/backup/amanda/file1 of=/dev/rmt/0b bs=512k count=4000
4000+0
Re: Backups to tape consistently under 60% tape capacity
On 24/10/14 08:30, Jean-Francois Malouin wrote:
Stepping in, I miss the earlier comments so maybe this is not
appropriate or OT, in which case just toss me in the dust bin.
Your length and speed are way off.
This is my tapetype for a HP Ultrium LTO-5
define tapetype tape-lto5 {
comment Created by amtapetype; compression disabled
length 1480900608 kbytes
filemark 3413 kbytes
speed 107063 kps
blocksize 2048 kbytes
part-size 100gb
part-cache-max-size 100gb
part-cache-type disk
part-cache-dir /holddisk
}
In this case, amtapetype disappointedly reported only ~100MBs (native
speed per specs is 140MBs) but in my local setup I frequently see
values up to 300MBs with 'averaged xfer rate' around the specs value,
eg, see the attached munin graph of my holdding disk performance from
last night run, the green line is data from the holdding disk to the
tape. LTO-5 will stream from 40MBs (I think) to 150Mbs, lower than
that you're shoe-shinning.
If the data xfer from the holdding disk to the drive can sustain the max
xfer data rate to the drive (140MBs) this suggest you have a dud or
experiencing other hardware issues. I would test the hardware directly
without amanda in the way using native OS tool, dd or whatever you
fancy.
Here are two files from the daily dump:
1021882368 Oct 22 21:02 slot4/00076.rook._data_data0.1
1094219252 Oct 17 20:32 slot7/00070.scion._home.1
Here's a tar directly to tape (run as amanda user):
$ time tar cvf /dev/rmt/0b slot7/00070.scion._home.1
slot4/00076.rook._data_data0.1
slot7/00070.scion._home.1
slot4/00076.rook._data_data0.1
real0m29.755s
user0m0.656s
sys 0m5.063s
$ echo '((1094219252+1021882368)/1000/1000)/29.755' | bc -l
71.11751369517728112922
Some speed tests with dd:
$ dd if=/dev/zero of=/data/backup/amanda/file1 bs=1024k count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 1.34568 s, 1.6 GB/s
$ time dd if=/data/backup/amanda/file1 of=/dev/rmt/0b bs=512k count=4000
4000+0 records in
4000+0 records out
2097152000 bytes (2.1 GB) copied, 13.4122 s, 156 MB/s
real0m13.427s
user0m0.015s
sys 0m1.519s
$ time dd if=/data/backup/amanda/file1 of=/dev/rmt/0b bs=512k count=4000
4000+0 records in
4000+0 records out
2097152000 bytes (2.1 GB) copied, 13.4404 s, 156 MB/s
real0m13.456s
user0m0.014s
sys 0m1.471s
$ time dd if=/dev/zero of=/dev/rmt/0b bs=512k count=4000
4000+0 records in
4000+0 records out
2097152000 bytes (2.1 GB) copied, 8.24686 s, 254 MB/s
real0m8.262s
user0m0.011s
sys 0m0.297s
$ time dd if=/dev/zero of=/dev/rmt/0b bs=512k count=4000
4000+0 records in
4000+0 records out
2097152000 bytes (2.1 GB) copied, 8.1345 s, 258 MB/s
real0m8.150s
user0m0.011s
sys 0m0.299s
Writing directly to tape with dd if much faster than reading from filesystem
and writing to tape. In
which case maybe I probably have a controller type issue?
signature.asc
Description: OpenPGP digital signature
Re: Backups to tape consistently under 60% tape capacity
On Fri, Oct 24, 2014 at 10:13:28 +1100, Tom Robinson wrote: upgraded the OS (prompted by issues unrelated to backup) but the tape performance was already suffering degradation before that (starting in August with degradation from 117% (2014-08-10) to 86% tape capacity the following week, then steadily declining weekly by week to 57.6%, 56.6%, 42.9%, 54.4%, etc., remaining at about 56%, This is a bit of a shot in the dark, but we had a situation with our LTO-2 drive where apparently the head-cleaning cartridge wasn't cleaning (though it acted normally when inserted). We were getting end-of-tape errors on our tapes with much less data than expected, in spite of various attempts to figure out what was wrong -- until we tried a new cleaning cartridge and suddenly tape capacities were more or less back to normal Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Backups to tape consistently under 60% tape capacity
On Thursday 23 October 2014 16:48:50 Jon LaBadie did opine And Gene did reply: On Thu, Oct 23, 2014 at 10:34:38AM -0400, Gene Heskett wrote: On Thursday 23 October 2014 01:28:01 Tom Robinson did opine ... If you are feeding the tape device compressed files, and the drives compressor is enabled too, this will quite often cause file expansions on the tape itself. The drives compressor, because it is intended to handle the compression on the fly, is generally not sophisticated enough to do any further compression and will add to the datasize, expanding what actually goes down the cable to the drives heads. Tom is using an LTO drive (-5 I think). Most modern tape drives, including all LTO's do not exhibit the bad behavior of the DDS drives with their run-length encoding scheme. IIRC, they have enough cpu smarts and memory to first collect the data in memory, try to compress it to another another memory buffer, and if it is enlarged the block is saved uncompressed. Note, instead of a flag at the start of the tape indicating compressed or uncompressed, there is a flag for each tape block. jl Interesting Jon. Finally some smarter drives. I'll try to keep that in mind. Cheers, Gene Heskett -- There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) Genes Web page http://geneslinuxbox.net:6309/gene US V Castleman, SCOTUS, Mar 2014 is grounds for Impeaching SCOTUS
