Re: amanda-3.4.5 does not fill one tape

[Jens Berg]

There was a discussion back in 2014 with subject "Backups to tape 
consistently under 60% tape capacity". I haven't read the whole lengthy 
thread but one participant mentioned that in his case a bad cleaning 
tape was found to be responsible for the capacity loss. Others reported 
that they had trouble with the compression settings. My gut feeling 
tells me that this is not the problem here but maybe your impression is 
a different one or you get a new idea if you read through the postings.


Jens

Am 13.05.2020 um 19:40 schrieb Debra S Baddorf:



On May 13, 2020, at 11:17 AM, Stefan G. Weichinger  wrote:

Now look at this run of

"amtapetype -b 128k /dev/nst0"

with another tape, FUJI instead of HP:

define tapetype LTO3-fuji {
comment "Created by amtapetype; compression disabled"
length 284180096 kbytes
filemark 20803 kbytes
speed 38376 kps
blocksize 128 kbytes
}

~290 GB ... faster, and large filemarks.

Maybe that drive is somehow failing .. ?



No real ideas here, but I *DO* recall some failing drives on this list
where the symptom was like this.  Tapes wouldn’t fill.

Anybody else still out there in the aether?

Deb Baddorf
Fermilab



--
Jens Berg  R  VTech IAD GmbH
Max-Giese-Str. 22, 24116 Kiel, Germany
Phone: +49 431 99081811
PGP key Id 0x45B55BB1
Location: Kiel, Local Court Kiel, HRB 10298 KI
Managing Director: Andreas Zipp
---
Please be informed that this email and any materials attached herewith
may contain confidential or legally privileged information that are
intended solely for the use by its named recipient. If you have
received this email in error, please notify us immediately by reply
email and delete this email from your system. Any disclosure, use,
copying, printing, distribution or reliance upon the contents of this
email is strictly prohibited. Thank you for your attention. - This mail
is sent via VTech IAD GmbH

Re: amanda-3.4.5 does not fill one tape

[Gene Heskett]

On Thursday 14 May 2020 17:30:53 Nathan Stratton Treadway wrote:

> On Thu, May 14, 2020 at 09:14:17 +0200, Stefan G. Weichinger wrote:
> > Interesting, how can a "dirty" drive trigger this behavior?
> >
> > I'd expect failures all along and not after ~200 or 300 GB written.
> >
> > I don't see any interrupted writing or so (until that End Of Tape).
>
> (We switched to disk-drive vtapes a long time ago so when I was last
> looking into the details of backup-tape-drive behavior it was probably
> for pre-LTO technology, but I would assume that for this discussion
> LTO is similar)
>
> For "modern" error-correcting tape drives, when the computer sends
> data out to the tape drive to be written to tape, the drive actually
> then uses the read head to immedately read back in the data it just
> wrote. If that read fails, the drive will automatically/transparently
> try the write again... repeating the process until it is able to
> achieve a successful confirmation read of that block of data.
>
> Normally this just happens once in a while, when there's a bad spot on
> the tape or some fluke of writing makes the data unreadable, and one
> doesn't even notice it's happening.
>
> However, if the drive head is dirty or the tape media in general is
> wearing out, then what happens is that many many many of the data
> blocks either will be written badly or will fail to read back in
> [depending on what exactly is dirty or failing], and the drive will
> have to re-write data multiple times before a succesful write/read
> cycle.
>
> When that happens, then lots of the linear space on the tape is used
> by all the repeated writes -- thus making the tape appear to have a
> lower capacity than you would expect -- and also all that re-writing
> means the data throughput from the server's point of view is much
> reduced.
>
> (Note that in this scenario the drive just keeps retrying to write a
> block up data until it succeeds... or until it hits the end of the
> tape. So that's why you don't get "interrupted writing" in the sense
> of having mid-tape write errors returned by the tape device the
> computer.  [But it is "interrupted" in the sense that a block takes
> much longer to write than it should so the computer has to wait a long
> time before it can sent the next block of data down to the drive.])
>
> Hope that makes sense.
>
>   Nathan
It makes perfect sense Nathan, what makes no sense is the drive 
continuing to try forever, continueing to wear out its heads when what 
it should if the 2nd or maybe the third write fails, it should at the 
very minimum start a blinking "clean me" led and abort the run. That at 
the very least would make me change the makers name on a frame rail 
someplace, its simply not excusable.

You buy these things looking for dependability, and I've never yet had a 
tape drive last over 1000 tape moving hours.

I have now removed it from service, but I've a 1t drive in a drawer I 
will likely use for my next linux install, still has the 25 re-allocated 
sectors it had the first time I asked it many years ago. That report 
caused me to update its firmware. And that spinning rust drive has over 
70,000 spinning hours on it now, retired because it was getting too 
small.

And people wonder why I use vtapes.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: amanda-3.4.5 does not fill one tape

[Jon LaBadie]

Glad you brought up this "feature" Nathan.  I had heard it before
but not using tape, promptly forgot it.

Jon


On Thu, May 14, 2020 at 05:30:53PM -0400, Nathan Stratton Treadway wrote:
> On Thu, May 14, 2020 at 09:14:17 +0200, Stefan G. Weichinger wrote:
> > Interesting, how can a "dirty" drive trigger this behavior?
> > 
> > I'd expect failures all along and not after ~200 or 300 GB written.
> > 
> > I don't see any interrupted writing or so (until that End Of Tape).
> 
> 
> (We switched to disk-drive vtapes a long time ago so when I was last
> looking into the details of backup-tape-drive behavior it was probably
> for pre-LTO technology, but I would assume that for this discussion LTO
> is similar)
> 
> For "modern" error-correcting tape drives, when the computer sends data
> out to the tape drive to be written to tape, the drive actually then
> uses the read head to immedately read back in the data it just wrote. 
> If that read fails, the drive will automatically/transparently try the
> write again... repeating the process until it is able to achieve a
> successful confirmation read of that block of data.
> 
> Normally this just happens once in a while, when there's a bad spot on
> the tape or some fluke of writing makes the data unreadable, and one
> doesn't even notice it's happening.  
> 
> However, if the drive head is dirty or the tape media in general is
> wearing out, then what happens is that many many many of the data blocks
> either will be written badly or will fail to read back in [depending on
> what exactly is dirty or failing], and the drive will have to re-write
> data multiple times before a succesful write/read cycle.
> 
> When that happens, then lots of the linear space on the tape is used by
> all the repeated writes -- thus making the tape appear to have a lower
> capacity than you would expect -- and also all that re-writing means the
> data throughput from the server's point of view is much reduced.
> 
> (Note that in this scenario the drive just keeps retrying to write a
> block up data until it succeeds... or until it hits the end of the tape. 
> So that's why you don't get "interrupted writing" in the sense of having
> mid-tape write errors returned by the tape device the computer.  [But it
> is "interrupted" in the sense that a block takes much longer to write
> than it should so the computer has to wait a long time before it can
> sent the next block of data down to the drive.])
> 
> Hope that makes sense.
> 
>   Nathan
> 
> 
> Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
> Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
>  GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
>  Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda-3.4.5 does not fill one tape

[Nathan Stratton Treadway]

On Thu, May 14, 2020 at 09:14:17 +0200, Stefan G. Weichinger wrote:
> Interesting, how can a "dirty" drive trigger this behavior?
> 
> I'd expect failures all along and not after ~200 or 300 GB written.
> 
> I don't see any interrupted writing or so (until that End Of Tape).


(We switched to disk-drive vtapes a long time ago so when I was last
looking into the details of backup-tape-drive behavior it was probably
for pre-LTO technology, but I would assume that for this discussion LTO
is similar)

For "modern" error-correcting tape drives, when the computer sends data
out to the tape drive to be written to tape, the drive actually then
uses the read head to immedately read back in the data it just wrote. 
If that read fails, the drive will automatically/transparently try the
write again... repeating the process until it is able to achieve a
successful confirmation read of that block of data.

Normally this just happens once in a while, when there's a bad spot on
the tape or some fluke of writing makes the data unreadable, and one
doesn't even notice it's happening.  

However, if the drive head is dirty or the tape media in general is
wearing out, then what happens is that many many many of the data blocks
either will be written badly or will fail to read back in [depending on
what exactly is dirty or failing], and the drive will have to re-write
data multiple times before a succesful write/read cycle.

When that happens, then lots of the linear space on the tape is used by
all the repeated writes -- thus making the tape appear to have a lower
capacity than you would expect -- and also all that re-writing means the
data throughput from the server's point of view is much reduced.

(Note that in this scenario the drive just keeps retrying to write a
block up data until it succeeds... or until it hits the end of the tape. 
So that's why you don't get "interrupted writing" in the sense of having
mid-tape write errors returned by the tape device the computer.  [But it
is "interrupted" in the sense that a block takes much longer to write
than it should so the computer has to wait a long time before it can
sent the next block of data down to the drive.])

Hope that makes sense.

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: amanda-3.4.5 does not fill one tape

[Jon LaBadie]

On Thu, May 14, 2020 at 09:14:17AM +0200, Stefan G. Weichinger wrote:
> Am 14.05.20 um 07:59 schrieb Jens Berg:
> > There was a discussion back in 2014 with subject "Backups to tape
> > consistently under 60% tape capacity". I haven't read the whole lengthy
> > thread but one participant mentioned that in his case a bad cleaning
> > tape was found to be responsible for the capacity loss. Others reported
> > that they had trouble with the compression settings. My gut feeling
> > tells me that this is not the problem here but maybe your impression is
> > a different one or you get a new idea if you read through the postings.
> 
> Thanks for the pointer.
> 
> Interesting, how can a "dirty" drive trigger this behavior?
> 

Because somewhere along the line, some piece of the software does
not distinguish an error return as distinct from an EOF or EOM.
By the time it gets to the human, it is an EOF even if it was an
error.

For example, some writing functions return only success/failure.

Is that failure the end of tape or some other error?  Does the code
assume one and not investigate further?  If the code does investigate
further, is there a way to tell the upstream code success/EOF/error
or just two statuses?

> But it's worth a try, the customer for sure hasn't used the cleaning
> tape for years.

Sounds like a plan.  Maybe two runs?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda-3.4.5 does not fill one tape

[Gene Heskett]

On Thursday 14 May 2020 01:32:05 Jon LaBadie wrote:

> On Wed, May 13, 2020 at 06:17:02PM +0200, Stefan G. Weichinger wrote:
> > Now look at this run of
> >
> > "amtapetype -b 128k /dev/nst0"
> >
> > with another tape, FUJI instead of HP:
> >
> > define tapetype LTO3-fuji {
> > comment "Created by amtapetype; compression disabled"
> > length 284180096 kbytes
> > filemark 20803 kbytes
> > speed 38376 kps
> > blocksize 128 kbytes
> > }
> >
> > ~290 GB ... faster, and large filemarks.
> >
> > Maybe that drive is somehow failing .. ?
>
> I wasn't sure what LEOM was.  I assume it is "Logical End of Media".
>
> Anyway I came across two references that said need for cleaning
> is one reason for getting early EOM.

At this point I would throw in that these drives work on the same 
principles as a video tape recorder, and when I walked in the door 
in '84 to be the CE at a tv station, hey were using freon TF to clean 
tape heads with, and doing it 2-3 times a day on every machine.  When 
the freon's started to get a bad rap, I switched to paint thinner 
alcohol, and made an amazing discovery, suddenly the machines were going 
a week before cleanings. But the tape decks used here, are wrapped up in 
the mechanics and not readily accessible for such q-tip based cleaning.

I've also observed that tape stored at 40F and 10% humidity is only about 
2% as abrasive as tape stored at room temp and humidity.  So putting a 
tape library in a small closet on an outside wall, with a too small AC 
running forever to not only keep it cold but suck all the humidity out 
of the air can be very helpfull.

At Nebraska ETV, half the state is in the mountain time zone, so at one 
of the tv stations they had 3 of the old 2" Ampex machines, serving as 
the 1 hour time delay, and that room was kept at around 55F with a 
precipitron equipt HVAC. People, but no smoking, food or drink allowed.

Head life in a normal environment for a 'soft' head on those machines is 
around 750 hours. After a while they had a transformer failure and had 
to swap heads for a fresh one.  It was at that point they read the old 
heads spinning hours meter and found it had 9700 hours on it.  Sent back 
to the rebuilder, he fixed the transformer and sent it back.  Put back 
in the machine it finally failed at nearly 12,000 hours. Pretty good for 
a head with a pro-rated 750 hour warranty. More than demoing the effect 
of environment on such stuff.

> I'm wondering also if this could be a case of Amanda tapes being
> labelled with the mode set to LTO-2 capacity.  I know you check
> the mode and it shows 44, but Amanda always reads the tape before
> writing.  Could this be setting the mode back to 42 because the
> tapes were initially labelled with the mode set incorrectly?

In which case the fix is to read the label block and save it, rewind the 
tape but do NOT remove it giving the drive a chance to re-read the tapes 
hidden header, use mtx or whatever to set it correctly and rewrite the 
label block.  The drive will then fix that bad ID in the header before 
it writes the label block. If the tape is removed you have to start all 
over again because the drive will scan the hidden header and restore the 
erronious settings.  BTDT, pain in the butt.

> What if you forget about amtapetype and simply use dd to see how
> much random data it will write to tape.

Since the above fix is not a full pass run, but only a few seconds if a 
script does it, write the script and do the above first.  Much faster.

> Jon



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 14.05.20 um 07:59 schrieb Jens Berg:
> There was a discussion back in 2014 with subject "Backups to tape
> consistently under 60% tape capacity". I haven't read the whole lengthy
> thread but one participant mentioned that in his case a bad cleaning
> tape was found to be responsible for the capacity loss. Others reported
> that they had trouble with the compression settings. My gut feeling
> tells me that this is not the problem here but maybe your impression is
> a different one or you get a new idea if you read through the postings.

Thanks for the pointer.

Interesting, how can a "dirty" drive trigger this behavior?

I'd expect failures all along and not after ~200 or 300 GB written.

I don't see any interrupted writing or so (until that End Of Tape).

But it's worth a try, the customer for sure hasn't used the cleaning
tape for years.

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 14.05.20 um 07:32 schrieb Jon LaBadie:

> I wasn't sure what LEOM was.  I assume it is "Logical End of Media".

Seems like, yes. I still haven't figured out how to set that, the
"device_property" line wasn't accepted by Amanda.

> Anyway I came across two references that said need for cleaning
> is one reason for getting early EOM.
> 
> I'm wondering also if this could be a case of Amanda tapes being
> labelled with the mode set to LTO-2 capacity.  I know you check
> the mode and it shows 44, but Amanda always reads the tape before
> writing.  Could this be setting the mode back to 42 because the
> tapes were initially labelled with the mode set incorrectly?

I used two new tapes in the last days. The labelling happened after
"stinit", and I even rm-ed the header with dd. etc etc

> What if you forget about amtapetype and simply use dd to see how
> much random data it will write to tape.

Would be a test, yes, but wouldn't help if amanda does things differently.

I mailed the customer and have them look for their cleaning tape.

After a cleaning run (if they have and find the tape) I try some dd-test
run, ok.