Re: bad tape?

[Olivier]

Gene Heskett  writes:

> Generally speaking, only because the disc is random access.

But a disk dedicated to vtapes should be doing a lot of sequetial
accesses: once it has been formatted and the slots have been assigned,
it is writting files the size of one Amanda's chunk. In fact, that would
be worth a study: the disk usage for vtapes vs. normal disk usage.

That is just gross figures but:

Users' home directories:
FilesystemSizeUsed   Avail Capacity  Mounted on
/dev/da1p12.9T851G1.8T31%/home
2565312 files, 223129681 used, 556890331 free
(564355 frags, 69540747 blocks, 0.1% fragmentation)

Amanda vtape disk:
FilesystemSizeUsed   Avail Capacity  Mounted on
/dev/ada5p1   2.6T2.2T269G89%/automnt/ada5
475 files, 582393950 used, 127171372 free
(84 frags, 15896411 blocks, 0.0% fragmentation)

The vtape disk is slightly older than the users' home, definitely fuller
and less fragmented, so I would guess big sequetial files with little
head movement.

Good luck with your health.

Olivier

Re: bad tape?

[Gene Heskett]

On Monday 26 August 2019 21:42:29 Olivier wrote:

> ghe  writes:
> > Stan and Debra have convinced me to bite the bullet and buy a new
> > tape. I've never been in this situation before (the DLT drive used
> > to fail every once in a while, but a couple hours with a jeweler's
> > screwdriver got it going again).
> >
> > Looks like I'm going to have a spare, mildly flaky, tape around.
>
> You mean that all your backups were on a single tape? That is beyond
> daring IMHO.
>
> Like Gene mentioned, I have moved to disk and vtapes, never had a
> problem with the disks holding the vtape, I had problems with the disk
> with the holding disk though, it started develop bad blocks in the
> holding disk partition, but vtapes? They are hardly used, if you
> unmount/stop them after the backups are written, they can last a long
> time.
>
> And when you need more space, you just upgrade with newer disks with
> more capacity and store the old disks, so you even have offline old
> stuff.
>
> Your slots need to be numbered starting from 1, but the tapes number
> can start from the next number in sequence in your tape naming scheme,
> so keep the old disks. You will change them because you need more
> backup storage, not because they fail (never had since I use disks,
> while I had to replace the tapes a number of times).
>
> I started with 7x512GB disks, then 7x1.5TB and am now at 7x3TB
> and never had the slightest problem. Admittedly, the size of the
> vtapes chosen at first looks a bit tiny nowdays, but with vtapes, it
> really adds very little overhead, largely compensated by avoiding half
> full tapes.
>
> When we were hit by the great flood and we had to move the datacenter,
> I took the disks, but not Amanda server, as I knew I could connect the
> disk directly inside any server to do a restore.
>
> And no fiddling with mounting and dismounting tapes every day, I have
> a system with all my vtapes available all the time.
>
> In a similar way to what Gene described, after the dump, I get a copy
> of the indexes, etc. and rsync that to a database server and mail
> myself a copy, I end up with 5 or 6 copies of that critical
> information (because database and mail are backed up too).
>
> And remeber, tapes do rub on the R/W head of the tape drive so doing
> an amcheckdump does double the wear and tear of the tape. 
That also doubles the head wear.  Since disk heads fly on a film of air 
that might be under a micron thick, if the disc is well finished, the 
only contact is at a power fail, when it has no choice but to land on 
the still spinning disc. Then the successfull restart depends on how 
long it takes to get power back.  Past a month or so sitting there, the 
head is likely stuck to the disc like a machinists set of joe's blocks 
and the disc is locked from spinning up again.

I have a pair of one GB scsi drives on a trs-80 color computer that 
haven't been shut down for more than a couple months since about 1989, 
and had to dismount them so the projected 1/2" from the front of the 
tower they are in, power restored, and an 8oz dead blow hammer applied 
to the front corner enough to bump it sideways 1/8", which is enough to 
rotate the housing about the disc, breaking the stiction. And once the 
disc has moved 1/4 turn its speed supplies the air cushion under the 
head.  That was about 2 years ago. I have a 20 kw standby thats 
typically up and running before the disc's have stopped, so a tap on the 
reset button reboots it normally. Those old drives have had since around 
1989, collecting spinning hours at 8766 a year, so have close to 166554 
spinning hours. Still functioning at 100%.  No bad sectors.

I think thats pretty decent testimony to convince anyone into never 
shutting a drive off

They will happily outlast me. I've just spent 2 weeks in the shop from a 
heart attack, and still look like your grandmothers pincushion from all 
the taps they put in to keep me going. Currrantly its pumping around 
30%, so I'm not pushing myself too hard just yet as they are building a 
new aorta valve they will install as a cath-lab outpatient some time 
next month.

> No physical 
> contact between the head and support on disk.
>
> Not to mention that disks are way faster...
>
Generally speaking, only because the disc is random access. Actual data 
rate to/from the head is rarely more than one magnitude faster for the 
disc. And that differential is more in the teenyness of the head 
allowing a higher frequency than in any other magic.

> Best regards,
>
> Olivier



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: bad tape?

[Charles Curley]

On Mon, 26 Aug 2019 16:01:58 -0400
Gene Heskett  wrote:

> But AFAIK, tapes don't maintain an allocation map, and we have no
> tape writing tools so organized as to be able to implement such a
> scheme.

Not entirely true.

Tape drives such as Colorado Memory Systems' QIC drives were random
access, because they used the floppy controller to access the tape.
Firmware on the drive translated head, track and sector selection into
tracks and sectors on the tape drive.

Also, many years ago, Forth, Inc wrote a driver for a DEC tape drive
that did random access. The client wanted *rugged* and tape drives in
those days were far more rugged than hard drives. Forth doesn't use
FATs; indeed, it doesn't use files.

Yes, it was slow, compared to hard drives even then. Both systems were
inclined toward a lot of shoe-shining, which wears out tapes.

Whether any modern tape drives have any analogous capabilities I cannot
say.

Today's computer trivia was brought to you by the Stonehenge Retro
Computing Club.

-- 
"When we talk of civilization, we are too apt to limit the meaning of
the word to its mere embellishments, such as arts and sciences; but
the true distinction between it and barbarism is, that the one
presents a state of society under the protection of just and
well-administered law, and the other is left to the chance government
of brute force."
- The Rev. James White, Eighteen Christian Centuries, 1889
Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
https://charlescurley.com

Re: bad tape?

[Olivier]

ghe  writes:

> Stan and Debra have convinced me to bite the bullet and buy a new tape.
> I've never been in this situation before (the DLT drive used to fail
> every once in a while, but a couple hours with a jeweler's screwdriver
> got it going again).
>
> Looks like I'm going to have a spare, mildly flaky, tape around.

You mean that all your backups were on a single tape? That is beyond
daring IMHO.

Like Gene mentioned, I have moved to disk and vtapes, never had a
problem with the disks holding the vtape, I had problems with the disk
with the holding disk though, it started develop bad blocks in the
holding disk partition, but vtapes? They are hardly used, if you
unmount/stop them after the backups are written, they can last a long
time.

And when you need more space, you just upgrade with newer disks with
more capacity and store the old disks, so you even have offline old
stuff.

Your slots need to be numbered starting from 1, but the tapes number can
start from the next number in sequence in your tape naming scheme, so
keep the old disks. You will change them because you need more backup
storage, not because they fail (never had since I use disks, while I had
to replace the tapes a number of times).

I started with 7x512GB disks, then 7x1.5TB and am now at 7x3TB
and never had the slightest problem. Admittedly, the size of the vtapes
chosen at first looks a bit tiny nowdays, but with vtapes, it really
adds very little overhead, largely compensated by avoiding half full
tapes.

When we were hit by the great flood and we had to move the datacenter, I
took the disks, but not Amanda server, as I knew I could connect the
disk directly inside any server to do a restore.

And no fiddling with mounting and dismounting tapes every day, I have a
system with all my vtapes available all the time.

In a similar way to what Gene described, after the dump, I get a copy of
the indexes, etc. and rsync that to a database server and mail myself a
copy, I end up with 5 or 6 copies of that critical information (because
database and mail are backed up too).

And remeber, tapes do rub on the R/W head of the tape drive so doing an
amcheckdump does double the wear and tear of the tape. No physical
contact between the head and support on disk.

Not to mention that disks are way faster...

Best regards,

Olivier

Re: bad tape?

[ghe]

On 8/26/19 5:20 PM, Nathan Stratton Treadway wrote:

> I haven't looked closely at tape drives in a few years, but others on
> this list certainly have in-depth experience with them.  I think some of
> the specific answers do depend on what tape-drive technology/generation
> you are using, so it might help for you to post that info.

It's an LTO-5.

> Also, "because of a tape error" is pretty vague; knowing the exact error
> message might make a difference in what advice we can give you

Well, it's been more than 30 minutes ago, so I can't remember exactly
what the massage was. But I think it was something about some of the
DLEs didn't copy because of a tape failure, but they're on the holding
disk. Nothing real specific.

> So, if you have a tape that is starting to "go bad" overall, usually the
> symptom is that Amanda hits the end of the tape after writing less data
> than the expected capacity for the tape -- rather than some sort of "bad
> block" error.

Yup. That's me.

> A related thing to keep in mind is that the writes can fail because
> specific spots on a specfic tape are bad...

That's what I think is going on.

> (amtapetype does have an "-l" option so you could tell it to re-create
> the label that is already in the header file on the tape, 

No, I wasn't wanting to validate the data. Just the tape itself, without
destroying the Amanda header -- I think the -l option to amtapetype is
what I'm looking for.

Stan and Debra have convinced me to bite the bullet and buy a new tape.
I've never been in this situation before (the DLT drive used to fail
every once in a while, but a couple hours with a jeweler's screwdriver
got it going again).

Looks like I'm going to have a spare, mildly flaky, tape around.

> (Denise mentioned the "amcheckdump" utility.  

My backup script rewinds and runs amcheckdump. I was told a long time
ago "An unverified backup isn't a backup"

Thanks all.

-- 
Glenn English

Re: bad tape?

[Debra S Baddorf]

Cheap is defined in comparison with “how long will it take to recreate all that 
data by hand,
if it’s lost”.  The value of that may depend on your reason for doing a backup.

Deb Baddorf
Fermilab

> On Aug 26, 2019, at 6:39 PM, ghe  wrote:
> 
> On 8/26/19 4:16 PM, stan wrote:
> 
>> Tapes are cheap! 
> 
> Define 'cheap' :-)
> 
> There's a significant difference between what Google and I consider cheap...
> 
>> What technology BTW.
> 
> LTO-5
> 
> I'm a newcomer from the DLT world. Those were reasonably cheap by my
> definition...
> 
> -- 
> Glenn English

Re: bad tape?

[ghe]

On 8/26/19 2:01 PM, Gene Heskett wrote:

> I did that once for a very old hard drive, permanently allocating about 
> 30 sectors to a file named badsectors.fd.  Worked great.

That's a clever idea...

> But AFAIK, tapes don't maintain an allocation map, and we have no tape 
> writing tools so organized as to be able to implement such a scheme.
> 
> Because tape is sequential, any such attempt would have to maintain such 
> a mapping on separate random access media.
> 
> The drive would have to be able to seek past the damaged tape to the next 
> block with a good checksum, something I've not found a drive capable of 
> doing in around a decade of trying, they all stop and error on the bad 
> spot.

Ada Lovelace era technology. That's what I suspected. But one can always
hope.

> The error correction in the average terabyte class spinning rust drive 
> has far exceeded that of tape, yes I'm talking about the sub $100 drive 
> here, and I haven't lost a single bit of data in the 15+ years since I 
> converted to vtape. 

I'm on tape instead of disk because it's disks we're backing up so we'll
still have data when the disk goes south. Or when the computer looses
it's mind and scribbles all over the disk(s).

> you can read without using anything in the 
> amanda ammo box at all if push comes to shove while doing a bare metal 
> recovery. 

Yeah, they claim that with tape too. I haven't yet had the pleasure.

-- 
Glenn English

Re: bad tape?

[ghe]

On 8/26/19 4:16 PM, stan wrote:

> Tapes are cheap! 

Define 'cheap' :-)

There's a significant difference between what Google and I consider cheap...

> What technology BTW.

LTO-5

I'm a newcomer from the DLT world. Those were reasonably cheap by my
definition...

-- 
Glenn English

Re: bad tape?

[Nathan Stratton Treadway]

On Fri, Aug 16, 2019 at 11:38:45 -0600, ghe wrote:
> I have reason to believe that one of my tapes isn't working properly
> (last night's backup died without finishing because of a tape error --
> the retry running right now successfully flushed last night's and wrote
> a new one).

I haven't looked closely at tape drives in a few years, but others on
this list certainly have in-depth experience with them.  I think some of
the specific answers do depend on what tape-drive technology/generation
you are using, so it might help for you to post that info.

Also, "because of a tape error" is pretty vague; knowing the exact error
message might make a difference in what advice we can give you

> Is there an Amanda utility that will validate a tape without destroying
> the Amanda header at the front of the tape? And maybe build a 'bad
> sector table' on the tape?

In my experence (which I guess was an LTO-2 drive), bad spots on the
tape were handled transparently by the drive. As Gene mentioned, tapes
are sequential, so there aren't fixed "blocks" like there are on disk
drives.  Instead, the drive just knows that it has some data it needs to
write, and writes out that data onto the bit of tape currently streaming
past the write head, then attempts to re-read that just-written data
with the read head as the tape goes by there.  If the read is
successful the drive considers that data to have been successfully
written and moves on to the next chunk of date; if the read fails, the
drive will write that same data again onto the next segment of tape,
repeating the process until the write/read cycle succeeds.

(Put another way, the exact spot on the tape that gigabyte number X is
written onto the tape will vary from run to run, depending on all sorts
of factors happening earlier in the the run.  But it doesn't matter,
because the drive never tries to find data based on how many centimeters
of tape have passed from the front of the tape, but instead scans along
the tape bit by bit reading data, or at least looking for file markers,
as it goes.)

So, if you have a tape that is starting to "go bad" overall, usually the
symptom is that Amanda hits the end of the tape after writing less data
than the expected capacity for the tape -- rather than some sort of "bad
block" error.

A related thing to keep in mind is that the writes can fail because
specific spots on a specfic tape are bad... or because e.g. the write
head is dirty, which has nothing to do with the specific tape in use.

So, what I did was watch my Amanda reports to see if a specific tape
consistently gave me an "out of tape" error each type it was used, and
when that happened I figured the tape was getting worn out and removed
it from the cycle.  (We ran the cleaning tape on the weekly cycle or
whatever the drive manufacturer recommended, which was not an even
fraction of the number of tapes in rotation, so from use to use a
particular tape would sometimes be used shortly after the heads were
cleaned -- and thus we could be pretty sure that it really was a bad
tape if the error happened consistently over many cycles.)

Regarding utilities to "validate" the tape: as far as I remember only
"amtapetype" is related to checking a tape to see its capacity.  So if
you wanted to more directly check that a particular tape had a much
lower than expected capacity you could use amtapetype to see but
really all that can do is confirm what Amanda is already seeing (i.e.
the expected dumps no longer fit on that tape).  And such a test would
overwrite any dumps currently on the tape, which you probably don't want
to do at least until enough time has passed that those dumps have all
become obsolete by later runs.  

(amtapetype does have an "-l" option so you could tell it to re-create
the label that is already in the header file on the tape, but off hand I
am not sure if using that will cause Amanda to properly delete the
previous run found on that tape from the index, etc)

(Denise mentioned the "amcheckdump" utility.  This reads through an
existing Amanda tape and verifies all the dumps found on it can be
successfully read back -- something that is a good idea to do if a tape
seems to be failing just in case it is failing so badly that the drive's
normal rewriting-data-until-the-test-read-succeeds doesn't actually
result in a tape that can be read back later -- but a different task
than attempting to write to the tape to see how much data it can
currently store.)

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: bad tape?

[Gene Heskett]

On Friday 16 August 2019 13:38:45 ghe wrote:

> I have reason to believe that one of my tapes isn't working properly
> (last night's backup died without finishing because of a tape error --
> the retry running right now successfully flushed last night's and
> wrote a new one).
>
> Is there an Amanda utility that will validate a tape without
> destroying the Amanda header at the front of the tape? And maybe build
> a 'bad sector table' on the tape?

I did that once for a very old hard drive, permanently allocating about 
30 sectors to a file named badsectors.fd.  Worked great.

But AFAIK, tapes don't maintain an allocation map, and we have no tape 
writing tools so organized as to be able to implement such a scheme.

Because tape is sequential, any such attempt would have to maintain such 
a mapping on separate random access media.

The drive would have to be able to seek past the damaged tape to the next 
block with a good checksum, something I've not found a drive capable of 
doing in around a decade of trying, they all stop and error on the bad 
spot.

The error correction in the average terabyte class spinning rust drive 
has far exceeded that of tape, yes I'm talking about the sub $100 drive 
here, and I haven't lost a single bit of data in the 15+ years since I 
converted to vtape. by having amanda write to virtual tapes which are 
nothing but directories you can read without using anything in the 
amanda ammo box at all if push comes to shove while doing a bare metal 
recovery. In fact, because the amanda index's are only valid after the 
run has been completed, I have written a wrapper script that collects 
that, and the configuration files, and writes then all into the 
directory on the hd that it just used as a virtual tape. That has the 
effect of, in case of bare metal on a clean drive, bringing the recovery 
to a day fresher status.

I have a machine that normally is part of the backup, with a lightning  
damaged psu, so that machine is missing from last nights backup. "data" 
is a softlink to the last runs vtape:
gene@coyote:~$ sudo ls -l /amandatapes/Dailys/data/
total 26361140
-rw--- 1 amanda amanda   32768 Aug 26 02:03 0.Dailys-35
-rw--- 1 amanda amanda  133105 Aug 26 02:03 
1.lathe._var_lib_amanda.1
-rw--- 1 amanda amanda  120881 Aug 26 02:03 
2.shop._var_lib_amanda.1
-rw--- 1 amanda amanda  103947 Aug 26 02:04 
3.lathe._usr_src.1
-rw--- 1 amanda amanda   99164 Aug 26 02:04 4.shop._home.1
-rw--- 1 amanda amanda   53242 Aug 26 02:04 5.lathe._etc.1
-rw--- 1 amanda amanda   53365 Aug 26 02:04 6.shop._etc.1
-rw--- 1 amanda amanda13327331 Aug 26 02:04 7.picnc._.1
-rw--- 1 amanda amanda   49184 Aug 26 02:04 8.lathe._home.1
-rw--- 1 amanda amanda   36715 Aug 26 02:04 
9.shop._lib_firmware.1
-rw--- 1 amanda amanda   34105 Aug 26 02:04 00010.picnc._boot.1
-rw--- 1 amanda amanda   36713 Aug 26 02:05 
00011.lathe._lib_firmware.1
-rw--- 1 amanda amanda   35314 Aug 26 02:05 00012.shop._root.1
-rw--- 1 amanda amanda   35244 Aug 26 02:05 00013.lathe._root.1
-rw--- 1 amanda amanda   33712 Aug 26 02:05 
00014.shop._usr_local.0
-rw--- 1 amanda amanda   33713 Aug 26 02:05 
00015.lathe._usr_local.0
-rw--- 1 amanda amanda   32938 Aug 26 02:05 
00016.shop._var_amanda.0
-rw--- 1 amanda amanda   32939 Aug 26 02:05 
00017.lathe._var_amanda.0
-rw--- 1 amanda amanda   33043 Aug 26 02:05 
00018.shop._usr_lib_amanda.1
-rw--- 1 amanda amanda   33044 Aug 26 02:06 
00019.lathe._usr_lib_amanda.1
-rw--- 1 amanda amanda 12046283958 Aug 26 02:17 
00020.coyote._home_gene_Download.0
-rw--- 1 amanda amanda   187740857 Aug 26 02:18 
00021.coyote._home_gene.3
-rw--- 1 amanda amanda10867557 Aug 26 02:18 
00022.coyote._home_gene_Mail.3
-rw--- 1 amanda amanda   33569 Aug 26 02:18 
00023.coyote.Downloadsnz.1
-rw--- 1 amanda amanda  840726 Aug 26 02:18 
00024.coyote._home_amanda.1
-rw--- 1 amanda amanda  252550 Aug 26 02:18 
00025.coyote._home_gene_src.1
-rw--- 1 amanda amanda  159619 Aug 26 02:18 
00026.coyote.Downloadsam.1
-rw--- 1 amanda amanda   37341 Aug 26 02:18 
00027.coyote._home_ups.1
-rw--- 1 amanda amanda   35159 Aug 26 02:18 
00028.coyote._home_nut.1
-rw--- 1 amanda amanda   638043961 Aug 26 02:20 
00029.coyote._GenesAmandaHelper-0.61_config-bak.3
-rw--- 1 amanda amanda   33308 Aug 26 02:20 
00030.coyote._GenesAmandaHelper-0.61.1
-rw--- 1 amanda amanda   348438528 Aug 26 02:20 00031.coyote._var.4
-rw--- 1 amanda amanda 13234614709 Aug 26 02:48 
00032.coyote.PublicAap.0
-rw--- 1 amanda amanda16479741 Aug 26 02:48 
00033.coyote._usr_local.1
-rw--- 1 amanda amanda11653827 Aug 26 02:48 00034.coyote._opt.1
-rw--- 1 amanda amanda  841528 Aug 26 02:48 
00035.coyote._usr_share.1
-rw--- 1 amanda amanda  811008 Aug 26 02:4

Re: bad tape?

[Debra S Baddorf]

Try  “man  amcheckdump”   It might give you what you want.

Deb Baddorf
Fermilab

> On Aug 16, 2019, at 12:38 PM, ghe  wrote:
> 
> I have reason to believe that one of my tapes isn't working properly
> (last night's backup died without finishing because of a tape error --
> the retry running right now successfully flushed last night's and wrote
> a new one).
> 
> Is there an Amanda utility that will validate a tape without destroying
> the Amanda header at the front of the tape? And maybe build a 'bad
> sector table' on the tape?
> 
> -- 
> Glenn English

bad tape?

[ghe]

I have reason to believe that one of my tapes isn't working properly
(last night's backup died without finishing because of a tape error --
the retry running right now successfully flushed last night's and wrote
a new one).

Is there an Amanda utility that will validate a tape without destroying
the Amanda header at the front of the tape? And maybe build a 'bad
sector table' on the tape?

-- 
Glenn English