Re: tape problem

[ghe2001]

amanda v 3.5.1, Debian AMD64 Buster, Quantum STO-5, LSI SAS2008 PCIe card

I think I've found my problem.  Amanda can't write big files to tape.

>From my Debian kernel.log (grep for st0):

Dec 13 15:41:34 sbox kernel: [ 2708.331836] st 0:0:2:0: [st0] Block limits 1 - 
16777215 bytes.
Dec 13 15:44:32 sbox kernel: [ 2885.931242] st 0:0:2:0: [st0] Error e 
(driver bt 0x0, host bt 0xe).
Dec 13 15:44:32 sbox kernel: [ 2885.931245] st 0:0:2:0: [st0] Error on write 
filemark.
Dec 13 15:44:38 sbox kernel: [ 2891.941684] st 0:0:3:0: Attached scsi tape st0
Dec 13 15:44:38 sbox kernel: [ 2891.941687] st 0:0:3:0: st0: try direct i/o: 
yes (alignment 4 B)
Dec 13 15:46:28 sbox kernel: [ 3001.878038] st 0:0:3:0: [st0] Block limits 1 - 
16777215 bytes.
Dec 14 14:06:33 sbox kernel: [2.424988] scsi host0: ata_piix
Dec 14 14:06:33 sbox kernel: [   11.462017] st 3:0:0:0: Attached scsi tape st0
Dec 14 14:06:33 sbox kernel: [   11.462018] st 3:0:0:0: st0: try direct i/o: 
yes (alignment 4 B)
Dec 14 15:53:45 sbox kernel: [ 6603.276972] st 3:0:0:0: [st0] Block limits 1 - 
16777215 bytes.
Dec 14 16:08:29 sbox kernel: [ 7487.326410] st 3:0:0:0: [st0] Error 40 (driver 
bt 0inx0, host bt 0x0).
Dec 14 16:09:05 sbox kernel: [ 7523.203323] st 3:0:1:0: Attached scsi tape st0
Dec 14 16:09:05 sbox kernel: [ 7523.203326] st 3:0:1:0: st0: try direct i/o: 
yes (alignment 4 B)

and on and on for pages.


dmseg says:  16777215

root@sbox:/var/log# dmesg | egrep st0
[2.424988] scsi host0: ata_piix
[   11.462017] st 3:0:0:0: Attached scsi tape st0
[   11.462018] st 3:0:0:0: st0: try direct i/o: yes (alignment 4 B)
[ 6603.276972] st 3:0:0:0: [st0] Block limits 1 - 16777215 bytes.
[ 7487.326410] st 3:0:0:0: [st0] Error 40 (driver bt 0x0, host bt 0xin0).
[ 7523.203323] st 3:0:1:0: Attached scsi tape st0
[ 7523.203326] st 3:0:1:0: st0: try direct i/o: yes (alignment 4 B)
[ 7570.604960] st 3:0:1:0: [st0] Block limits 1 - 16777215 bytes.
[ 7844.727097] st 3:0:2:0: Attached scsi tape st0
[ 7844.727101] st 3:0:2:0: st0: try direct i/o: yes (alignment 4 B)
[ 7955.289508] st 3:0:2:0: [st0] Block limits 1 - 16777215 bytes.
[130700.591001] st 3:0:2:0: [st0] Error e (driver bt 0x0, host bt 0xe).
[130702.840754] st 3:0:2:0: [st0] Error 1 (driver bt 0x0, host bt 
0x1).16777215
[130706.850766] st 3:0:3:0: Attached scsi tape st0
[130706.850787] st 3:0:3:0: st0: try direct i/o: yes (alignment 4 B)
[131166.864674] st 3:0:3:0: [st0] Block limits 1 - 16777215 bytes.

It looks to me like amanda's trying to write data that's too big for something 
in the kernel (there are 1G files in the holding disk).  If that's correct, it 
looks like a simple change in some config should fix it.  I've looked for a 
config to change, but can't find it.

grep tells me that '16777215' (aka 0xff) isn't in any file in /boot.

On the web, it looks like others are having similar troubles in other Linux 
distributions and with other backup software.  That says to me that there was a 
change in my most recent update of the Linux kernel -- and that (probably) 
means the size definition is somewhere in the source (or maybe a binary number 
somewhere in /boot).

I've compiled kernels before, so I think I could probably find this setting 
somewhere in the source, but the source is huge.  Does anyone know where that 
16777215 number is?  Is it accessible in the make config file?  Is it in one of 
the C include files?  Do the amanda developers have a patch to deal with this?  
Is there a more civilized way to do this?  Am I looking at the wrong thing?

--
Glenn English

Re: tape problem

[Stefan G. Weichinger]

Am 06.12.21 um 15:39 schrieb David Simpson:

Further to Jose's questions.

Some config excerpt may help.

On blocksize, that is defined within the tapetype.

 blocksize 512 kbytes



And maybe try a run with amtapetype.

RE: tape problem

[David Simpson]

Further to Jose's questions.

Some config excerpt may help.

On blocksize, that is defined within the tapetype.

blocksize 512 kbytes

-
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB   
    

David Simpson - peiriannydd uwch systemau
ARCCA, Adeilad Redwood,
King Edward VII Avenue,
Caerdydd, CF10 3NB

-Original Message-
From: owner-amanda-us...@amanda.org  On Behalf 
Of Jose M Calhariz
Sent: 06 December 2021 11:17
To: ghe2001 
Cc: amanda-users@amanda.org
Subject: Re: tape problem


I think the explanation for most of your problems is explained the amanda 
report.  I have some questions that may help find the problem:

  - how much free space do you have on holding disk, how much space is
used?

  - How big are the backups every day?

  - Do you use hardware compression on the tape?


Kind regards
Jose M Calhariz


On Fri, Dec 03, 2021 at 11:18:08PM +, ghe2001 wrote:
> amanda version: amadmin-3.5.1
> OS: Debian Linux, Buster
> Host: Supermicro
> PCI card: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT 
> SAS-2 [Falcon] (rev 03) Tape drive: Quantum LTO-5
> Tapes: Quantum Ultrium 5
> 
> Problem: Over the years, I've written a bunch of amanda scripts.  Here's what 
> the one that displays what's going on during a backup says at one point:
> 
> gobook3.slsware.lan:/usr   20211203133758 0   4519450k 
> dump done, wait for writing
> pi.slsware.lan:/lib20211203133758 0   1720720k 
> dump done, writing (1518944k done (88.27%)) (14:03:16)
> pi.slsware.lan:/toshiba1   20211203133758 0   5089430k 
> dump done, wait for writing
> pi.slsware.lan:/usr20211203133758 0   3108660k 
> dump done, wait for writing
> sbox.slsware.lan:/blackHole/amanda 20211203133758 0   2251540k 
> dump done, wait for writing
> sbox.slsware.lan:/home 20211203133758 0  74493810k 
> dump done, wait for writing
> sbox.slsware.lan:/usr  20211203133758 0   5144210k 
> dump done, wait for writing
>0: writing (pi.slsware.lan:/lib) === Fri Dec 03, 2021 
> 02:03:35 PM
> 
>   This is normal.  A lot of flushing's been done, and all the small files.  
> The script futzes with amstatus and displays every 5 seconds.
> 
> .
> 
>   The '.' means amstatus has written the same line, and nothing of interest 
> has happened.
> 
> gobook3.slsware.lan:/usr   20211203133758 0   4519450k 
> dump done, wait for writing
> pi.slsware.lan:/toshiba1   20211203133758 0   5089430k 
> dump done, wait for writing
> pi.slsware.lan:/usr20211203133758 0   3108660k 
> dump done, wait for writing
> sbox.slsware.lan:/blackHole/amanda 20211203133758 0   2251540k 
> dump done, wait for writing
> sbox.slsware.lan:/home 20211203133758 0  74493810k 
> dump done, wait for writing
> sbox.slsware.lan:/usr  20211203133758 0   5144210k 
> dump done, wait for writing
>0: tape error: Couldn't rewind device to finish: No 
> such device, splitting not enabled (pi.slsware.lan:/lib) === Fri Dec 
> 03, 2021 02:03:52 PM
> 
>   Notice the than ~30 seconds from the last display.  And who's trying to 
> rewind and finding no device (/dev/nst0 I suppose).
> 
>   After this, every line says "terminated while waiting for writing."
> 
>   This looks to me like maybe something can't deal with files larger that 1G. 
>  But I've seen it get to 55G or so.
> 
> 
> This morning, Logwatch said:
> 
> WARNING:  Kernel Errors Present
> st 0:0:3:0: [st0] Error 1 (driver bt ...:  2 Time(s)
> st 0:0:3:0: [st0] Error e (driver bt ...:  1 Time(s)
> st 0:0:4:0: [st0] Error 1 (driver bt ...:  1 Time(s)
> st 0:0:4:0: [st0] Error e (driver bt ...:  1 Time(s)
> st 0:0:5:0: [st0] Error 1 (driver bt ...:  1 Time(s)
> st 0:0:5:0: [st0] Error e (driver bt ...:  1 Time(s)
> st 0:0:6:0: [st0] Error 1 (driver bt ...:  1 Time(s)
> st 0:0:6:0: [st0] Error e (driver bt ...:  1 Time(s)
> st 0:0:7:0: [st0] Error 1 (driver bt ...:  1 Time(s)
> st 0:0:7:0: [st0] Error 7 (driver bt ...:  1 Time(s)
> st 0:0:7:0: [st0] Error e (driver bt ...:  1 Time(s)
> 
> st0 is the non-rewinding tape drive.  As best I know, nobody does anything 
> with st0 -- nst0 is what I, and amanda, use.
> 
> This looks like there might be a bent driver.
> 
> 
> Amcheck:
> 
> slot 1: volume 'sls-9'
> Will write to volume 'sls-9

Re: tape problem

[Jose M Calhariz]

I think the explanation for most of your problems is explained the
amanda report.  I have some questions that may help find the problem:

  - how much free space do you have on holding disk, how much space is
used?

  - How big are the backups every day?

  - Do you use hardware compression on the tape?


Kind regards
Jose M Calhariz


On Fri, Dec 03, 2021 at 11:18:08PM +, ghe2001 wrote:
> amanda version: amadmin-3.5.1
> OS: Debian Linux, Buster
> Host: Supermicro
> PCI card: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 
> [Falcon] (rev 03)
> Tape drive: Quantum LTO-5
> Tapes: Quantum Ultrium 5
> 
> Problem: Over the years, I've written a bunch of amanda scripts.  Here's what 
> the one that displays what's going on during a backup says at one point:
> 
> gobook3.slsware.lan:/usr   20211203133758 0   4519450k 
> dump done, wait for writing
> pi.slsware.lan:/lib20211203133758 0   1720720k 
> dump done, writing (1518944k done (88.27%)) (14:03:16)
> pi.slsware.lan:/toshiba1   20211203133758 0   5089430k 
> dump done, wait for writing
> pi.slsware.lan:/usr20211203133758 0   3108660k 
> dump done, wait for writing
> sbox.slsware.lan:/blackHole/amanda 20211203133758 0   2251540k 
> dump done, wait for writing
> sbox.slsware.lan:/home 20211203133758 0  74493810k 
> dump done, wait for writing
> sbox.slsware.lan:/usr  20211203133758 0   5144210k 
> dump done, wait for writing
>0: writing (pi.slsware.lan:/lib)
> === Fri Dec 03, 2021 02:03:35 PM
> 
>   This is normal.  A lot of flushing's been done, and all the small files.  
> The script futzes with amstatus and displays every 5 seconds.
> 
> .
> 
>   The '.' means amstatus has written the same line, and nothing of interest 
> has happened.
> 
> gobook3.slsware.lan:/usr   20211203133758 0   4519450k 
> dump done, wait for writing
> pi.slsware.lan:/toshiba1   20211203133758 0   5089430k 
> dump done, wait for writing
> pi.slsware.lan:/usr20211203133758 0   3108660k 
> dump done, wait for writing
> sbox.slsware.lan:/blackHole/amanda 20211203133758 0   2251540k 
> dump done, wait for writing
> sbox.slsware.lan:/home 20211203133758 0  74493810k 
> dump done, wait for writing
> sbox.slsware.lan:/usr  20211203133758 0   5144210k 
> dump done, wait for writing
>0: tape error: Couldn't rewind device to finish: No such 
> device, splitting not enabled (pi.slsware.lan:/lib)
> === Fri Dec 03, 2021 02:03:52 PM
> 
>   Notice the than ~30 seconds from the last display.  And who's trying to 
> rewind and finding no device (/dev/nst0 I suppose).
> 
>   After this, every line says "terminated while waiting for writing."
> 
>   This looks to me like maybe something can't deal with files larger that 1G. 
>  But I've seen it get to 55G or so.
> 
> 
> This morning, Logwatch said:
> 
> WARNING:  Kernel Errors Present
> st 0:0:3:0: [st0] Error 1 (driver bt ...:  2 Time(s)
> st 0:0:3:0: [st0] Error e (driver bt ...:  1 Time(s)
> st 0:0:4:0: [st0] Error 1 (driver bt ...:  1 Time(s)
> st 0:0:4:0: [st0] Error e (driver bt ...:  1 Time(s)
> st 0:0:5:0: [st0] Error 1 (driver bt ...:  1 Time(s)
> st 0:0:5:0: [st0] Error e (driver bt ...:  1 Time(s)
> st 0:0:6:0: [st0] Error 1 (driver bt ...:  1 Time(s)
> st 0:0:6:0: [st0] Error e (driver bt ...:  1 Time(s)
> st 0:0:7:0: [st0] Error 1 (driver bt ...:  1 Time(s)
> st 0:0:7:0: [st0] Error 7 (driver bt ...:  1 Time(s)
> st 0:0:7:0: [st0] Error e (driver bt ...:  1 Time(s)
> 
> st0 is the non-rewinding tape drive.  As best I know, nobody does anything 
> with st0 -- nst0 is what I, and amanda, use.
> 
> This looks like there might be a bent driver.
> 
> 
> Amcheck:
> 
> slot 1: volume 'sls-9'
> Will write to volume 'sls-9' in slot 1.
> Writing label 'sls-9' to check writability
> Volume 'sls-9' is writeable.
> Server check took 25.073 seconds
> (brought to you by Amanda 3.5.1)
> 
> 
> I'm at a loss.  Amanda worked for a decade or so with the old DLT drive, then 
> for 3 or 4 years with the LTO -- then started breaking in every backup.  I 
> trusted Quantum and LSI -- I ran tar to copy my entire / directory to the 
> Quantum tape that came with the drive, and it worked -- that ruled out the 
> card and the drive, so I bought a new collection of Quantum tapes to replace 
> the HPs I had been using.  No joy.
> 
>

Re: tape problem

[Charles Curley]

On Fri, 03 Dec 2021 23:18:08 +
ghe2001  wrote:

> Any ideas, explanations, fixes?

Maybe it's time to move to Bullseye?

Short of that, try a kernel from backports.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

RE: LTO8 tape profile - amtapetype

[David Simpson]

Increasing the blocksize to 1024k achieved 284MB/sec uncompressed with 
amtapetype

Have not changed kernel module (st) parameters or anything like that yet

Currently looking at the HP Library & Tape Tools ... it seems to want to test 
with blocksizes of 32k-128k only

-
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB   
    

David Simpson - peiriannydd uwch systemau
ARCCA, Adeilad Redwood,
King Edward VII Avenue,
Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk
+44 29208 74657

-Original Message-
From: owner-amanda-us...@amanda.org  On Behalf 
Of David Simpson
Sent: 28 September 2021 21:38
To: Luc Lalonde ; AMANDA users 
Subject: RE: LTO8 tape profile - amtapetype

Thanks. I noticed your blocksize... I may need to experiment more with that.

So far I've not been able to turn drive HW compression off... though I'm happy 
to use it in production as opposed to software compression.

-
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB   
    

David Simpson - peiriannydd uwch systemau ARCCA, Adeilad Redwood, King Edward 
VII Avenue, Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk
+44 29208 74657

-Original Message-
From: owner-amanda-us...@amanda.org  On Behalf 
Of Luc Lalonde
Sent: 28 September 2021 20:58
To: AMANDA users 
Subject: Re: LTO8 tape profile - amtapetype

External email to Cardiff University - Take care when replying/opening 
attachments or links.
Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor 
atodiadau neu ddolenni.

RE: LTO8 tape profile - amtapetype

[David Simpson]

Thanks. I noticed your blocksize... I may need to experiment more with that.

So far I've not been able to turn drive HW compression off... though I'm happy 
to use it in production as opposed to software compression.

-
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB   
    

David Simpson - peiriannydd uwch systemau
ARCCA, Adeilad Redwood,
King Edward VII Avenue,
Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk
+44 29208 74657

-Original Message-
From: owner-amanda-us...@amanda.org  On Behalf 
Of Luc Lalonde
Sent: 28 September 2021 20:58
To: AMANDA users 
Subject: Re: LTO8 tape profile - amtapetype

External email to Cardiff University - Take care when replying/opening 
attachments or links.
Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor 
atodiadau neu ddolenni.

RE: LTO8 tape profile - amtapetype

[David Simpson]

What tweaks did you do?

Yes, should of mentioned the server is dedicated for this purpose and is 
reasonably resourced:
-dedicated SAS card for tape drives
-64GB RAM
-2x Xeon E5-2640 v3, so plenty of cores
-dedicated holding disk areas (RAID6) .. two of these coming in at approx. 
32.7TB, which will match up to LTO8 quite nicely hopefully. Comprised 8x 6TB 
disks.

By the sounds of it, not too dissimilar to your setup.


-
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB   
    

David Simpson - peiriannydd uwch systemau
ARCCA, Adeilad Redwood,
King Edward VII Avenue,
Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk
+44 29208 74657

-Original Message-
From: Chris Hoogendyk  
Sent: 28 September 2021 20:00
To: David Simpson ; AMANDA users 

Subject: Re: LTO8 tape profile - amtapetype

External email to Cardiff University - Take care when replying/opening 
attachments or links.
Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor 
atodiadau neu ddolenni.



You should get substantially faster. That said, throughput to the tape will 
depend on a lot of things about your system and where the bottleneck actually 
is. The maximum throughput of dual SAS2 is 6Gbit/s, which would be 750MB/s. So, 
you aren't likely to come close to the 900MB/s for compressible data that LTO8 
is rated for. If there are other things going on using the SAS bandwidth, like 
your hard drive systems, then that bites into what's available for pushing out 
to tape. Then there is the cpu for generating the data stream.

I've only gotten up to LTO7. However, it took some tweaking of my supermicro 
servers to approach the rated speed for the LTO7. The tape drives are on a 
dedicated SAS card. The disk drives are on a separate SAS card. My servers have 
two cpus with I think 16 core each, and 64GB memory. So, even though they are 
doing a lot of other stuff, Amanda can still push the LTO7 approaching its 
rated speed.


On 9/28/21 5:10 AM, David Simpson wrote:
>
> Just ran amtapetype against an LTO8 tape + HP MSL3040 (with SAS drives). 
> Debian 11.
>
> Some observations:
>
> -compression on, expected
> -length looks ok
>
> -speed .. should I have expected more? Seemed a bit low I thought..
>
> -no LEOM.. I don't know if it's the hardware or the Debian 11 kernel 
> or both, though understand this is not essential
>
>
>
> Checking for FSF_AFTER_FILEMARK requirement
>
> Applying heuristic check for compression.
>
> Wrote random (uncompressible) data at 96137551.7377049 bytes/sec
>
> Wrote fixed (compressible) data at 167554018.742857 bytes/sec
>
> Compression: enabled
>
> Writing one file to fill the volume.
>
> Wrote 12011529469952 bytes at 94902 kb/sec
>
> Writing smaller files (120115265536 bytes) to determine filemark.
>
> define tapetype unknown-tapetype {
>
> comment "Created by amtapetype; compression enabled"
>
> length 11730009248 kbytes
>
> filemark 4132 kbytes
>
> speed 94902 kps
>
> blocksize 32 kbytes
>
> }
>
> # LEOM is not supported for this drive and kernel
>
>
>
>
>
> -
>
> David Simpson - Senior Systems Engineer
>
> ARCCA, Redwood Building,
>
> King Edward VII Avenue,
>
> Cardiff, CF10 3NB
>
> David Simpson - peiriannydd uwch systemau
>
> ARCCA, Adeilad Redwood,
>
> King Edward VII Avenue,
>
> Caerdydd, CF10 3NB
>
> simpso...@cardiff.ac.uk <mailto:simpso...@cardiff.ac.uk>
>
> +44 29208 74657
>
--
---

Chris Hoogendyk

-
O__   Systems Administrator, Retired
   c/ /'_ --- Biology & Geosciences Departments
  (*) \(*) -- 315 Morrill Science Center III ~~ - University of 
Massachusetts, Amherst



---

Erdös 4

Re: LTO8 tape profile - amtapetype

[Luc Lalonde]

Here's mine:

define tapetype LTO8-compoff {
    comment "Created by amtapetype; compression disabled"
    comment "/usr/sbin/amtapetype -f -b 1024k /dev/nst2"
    length 11732915200 kbytes
    filemark 5006 kbytes
    speed 172710 kps
    blocksize 1024 kbytes
}

It's an FC drive on a Dell ML3 Tape Library.

On 2021-09-28 3:00 p.m., Chris Hoogendyk wrote:
You should get substantially faster. That said, throughput to the tape 
will depend on a lot of things about your system and where the 
bottleneck actually is. The maximum throughput of dual SAS2 is 
6Gbit/s, which would be 750MB/s. So, you aren't likely to come close 
to the 900MB/s for compressible data that LTO8 is rated for. If there 
are other things going on using the SAS bandwidth, like your hard 
drive systems, then that bites into what's available for pushing out 
to tape. Then there is the cpu for generating the data stream.


I've only gotten up to LTO7. However, it took some tweaking of my 
supermicro servers to approach the rated speed for the LTO7. The tape 
drives are on a dedicated SAS card. The disk drives are on a separate 
SAS card. My servers have two cpus with I think 16 core each, and 64GB 
memory. So, even though they are doing a lot of other stuff, Amanda 
can still push the LTO7 approaching its rated speed.



On 9/28/21 5:10 AM, David Simpson wrote:


Just ran amtapetype against an LTO8 tape + HP MSL3040 (with SAS 
drives). Debian 11.


Some observations:

-compression on, expected
-length looks ok

-speed .. should I have expected more? Seemed a bit low I thought..

-no LEOM.. I don’t know if it’s the hardware or the Debian 11 kernel 
or both, though understand this is not essential




Checking for FSF_AFTER_FILEMARK requirement

Applying heuristic check for compression.

Wrote random (uncompressible) data at 96137551.7377049 bytes/sec

Wrote fixed (compressible) data at 167554018.742857 bytes/sec

Compression: enabled

Writing one file to fill the volume.

Wrote 12011529469952 bytes at 94902 kb/sec

Writing smaller files (120115265536 bytes) to determine filemark.

define tapetype unknown-tapetype {

comment "Created by amtapetype; compression enabled"

length 11730009248 kbytes

filemark 4132 kbytes

speed 94902 kps

blocksize 32 kbytes

}

# LEOM is not supported for this drive and kernel





-

David Simpson - Senior Systems Engineer

ARCCA, Redwood Building,

King Edward VII Avenue,

Cardiff, CF10 3NB

David Simpson - peiriannydd uwch systemau

ARCCA, Adeilad Redwood,

King Edward VII Avenue,

Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk <mailto:simpso...@cardiff.ac.uk>

+44 29208 74657


--
Luc Lalonde, analyste
-
Département de génie informatique:
École polytechnique de MTL
(514) 340-4711 x5049
luc.lalo...@polymtl.ca
-




OpenPGP_signature
Description: OpenPGP digital signature

Re: LTO8 tape profile - amtapetype

[Chris Hoogendyk]

You should get substantially faster. That said, throughput to the tape will depend on a lot of 
things about your system and where the bottleneck actually is. The maximum throughput of dual SAS2 
is 6Gbit/s, which would be 750MB/s. So, you aren't likely to come close to the 900MB/s for 
compressible data that LTO8 is rated for. If there are other things going on using the SAS 
bandwidth, like your hard drive systems, then that bites into what's available for pushing out to 
tape. Then there is the cpu for generating the data stream.


I've only gotten up to LTO7. However, it took some tweaking of my supermicro servers to approach the 
rated speed for the LTO7. The tape drives are on a dedicated SAS card. The disk drives are on a 
separate SAS card. My servers have two cpus with I think 16 core each, and 64GB memory. So, even 
though they are doing a lot of other stuff, Amanda can still push the LTO7 approaching its rated speed.



On 9/28/21 5:10 AM, David Simpson wrote:


Just ran amtapetype against an LTO8 tape + HP MSL3040 (with SAS drives). Debian 
11.

Some observations:

-compression on, expected
-length looks ok

-speed .. should I have expected more? Seemed a bit low I thought..

-no LEOM.. I don’t know if it’s the hardware or the Debian 11 kernel or both, though understand 
this is not essential




Checking for FSF_AFTER_FILEMARK requirement

Applying heuristic check for compression.

Wrote random (uncompressible) data at 96137551.7377049 bytes/sec

Wrote fixed (compressible) data at 167554018.742857 bytes/sec

Compression: enabled

Writing one file to fill the volume.

Wrote 12011529469952 bytes at 94902 kb/sec

Writing smaller files (120115265536 bytes) to determine filemark.

define tapetype unknown-tapetype {

comment "Created by amtapetype; compression enabled"

length 11730009248 kbytes

filemark 4132 kbytes

speed 94902 kps

blocksize 32 kbytes

}

# LEOM is not supported for this drive and kernel





-

David Simpson - Senior Systems Engineer

ARCCA, Redwood Building,

King Edward VII Avenue,

Cardiff, CF10 3NB

David Simpson - peiriannydd uwch systemau

ARCCA, Adeilad Redwood,

King Edward VII Avenue,

Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk <mailto:simpso...@cardiff.ac.uk>

+44 29208 74657


--
---

Chris Hoogendyk

-
   O__   Systems Administrator, Retired
  c/ /'_ --- Biology & Geosciences Departments
 (*) \(*) -- 315 Morrill Science Center III
~~ - University of Massachusetts, Amherst



---

Erdös 4

LTO8 tape profile - amtapetype

[David Simpson]

Just ran amtapetype against an LTO8 tape + HP MSL3040 (with SAS drives). Debian 
11.

Some observations:
-compression on, expected
-length looks ok
-speed .. should I have expected more? Seemed a bit low I thought..
-no LEOM.. I don't know if it's the hardware or the Debian 11 kernel or both, 
though understand this is not essential



Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 96137551.7377049 bytes/sec
Wrote fixed (compressible) data at 167554018.742857 bytes/sec
Compression: enabled
Writing one file to fill the volume.
Wrote 12011529469952 bytes at 94902 kb/sec
Writing smaller files (120115265536 bytes) to determine filemark.
define tapetype unknown-tapetype {
comment "Created by amtapetype; compression enabled"
length 11730009248 kbytes
filemark 4132 kbytes
speed 94902 kps
blocksize 32 kbytes
}
# LEOM is not supported for this drive and kernel






-
David Simpson - Senior Systems Engineer
ARCCA, Redwood Building,
King Edward VII Avenue,
Cardiff, CF10 3NB

David Simpson - peiriannydd uwch systemau
ARCCA, Adeilad Redwood,
King Edward VII Avenue,
Caerdydd, CF10 3NB

simpso...@cardiff.ac.uk<mailto:simpso...@cardiff.ac.uk>
+44 29208 74657

Tape partitionning

[Luc Lalonde]

Hello Folks,

Since LTO-4, we're supposed to be able to partition the tapes.

Is this supported by Amanda?   If so, what tools are you using to 
partition the tapes?


Thank You!

--
Luc Lalonde, analyste
-
Département de génie informatique:
École polytechnique de MTL
(514) 340-4711 x5049
luc.lalo...@polymtl.ca
-




OpenPGP_signature
Description: OpenPGP digital signature

Re: How to "unlable" a tape

[Olivier]

Nathan Stratton Treadway  writes:

> On Fri, Sep 18, 2020 at 10:53:44 +0700, Olivier wrote:
>> I know there is amadmin no-reuse, but suppose the tape had been
>> completely destroyed and is not readable anymore, there should be a way
>> to tell Amanda it should completely forget about that tape, remove any
>> index it can have aboutt he tape.
>> 
>> What would be the command then?
>
> You're looking for "amrmtape" .

Thank you, that was of course what I was looking for,

Best regards,

Olivier

>
>   Nathan
>
> 
> Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
> Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
>  GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
>  Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239
>

-- 

Re: How to "unlable" a tape

[Nathan Stratton Treadway]

On Fri, Sep 18, 2020 at 10:53:44 +0700, Olivier wrote:
> I know there is amadmin no-reuse, but suppose the tape had been
> completely destroyed and is not readable anymore, there should be a way
> to tell Amanda it should completely forget about that tape, remove any
> index it can have aboutt he tape.
> 
> What would be the command then?

You're looking for "amrmtape" .

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: How to "unlable" a tape

[J Martin Rushton]

Probably not the official way, but you can delete it from 
/etc/amanda//tapelist  I realise that won't remove the index 
information, but it will stop Amanda from wanting to consider the tape.


HTH,
Martin

On 18/09/2020 04:53, Olivier wrote:

Hi,

I have been trying to find the command opposite to amlabel: completely
remove a tape from Amanda.

I know there is amadmin no-reuse, but suppose the tape had been
completely destroyed and is not readable anymore, there should be a way
to tell Amanda it should completely forget about that tape, remove any
index it can have aboutt he tape.

What would be the command then?

TIA,

Olivier



--
J Martin Rushton MBCS

Re: How to "unlable" a tape

[Kamil Jońca]

Olivier  writes:

> Hi,
>
> I have been trying to find the command opposite to amlabel: completely
> remove a tape from Amanda.
>
> I know there is amadmin no-reuse, but suppose the tape had been
> completely destroyed and is not readable anymore, there should be a way
> to tell Amanda it should completely forget about that tape, remove any
> index it can have aboutt he tape.
>
> What would be the command then?

man amrmtape

KJ


-- 
http://wolnelektury.pl/wesprzyj/teraz/
You will be married within a year.

How to "unlable" a tape

[Olivier]

Hi,

I have been trying to find the command opposite to amlabel: completely
remove a tape from Amanda.

I know there is amadmin no-reuse, but suppose the tape had been
completely destroyed and is not readable anymore, there should be a way
to tell Amanda it should completely forget about that tape, remove any
index it can have aboutt he tape.

What would be the command then?

TIA,

Olivier
-- 

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 15.05.20 um 13:33 schrieb Nathan Stratton Treadway:
> On Fri, May 15, 2020 at 10:59:49 +0200, Stefan G. Weichinger wrote:
>> I am gonna restore as well just to check if there are hidden write
>> errors (doesn't look like that to me so far ...)
> 
> 
> That reminded me that (at least on our Ubuntu Linux system) the
> smartmontools package's "smartctl" let us read error statistics
> information from the SCSI tape drive.  I put
> 
>smartctl -l error -H $TAPEDEV

Oh, interesting, I didn't know about this feature of smartctl.

Will try as soon as my n-th amtapetype finishes.

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 15.05.20 um 13:59 schrieb Andreas Haumer:

> I think this was not mentioned on this thread yet, but
> there is also the HPE Library & Tape Tools utility ("ltt"),
> which allows to test many aspects of a tape drive.
> This software runs fine under Linux and can be used with tape
> drives of other brands, too (not only HP drives).
> I used it successfully on several occasions over the years,
> with drive types from DDS2, DLT to LTO-6 and drive brands like
> HP, Quantum or Tandberg.

Nice reminder, thanks. Used that tool earlier as well.

Unfortunately it's hard to install under Gentoo. At first I had to
convert the rpm to a tar .. then it complains about libtinfo.so.5 ...
which is part of ncurses. Tried various symlinks without success (wrong
ELF header or plain crashes).

> The user interface of this tool is, well, "interesting", though... :-)

Something else to look forward to ;-)

thanks

Re: amanda-3.4.5 does not fill one tape

[Andreas Haumer]

Hi!

Am 15.05.20 um 13:33 schrieb Nathan Stratton Treadway:
> On Fri, May 15, 2020 at 10:59:49 +0200, Stefan G. Weichinger wrote:
>> I am gonna restore as well just to check if there are hidden write
>> errors (doesn't look like that to me so far ...)
> 
> 
> That reminded me that (at least on our Ubuntu Linux system) the
> smartmontools package's "smartctl" let us read error statistics
> information from the SCSI tape drive.  I put
> 
>smartctl -l error -H $TAPEDEV
> 
> in the cron script which ran Amanda, and it would produce output like
> this:
> 

I think this was not mentioned on this thread yet, but
there is also the HPE Library & Tape Tools utility ("ltt"),
which allows to test many aspects of a tape drive.
This software runs fine under Linux and can be used with tape
drives of other brands, too (not only HP drives).
I used it successfully on several occasions over the years,
with drive types from DDS2, DLT to LTO-6 and drive brands like
HP, Quantum or Tandberg.

The user interface of this tool is, well, "interesting", though... :-)

HTH

- andreas

-- 
Andreas Haumer | mailto:andr...@xss.co.at
*x Software + Systeme  | http://www.xss.co.at/
Karmarschgasse 51/2/20 | Tel: +43-1-6060114-0
A-1100 Vienna, Austria | Fax: +43-1-6060114-71



signature.asc
Description: OpenPGP digital signature

Re: amanda-3.4.5 does not fill one tape

[Nathan Stratton Treadway]

On Fri, May 15, 2020 at 10:59:49 +0200, Stefan G. Weichinger wrote:
> I am gonna restore as well just to check if there are hidden write
> errors (doesn't look like that to me so far ...)


That reminded me that (at least on our Ubuntu Linux system) the
smartmontools package's "smartctl" let us read error statistics
information from the SCSI tape drive.  I put

   smartctl -l error -H $TAPEDEV

in the cron script which ran Amanda, and it would produce output like
this:

==
smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen  
Home page is http://smartmontools.sourceforge.net/  

TapeAlert: OK   

Error counter log:  
   Errors Corrected by   Total   Correction Gigabytes   
Total  
   ECC  rereads/errors   algorithm  processed   
uncorrected
   fast | delayed   rewrites  corrected  invocations   [10^9 bytes] 
errors 
read:  00 0 0  0  0.000 
0  
write:  63040  6304  6304   6304  0.000 
0  
==

With that output at the end of the Amanda-running script I could keep an
eye on the numbers for each particular tape as it came through the
rotation cycle.  (A few thousand seemed normal, at least by the time I
implemented this monitoring [since by then the tapes were several years
old]; when tapes were starting to really go bad I saw error counts in
the tens of thousands, etc.)


When the drive detected that the tape heads needed to be cleaned, the
output looked like
==
smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen  
Home page is http://smartmontools.sourceforge.net/  

TapeAlert Errors (C=Critical, W=Warning, I=Informational):  
[0x14] C: The tape drive needs cleaning:
  1. If the operation has stopped, eject the tape and clean the drive.  
  2. If the operation has not stopped, wait for it to finish and then   
  clean the drive.      
  Check the tape drive users manual for device specific cleaning instructions.  

Error counter log:  
   Errors Corrected by   Total   Correction Gigabytes   
Total  
   ECC  rereads/errors   algorithm  processed   
uncorrected
   fast | delayed   rewrites  corrected  invocations   [10^9 bytes] 
errors 

read:  00 0 0  0  0.000 
0  
write:  33901  3392  3389   3390  0.000 
1  
==


(You should double-check the behavior if you are going to rely on it,
but as I recall the stats are cleared when you swap a tape and when you
run the above smartctl command.)


Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 13.05.20 um 18:17 schrieb Stefan G. Weichinger:
> Now look at this run of
> 
> "amtapetype -b 128k /dev/nst0"
> 
> with another tape, FUJI instead of HP:
> 
> define tapetype LTO3-fuji {
> comment "Created by amtapetype; compression disabled"
> length 284180096 kbytes
> filemark 20803 kbytes
> speed 38376 kps
> blocksize 128 kbytes
> }
> 
> ~290 GB ... faster, and large filemarks.
> 
> Maybe that drive is somehow failing .. ?

Thanks all for you replies.

I let them re-clean today and we use the ~290GB tapetype for now and
look if things work.

I am gonna restore as well just to check if there are hidden write
errors (doesn't look like that to me so far ...)

Spent many hours with this over the last few days. Maybe we should
consider moving over to vtapes as well (although this brings up other
issues).

ah btw:

with a plain dd the tape was also over at around 200 GB

I suspect a wrong setting somewhere but my googling for stinit settings
doesn't give me good infos.

Back in 2019 I find reports of exactly this setup writing:

323403M
323642M
354610M
345766M
343395M
343781M

and around new year it started complaining

Maybe another few cleaning runs help.

Re: amanda-3.4.5 does not fill one tape

[Jens Berg]

There was a discussion back in 2014 with subject "Backups to tape 
consistently under 60% tape capacity". I haven't read the whole lengthy 
thread but one participant mentioned that in his case a bad cleaning 
tape was found to be responsible for the capacity loss. Others reported 
that they had trouble with the compression settings. My gut feeling 
tells me that this is not the problem here but maybe your impression is 
a different one or you get a new idea if you read through the postings.


Jens

Am 13.05.2020 um 19:40 schrieb Debra S Baddorf:



On May 13, 2020, at 11:17 AM, Stefan G. Weichinger  wrote:

Now look at this run of

"amtapetype -b 128k /dev/nst0"

with another tape, FUJI instead of HP:

define tapetype LTO3-fuji {
comment "Created by amtapetype; compression disabled"
length 284180096 kbytes
filemark 20803 kbytes
speed 38376 kps
blocksize 128 kbytes
}

~290 GB ... faster, and large filemarks.

Maybe that drive is somehow failing .. ?



No real ideas here, but I *DO* recall some failing drives on this list
where the symptom was like this.  Tapes wouldn’t fill.

Anybody else still out there in the aether?

Deb Baddorf
Fermilab



--
Jens Berg  R  VTech IAD GmbH
Max-Giese-Str. 22, 24116 Kiel, Germany
Phone: +49 431 99081811
PGP key Id 0x45B55BB1
Location: Kiel, Local Court Kiel, HRB 10298 KI
Managing Director: Andreas Zipp
---
Please be informed that this email and any materials attached herewith
may contain confidential or legally privileged information that are
intended solely for the use by its named recipient. If you have
received this email in error, please notify us immediately by reply
email and delete this email from your system. Any disclosure, use,
copying, printing, distribution or reliance upon the contents of this
email is strictly prohibited. Thank you for your attention. - This mail
is sent via VTech IAD GmbH

Re: amanda-3.4.5 does not fill one tape

[Gene Heskett]

On Thursday 14 May 2020 17:30:53 Nathan Stratton Treadway wrote:

> On Thu, May 14, 2020 at 09:14:17 +0200, Stefan G. Weichinger wrote:
> > Interesting, how can a "dirty" drive trigger this behavior?
> >
> > I'd expect failures all along and not after ~200 or 300 GB written.
> >
> > I don't see any interrupted writing or so (until that End Of Tape).
>
> (We switched to disk-drive vtapes a long time ago so when I was last
> looking into the details of backup-tape-drive behavior it was probably
> for pre-LTO technology, but I would assume that for this discussion
> LTO is similar)
>
> For "modern" error-correcting tape drives, when the computer sends
> data out to the tape drive to be written to tape, the drive actually
> then uses the read head to immedately read back in the data it just
> wrote. If that read fails, the drive will automatically/transparently
> try the write again... repeating the process until it is able to
> achieve a successful confirmation read of that block of data.
>
> Normally this just happens once in a while, when there's a bad spot on
> the tape or some fluke of writing makes the data unreadable, and one
> doesn't even notice it's happening.
>
> However, if the drive head is dirty or the tape media in general is
> wearing out, then what happens is that many many many of the data
> blocks either will be written badly or will fail to read back in
> [depending on what exactly is dirty or failing], and the drive will
> have to re-write data multiple times before a succesful write/read
> cycle.
>
> When that happens, then lots of the linear space on the tape is used
> by all the repeated writes -- thus making the tape appear to have a
> lower capacity than you would expect -- and also all that re-writing
> means the data throughput from the server's point of view is much
> reduced.
>
> (Note that in this scenario the drive just keeps retrying to write a
> block up data until it succeeds... or until it hits the end of the
> tape. So that's why you don't get "interrupted writing" in the sense
> of having mid-tape write errors returned by the tape device the
> computer.  [But it is "interrupted" in the sense that a block takes
> much longer to write than it should so the computer has to wait a long
> time before it can sent the next block of data down to the drive.])
>
> Hope that makes sense.
>
>   Nathan
It makes perfect sense Nathan, what makes no sense is the drive 
continuing to try forever, continueing to wear out its heads when what 
it should if the 2nd or maybe the third write fails, it should at the 
very minimum start a blinking "clean me" led and abort the run. That at 
the very least would make me change the makers name on a frame rail 
someplace, its simply not excusable.

You buy these things looking for dependability, and I've never yet had a 
tape drive last over 1000 tape moving hours.

I have now removed it from service, but I've a 1t drive in a drawer I 
will likely use for my next linux install, still has the 25 re-allocated 
sectors it had the first time I asked it many years ago. That report 
caused me to update its firmware. And that spinning rust drive has over 
70,000 spinning hours on it now, retired because it was getting too 
small.

And people wonder why I use vtapes.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: amanda-3.4.5 does not fill one tape

[Jon LaBadie]

Glad you brought up this "feature" Nathan.  I had heard it before
but not using tape, promptly forgot it.

Jon


On Thu, May 14, 2020 at 05:30:53PM -0400, Nathan Stratton Treadway wrote:
> On Thu, May 14, 2020 at 09:14:17 +0200, Stefan G. Weichinger wrote:
> > Interesting, how can a "dirty" drive trigger this behavior?
> > 
> > I'd expect failures all along and not after ~200 or 300 GB written.
> > 
> > I don't see any interrupted writing or so (until that End Of Tape).
> 
> 
> (We switched to disk-drive vtapes a long time ago so when I was last
> looking into the details of backup-tape-drive behavior it was probably
> for pre-LTO technology, but I would assume that for this discussion LTO
> is similar)
> 
> For "modern" error-correcting tape drives, when the computer sends data
> out to the tape drive to be written to tape, the drive actually then
> uses the read head to immedately read back in the data it just wrote. 
> If that read fails, the drive will automatically/transparently try the
> write again... repeating the process until it is able to achieve a
> successful confirmation read of that block of data.
> 
> Normally this just happens once in a while, when there's a bad spot on
> the tape or some fluke of writing makes the data unreadable, and one
> doesn't even notice it's happening.  
> 
> However, if the drive head is dirty or the tape media in general is
> wearing out, then what happens is that many many many of the data blocks
> either will be written badly or will fail to read back in [depending on
> what exactly is dirty or failing], and the drive will have to re-write
> data multiple times before a succesful write/read cycle.
> 
> When that happens, then lots of the linear space on the tape is used by
> all the repeated writes -- thus making the tape appear to have a lower
> capacity than you would expect -- and also all that re-writing means the
> data throughput from the server's point of view is much reduced.
> 
> (Note that in this scenario the drive just keeps retrying to write a
> block up data until it succeeds... or until it hits the end of the tape. 
> So that's why you don't get "interrupted writing" in the sense of having
> mid-tape write errors returned by the tape device the computer.  [But it
> is "interrupted" in the sense that a block takes much longer to write
> than it should so the computer has to wait a long time before it can
> sent the next block of data down to the drive.])
> 
> Hope that makes sense.
> 
>   Nathan
> 
> 
> Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
> Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
>  GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
>  Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda-3.4.5 does not fill one tape

[Nathan Stratton Treadway]

On Thu, May 14, 2020 at 09:14:17 +0200, Stefan G. Weichinger wrote:
> Interesting, how can a "dirty" drive trigger this behavior?
> 
> I'd expect failures all along and not after ~200 or 300 GB written.
> 
> I don't see any interrupted writing or so (until that End Of Tape).


(We switched to disk-drive vtapes a long time ago so when I was last
looking into the details of backup-tape-drive behavior it was probably
for pre-LTO technology, but I would assume that for this discussion LTO
is similar)

For "modern" error-correcting tape drives, when the computer sends data
out to the tape drive to be written to tape, the drive actually then
uses the read head to immedately read back in the data it just wrote. 
If that read fails, the drive will automatically/transparently try the
write again... repeating the process until it is able to achieve a
successful confirmation read of that block of data.

Normally this just happens once in a while, when there's a bad spot on
the tape or some fluke of writing makes the data unreadable, and one
doesn't even notice it's happening.  

However, if the drive head is dirty or the tape media in general is
wearing out, then what happens is that many many many of the data blocks
either will be written badly or will fail to read back in [depending on
what exactly is dirty or failing], and the drive will have to re-write
data multiple times before a succesful write/read cycle.

When that happens, then lots of the linear space on the tape is used by
all the repeated writes -- thus making the tape appear to have a lower
capacity than you would expect -- and also all that re-writing means the
data throughput from the server's point of view is much reduced.

(Note that in this scenario the drive just keeps retrying to write a
block up data until it succeeds... or until it hits the end of the tape. 
So that's why you don't get "interrupted writing" in the sense of having
mid-tape write errors returned by the tape device the computer.  [But it
is "interrupted" in the sense that a block takes much longer to write
than it should so the computer has to wait a long time before it can
sent the next block of data down to the drive.])

Hope that makes sense.

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: amanda-3.4.5 does not fill one tape

[Jon LaBadie]

On Thu, May 14, 2020 at 09:14:17AM +0200, Stefan G. Weichinger wrote:
> Am 14.05.20 um 07:59 schrieb Jens Berg:
> > There was a discussion back in 2014 with subject "Backups to tape
> > consistently under 60% tape capacity". I haven't read the whole lengthy
> > thread but one participant mentioned that in his case a bad cleaning
> > tape was found to be responsible for the capacity loss. Others reported
> > that they had trouble with the compression settings. My gut feeling
> > tells me that this is not the problem here but maybe your impression is
> > a different one or you get a new idea if you read through the postings.
> 
> Thanks for the pointer.
> 
> Interesting, how can a "dirty" drive trigger this behavior?
> 

Because somewhere along the line, some piece of the software does
not distinguish an error return as distinct from an EOF or EOM.
By the time it gets to the human, it is an EOF even if it was an
error.

For example, some writing functions return only success/failure.

Is that failure the end of tape or some other error?  Does the code
assume one and not investigate further?  If the code does investigate
further, is there a way to tell the upstream code success/EOF/error
or just two statuses?

> But it's worth a try, the customer for sure hasn't used the cleaning
> tape for years.

Sounds like a plan.  Maybe two runs?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda-3.4.5 does not fill one tape

[Gene Heskett]

On Thursday 14 May 2020 01:32:05 Jon LaBadie wrote:

> On Wed, May 13, 2020 at 06:17:02PM +0200, Stefan G. Weichinger wrote:
> > Now look at this run of
> >
> > "amtapetype -b 128k /dev/nst0"
> >
> > with another tape, FUJI instead of HP:
> >
> > define tapetype LTO3-fuji {
> > comment "Created by amtapetype; compression disabled"
> > length 284180096 kbytes
> > filemark 20803 kbytes
> > speed 38376 kps
> > blocksize 128 kbytes
> > }
> >
> > ~290 GB ... faster, and large filemarks.
> >
> > Maybe that drive is somehow failing .. ?
>
> I wasn't sure what LEOM was.  I assume it is "Logical End of Media".
>
> Anyway I came across two references that said need for cleaning
> is one reason for getting early EOM.

At this point I would throw in that these drives work on the same 
principles as a video tape recorder, and when I walked in the door 
in '84 to be the CE at a tv station, hey were using freon TF to clean 
tape heads with, and doing it 2-3 times a day on every machine.  When 
the freon's started to get a bad rap, I switched to paint thinner 
alcohol, and made an amazing discovery, suddenly the machines were going 
a week before cleanings. But the tape decks used here, are wrapped up in 
the mechanics and not readily accessible for such q-tip based cleaning.

I've also observed that tape stored at 40F and 10% humidity is only about 
2% as abrasive as tape stored at room temp and humidity.  So putting a 
tape library in a small closet on an outside wall, with a too small AC 
running forever to not only keep it cold but suck all the humidity out 
of the air can be very helpfull.

At Nebraska ETV, half the state is in the mountain time zone, so at one 
of the tv stations they had 3 of the old 2" Ampex machines, serving as 
the 1 hour time delay, and that room was kept at around 55F with a 
precipitron equipt HVAC. People, but no smoking, food or drink allowed.

Head life in a normal environment for a 'soft' head on those machines is 
around 750 hours. After a while they had a transformer failure and had 
to swap heads for a fresh one.  It was at that point they read the old 
heads spinning hours meter and found it had 9700 hours on it.  Sent back 
to the rebuilder, he fixed the transformer and sent it back.  Put back 
in the machine it finally failed at nearly 12,000 hours. Pretty good for 
a head with a pro-rated 750 hour warranty. More than demoing the effect 
of environment on such stuff.

> I'm wondering also if this could be a case of Amanda tapes being
> labelled with the mode set to LTO-2 capacity.  I know you check
> the mode and it shows 44, but Amanda always reads the tape before
> writing.  Could this be setting the mode back to 42 because the
> tapes were initially labelled with the mode set incorrectly?

In which case the fix is to read the label block and save it, rewind the 
tape but do NOT remove it giving the drive a chance to re-read the tapes 
hidden header, use mtx or whatever to set it correctly and rewrite the 
label block.  The drive will then fix that bad ID in the header before 
it writes the label block. If the tape is removed you have to start all 
over again because the drive will scan the hidden header and restore the 
erronious settings.  BTDT, pain in the butt.

> What if you forget about amtapetype and simply use dd to see how
> much random data it will write to tape.

Since the above fix is not a full pass run, but only a few seconds if a 
script does it, write the script and do the above first.  Much faster.

> Jon



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 14.05.20 um 07:59 schrieb Jens Berg:
> There was a discussion back in 2014 with subject "Backups to tape
> consistently under 60% tape capacity". I haven't read the whole lengthy
> thread but one participant mentioned that in his case a bad cleaning
> tape was found to be responsible for the capacity loss. Others reported
> that they had trouble with the compression settings. My gut feeling
> tells me that this is not the problem here but maybe your impression is
> a different one or you get a new idea if you read through the postings.

Thanks for the pointer.

Interesting, how can a "dirty" drive trigger this behavior?

I'd expect failures all along and not after ~200 or 300 GB written.

I don't see any interrupted writing or so (until that End Of Tape).

But it's worth a try, the customer for sure hasn't used the cleaning
tape for years.

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 14.05.20 um 07:32 schrieb Jon LaBadie:

> I wasn't sure what LEOM was.  I assume it is "Logical End of Media".

Seems like, yes. I still haven't figured out how to set that, the
"device_property" line wasn't accepted by Amanda.

> Anyway I came across two references that said need for cleaning
> is one reason for getting early EOM.
> 
> I'm wondering also if this could be a case of Amanda tapes being
> labelled with the mode set to LTO-2 capacity.  I know you check
> the mode and it shows 44, but Amanda always reads the tape before
> writing.  Could this be setting the mode back to 42 because the
> tapes were initially labelled with the mode set incorrectly?

I used two new tapes in the last days. The labelling happened after
"stinit", and I even rm-ed the header with dd. etc etc

> What if you forget about amtapetype and simply use dd to see how
> much random data it will write to tape.

Would be a test, yes, but wouldn't help if amanda does things differently.

I mailed the customer and have them look for their cleaning tape.

After a cleaning run (if they have and find the tape) I try some dd-test
run, ok.

Re: amanda-3.4.5 does not fill one tape

[Jon LaBadie]

On Wed, May 13, 2020 at 06:17:02PM +0200, Stefan G. Weichinger wrote:
> Now look at this run of
> 
> "amtapetype -b 128k /dev/nst0"
> 
> with another tape, FUJI instead of HP:
> 
> define tapetype LTO3-fuji {
> comment "Created by amtapetype; compression disabled"
> length 284180096 kbytes
> filemark 20803 kbytes
> speed 38376 kps
> blocksize 128 kbytes
> }
> 
> ~290 GB ... faster, and large filemarks.
> 
> Maybe that drive is somehow failing .. ?

I wasn't sure what LEOM was.  I assume it is "Logical End of Media".

Anyway I came across two references that said need for cleaning
is one reason for getting early EOM.

I'm wondering also if this could be a case of Amanda tapes being
labelled with the mode set to LTO-2 capacity.  I know you check
the mode and it shows 44, but Amanda always reads the tape before
writing.  Could this be setting the mode back to 42 because the
tapes were initially labelled with the mode set incorrectly?

What if you forget about amtapetype and simply use dd to see how
much random data it will write to tape.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda-3.4.5 does not fill one tape

[Debra S Baddorf]

> On May 13, 2020, at 11:17 AM, Stefan G. Weichinger  wrote:
> 
> Now look at this run of
> 
> "amtapetype -b 128k /dev/nst0"
> 
> with another tape, FUJI instead of HP:
> 
> define tapetype LTO3-fuji {
>comment "Created by amtapetype; compression disabled"
>length 284180096 kbytes
>filemark 20803 kbytes
>speed 38376 kps
>blocksize 128 kbytes
> }
> 
> ~290 GB ... faster, and large filemarks.
> 
> Maybe that drive is somehow failing .. ?


No real ideas here, but I *DO* recall some failing drives on this list
where the symptom was like this.  Tapes wouldn’t fill.

Anybody else still out there in the aether?

Deb Baddorf
Fermilab

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Now look at this run of

"amtapetype -b 128k /dev/nst0"

with another tape, FUJI instead of HP:

define tapetype LTO3-fuji {
comment "Created by amtapetype; compression disabled"
length 284180096 kbytes
filemark 20803 kbytes
speed 38376 kps
blocksize 128 kbytes
}

~290 GB ... faster, and large filemarks.

Maybe that drive is somehow failing .. ?

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 13.05.20 um 09:03 schrieb Stefan G. Weichinger:

> It looks like a LTO2 tape ... although the customer told me it says LTO3
> on the cartridge (and has a correct product number).
> 
> mt status detects it as density=0x44 as well, but the capacity and speed
> looks like LTO2.
> 
> Strange.

Some more info:

# tapeinfo  -f /dev/nst0
Product Type: Tape Drive
Vendor ID: 'HP  '
Product ID: 'Ultrium 3-SCSI  '
Revision: 'Q51D'
Attached Changer API: No
SerialNumber: 'HU11339W0G'
MinBlock: 1
MaxBlock: 16777215
SCSI ID: 4
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x44
BlockSize: 0
DataCompEnabled: no
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
Block Position: 1
Partition 0 Remaining Kbytes: 400308
Partition 0 Size in Kbytes: 400308
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 0

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Retried amtatpetype with another new tape yesterday.

No success.

Labelled and tested with 64k blocksize, still about 200 GB size only.

Could the older kernel somehow play a role here?

The output of amtapetype says:

"LEOM is not supported for this drive and kernel"

I try to set "LEOM false" now explicitly.

Linux version 3.12.13-gentoo ... yeah, I know ...

But the server is ~3 hrs away and old, reinstallation without guarantees
is somehow inefficient.

(No IRMC/ILO remote console for doing kernel change easily)

-

It looks like a LTO2 tape ... although the customer told me it says LTO3
on the cartridge (and has a correct product number).

mt status detects it as density=0x44 as well, but the capacity and speed
looks like LTO2.

Strange.

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 12.05.20 um 14:10 schrieb Stefan G. Weichinger:

> Maybe they bought LTO2 tapes, I check that asap.

No, the tapes are OK.

HP LTO3 C7973A

I googled some stinit.def and set:

# cat /etc/stinit.def
# HP StorageWorks Ultrium 960/920 SAS LTO-3
manufacturer="HP" model = "Ultrium 3-SCSI" {
scsi2logical=1
can-bsr=1
auto-lock=0
two-fms=0
drive-buffering=1
buffer-writes
read-ahead=1
async-writes=1
can-partitions=0
fast-eom=1
blocksize=0
sili=1
#
# If your stinit supports the timeouts:
timeout=900 # 15 min
long-timeout=14400 # 4 hours
#
# Drive is backwards write compatible to LTO-2 media and read compatible
to LTO-1 media. Mode settings are only required for writing, so we need
only the LTO-3/LTO-2 modes here.
mode1 density=0x44 compression=0# LTO3 400 GB
mode2 density=0x44 compression=1# LTO3 400 GB + compression
mode3 density=0x42 compression=0# LTO2 220 GB
mode4 density=0x42 compression=1# LTO2 220 GB + compression
}


I try /dev/st0a etc now

Re: amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Am 12.05.20 um 08:34 schrieb Stefan G. Weichinger:

> backups run to a LTO3 tape (remember, 400 GB uncompressed space per
> definition)

I ran amtapetpye and only get half of the expected capacity!

why that  ...

$ amtapetype -t LTO3_2020 -f /dev/nst0

Checking for FSF_AFTER_FILEMARK requirement
Applying heuristic check for compression.
Wrote random (uncompressible) data at 27134018.0645161 bytes/sec
Wrote fixed (compressible) data at 27578838.0327869 bytes/sec
Compression: disabled
Writing one file to fill the volume.
Wrote 209430315008 bytes at 27608 kb/sec
Writing smaller files (2094301184 bytes) to determine filemark.
define tapetype LTO3_2020 {
comment "Created by amtapetype; compression disabled"
length 204521792 kbytes
filemark 0 kbytes
speed 27608 kps
blocksize 32 kbytes
}
# LEOM is not supported for this drive and kernel

amanda@amanda ~ $ mt -f /dev/st0 status
SCSI 2 tape drive:
File number=102, block number=0, partition=0.
Tape block size 0 bytes. Density code 0x44 (LTO-3).
Soft error count since last status=0
General status bits on (8101):
 EOF ONLINE IM_REP_EN


amanda@amanda ~ $ cat /proc/scsi/scsi
Attached devices:

[..]

  Vendor: HP   Model: Ultrium 3-SCSI   Rev: Q51D
  Type:   Sequential-AccessANSI  SCSI revision: 05


Maybe they bought LTO2 tapes, I check that asap.

amanda-3.4.5 does not fill one tape

[Stefan G. Weichinger]

Over the last weeks I fiddle with an installation with the following
behavior:

The amanda server is an older gentoo linux box and runs amanda-3.4.5r1
(gentoo package number).

The reason I keep this older version is that I have to talk to an amanda
client which is way older -> a VM with ancient SUSE Linux and
amanda-2.4.0 ... that VM is "read only" in terms of upgrades etc

Things work so far, and I assume my issue is not related to the VM.

The point is:

backups run to a LTO3 tape (remember, 400 GB uncompressed space per
definition)

And when I run a "weekly" run, which uses "dumpcycle 0" it fills the
tape up to ~60% = 239944M and lets a DLE in the holding disk which is
small enough to fit on the tape ...

( a 72 GB DLE )

-

I disabled hw compression again, and added that to the cron job.

I will run amtapetype on a new tape today to determine "my" LTO3 tapetype.

Any ideas?

Re: bad tape?

[Olivier]

Jens Berg  writes:

> On 8/28/2019 9:14 AM, Olivier wrote:
>> I have a dedicated server that runs Amanda, with 8 bays, I never
>> disconnect the disks unless it is time to replace them with a newer
>> and biger one.
>
> Don't you have Off-Site backups?

No.

> Since I switched from LTO to vtapes, I'm using USB drives for that which
> I exchange weekly with a set I store at home. Never had issues with that
> in the last years.

My Amanda server is an old machine, no USB 3, so USB drives is not realistic.

> I'd love to use a cloud service instead but the price for a reasonable
> fast Internet connection together with a sufficiently large cloud
> storage is way too high.

Depending on the amount of cloud storage you need, how long you want to
keep it, etc. I was considering Blackblaze B2 storage or Amazon glacier
(is it Amazon) as it is storage that you hopefully do not need to access
much, so in the cloud it can drift to cheaper 3rd tier storage. From
what I looked, their prices would be similar. I'd favor Blackblaze for
their no non-sense, very simple approach.

If I remember well, I needed a round robin of about 10TB, (new backup
added every week, but older backup removed at the same rate, so the
total storage does not grow much) and it adds up to 5$/month/TB. My
current limitation is the network bandwidth, until there is a cloud
installed in Thailand, I would use all my current weekly b/w just to
upload my weekly level 0 backup.

I had been considering other solutions, that would offer "unlimited"
cloud storage for personal use, but my use is not personal and it is
often "backup storage" so you have to conform to their file system, use
their application to do the storage (not API, but backup software)
etc. And you have never guarantee as when they will start limiting their
storage capacity.

One last solution is my own colocated server at a datacenter we have
good connection with, but so far, monthly running cost would not be
cheaper than storage space on the cloud and you must factor the
procurement and maintenance cost. Maybe if I replace the current Amanda
server by a brand new system, that current one could go to coloc.

Best regards,

Olivier

-- 

Re: bad tape?

[Jens Berg]

On 8/28/2019 9:14 AM, Olivier wrote:
> I have a dedicated server that runs Amanda, with 8 bays, I never
> disconnect the disks unless it is time to replace them with a newer
> and biger one.

Don't you have Off-Site backups?

Since I switched from LTO to vtapes, I'm using USB drives for that which
I exchange weekly with a set I store at home. Never had issues with that
in the last years.
I'd love to use a cloud service instead but the price for a reasonable
fast Internet connection together with a sufficiently large cloud
storage is way too high.

Jens

Re: bad tape?

[Olivier]

Diego Zuccato  writes:

> Il 28/08/19 03:34, Olivier ha scritto:
>
>> Or write another device for Amanda to use, it would not be vtape, it
>> would be ... something.
> Could be 'rawdisk'. :)
>
> But better plan for some redundancy to compensate for silent corruption.
>
> And consider that SATA connectors have a limited life (about 500 mating
> cycles, IIRC): for frequently-removed disks it's better to use a
> dedicated caddy, maybe an eSata one.

I have a dedicated server that runs Amanda, with 8 bays, I never
disconnect the disks unless it is time to replace them with a newer
and biger one.

Olivier

-- 

Re: bad tape?

[Diego Zuccato]

Il 28/08/19 03:34, Olivier ha scritto:

> Or write another device for Amanda to use, it would not be vtape, it
> would be ... something.
Could be 'rawdisk'. :)

But better plan for some redundancy to compensate for silent corruption.

And consider that SATA connectors have a limited life (about 500 mating
cycles, IIRC): for frequently-removed disks it's better to use a
dedicated caddy, maybe an eSata one.

-- 
Diego Zuccato
Servizi Informatici
Dip. di Fisica e Astronomia (DIFA) - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
mail: diego.zucc...@unibo.it

Re: bad tape?

[Olivier]

Gene Heskett  writes:

> On Monday 26 August 2019 23:55:31 Olivier wrote:
>
>> Gene Heskett  writes:
>> > Generally speaking, only because the disc is random access.
>>
>> But a disk dedicated to vtapes should be doing a lot of sequetial
>> accesses: once it has been formatted and the slots have been assigned,
>> it is writting files the size of one Amanda's chunk. In fact, that
>> would be worth a study: the disk usage for vtapes vs. normal disk
>> usage.
>>
>> That is just gross figures but:
>>
>> Users' home directories:
>> FilesystemSizeUsed   Avail Capacity  Mounted on
>> /dev/da1p12.9T851G1.8T31%/home
>> 2565312 files, 223129681 used, 556890331 free
>> (564355 frags, 69540747 blocks, 0.1% fragmentation)
>
> My Mail dir is on /home, dates back to about my 2nd install in 2001, with 
> probably north of 20 GB of maildirs, so I'd expect that is more than 1% 
> fragmented, but except for tde's kmail occasionally bucking about it, 
> has not been a major problem. Copying it to a new Maildir usually fixes 
> it for that particular list.  Reducing the keep time for those lists 
> deemed not so important has also helped. About 3 lists I keep forever, 
> but 50 more are expired every 3 or 6 months.

That was given as an example only, to show that fragmentation grows
faster on a file system supporting users' home directory tha on a file
system supporting Amanda'a vtapes. I had rebuild the users' home
directory on a new RAID array recently, that is why it is so little
fragmented, but still much more than an Amanda disk that is much older.

>> Amanda vtape disk:
>> FilesystemSizeUsed   Avail Capacity  Mounted on
>> /dev/ada5p1   2.6T2.2T269G89%/automnt/ada5
>> 475 files, 582393950 used, 127171372 free
>> (84 frags, 15896411 blocks, 0.0% fragmentation)
>>
>> The vtape disk is slightly older than the users' home, definitely
>> fuller and less fragmented, so I would guess big sequetial files with
>> little head movement.
>>
> I wondered about fragmentation myself, but it has never reared its head 
> in well over 15 years.

Best regards,

Olivier


-- 

Re: bad tape?

[Olivier]

Diego Zuccato  writes:

> Il 27/08/19 05:55, Olivier ha scritto:
>
>> But a disk dedicated to vtapes should be doing a lot of sequetial
>> accesses: once it has been formatted and the slots have been assigned,
>> it is writting files the size of one Amanda's chunk. In fact, that would
>> be worth a study: the disk usage for vtapes vs. normal disk usage.
> That's the *perfect* use-case for SMR drives. But they'd need either no
> filesystem (like tapes :) ) or a dedicated one.
> Is it possible to use raw devices as vtapes?

vtape do rely on a Unix file system, it needs a directory and a couple
of sub-directory per vtape.

Olivier
-- 

Re: bad tape?

[Olivier]

Jon LaBadie  writes:

> On Tue, Aug 27, 2019 at 11:44:02AM +0200, Diego Zuccato wrote:
>> Il 27/08/19 05:55, Olivier ha scritto:
>> 
>> > But a disk dedicated to vtapes should be doing a lot of sequetial
>> > accesses: once it has been formatted and the slots have been assigned,
>> > it is writting files the size of one Amanda's chunk. In fact, that would
>> > be worth a study: the disk usage for vtapes vs. normal disk usage.
>> That's the *perfect* use-case for SMR drives. But they'd need either no
>> filesystem (like tapes :) ) or a dedicated one.
>> Is it possible to use raw devices as vtapes?
>
> My guess is that vtapes are expecting normal *nix file
> names and system/library calls.

They do, indeed.

> I'm pretty sure it would be possible to write a FUSE (file system
> in user space extension) to implement what you imagine.

Or write another device for Amanda to use, it would not be vtape, it
would be ... something.

Olivier
-- 

Re: bad tape?

[Jon LaBadie]

On Tue, Aug 27, 2019 at 11:44:02AM +0200, Diego Zuccato wrote:
> Il 27/08/19 05:55, Olivier ha scritto:
> 
> > But a disk dedicated to vtapes should be doing a lot of sequetial
> > accesses: once it has been formatted and the slots have been assigned,
> > it is writting files the size of one Amanda's chunk. In fact, that would
> > be worth a study: the disk usage for vtapes vs. normal disk usage.
> That's the *perfect* use-case for SMR drives. But they'd need either no
> filesystem (like tapes :) ) or a dedicated one.
> Is it possible to use raw devices as vtapes?

My guess is that vtapes are expecting normal *nix file
names and system/library calls.

I'm pretty sure it would be possible to write a FUSE (file system
in user space extension) to implement what you imagine.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: bad tape?

[Diego Zuccato]

Il 27/08/19 05:55, Olivier ha scritto:

> But a disk dedicated to vtapes should be doing a lot of sequetial
> accesses: once it has been formatted and the slots have been assigned,
> it is writting files the size of one Amanda's chunk. In fact, that would
> be worth a study: the disk usage for vtapes vs. normal disk usage.
That's the *perfect* use-case for SMR drives. But they'd need either no
filesystem (like tapes :) ) or a dedicated one.
Is it possible to use raw devices as vtapes?

-- 
Diego Zuccato
Servizi Informatici
Dip. di Fisica e Astronomia (DIFA) - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
mail: diego.zucc...@unibo.it

Re: bad tape?

[Gene Heskett]

On Monday 26 August 2019 23:55:31 Olivier wrote:

> Gene Heskett  writes:
> > Generally speaking, only because the disc is random access.
>
> But a disk dedicated to vtapes should be doing a lot of sequetial
> accesses: once it has been formatted and the slots have been assigned,
> it is writting files the size of one Amanda's chunk. In fact, that
> would be worth a study: the disk usage for vtapes vs. normal disk
> usage.
>
> That is just gross figures but:
>
> Users' home directories:
> FilesystemSizeUsed   Avail Capacity  Mounted on
> /dev/da1p12.9T851G1.8T31%/home
> 2565312 files, 223129681 used, 556890331 free
> (564355 frags, 69540747 blocks, 0.1% fragmentation)

My Mail dir is on /home, dates back to about my 2nd install in 2001, with 
probably north of 20 GB of maildirs, so I'd expect that is more than 1% 
fragmented, but except for tde's kmail occasionally bucking about it, 
has not been a major problem. Copying it to a new Maildir usually fixes 
it for that particular list.  Reducing the keep time for those lists 
deemed not so important has also helped. About 3 lists I keep forever, 
but 50 more are expired every 3 or 6 months.

> Amanda vtape disk:
> FilesystemSizeUsed   Avail Capacity  Mounted on
> /dev/ada5p1   2.6T2.2T269G89%/automnt/ada5
> 475 files, 582393950 used, 127171372 free
> (84 frags, 15896411 blocks, 0.0% fragmentation)
>
> The vtape disk is slightly older than the users' home, definitely
> fuller and less fragmented, so I would guess big sequetial files with
> little head movement.
>
I wondered about fragmentation myself, but it has never reared its head 
in well over 15 years.

> Good luck with your health.
>
Thank you.

> Olivier



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: bad tape?

[Olivier]

Gene Heskett  writes:

> Generally speaking, only because the disc is random access.

But a disk dedicated to vtapes should be doing a lot of sequetial
accesses: once it has been formatted and the slots have been assigned,
it is writting files the size of one Amanda's chunk. In fact, that would
be worth a study: the disk usage for vtapes vs. normal disk usage.

That is just gross figures but:

Users' home directories:
FilesystemSizeUsed   Avail Capacity  Mounted on
/dev/da1p12.9T851G1.8T31%/home
2565312 files, 223129681 used, 556890331 free
(564355 frags, 69540747 blocks, 0.1% fragmentation)

Amanda vtape disk:
FilesystemSizeUsed   Avail Capacity  Mounted on
/dev/ada5p1   2.6T2.2T269G89%/automnt/ada5
475 files, 582393950 used, 127171372 free
(84 frags, 15896411 blocks, 0.0% fragmentation)

The vtape disk is slightly older than the users' home, definitely fuller
and less fragmented, so I would guess big sequetial files with little
head movement.

Good luck with your health.

Olivier

Re: bad tape?

[Gene Heskett]

On Monday 26 August 2019 21:42:29 Olivier wrote:

> ghe  writes:
> > Stan and Debra have convinced me to bite the bullet and buy a new
> > tape. I've never been in this situation before (the DLT drive used
> > to fail every once in a while, but a couple hours with a jeweler's
> > screwdriver got it going again).
> >
> > Looks like I'm going to have a spare, mildly flaky, tape around.
>
> You mean that all your backups were on a single tape? That is beyond
> daring IMHO.
>
> Like Gene mentioned, I have moved to disk and vtapes, never had a
> problem with the disks holding the vtape, I had problems with the disk
> with the holding disk though, it started develop bad blocks in the
> holding disk partition, but vtapes? They are hardly used, if you
> unmount/stop them after the backups are written, they can last a long
> time.
>
> And when you need more space, you just upgrade with newer disks with
> more capacity and store the old disks, so you even have offline old
> stuff.
>
> Your slots need to be numbered starting from 1, but the tapes number
> can start from the next number in sequence in your tape naming scheme,
> so keep the old disks. You will change them because you need more
> backup storage, not because they fail (never had since I use disks,
> while I had to replace the tapes a number of times).
>
> I started with 7x512GB disks, then 7x1.5TB and am now at 7x3TB
> and never had the slightest problem. Admittedly, the size of the
> vtapes chosen at first looks a bit tiny nowdays, but with vtapes, it
> really adds very little overhead, largely compensated by avoiding half
> full tapes.
>
> When we were hit by the great flood and we had to move the datacenter,
> I took the disks, but not Amanda server, as I knew I could connect the
> disk directly inside any server to do a restore.
>
> And no fiddling with mounting and dismounting tapes every day, I have
> a system with all my vtapes available all the time.
>
> In a similar way to what Gene described, after the dump, I get a copy
> of the indexes, etc. and rsync that to a database server and mail
> myself a copy, I end up with 5 or 6 copies of that critical
> information (because database and mail are backed up too).
>
> And remeber, tapes do rub on the R/W head of the tape drive so doing
> an amcheckdump does double the wear and tear of the tape. 
That also doubles the head wear.  Since disk heads fly on a film of air 
that might be under a micron thick, if the disc is well finished, the 
only contact is at a power fail, when it has no choice but to land on 
the still spinning disc. Then the successfull restart depends on how 
long it takes to get power back.  Past a month or so sitting there, the 
head is likely stuck to the disc like a machinists set of joe's blocks 
and the disc is locked from spinning up again.

I have a pair of one GB scsi drives on a trs-80 color computer that 
haven't been shut down for more than a couple months since about 1989, 
and had to dismount them so the projected 1/2" from the front of the 
tower they are in, power restored, and an 8oz dead blow hammer applied 
to the front corner enough to bump it sideways 1/8", which is enough to 
rotate the housing about the disc, breaking the stiction. And once the 
disc has moved 1/4 turn its speed supplies the air cushion under the 
head.  That was about 2 years ago. I have a 20 kw standby thats 
typically up and running before the disc's have stopped, so a tap on the 
reset button reboots it normally. Those old drives have had since around 
1989, collecting spinning hours at 8766 a year, so have close to 166554 
spinning hours. Still functioning at 100%.  No bad sectors.

I think thats pretty decent testimony to convince anyone into never 
shutting a drive off

They will happily outlast me. I've just spent 2 weeks in the shop from a 
heart attack, and still look like your grandmothers pincushion from all 
the taps they put in to keep me going. Currrantly its pumping around 
30%, so I'm not pushing myself too hard just yet as they are building a 
new aorta valve they will install as a cath-lab outpatient some time 
next month.

> No physical 
> contact between the head and support on disk.
>
> Not to mention that disks are way faster...
>
Generally speaking, only because the disc is random access. Actual data 
rate to/from the head is rarely more than one magnitude faster for the 
disc. And that differential is more in the teenyness of the head 
allowing a higher frequency than in any other magic.

> Best regards,
>
> Olivier



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: bad tape?

[Charles Curley]

On Mon, 26 Aug 2019 16:01:58 -0400
Gene Heskett  wrote:

> But AFAIK, tapes don't maintain an allocation map, and we have no
> tape writing tools so organized as to be able to implement such a
> scheme.

Not entirely true.

Tape drives such as Colorado Memory Systems' QIC drives were random
access, because they used the floppy controller to access the tape.
Firmware on the drive translated head, track and sector selection into
tracks and sectors on the tape drive.

Also, many years ago, Forth, Inc wrote a driver for a DEC tape drive
that did random access. The client wanted *rugged* and tape drives in
those days were far more rugged than hard drives. Forth doesn't use
FATs; indeed, it doesn't use files.

Yes, it was slow, compared to hard drives even then. Both systems were
inclined toward a lot of shoe-shining, which wears out tapes.

Whether any modern tape drives have any analogous capabilities I cannot
say.

Today's computer trivia was brought to you by the Stonehenge Retro
Computing Club.

-- 
"When we talk of civilization, we are too apt to limit the meaning of
the word to its mere embellishments, such as arts and sciences; but
the true distinction between it and barbarism is, that the one
presents a state of society under the protection of just and
well-administered law, and the other is left to the chance government
of brute force."
- The Rev. James White, Eighteen Christian Centuries, 1889
Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
https://charlescurley.com

Re: bad tape?

[Olivier]

ghe  writes:

> Stan and Debra have convinced me to bite the bullet and buy a new tape.
> I've never been in this situation before (the DLT drive used to fail
> every once in a while, but a couple hours with a jeweler's screwdriver
> got it going again).
>
> Looks like I'm going to have a spare, mildly flaky, tape around.

You mean that all your backups were on a single tape? That is beyond
daring IMHO.

Like Gene mentioned, I have moved to disk and vtapes, never had a
problem with the disks holding the vtape, I had problems with the disk
with the holding disk though, it started develop bad blocks in the
holding disk partition, but vtapes? They are hardly used, if you
unmount/stop them after the backups are written, they can last a long
time.

And when you need more space, you just upgrade with newer disks with
more capacity and store the old disks, so you even have offline old
stuff.

Your slots need to be numbered starting from 1, but the tapes number can
start from the next number in sequence in your tape naming scheme, so
keep the old disks. You will change them because you need more backup
storage, not because they fail (never had since I use disks, while I had
to replace the tapes a number of times).

I started with 7x512GB disks, then 7x1.5TB and am now at 7x3TB
and never had the slightest problem. Admittedly, the size of the vtapes
chosen at first looks a bit tiny nowdays, but with vtapes, it really
adds very little overhead, largely compensated by avoiding half full
tapes.

When we were hit by the great flood and we had to move the datacenter, I
took the disks, but not Amanda server, as I knew I could connect the
disk directly inside any server to do a restore.

And no fiddling with mounting and dismounting tapes every day, I have a
system with all my vtapes available all the time.

In a similar way to what Gene described, after the dump, I get a copy of
the indexes, etc. and rsync that to a database server and mail myself a
copy, I end up with 5 or 6 copies of that critical information (because
database and mail are backed up too).

And remeber, tapes do rub on the R/W head of the tape drive so doing an
amcheckdump does double the wear and tear of the tape. No physical
contact between the head and support on disk.

Not to mention that disks are way faster...

Best regards,

Olivier

Re: bad tape?

[ghe]

On 8/26/19 5:20 PM, Nathan Stratton Treadway wrote:

> I haven't looked closely at tape drives in a few years, but others on
> this list certainly have in-depth experience with them.  I think some of
> the specific answers do depend on what tape-drive technology/generation
> you are using, so it might help for you to post that info.

It's an LTO-5.

> Also, "because of a tape error" is pretty vague; knowing the exact error
> message might make a difference in what advice we can give you

Well, it's been more than 30 minutes ago, so I can't remember exactly
what the massage was. But I think it was something about some of the
DLEs didn't copy because of a tape failure, but they're on the holding
disk. Nothing real specific.

> So, if you have a tape that is starting to "go bad" overall, usually the
> symptom is that Amanda hits the end of the tape after writing less data
> than the expected capacity for the tape -- rather than some sort of "bad
> block" error.

Yup. That's me.

> A related thing to keep in mind is that the writes can fail because
> specific spots on a specfic tape are bad...

That's what I think is going on.

> (amtapetype does have an "-l" option so you could tell it to re-create
> the label that is already in the header file on the tape, 

No, I wasn't wanting to validate the data. Just the tape itself, without
destroying the Amanda header -- I think the -l option to amtapetype is
what I'm looking for.

Stan and Debra have convinced me to bite the bullet and buy a new tape.
I've never been in this situation before (the DLT drive used to fail
every once in a while, but a couple hours with a jeweler's screwdriver
got it going again).

Looks like I'm going to have a spare, mildly flaky, tape around.

> (Denise mentioned the "amcheckdump" utility.  

My backup script rewinds and runs amcheckdump. I was told a long time
ago "An unverified backup isn't a backup"

Thanks all.

-- 
Glenn English

Re: bad tape?

[Debra S Baddorf]

Cheap is defined in comparison with “how long will it take to recreate all that 
data by hand,
if it’s lost”.  The value of that may depend on your reason for doing a backup.

Deb Baddorf
Fermilab

> On Aug 26, 2019, at 6:39 PM, ghe  wrote:
> 
> On 8/26/19 4:16 PM, stan wrote:
> 
>> Tapes are cheap! 
> 
> Define 'cheap' :-)
> 
> There's a significant difference between what Google and I consider cheap...
> 
>> What technology BTW.
> 
> LTO-5
> 
> I'm a newcomer from the DLT world. Those were reasonably cheap by my
> definition...
> 
> -- 
> Glenn English

Re: bad tape?

[ghe]

On 8/26/19 2:01 PM, Gene Heskett wrote:

> I did that once for a very old hard drive, permanently allocating about 
> 30 sectors to a file named badsectors.fd.  Worked great.

That's a clever idea...

> But AFAIK, tapes don't maintain an allocation map, and we have no tape 
> writing tools so organized as to be able to implement such a scheme.
> 
> Because tape is sequential, any such attempt would have to maintain such 
> a mapping on separate random access media.
> 
> The drive would have to be able to seek past the damaged tape to the next 
> block with a good checksum, something I've not found a drive capable of 
> doing in around a decade of trying, they all stop and error on the bad 
> spot.

Ada Lovelace era technology. That's what I suspected. But one can always
hope.

> The error correction in the average terabyte class spinning rust drive 
> has far exceeded that of tape, yes I'm talking about the sub $100 drive 
> here, and I haven't lost a single bit of data in the 15+ years since I 
> converted to vtape. 

I'm on tape instead of disk because it's disks we're backing up so we'll
still have data when the disk goes south. Or when the computer looses
it's mind and scribbles all over the disk(s).

> you can read without using anything in the 
> amanda ammo box at all if push comes to shove while doing a bare metal 
> recovery. 

Yeah, they claim that with tape too. I haven't yet had the pleasure.

-- 
Glenn English

Re: bad tape?

[ghe]

On 8/26/19 4:16 PM, stan wrote:

> Tapes are cheap! 

Define 'cheap' :-)

There's a significant difference between what Google and I consider cheap...

> What technology BTW.

LTO-5

I'm a newcomer from the DLT world. Those were reasonably cheap by my
definition...

-- 
Glenn English

Re: bad tape?

[Nathan Stratton Treadway]

On Fri, Aug 16, 2019 at 11:38:45 -0600, ghe wrote:
> I have reason to believe that one of my tapes isn't working properly
> (last night's backup died without finishing because of a tape error --
> the retry running right now successfully flushed last night's and wrote
> a new one).

I haven't looked closely at tape drives in a few years, but others on
this list certainly have in-depth experience with them.  I think some of
the specific answers do depend on what tape-drive technology/generation
you are using, so it might help for you to post that info.

Also, "because of a tape error" is pretty vague; knowing the exact error
message might make a difference in what advice we can give you

> Is there an Amanda utility that will validate a tape without destroying
> the Amanda header at the front of the tape? And maybe build a 'bad
> sector table' on the tape?

In my experence (which I guess was an LTO-2 drive), bad spots on the
tape were handled transparently by the drive. As Gene mentioned, tapes
are sequential, so there aren't fixed "blocks" like there are on disk
drives.  Instead, the drive just knows that it has some data it needs to
write, and writes out that data onto the bit of tape currently streaming
past the write head, then attempts to re-read that just-written data
with the read head as the tape goes by there.  If the read is
successful the drive considers that data to have been successfully
written and moves on to the next chunk of date; if the read fails, the
drive will write that same data again onto the next segment of tape,
repeating the process until the write/read cycle succeeds.

(Put another way, the exact spot on the tape that gigabyte number X is
written onto the tape will vary from run to run, depending on all sorts
of factors happening earlier in the the run.  But it doesn't matter,
because the drive never tries to find data based on how many centimeters
of tape have passed from the front of the tape, but instead scans along
the tape bit by bit reading data, or at least looking for file markers,
as it goes.)

So, if you have a tape that is starting to "go bad" overall, usually the
symptom is that Amanda hits the end of the tape after writing less data
than the expected capacity for the tape -- rather than some sort of "bad
block" error.

A related thing to keep in mind is that the writes can fail because
specific spots on a specfic tape are bad... or because e.g. the write
head is dirty, which has nothing to do with the specific tape in use.

So, what I did was watch my Amanda reports to see if a specific tape
consistently gave me an "out of tape" error each type it was used, and
when that happened I figured the tape was getting worn out and removed
it from the cycle.  (We ran the cleaning tape on the weekly cycle or
whatever the drive manufacturer recommended, which was not an even
fraction of the number of tapes in rotation, so from use to use a
particular tape would sometimes be used shortly after the heads were
cleaned -- and thus we could be pretty sure that it really was a bad
tape if the error happened consistently over many cycles.)

Regarding utilities to "validate" the tape: as far as I remember only
"amtapetype" is related to checking a tape to see its capacity.  So if
you wanted to more directly check that a particular tape had a much
lower than expected capacity you could use amtapetype to see but
really all that can do is confirm what Amanda is already seeing (i.e.
the expected dumps no longer fit on that tape).  And such a test would
overwrite any dumps currently on the tape, which you probably don't want
to do at least until enough time has passed that those dumps have all
become obsolete by later runs.  

(amtapetype does have an "-l" option so you could tell it to re-create
the label that is already in the header file on the tape, but off hand I
am not sure if using that will cause Amanda to properly delete the
previous run found on that tape from the index, etc)

(Denise mentioned the "amcheckdump" utility.  This reads through an
existing Amanda tape and verifies all the dumps found on it can be
successfully read back -- something that is a good idea to do if a tape
seems to be failing just in case it is failing so badly that the drive's
normal rewriting-data-until-the-test-read-succeeds doesn't actually
result in a tape that can be read back later -- but a different task
than attempting to write to the tape to see how much data it can
currently store.)

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: bad tape?

[Gene Heskett]

On Friday 16 August 2019 13:38:45 ghe wrote:

> I have reason to believe that one of my tapes isn't working properly
> (last night's backup died without finishing because of a tape error --
> the retry running right now successfully flushed last night's and
> wrote a new one).
>
> Is there an Amanda utility that will validate a tape without
> destroying the Amanda header at the front of the tape? And maybe build
> a 'bad sector table' on the tape?

I did that once for a very old hard drive, permanently allocating about 
30 sectors to a file named badsectors.fd.  Worked great.

But AFAIK, tapes don't maintain an allocation map, and we have no tape 
writing tools so organized as to be able to implement such a scheme.

Because tape is sequential, any such attempt would have to maintain such 
a mapping on separate random access media.

The drive would have to be able to seek past the damaged tape to the next 
block with a good checksum, something I've not found a drive capable of 
doing in around a decade of trying, they all stop and error on the bad 
spot.

The error correction in the average terabyte class spinning rust drive 
has far exceeded that of tape, yes I'm talking about the sub $100 drive 
here, and I haven't lost a single bit of data in the 15+ years since I 
converted to vtape. by having amanda write to virtual tapes which are 
nothing but directories you can read without using anything in the 
amanda ammo box at all if push comes to shove while doing a bare metal 
recovery. In fact, because the amanda index's are only valid after the 
run has been completed, I have written a wrapper script that collects 
that, and the configuration files, and writes then all into the 
directory on the hd that it just used as a virtual tape. That has the 
effect of, in case of bare metal on a clean drive, bringing the recovery 
to a day fresher status.

I have a machine that normally is part of the backup, with a lightning  
damaged psu, so that machine is missing from last nights backup. "data" 
is a softlink to the last runs vtape:
gene@coyote:~$ sudo ls -l /amandatapes/Dailys/data/
total 26361140
-rw--- 1 amanda amanda   32768 Aug 26 02:03 0.Dailys-35
-rw--- 1 amanda amanda  133105 Aug 26 02:03 
1.lathe._var_lib_amanda.1
-rw--- 1 amanda amanda  120881 Aug 26 02:03 
2.shop._var_lib_amanda.1
-rw--- 1 amanda amanda  103947 Aug 26 02:04 
3.lathe._usr_src.1
-rw--- 1 amanda amanda   99164 Aug 26 02:04 4.shop._home.1
-rw--- 1 amanda amanda   53242 Aug 26 02:04 5.lathe._etc.1
-rw--- 1 amanda amanda   53365 Aug 26 02:04 6.shop._etc.1
-rw--- 1 amanda amanda13327331 Aug 26 02:04 7.picnc._.1
-rw--- 1 amanda amanda   49184 Aug 26 02:04 8.lathe._home.1
-rw--- 1 amanda amanda   36715 Aug 26 02:04 
9.shop._lib_firmware.1
-rw--- 1 amanda amanda   34105 Aug 26 02:04 00010.picnc._boot.1
-rw--- 1 amanda amanda   36713 Aug 26 02:05 
00011.lathe._lib_firmware.1
-rw--- 1 amanda amanda   35314 Aug 26 02:05 00012.shop._root.1
-rw--- 1 amanda amanda   35244 Aug 26 02:05 00013.lathe._root.1
-rw--- 1 amanda amanda   33712 Aug 26 02:05 
00014.shop._usr_local.0
-rw--- 1 amanda amanda   33713 Aug 26 02:05 
00015.lathe._usr_local.0
-rw--- 1 amanda amanda   32938 Aug 26 02:05 
00016.shop._var_amanda.0
-rw--- 1 amanda amanda   32939 Aug 26 02:05 
00017.lathe._var_amanda.0
-rw--- 1 amanda amanda   33043 Aug 26 02:05 
00018.shop._usr_lib_amanda.1
-rw--- 1 amanda amanda   33044 Aug 26 02:06 
00019.lathe._usr_lib_amanda.1
-rw--- 1 amanda amanda 12046283958 Aug 26 02:17 
00020.coyote._home_gene_Download.0
-rw--- 1 amanda amanda   187740857 Aug 26 02:18 
00021.coyote._home_gene.3
-rw--- 1 amanda amanda10867557 Aug 26 02:18 
00022.coyote._home_gene_Mail.3
-rw--- 1 amanda amanda   33569 Aug 26 02:18 
00023.coyote.Downloadsnz.1
-rw--- 1 amanda amanda  840726 Aug 26 02:18 
00024.coyote._home_amanda.1
-rw--- 1 amanda amanda  252550 Aug 26 02:18 
00025.coyote._home_gene_src.1
-rw--- 1 amanda amanda  159619 Aug 26 02:18 
00026.coyote.Downloadsam.1
-rw--- 1 amanda amanda   37341 Aug 26 02:18 
00027.coyote._home_ups.1
-rw--- 1 amanda amanda   35159 Aug 26 02:18 
00028.coyote._home_nut.1
-rw--- 1 amanda amanda   638043961 Aug 26 02:20 
00029.coyote._GenesAmandaHelper-0.61_config-bak.3
-rw--- 1 amanda amanda   33308 Aug 26 02:20 
00030.coyote._GenesAmandaHelper-0.61.1
-rw--- 1 amanda amanda   348438528 Aug 26 02:20 00031.coyote._var.4
-rw--- 1 amanda amanda 13234614709 Aug 26 02:48 
00032.coyote.PublicAap.0
-rw--- 1 amanda amanda16479741 Aug 26 02:48 
00033.coyote._usr_local.1
-rw--- 1 amanda amanda11653827 Aug 26 02:48 00034.coyote._opt.1
-rw--- 1 amanda amanda  841528 Aug 26 02:48 
00035.coyote._usr_share.1
-rw--- 1 a

Re: bad tape?

[Debra S Baddorf]

Try  “man  amcheckdump”   It might give you what you want.

Deb Baddorf
Fermilab

> On Aug 16, 2019, at 12:38 PM, ghe  wrote:
> 
> I have reason to believe that one of my tapes isn't working properly
> (last night's backup died without finishing because of a tape error --
> the retry running right now successfully flushed last night's and wrote
> a new one).
> 
> Is there an Amanda utility that will validate a tape without destroying
> the Amanda header at the front of the tape? And maybe build a 'bad
> sector table' on the tape?
> 
> -- 
> Glenn English

bad tape?

[ghe]

I have reason to believe that one of my tapes isn't working properly
(last night's backup died without finishing because of a tape error --
the retry running right now successfully flushed last night's and wrote
a new one).

Is there an Amanda utility that will validate a tape without destroying
the Amanda header at the front of the tape? And maybe build a 'bad
sector table' on the tape?

-- 
Glenn English

Re: eject tape after dump

[Chris Hoogendyk]

Check `man amanda.conf`. There is an option "eject-volume" which defaults to 
no, but can be set to yes.


On 7/28/19 6:15 AM, Stefan G. Weichinger wrote:

What's the current howto here?

I don't see a matching PROPERTY for the device "Tape device" here

https://wiki.zmanda.com/man/amanda-devices.7.html

Customer wishes to have all tapes ejected in the morning.

OK, I could run a cron job, but what if amdump isn't done yet?

something like

amdump config && mt -f /dev/nst0 offl

?




--
---

Chris Hoogendyk

-
   O__   Systems Administrator
  c/ /'_ --- Biology & Geosciences Departments
 (*) \(*) -- 315 Morrill Science Center
~~ - University of Massachusetts, Amherst



---

Erdös 4

Re: eject tape after dump

[ghe]

On 7/28/19 4:15 AM, Stefan G. Weichinger wrote:

> Customer wishes to have all tapes ejected in the morning.

I've written a shell script that, among other things, runs amdump then
mt eject (one tape). It's done what you're looking to do for a couple
decades.

-- 
Glenn English

eject tape after dump

[Stefan G. Weichinger]

What's the current howto here?

I don't see a matching PROPERTY for the device "Tape device" here

https://wiki.zmanda.com/man/amanda-devices.7.html

Customer wishes to have all tapes ejected in the morning.

OK, I could run a cron job, but what if amdump isn't done yet?

something like

amdump config && mt -f /dev/nst0 offl

?

Re: Caveat on the Tape Extraction Command

[Gene Heskett]

On Wednesday 29 May 2019 06:37:49 pm Charles Curley wrote:

> On Wed, 29 May 2019 17:49:56 -0400
>
> Gene Heskett  wrote:
> > On Wednesday 29 May 2019 05:00:22 pm Charles Curley wrote:
> > > Gene's query on the recipe for extracting in the tape headers got
> > > me curious. The few I looked at were complete backups. However, I
> > > also split my backups into chunks. Those did not have a suitable
> > > command line, and I'm not sure you could automate one for
> > > inclusion in the header.
> >
> > The difference would seem to be that while I chunk it on the holding
> > disk, is merged on it way to the backup media, so I only get one
> > file per dle on the backup media.
> >
> > But I'm not at all sure how it would be told to back each 2G hunk as
> > an individual file as it been a decade or more since I set it up to
> > do that in order to not confuse a 32 bit file system, back when my
> > hair more closely resembled the pix of me on my web page  Now its
> > whiter and thinner :(
>
> If it works, don't fix it!

Absolutely. kmail OTOH has crashed 4x today. Last time was clicking on 
answer list mail for this message.
>
> But for the terminally curious, it's in the definition of your
> tapetype, plus setting "allow-split" in your disktype.
>
> define tapetype HARD-DISK {
>   comment "Dump onto hard disk"
>   length 15360 mbytes # specified in mbytes to get the exact 
> size of
> 15 GB 2015-02-17
>
> # Version 3.2.x splitting parameters. Set "allow-split yes" in the
> # disktype.
> part-cache-type disk
> part-cache-dir "/crc/back/amanda/part-cache"
> part-size 900 mb
> part-cache-max-size 900 mb
>   }
I do 2GB in the dumps dir, but its all reassembled to the vtape. Now that 
I've done a 64 bit install, I might remove that breakup, but not before 
I find out why I was able to run it as amanda from a cli, but amanda's 
crontab attempt dies in 11 seconds. I've added some more echos to the 
script which should show me whats fubar in the crontab launch,  I hope.  
And I've fixed I hope, why I haven't been getting the emails from 
previous attempts since the crontabs were all copied over from the 
wheezy 32 bit install.

Take care Charles.

Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: Caveat on the Tape Extraction Command

[Charles Curley]

On Wed, 29 May 2019 17:49:56 -0400
Gene Heskett  wrote:

> On Wednesday 29 May 2019 05:00:22 pm Charles Curley wrote:
> 
> > Gene's query on the recipe for extracting in the tape headers got me
> > curious. The few I looked at were complete backups. However, I also
> > split my backups into chunks. Those did not have a suitable command
> > line, and I'm not sure you could automate one for inclusion in the
> > header.
> >  
> The difference would seem to be that while I chunk it on the holding 
> disk, is merged on it way to the backup media, so I only get one file 
> per dle on the backup media.
> 
> But I'm not at all sure how it would be told to back each 2G hunk as
> an individual file as it been a decade or more since I set it up to
> do that in order to not confuse a 32 bit file system, back when my
> hair more closely resembled the pix of me on my web page  Now its
> whiter and thinner :(

If it works, don't fix it!

But for the terminally curious, it's in the definition of your
tapetype, plus setting "allow-split" in your disktype.

define tapetype HARD-DISK {
comment "Dump onto hard disk"
length 15360 mbytes # specified in mbytes to get the exact 
size of 15 GB 2015-02-17

# Version 3.2.x splitting parameters. Set "allow-split yes" in the
# disktype.
part-cache-type disk
part-cache-dir "/crc/back/amanda/part-cache"
part-size 900 mb
part-cache-max-size 900 mb
}



-- 
"When we talk of civilization, we are too apt to limit the meaning of
the word to its mere embellishments, such as arts and sciences; but
the true distinction between it and barbarism is, that the one
presents a state of society under the protection of just and
well-administered law, and the other is left to the chance government
of brute force."
- The Rev. James White, Eighteen Christian Centuries, 1889
Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
https://charlescurley.com

Re: Caveat on the Tape Extraction Command

[Gene Heskett]

On Wednesday 29 May 2019 05:00:22 pm Charles Curley wrote:

> Gene's query on the recipe for extracting in the tape headers got me
> curious. The few I looked at were complete backups. However, I also
> split my backups into chunks. Those did not have a suitable command
> line, and I'm not sure you could automate one for inclusion in the
> header.
>
The difference would seem to be that while I chunk it on the holding 
disk, is merged on it way to the backup media, so I only get one file 
per dle on the backup media.

But I'm not at all sure how it would be told to back each 2G hunk as an 
individual file as it been a decade or more since I set it up to do that 
in order to not confuse a 32 bit file system, back when my hair more 
closely resembled the pix of me on my web page  Now its whiter and 
thinner :(

> The algorithm would be:
>
> * extract from the tape and decompress the chunks.
>
> * cat the chunks into one monster file.
>
> * run tar on the monster file.
>
> That would take a lot of disk space, fortunately cheap these days.
> Still, I wonder if you could cut that down with an appropriate shell
> script loop.
>
> Is this documented? A quick search on amanda.org did not turn it up.



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Caveat on the Tape Extraction Command

[Charles Curley]

Gene's query on the recipe for extracting in the tape headers got me
curious. The few I looked at were complete backups. However, I also
split my backups into chunks. Those did not have a suitable command
line, and I'm not sure you could automate one for inclusion in the
header.

The algorithm would be:

* extract from the tape and decompress the chunks.

* cat the chunks into one monster file.

* run tar on the monster file.

That would take a lot of disk space, fortunately cheap these days.
Still, I wonder if you could cut that down with an appropriate shell
script loop.

Is this documented? A quick search on amanda.org did not turn it up.

-- 
"When we talk of civilization, we are too apt to limit the meaning of
the word to its mere embellishments, such as arts and sciences; but
the true distinction between it and barbarism is, that the one
presents a state of society under the protection of just and
well-administered law, and the other is left to the chance government
of brute force."
- The Rev. James White, Eighteen Christian Centuries, 1889
Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
https://charlescurley.com

Re: amanda tape header stripped, why?

[Gene Heskett]

On Tuesday 28 May 2019 11:09:43 pm Nathan Stratton Treadway wrote:

> On Tue, May 28, 2019 at 21:19:55 -0400, Gene Heskett wrote:
> > Mine, when running 3.3.7p1 on wheezy, had this good header in every
> > vtape "header"
>
> Note that the "0.*" file is simply the table label written by
> amlabel -- there is no "data" in it, the only point of the file is to
> contain the volume label.  You'll see that they are always exactly
> 32kiB (32768 bytes), and they contain just the the label info 
> followed by NUL bytes padding out to 32kiB...
>
> =
> root@tumhalad:~/rushey_etc_git# hd
> /vtapes/TestBackup/slot1/0.TESTBACKUP-01   41 4d 41 4e 44
> 41 3a 20  54 41 50 45 53 54 41 52  |AMANDA: TAPESTAR| 0010  54 20
> 44 41 54 45 20 32  30 31 38 31 31 30 38 30  |T DATE 201811080|
> 0020  37 30 31 30 33 20 54 41  50 45 20 54 45 53 54 42  |70103
> TAPE TESTB| 0030  41 43 4b 55 50 2d 30 31  0a 0c 0a 00 00 00 00 00
>  |ACKUP-01| 0040  00 00 00 00 00 00 00 00  00 00 00 00 00
> 00 00 00  || *
> 8000
> =
>
> Thus, the usual start-of-the-recovery-command-pipeline command, "dd
> if= bs=32k skip=1 |", would skip over the entire file and have
> nothing to pass to the later commands in the pipeline.
>
> (The vtape files after "0", on the other hand, all do have dump
> data following a 32kiB header block)
>
>   Nathan
>

The more I think on it, the righter you all are. So lets drop this based 
on me miss-remembering that it was the file header, not the tape header.
>
>
> --
>-- Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic
> region Ray Ontko & Co.  -  Software consulting services  -  
> http://www.ontko.com/ GPG Key:
> http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239 Key
> fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: amanda tape header stripped, why?

[Nathan Stratton Treadway]

On Tue, May 28, 2019 at 21:19:55 -0400, Gene Heskett wrote:
> Mine, when running 3.3.7p1 on wheezy, had this good header in every 
> vtape "header"

Note that the "0.*" file is simply the table label written by
amlabel -- there is no "data" in it, the only point of the file is to
contain the volume label.  You'll see that they are always exactly 32kiB
(32768 bytes), and they contain just the the label info  followed by NUL
bytes padding out to 32kiB...

=
root@tumhalad:~/rushey_etc_git# hd /vtapes/TestBackup/slot1/0.TESTBACKUP-01 
  41 4d 41 4e 44 41 3a 20  54 41 50 45 53 54 41 52  |AMANDA: TAPESTAR|
0010  54 20 44 41 54 45 20 32  30 31 38 31 31 30 38 30  |T DATE 201811080|
0020  37 30 31 30 33 20 54 41  50 45 20 54 45 53 54 42  |70103 TAPE TESTB|
0030  41 43 4b 55 50 2d 30 31  0a 0c 0a 00 00 00 00 00  |ACKUP-01|
0040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ||
*
8000
=

Thus, the usual start-of-the-recovery-command-pipeline command, "dd
if= bs=32k skip=1 |", would skip over the entire file and have
nothing to pass to the later commands in the pipeline.

(The vtape files after "0", on the other hand, all do have dump data
following a 32kiB header block)

Nathan




Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: amanda tape header stripped, why?

[Gene Heskett]

On Tuesday 28 May 2019 12:49:05 pm Charles Curley wrote:

> On Tue, 28 May 2019 11:40:33 -0400
>
> Gene Heskett  wrote:
> > In my build on stretch the tape header, with normally contains
> > instructions for a gzip/tar only system, the recipe for bare
> > recovery has disappeared.  It now looks like this:
> > AMANDA: TAPESTART DATE 20190527162707 TAPE Dailys-1
> >
> > There is supposed to be a 2nd line, showing how to unpack the
> > archive with nothing but gzip and tar,
> >
> >
> > Cheers, Gene Heskett
>
> I can confirm that for the first file per VTAPE on my build:
>
> root@amanda2:/var/amanda/vtapes/slot1# od -N 96 -A x -t x1z -v
> 0.DailySet1-001 00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54
> 41 52  >AMANDA: TAPESTAR< 10 54 20 44 41 54 45 20 32 30 31 39 30
> 35 32 37 31  >T DATE 201905271< 20 33 30 38 34 35 20 54 41 50 45
> 20 44 61 69 6c 79  >30845 TAPE Daily< 30 53 65 74 31 2d 30 30 31
> 0a 0c 0a 00 00 00 00 00  >Set1-001< 40 00 00 00 00 00 00
> 00 00 00 00 00 00 00 00 00 00  >< 50 00 00 00 00
> 00 00 00 00 00 00 00 00 00 00 00 00  >< 60
> root@amanda2:/var/amanda/vtapes/slot1#
>
> However it also seems to be the case for Amanda 3.3.9-5, the current
> stable version on debian:
>
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 96 -A x -t x1z
> -v 0.DailySet1_01 00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54
> 41 52  >AMANDA: TAPESTAR< 10 54 20 44 41 54 45 20 32 30 31 39 30
> 34 31 32 30  >T DATE 201904120< 20 30 30 31 30 31 20 54 41 50 45
> 20 44 61 69 6c 79  >00101 TAPE Daily< 30 53 65 74 31 5f 30 31 0a
> 0c 0a 00 00 00 00 00 00  >Set1_01.< 40 00 00 00 00 00 00
> 00 00 00 00 00 00 00 00 00 00  >< 50 00 00 00 00
> 00 00 00 00 00 00 00 00 00 00 00 00  >< 60
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1#
>
> But if you look at the next one, the first real backup file,
> starting at about 0x190 or so...
>
Mine, when running 3.3.7p1 on wheezy, had this good header in every 
vtape "header"

> root@amanda2:/var/amanda/vtapes/slot1# od -N 544 -A x -t x1z -v
> 1.localhost._root.0 00 41 4d 41 4e 44 41 3a 20 53 50 4c 49 54
> 5f 46 49  >AMANDA: SPLIT_FI< 10 4c 45 20 32 30 31 39 30 35 32 37
> 31 33 30 38 34  >LE 2019052713084< 20 35 20 6c 6f 63 61 6c 68 6f
> 73 74 20 2f 72 6f 6f  >5 localhost /roo< 30 74 20 20 70 61 72 74
> 20 31 2f 2d 31 20 20 6c 65  >t  part 1/-1  le< 40 76 20 30 20 63
> 6f 6d 70 20 2e 67 7a 20 70 72 6f  >v 0 comp .gz pro< 50 67 72 61
> 6d 20 2f 62 69 6e 2f 74 61 72 0a 4f 52  >gram /bin/tar.OR< 60 49
> 47 53 49 5a 45 3d 34 30 0a 4e 41 54 49 56 45  >IGSIZE=40.NATIVE<
> 70 2d 43 52 43 3d 32 61 61 38 62 32 65 35 3a 34 30 
> >-CRC=2aa8b2e5:40< 80 39 36 30 0a 43 4c 49 45 4e 54 2d 43 52 43 3d
> 37  >960.CLIENT-CRC=7< 90 66 36 30 33 61 32 66 3a 39 35 33 31 0a
> 53 45 52  >f603a2f:9531.SER< a0 56 45 52 2d 43 52 43 3d 37 66 36
> 30 33 61 32 66  >VER-CRC=7f603a2f< b0 3a 39 35 33 31 0a 44 4c 45
> 3d 3c 3c 45 4e 44 44  >:9531.DLE=< 3e 0a 20 20 3c 70 72 6f 67  >LE..   4e 55 54 41 52 3c 2f 70 72 6f 67  >ram>GNUTAR 3e 0a 20 20 3c 64 69 73 6b 3e 2f 72 6f  >ram>.  /ro< f0 6f
> 74 3c 2f 64 69 73 6b 3e 0a 20 20 3c 6c 65 76  >ot.   000100 65 6c 3e 30 3c 2f 6c 65 76 65 6c 3e 0a 20 20 3c  >el>0.
>  << 000110 61 75 74 68 3e 42 53 44 54 43 50 3c 2f 61 75 74 
> >auth>BSDTCP 46  >h>.  F< 000130 41 53 54 3c 2f 63 6f 6d 70 72 65 73 73
> 3e 0a 20  >AST. < 000140 20 3c 72 65 63 6f 72 64 3e 59 45
> 53 3c 2f 72 65  > YES 69 6e 64 65 78 3e 59  >cord>.  Y< 000160 45 53 3c 2f 69 6e 64
> 65 78 3e 0a 20 20 3c 64 61  >ES.   68 3e 41 4d 41 4e 44 41 3c 2f 64  >tapath>AMANDA 70 61 74 68 3e 0a 3c 2f 64 6c 65 3e 0a  >atapath>..< 000190 45
> 4e 44 44 4c 45 0a 54 6f 20 72 65 73 74 6f 72  >ENDDLE.To restor<
> 0001a0 65 2c 20 70 6f 73 69 74 69 6f 6e 20 74 61 70 65  >e, position
> tape< 0001b0 20 61 74 20 73 74 61 72 74 20 6f 66 20 66 69 6c  > at
> start of fil< 0001c0 65 20 61 6e 64 20 72 75 6e 3a 0a 09 64 64 20 69 
> >e and run:..dd i< 0001d0 66 3d 3c 74 61 70 65 3e 20 62 73 3d 33 32 6b
> 20  >f= bs=32k < 0001e0 73 6b 69 70 3d 31 20 7c 20 2f 62 69 6e
> 2f 67 7a  >skip=1 | /bin/gz< 0001f0 69 70 20 2d 64 63 20 7c 20 2f 62
> 69 6e 2f 74 61  >ip -dc | /bin/ta< 000200 72 20 2d 78 70 47 66 20 2d
> 20 2e 2e 2e 20 0a 0c  >r -xpGf - ... ..< 000210 0a 00 00 00 00 00 00
> 00 00 00 00 00 00 00 00 00  >...

Re: amanda tape header stripped, why?

[Debra S Baddorf]

Agreed - the short form is:  the tape header needs no instructions,
and each DLE file might be done with a different mechanism  (true for me)  so
each DLE backup has the instructions in the header.

Deb Baddorf
Fermilab



> On May 28, 2019, at 11:49 AM, Charles Curley 
>  wrote:
> 
> On Tue, 28 May 2019 11:40:33 -0400
> Gene Heskett  wrote:
> 
>> In my build on stretch the tape header, with normally contains 
>> instructions for a gzip/tar only system, the recipe for bare recovery 
>> has disappeared.  It now looks like this:
>> AMANDA: TAPESTART DATE 20190527162707 TAPE Dailys-1
>> 
>> There is supposed to be a 2nd line, showing how to unpack the archive 
>> with nothing but gzip and tar,
>> 
>> 
>> Cheers, Gene Heskett
> 
> I can confirm that for the first file per VTAPE on my build:
> 
> root@amanda2:/var/amanda/vtapes/slot1# od -N 96 -A x -t x1z -v 
> 0.DailySet1-001 
> 00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54 41 52  >AMANDA: TAPESTAR<
> 10 54 20 44 41 54 45 20 32 30 31 39 30 35 32 37 31  >T DATE 201905271<
> 20 33 30 38 34 35 20 54 41 50 45 20 44 61 69 6c 79  >30845 TAPE Daily<
> 30 53 65 74 31 2d 30 30 31 0a 0c 0a 00 00 00 00 00  >Set1-001<
> 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
> 50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
> 60
> root@amanda2:/var/amanda/vtapes/slot1# 
> 
> However it also seems to be the case for Amanda 3.3.9-5, the current
> stable version on debian:
> 
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 96 -A x -t x1z -v 
> 0.DailySet1_01 
> 00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54 41 52  >AMANDA: TAPESTAR<
> 10 54 20 44 41 54 45 20 32 30 31 39 30 34 31 32 30  >T DATE 201904120<
> 20 30 30 31 30 31 20 54 41 50 45 20 44 61 69 6c 79  >00101 TAPE Daily<
> 30 53 65 74 31 5f 30 31 0a 0c 0a 00 00 00 00 00 00  >Set1_01.<
> 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
> 50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
> 60
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# 
> 
> But if you look at the next one, the first real backup file,
> starting at about 0x190 or so...
> 
> root@amanda2:/var/amanda/vtapes/slot1# od -N 544 -A x -t x1z -v 
> 1.localhost._root.0 
> 00 41 4d 41 4e 44 41 3a 20 53 50 4c 49 54 5f 46 49  >AMANDA: SPLIT_FI<
> 10 4c 45 20 32 30 31 39 30 35 32 37 31 33 30 38 34  >LE 2019052713084<
> 20 35 20 6c 6f 63 61 6c 68 6f 73 74 20 2f 72 6f 6f  >5 localhost /roo<
> 30 74 20 20 70 61 72 74 20 31 2f 2d 31 20 20 6c 65  >t  part 1/-1  le<
> 40 76 20 30 20 63 6f 6d 70 20 2e 67 7a 20 70 72 6f  >v 0 comp .gz pro<
> 50 67 72 61 6d 20 2f 62 69 6e 2f 74 61 72 0a 4f 52  >gram /bin/tar.OR<
> 60 49 47 53 49 5a 45 3d 34 30 0a 4e 41 54 49 56 45  >IGSIZE=40.NATIVE<
> 70 2d 43 52 43 3d 32 61 61 38 62 32 65 35 3a 34 30  >-CRC=2aa8b2e5:40<
> 80 39 36 30 0a 43 4c 49 45 4e 54 2d 43 52 43 3d 37  >960.CLIENT-CRC=7<
> 90 66 36 30 33 61 32 66 3a 39 35 33 31 0a 53 45 52  >f603a2f:9531.SER<
> a0 56 45 52 2d 43 52 43 3d 37 66 36 30 33 61 32 66  >VER-CRC=7f603a2f<
> b0 3a 39 35 33 31 0a 44 4c 45 3d 3c 3c 45 4e 44 44  >:9531.DLE=< c0 4c 45 0a 3c 64 6c 65 3e 0a 20 20 3c 70 72 6f 67  >LE..   d0 72 61 6d 3e 47 4e 55 54 41 52 3c 2f 70 72 6f 67  >ram>GNUTAR e0 72 61 6d 3e 0a 20 20 3c 64 69 73 6b 3e 2f 72 6f  >ram>.  /ro<
> f0 6f 74 3c 2f 64 69 73 6b 3e 0a 20 20 3c 6c 65 76  >ot.   000100 65 6c 3e 30 3c 2f 6c 65 76 65 6c 3e 0a 20 20 3c  >el>0.  <<
> 000110 61 75 74 68 3e 42 53 44 54 43 50 3c 2f 61 75 74  >auth>BSDTCP 000120 68 3e 0a 20 20 3c 63 6f 6d 70 72 65 73 73 3e 46  >h>.  F<
> 000130 41 53 54 3c 2f 63 6f 6d 70 72 65 73 73 3e 0a 20  >AST. <
> 000140 20 3c 72 65 63 6f 72 64 3e 59 45 53 3c 2f 72 65  > YES 000150 63 6f 72 64 3e 0a 20 20 3c 69 6e 64 65 78 3e 59  >cord>.  Y<
> 000160 45 53 3c 2f 69 6e 64 65 78 3e 0a 20 20 3c 64 61  >ES.   000170 74 61 70 61 74 68 3e 41 4d 41 4e 44 41 3c 2f 64  >tapath>AMANDA 000180 61 74 61 70 61 74 68 3e 0a 3c 2f 64 6c 65 3e 0a  >atapath>..<
> 000190 45 4e 44 44 4c 45 0a 54 6f 20 72 65 73 74 6f 72  >ENDDLE.To restor<
> 0001a0 65 2c 20 70 6f 73 69 74 69 6f 6e 20 74 61 70 65  >e, position tape<
> 0001b0 20 61 74 20 73 74 61 72 74 20 6f 66 20 66 69 6c  > at start of fil<
> 0001c0 65 20 61 6e 64 20 72 75 6e 3a 0a 09 64 64 20 69  >e and run:..dd i<
> 0001d0 66 3d 3c 74 61 70 65 3e 20 62 73 3d 33 32 6b 20  >f= bs=32k &l

Re: amanda tape header stripped, why?

[Charles Curley]

On Tue, 28 May 2019 11:40:33 -0400
Gene Heskett  wrote:

> In my build on stretch the tape header, with normally contains 
> instructions for a gzip/tar only system, the recipe for bare recovery 
> has disappeared.  It now looks like this:
> AMANDA: TAPESTART DATE 20190527162707 TAPE Dailys-1
> 
> There is supposed to be a 2nd line, showing how to unpack the archive 
> with nothing but gzip and tar,
> 
> 
> Cheers, Gene Heskett

I can confirm that for the first file per VTAPE on my build:

root@amanda2:/var/amanda/vtapes/slot1# od -N 96 -A x -t x1z -v 
0.DailySet1-001 
00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54 41 52  >AMANDA: TAPESTAR<
10 54 20 44 41 54 45 20 32 30 31 39 30 35 32 37 31  >T DATE 201905271<
20 33 30 38 34 35 20 54 41 50 45 20 44 61 69 6c 79  >30845 TAPE Daily<
30 53 65 74 31 2d 30 30 31 0a 0c 0a 00 00 00 00 00  >Set1-001<
40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
60
root@amanda2:/var/amanda/vtapes/slot1# 

However it also seems to be the case for Amanda 3.3.9-5, the current
stable version on debian:

root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 96 -A x -t x1z -v 
0.DailySet1_01 
00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54 41 52  >AMANDA: TAPESTAR<
10 54 20 44 41 54 45 20 32 30 31 39 30 34 31 32 30  >T DATE 201904120<
20 30 30 31 30 31 20 54 41 50 45 20 44 61 69 6c 79  >00101 TAPE Daily<
30 53 65 74 31 5f 30 31 0a 0c 0a 00 00 00 00 00 00  >Set1_01.<
40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
60
root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# 

But if you look at the next one, the first real backup file,
starting at about 0x190 or so...

root@amanda2:/var/amanda/vtapes/slot1# od -N 544 -A x -t x1z -v 
1.localhost._root.0 
00 41 4d 41 4e 44 41 3a 20 53 50 4c 49 54 5f 46 49  >AMANDA: SPLIT_FI<
10 4c 45 20 32 30 31 39 30 35 32 37 31 33 30 38 34  >LE 2019052713084<
20 35 20 6c 6f 63 61 6c 68 6f 73 74 20 2f 72 6f 6f  >5 localhost /roo<
30 74 20 20 70 61 72 74 20 31 2f 2d 31 20 20 6c 65  >t  part 1/-1  le<
40 76 20 30 20 63 6f 6d 70 20 2e 67 7a 20 70 72 6f  >v 0 comp .gz pro<
50 67 72 61 6d 20 2f 62 69 6e 2f 74 61 72 0a 4f 52  >gram /bin/tar.OR<
60 49 47 53 49 5a 45 3d 34 30 0a 4e 41 54 49 56 45  >IGSIZE=40.NATIVE<
70 2d 43 52 43 3d 32 61 61 38 62 32 65 35 3a 34 30  >-CRC=2aa8b2e5:40<
80 39 36 30 0a 43 4c 49 45 4e 54 2d 43 52 43 3d 37  >960.CLIENT-CRC=7<
90 66 36 30 33 61 32 66 3a 39 35 33 31 0a 53 45 52  >f603a2f:9531.SER<
a0 56 45 52 2d 43 52 43 3d 37 66 36 30 33 61 32 66  >VER-CRC=7f603a2f<
b0 3a 39 35 33 31 0a 44 4c 45 3d 3c 3c 45 4e 44 44  >:9531.DLE=<LE..  ram>GNUTARram>.  /ro<
f0 6f 74 3c 2f 64 69 73 6b 3e 0a 20 20 3c 6c 65 76  >ot.  el>0.  <<
000110 61 75 74 68 3e 42 53 44 54 43 50 3c 2f 61 75 74  >auth>BSDTCPh>.  F<
000130 41 53 54 3c 2f 63 6f 6d 70 72 65 73 73 3e 0a 20  >AST. <
000140 20 3c 72 65 63 6f 72 64 3e 59 45 53 3c 2f 72 65  > YEScord>.  Y<
000160 45 53 3c 2f 69 6e 64 65 78 3e 0a 20 20 3c 64 61  >ES.  tapath>AMANDAatapath>..<
000190 45 4e 44 44 4c 45 0a 54 6f 20 72 65 73 74 6f 72  >ENDDLE.To restor<
0001a0 65 2c 20 70 6f 73 69 74 69 6f 6e 20 74 61 70 65  >e, position tape<
0001b0 20 61 74 20 73 74 61 72 74 20 6f 66 20 66 69 6c  > at start of fil<
0001c0 65 20 61 6e 64 20 72 75 6e 3a 0a 09 64 64 20 69  >e and run:..dd i<
0001d0 66 3d 3c 74 61 70 65 3e 20 62 73 3d 33 32 6b 20  >f= bs=32k <
0001e0 73 6b 69 70 3d 31 20 7c 20 2f 62 69 6e 2f 67 7a  >skip=1 | /bin/gz<
0001f0 69 70 20 2d 64 63 20 7c 20 2f 62 69 6e 2f 74 61  >ip -dc | /bin/ta<
000200 72 20 2d 78 70 47 66 20 2d 20 2e 2e 2e 20 0a 0c  >r -xpGf - ... ..<
000210 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
000220
root@amanda2:/var/amanda/vtapes/slot1# 

And similarly for 3.3.9-5:

root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 0x270 -A x -t x1z -v 
1.hawk.localdomain._home_charles_projects.1 
00 41 4d 41 4e 44 41 3a 20 53 50 4c 49 54 5f 46 49  >AMANDA: SPLIT_FI<
10 4c 45 20 32 30 31 39 30 34 31 32 30 30 30 31 30  >LE 2019041200010<
20 31 20 68 61 77 6b 2e 6c 6f 63 61 6c 64 6f 6d 61  >1 hawk.localdoma<

...

0001b0 3e 0a 20 20 3c 64 61 74 61 70 61 74 68 3e 41 4d  >>.  AM<
0001c0 41 4e 44 41 3c 2f 64 61 74 61 70 61 74 68 3e 0a  >ANDA.<
0001d0 3c 2f 64 6c 65 3e 0a 45 4e 44 44 4c 45 0a 54 6f  >.ENDDLE.To<
0001e0 20 72 65 73 74 6f 72 65 2c 20 70 6f 73 69 74 69  > restore, positi<
0001f0 6f 6e 20 74 61 70 65 20

amanda tape header stripped, why?

[Gene Heskett]

In my build on stretch the tape header, with normally contains 
instructions for a gzip/tar only system, the recipe for bare recovery 
has disappeared.  It now looks like this:
AMANDA: TAPESTART DATE 20190527162707 TAPE Dailys-1

There is supposed to be a 2nd line, showing how to unpack the archive 
with nothing but gzip and tar,


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: Installing amanda on Debian 9 from git: amrecover tape device issue

[Charles Curley]

On Wed, 15 May 2019 22:18:42 -0400
Nathan Stratton Treadway  wrote:

> On Wed, May 15, 2019 at 19:58:52 -0600, Charles Curley wrote:
> > On Wed, 15 May 2019 20:48:46 -0400
> > Nathan Stratton Treadway  wrote:
> >   
> > > On Wed, May 15, 2019 at 17:02:57 -0600, Charles Curley wrote:  
> > 
> > Here's what I do to customize:
> > 
> > cp -rp /var/lib/amanda/example/amanda.conf ${confDir}
> > 
> > # Now adjust the configuration to suit.
> > cd ${confDir}
> > sed -i -e
> > "s/tape:\/dev\/YOUR-TAPE-DEVICE-HERE/chg-disk:\/var\/amanda\/vtapes/"
> > amanda.conf sed -i -e 's/tapetype HP-DAT/tapetype HARD-DISK/'
> > amanda.conf
> > 
> > And that seems to works. It does produce one spurious commented out
> > change, on line 118.  
> 
> When you say "that seems to work", do you mean it fixes amrecover, or
> just that the sed commands do edit the lines in the amanda.conf file?

Sorry. I mean that the sed commands do edit the lines.

> 
> As far as I understand, amrecover looks at amanda-client.conf (and not
> amanda.conf), so to get amrecover to work without the "-d" command
> line option you'd need to do do some editing of the -client file.

Bingo.

> 
> (But given that the official Debian packages work the way you want
> without any -client file at all, I'm thinking that it makes sense to
> simply comment out the tapdev line in the -client file, rather than
> edit it to make it match the amanda.conf file [though as long as the
> two lines are in sync the net effect is probably the same].)

Indeed, that is the case on all the hosts in my existing setup. The
tapedev line is commented out. That works on the testbed as
well. Methinks that's a better solution, as it would *usually* require
that changes be made only in one place.



-- 
"When we talk of civilization, we are too apt to limit the meaning of
the word to its mere embellishments, such as arts and sciences; but
the true distinction between it and barbarism is, that the one
presents a state of society under the protection of just and
well-administered law, and the other is left to the chance government
of brute force."
- The Rev. James White, Eighteen Christian Centuries, 1889
Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
https://charlescurley.com

Re: Installing amanda on Debian 9 from git: amrecover tape device issue

[Nathan Stratton Treadway]

On Wed, May 15, 2019 at 19:58:52 -0600, Charles Curley wrote:
> On Wed, 15 May 2019 20:48:46 -0400
> Nathan Stratton Treadway  wrote:
> 
> > On Wed, May 15, 2019 at 17:02:57 -0600, Charles Curley wrote:
> 
> Here's what I do to customize:
> 
> cp -rp /var/lib/amanda/example/amanda.conf ${confDir}
> 
> # Now adjust the configuration to suit.
> cd ${confDir}
> sed -i -e 
> "s/tape:\/dev\/YOUR-TAPE-DEVICE-HERE/chg-disk:\/var\/amanda\/vtapes/" 
> amanda.conf
> sed -i -e 's/tapetype HP-DAT/tapetype HARD-DISK/' amanda.conf
> 
> And that seems to works. It does produce one spurious commented out
> change, on line 118.

When you say "that seems to work", do you mean it fixes amrecover, or
just that the sed commands do edit the lines in the amanda.conf file?

As far as I understand, amrecover looks at amanda-client.conf (and not
amanda.conf), so to get amrecover to work without the "-d" command line
option you'd need to do do some editing of the -client file.  

(But given that the official Debian packages work the way you want
without any -client file at all, I'm thinking that it makes sense to
simply comment out the tapdev line in the -client file, rather than edit
it to make it match the amanda.conf file [though as long as the two
lines are in sync the net effect is probably the same].)


Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: Installing amanda on Debian 9 from git: amrecover tape device issue

[Charles Curley]

On Wed, 15 May 2019 20:48:46 -0400
Nathan Stratton Treadway  wrote:

> On Wed, May 15, 2019 at 17:02:57 -0600, Charles Curley wrote:
> > Using the official debian packages, I am accustomed to CDing to the
> > desired directory, and entering "amrecover DailySet1". I go through
> > the process with no issue as to which tape device to use and get a
> > good extraction.
> > 
> > Using the git package, I have to specify the tape device like so:
> > 
> > amrecover -d chg-disk:/var/amanda/vtapes -C DailySet1
> > 
> > If I don't specify the device, I get:
> > 
> > root@amanda:~/tmp/amanda/home# amrecover   
> [...] 
> > Extracting files using tape drive tape:/dev/YOUR-TAPE-DEVICE-HERE
> > on host localhost. The following tapes are needed: DailySet1-001  
> 
> 
> amrecover seems to pull the initial value of the -d setting from
> the "tapedev" setting in the amanda-client.conf file.
> 
> The official Debian Amanda packages do not install a "live"
> version of that file (rather, they install only some copies under
> /usr/share/doc/amanda-*/ )...

These go to /var/lib/amanda/example, but same idea.

> 
> ... but looking at the source repository it seems the Zmanda packages'
> postinst scripts copy the example file
> to /etc/amanda/amanda-client.conf

I didn't even look to see. I copy it in anyway; see below.

> 
> If that's true (and especially if that file contains the line 
>   tapedev "tape:/dev/YOUR-TAPE-DEVICE-HERE"
> ), then I'm guessing that commenting out that line -- or even just
> renaming the file to disable it completely [*] -- will allow amrecover
> to work as expected

Here's what I do to customize:

cp -rp /var/lib/amanda/example/amanda.conf ${confDir}

# Now adjust the configuration to suit.
cd ${confDir}
sed -i -e "s/tape:\/dev\/YOUR-TAPE-DEVICE-HERE/chg-disk:\/var\/amanda\/vtapes/" 
amanda.conf
sed -i -e 's/tapetype HP-DAT/tapetype HARD-DISK/' amanda.conf

And that seems to works. It does produce one spurious commented out
change, on line 118.

--
root@amanda:/etc/amanda/DailySet1# diff amanda.conf 
/var/lib/amanda/example/amanda.conf 
107c107
< tapedev "chg-disk:/var/amanda/vtapes" # tape changer or device to use
---
> tapedev "tape:/dev/YOUR-TAPE-DEVICE-HERE" # tape changer or device to use
118c118
< #   tapedev "tape:chg-disk:/var/amanda/vtapes"# your tape drive 
device file
---
> #   tapedev "tape:tape:/dev/YOUR-TAPE-DEVICE-HERE"# your tape drive 
> device file
130c130
< tapetype HARD-DISK# what kind of tape it is (see tapetypes below)
---
> tapetype HP-DAT   # what kind of tape it is (see tapetypes below)
312c312
< define tapetype HARD-DISK {
---
> define tapetype HP-DAT {
root@amanda:/etc/amanda/DailySet1#
--



> 
> [*] though if you rename it, presumably it would be auto-recreated on
> package upgrade
> 
>   Nathan
> 
> 
> Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic
> region Ray Ontko & Co.  -  Software consulting services  -
> http://www.ontko.com/ GPG Key:
> http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239 Key
> fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239



-- 
"When we talk of civilization, we are too apt to limit the meaning of
the word to its mere embellishments, such as arts and sciences; but
the true distinction between it and barbarism is, that the one
presents a state of society under the protection of just and
well-administered law, and the other is left to the chance government
of brute force."
- The Rev. James White, Eighteen Christian Centuries, 1889
Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
https://charlescurley.com

Re: Installing amanda on Debian 9 from git: amrecover tape device issue

[Nathan Stratton Treadway]

On Wed, May 15, 2019 at 17:02:57 -0600, Charles Curley wrote:
> Using the official debian packages, I am accustomed to CDing to the
> desired directory, and entering "amrecover DailySet1". I go through the
> process with no issue as to which tape device to use and get a good
> extraction.
> 
> Using the git package, I have to specify the tape device like so:
> 
> amrecover -d chg-disk:/var/amanda/vtapes -C DailySet1
> 
> If I don't specify the device, I get:
> 
> root@amanda:~/tmp/amanda/home# amrecover 
[...] 
> Extracting files using tape drive tape:/dev/YOUR-TAPE-DEVICE-HERE on host 
> localhost.
> The following tapes are needed: DailySet1-001


amrecover seems to pull the initial value of the -d setting from
the "tapedev" setting in the amanda-client.conf file.

The official Debian Amanda packages do not install a "live"
version of that file (rather, they install only some copies under
/usr/share/doc/amanda-*/ )...

... but looking at the source repository it seems the Zmanda packages'
postinst scripts copy the example file to /etc/amanda/amanda-client.conf

If that's true (and especially if that file contains the line 
  tapedev "tape:/dev/YOUR-TAPE-DEVICE-HERE"
), then I'm guessing that commenting out that line -- or even just
renaming the file to disable it completely [*] -- will allow amrecover
to work as expected

[*] though if you rename it, presumably it would be auto-recreated on
package upgrade

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Installing amanda on Debian 9 from git: amrecover tape device issue

[Charles Curley]

Using the official debian packages, I am accustomed to CDing to the
desired directory, and entering "amrecover DailySet1". I go through the
process with no issue as to which tape device to use and get a good
extraction.

Using the git package, I have to specify the tape device like so:

amrecover -d chg-disk:/var/amanda/vtapes -C DailySet1

If I don't specify the device, I get:

root@amanda:~/tmp/amanda/home# amrecover 
AMRECOVER Version 3.5.1.git.19364c7b. Contacting server on localhost ...
220 amanda AMANDA index server (3.5.1.git.19364c7b) ready.
Setting restore date to today (2019-05-15)
200 Working date set to 2019-05-15.
200 Config set to DailySet1.
200 Dump host set to amanda.
Use the setdisk command to choose dump disk to recover
amrecover> setdisk /home
200 Disk set to /home.
amrecover> add charles
Added dir /charles/ at date 2019-05-15-14-17-36
amrecover> extract

Extracting files using tape drive tape:/dev/YOUR-TAPE-DEVICE-HERE on host 
localhost.
The following tapes are needed: DailySet1-001

Extracting files using tape drive tape:/dev/YOUR-TAPE-DEVICE-HERE on host 
localhost.
Load tape DailySet1-001 now
Continue [?/Y/n/s/d]? y
Can't open tape device /dev/YOUR-TAPE-DEVICE-HERE: No such file or directory
Load tape DailySet1-001 now
Continue [?/Y/n/d]? n
Got no header and data from server, check in amidxtaped.*.debug and 
amandad.*.debug files on server
amrecover> exit
200 Good bye.
root@amanda:~/tmp/amanda/home#
...


-- 
"When we talk of civilization, we are too apt to limit the meaning of
the word to its mere embellishments, such as arts and sciences; but
the true distinction between it and barbarism is, that the one
presents a state of society under the protection of just and
well-administered law, and the other is left to the chance government
of brute force."
- The Rev. James White, Eighteen Christian Centuries, 1889
Key fingerprint = CE5C 6645 A45A 64E4 94C0  809C FFF6 4C48 4ECD DFDB
https://charlescurley.com

Re: script for loading the tape magazines

[Stefan G. Weichinger]

Am 22.01.19 um 18:50 schrieb Stefan G. Weichinger:
> 
> I might have written about that already some years(?) ago, but anyway:
> 
> I have a small shell-script to ask "amadmin config tape --days X" for
> the next tapes to put into the magazines.
> 
> That script is run via cron and emails the local admin at the customer
> with the requested tapes.
> 
> A bit ugly but it works.

maybe someone is interested and wants to comment or contribute:

https://github.com/stefangweichinger/amanda-helpers/blob/master/amanda_magazines.sh

early work in progress, still mostly hardcoded, doesn't read variables
from the commandline yet.

script for loading the tape magazines

[Stefan G. Weichinger]

I might have written about that already some years(?) ago, but anyway:

I have a small shell-script to ask "amadmin config tape --days X" for
the next tapes to put into the magazines.

That script is run via cron and emails the local admin at the customer
with the requested tapes.

A bit ugly but it works.

I want to improve this and do some comparison in the script:

amtape config inventory

vs.

amadmin config tape

because related to skipped runs or smaller runs (skipping some DLEs for
example) one run uses N tapes and another one uses N-1 tapes.

So the script might already tell me:

slot 3: OK (=inserted tape usable)
...
slot 4: tape007 out - tape013 in

or so.

maybe some of you already have such a script and we could share or learn
from each other?

like, you know, in an open source software project ;-)

Re: Dump to tape AND keep copy on disk?

[Jose M Calhariz]

On Tue, Jan 15, 2019 at 05:33:55PM -0500, Jon LaBadie wrote:
> On Tue, Jan 15, 2019 at 07:38:37PM +, Debra S Baddorf wrote:
> > Amanda people:  does the newer amanda have any means to dump to tape
> > while still retaining a copy on the holding disk?
> > 
> > Dumping to tape, and then vaulting BACK to a disk area seems excessive,
> > if there is a better way?
> > 
> > Deb Baddorf   (asking for Stefan Weichinger)
> > 
> 
> I've never used it, but ISTR amanda had a RAID feature
> where you could tape to multiple devices simultaneously.

The last time I tried to use "RAIT" I was told it was deprecated and
should use vaulting.   I have a limited experiencie with vaulting
where one copy was on vTapes on disk and a second copy was on S3, that
I can share.


> 
> 
> jon


Kind regards
Jose M Calhariz


> > 
> > 
> > > On Jan 15, 2019, at 1:34 PM, Stefan G. Weichinger  wrote:
> > > 
> > > Am 14.01.19 um 19:05 schrieb Debra S Baddorf:
> > > 
> > >> You can let the normal job send the dumps to tape,  and then
> > >> “vault”  a copy back onto a “vtape”  on disk  (not your holding disk,
> > >> but that’s semantics).
> > >> That might suit your need.
> > >> 
> > >> I’ve done vaulting from a tape onto a disk area …..  moved the
> > >> disk area to another node,  and vaulted again to copy it to a
> > >> different flavor of tape drive.   So the very first part of my task
> > >> might suit you to a tee.
> > > 
> > > I understand that this might work but it seems to way more steps than
> > > having a second "storage" to flush the holdingdisk content to. If that
> > > works (and the "storage" doesn't allow to filter the lev0 afaik).
> > > 
> > > I haven't yet found the time and the brain to continue testing that so 
> > > far.
> >>> End of included message <<<
> 

-- 
--
"A perda das vidas será irreversível."

-- Dan Quayle, ex-vice-presidente americano


signature.asc
Description: PGP signature

Re: Dump to tape AND keep copy on disk?

[Jon LaBadie]

On Tue, Jan 15, 2019 at 07:38:37PM +, Debra S Baddorf wrote:
> Amanda people:  does the newer amanda have any means to dump to tape
> while still retaining a copy on the holding disk?
> 
> Dumping to tape, and then vaulting BACK to a disk area seems excessive,
> if there is a better way?
> 
> Deb Baddorf   (asking for Stefan Weichinger)
> 

I've never used it, but ISTR amanda had a RAID feature
where you could tape to multiple devices simultaneously.


jon
> 
> 
> > On Jan 15, 2019, at 1:34 PM, Stefan G. Weichinger  wrote:
> > 
> > Am 14.01.19 um 19:05 schrieb Debra S Baddorf:
> > 
> >> You can let the normal job send the dumps to tape,  and then
> >> “vault”  a copy back onto a “vtape”  on disk  (not your holding disk,
> >> but that’s semantics).
> >> That might suit your need.
> >> 
> >> I’ve done vaulting from a tape onto a disk area …..  moved the
> >> disk area to another node,  and vaulted again to copy it to a
> >> different flavor of tape drive.   So the very first part of my task
> >> might suit you to a tee.
> > 
> > I understand that this might work but it seems to way more steps than
> > having a second "storage" to flush the holdingdisk content to. If that
> > works (and the "storage" doesn't allow to filter the lev0 afaik).
> > 
> > I haven't yet found the time and the brain to continue testing that so far.
>>> End of included message <<<

-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Dump to tape AND keep copy on disk?

[Debra S Baddorf]

Amanda people:  does the newer amanda have any means to dump to tape
while still retaining a copy on the holding disk?

Dumping to tape, and then vaulting BACK to a disk area seems excessive,
if there is a better way?

Deb Baddorf   (asking for Stefan Weichinger)



> On Jan 15, 2019, at 1:34 PM, Stefan G. Weichinger  wrote:
> 
> Am 14.01.19 um 19:05 schrieb Debra S Baddorf:
> 
>> You can let the normal job send the dumps to tape,  and then
>> “vault”  a copy back onto a “vtape”  on disk  (not your holding disk,
>> but that’s semantics).
>> That might suit your need.
>> 
>> I’ve done vaulting from a tape onto a disk area …..  moved the
>> disk area to another node,  and vaulted again to copy it to a
>> different flavor of tape drive.   So the very first part of my task
>> might suit you to a tee.
> 
> I understand that this might work but it seems to way more steps than
> having a second "storage" to flush the holdingdisk content to. If that
> works (and the "storage" doesn't allow to filter the lev0 afaik).
> 
> I haven't yet found the time and the brain to continue testing that so far.

Re: tape pools, wrapping my mind around them

[Jean-Louis Martineau]

You want 3 storage
A vtape can only contains backup from one storage.
You use the storage 'dump_selection' and the dumptype 'tag' to specify
which dle go to that storage

You could use a different pool for each storage.

But using the same pool for all storage have a big benefits:

   - no need to allocate a specific range of slot for each storage

sales use vtape-001, after a few days it is no longer needed (from the
policy), then research or support could use vtape-001

Jean-Louis


Le jeu. 15 nov. 2018, à 15 h 30, Jon LaBadie  a écrit :

> I currently have 240 Vtapes in a single changer.
> Suppose I have 3 "departments", sales, research,
> and support.
>
> Could I create 3 dumptype definitions so that
> all "sales" DLEs are backed up to a tape pool
> consisting of vtapes 1-80, "research" to vtapes
> 81-160, and "support" to vtapes 161-240?
>
> If so, then a single config could handle keeping
> the various data separated while still maintaining
> balance etc.  Of course, is the balance on a tape
> pool basis or on a total configuration basis?
>
> Jon
> --
> Jon H. LaBadie j...@jgcomp.com
>  11226 South Shore Rd.  (703) 787-0688 (H)
>  Reston, VA  20190  (703) 935-6720 (C)
>

RE: tape pools, wrapping my mind around them

[Cuttler, Brian R (HEALTH)]

I believe that you would define tape pool of 80 for each, with regular 
expressions you may be able to have the same label string format but different 
tape number ranges.
Maybe like this?

Sales POOL1[0-9][0-9]
Support POOL2[0-9][0-9]
Research POOL3[0-9][0-9]

For sanity I think I'd define 3 tape pools in the one changer. You spend time 
loading and unloading tapes to find the pool and tapes you want, but since this 
is done without having to access a physical tape drive or robot it will take 
virtually no time at all.

-Original Message-
From: owner-amanda-us...@amanda.org  On Behalf 
Of Jon LaBadie
Sent: Thursday, November 15, 2018 3:25 PM
To: amanda-users@amanda.org
Subject: tape pools, wrapping my mind around them

ATTENTION: This email came from an external source. Do not open attachments or 
click on links from unknown senders or unexpected emails.


I currently have 240 Vtapes in a single changer.
Suppose I have 3 "departments", sales, research, and support.

Could I create 3 dumptype definitions so that all "sales" DLEs are backed up to 
a tape pool consisting of vtapes 1-80, "research" to vtapes 81-160, and 
"support" to vtapes 161-240?

If so, then a single config could handle keeping the various data separated 
while still maintaining balance etc.  Of course, is the balance on a tape pool 
basis or on a total configuration basis?

Jon
--
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

tape pools, wrapping my mind around them

[Jon LaBadie]

I currently have 240 Vtapes in a single changer.
Suppose I have 3 "departments", sales, research,
and support.

Could I create 3 dumptype definitions so that
all "sales" DLEs are backed up to a tape pool
consisting of vtapes 1-80, "research" to vtapes
81-160, and "support" to vtapes 161-240?

If so, then a single config could handle keeping
the various data separated while still maintaining
balance etc.  Of course, is the balance on a tape
pool basis or on a total configuration basis?

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: executing tape drive commands before amdump/amcheck/amflush starts

[Ian Turner]

You can set the compression and block size natively in Amanda, no need  
to run mt.


See here: ‎https://wiki.zmanda.com/man/amanda-devices.7.html

  Original Message  
From: Charles Stroom
Sent: Friday, June 1, 2018 6:35 PM
To: amanda-users@amanda.org
Reply To: Charles Stroom
Subject: executing tape drive commands before amdump/amcheck/amflush starts


Hi,

before above amanda programs are started, 2 mt(1) commands are executed
("datcompression 0" & "setblk 32768").  At present, I do this with cron
jobs just before the amanda program starts (or manually in other cases),
but is there a simple way to integrate this somewhere in amanda?  E.g.
amanda.conf?

I looked at the "script" section, but it seems to be designed
for other things, but maybe I am wrong.  It also seems that
"datcompression" requires root execution, complicating matters a bit.

I guess I could make a script something like "my_amdump" etc with the
commands, but this has not my preference.

Thanks, Charles

--
Charles Stroom
email: charles at no-spam.stremen.xs4all.nl (remove the "no-spam.")

executing tape drive commands before amdump/amcheck/amflush starts

[Charles Stroom]

Hi,

before above amanda programs are started, 2 mt(1) commands are executed
("datcompression 0" & "setblk 32768").  At present, I do this with cron
jobs just before the amanda program starts (or manually in other cases),
but is there a simple way to integrate this somewhere in amanda?  E.g.
amanda.conf?

I looked at the "script" section, but it seems to be designed
for other things, but maybe I am wrong.  It also seems that
"datcompression" requires root execution, complicating matters a bit.

I guess I could make a script something like "my_amdump" etc with the
commands, but this has not my preference.

Thanks, Charles

-- 
Charles Stroom
email: charles at no-spam.stremen.xs4all.nl (remove the "no-spam.")

*** A TAPE ERROR OCCURRED: [no drives available] - tapelist.last_write symlink into void

[Tobias Köck]

I have upgraded to Amanda Server version 3.5.1-1~bpo9+6



The problem is that amanda-server creates a



tapelist.last_write -> 7862



link to a missing file (7862) and don't unlock the drive by deleting



tapelist.lock



and don't delete them after finishing.



So the first backup run works and the later ones stop working with



---

Hostname: chronos

Org : a2g-2

Config  : a2g-2

Date: May 17, 2018



*** A TAPE ERROR OCCURRED: [no drives available].

Some dumps may have been left in the holding disk.



The next 3 tapes Amanda expects to use are: a2g-2-03, a2g-2-09, a2g-2-04.

---

I am wondering if it is misconfigured? Or what's the problem?

It worked with version 3.3.9-5 included in Debian stretch but this version had 
some problems after Debian package updates.

Greetings and thanks,
Tobias

Re: sendbackup deadlock for my direct-to-tape dump

[Ian Turner]

That worked fine, thanks.

Ian


On 04/12/2018 05:59 PM, Jean-Louis Martineau wrote:
> Ian,
>
> Can you try to use the "amgtar" application instead of the GNUTAR program?
>
> Jean-Louis

Re: sendbackup deadlock for my direct-to-tape dump

[Jean-Louis Martineau]

Ian,

Can you try to use the "amgtar" application instead of the GNUTAR program?

Jean-Louis

From: owner-amanda-us...@amanda.org <owner-amanda-us...@amanda.org> on behalf 
of Ian Turner <vec...@vectro.org>
Sent: April 12, 2018 5:13 PM
To: amanda-users@amanda.org
Subject: sendbackup deadlock for my direct-to-tape dump

So I switched out my backup server to a new host running Ubuntu 18.04 (Bionic 
Beaver). All the Amanda stuff is working OK except for the localhost root 
filesystem dump, which is done direct to tape (no holding disk). That one hangs 
for a while, then fails with the error "sendbackup: critical (fatal): index tee 
cannot write [Broken pipe]".

The problem appears to be that sendbackup is deadlocked. This is what I observe:

Thu Apr 12 16:45:18 vectro@cue:~ {1}$ pstree -ap 28078
amdump,28078 /usr/sbin/amdump vectro.org<http://vectro.org>
  └─driver,28084 vectro.org<http://vectro.org> --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  ├─dumper,28086 vectro.org<http://vectro.org> --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  │   ├─amandad,28356 -auth=local
  │   │   ├─(amandad,28363)
  │   │   └─sendbackup,28362 amandad local --shm-name 
/amanda_shm_control-28085-0
  │   │   ├─sendbackup,28364 amandad local --shm-name 
/amanda_shm_control-28085-0
  │   │   │   └─sh,28369 -c /bin/tar -tf - 2>/dev/null | sed -e 
's/^\\.//'
  │   │   │   ├─sed,28371 -e s/^\\.//
  │   │   │   └─tar,28370 -tf -
  │   │   ├─tar,28367 --create --file - --directory / --one-file-system 
--listed-incremental /var/lib/amanda/gnutar-lists/localhost__0.new ...
  │   │   │   └─(sh,28372)
  │   │   └─{sendbackup},28368
  │   ├─gzip,28365 --best
  │   └─{dumper},28286
  ├─dumper,28087 vectro.org<http://vectro.org> --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  ├─dumper,28088 vectro.org<http://vectro.org> --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  ├─dumper,28089 vectro.org<http://vectro.org> --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  └─taper,28085 /usr/lib/amanda/taper vectro.org<http://vectro.org> 
--storage vectro.org<http://vectro.org> --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  ├─{taper},28351
  └─{taper},28352



The deadlocks are pipe and process based, as follows (I can make a diagram if 
needed):

  *   Process 28362 waits to read pipe 483972 (fd 0). The other end of this 
pipe is held as fd 2 by pids 28367, 28369, and 28371.
  *   Process 28367 waits to write pipe 483946 (fd 1). The other end of this 
pipe is held as fd 1 by pid 28364.
  *   Process 28364 waits to write pipe 483943 (fd 3). The other end of this 
pipe is held as fd 4 by pid 28362.
  *   Process 28369 is just waiting to reap its children (28371 and 28370).
  *   Process 28371 waits to read pipe 485744 (fd 0). The other end of this 
pipe is held as fd 1 by pid 28370.
  *   Process 28370 waits to read pipe 485743 (fd 0). The other end of this 
pipe is held as fd 5 by pid 28364.

Here are backtraces for the sendbackup processes:

Attaching to process 28362
0x7fd7116a9384 in __libc_read (fd=0, buf=0x55d8aa4c7090, nbytes=8192) at 
../sysdeps/unix/sysv/linux/read.c:27
27  ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
#0  0x7fd7116a9384 in __libc_read (fd=0, buf=0x55d8aa4c7090, nbytes=8192) 
at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x7fd711c073c3 in debug_areads () from 
/usr/lib/x86_64-linux-gnu/amanda/libamanda-3.5.1.so
#2  0x55d8a8be12d3 in parse_backup_messages ()
#3  0x55d8a8bde591 in main ()
Detaching from program: /usr/lib/amanda/sendbackup, process 28362

Attaching to process 28364
0x7fd7116a9281 in __libc_write (fd=3, buf=0x7fff8c6b19d0, nbytes=8192) at 
../sysdeps/unix/sysv/linux/write.c:27
27  ../sysdeps/unix/sysv/linux/write.c: No such file or directory.
#0  0x7fd7116a9281 in __libc_write (fd=3, buf=0x7fff8c6b19d0, nbytes=8192) 
at ../sysdeps/unix/sysv/linux/write.c:27
#1  0x7fd711c27556 in safe_write () from 
/usr/lib/x86_64-linux-gnu/amanda/libamanda-3.5.1.so
#2  0x7fd711c26d7e in full_write () from 
/usr/lib/x86_64-linux-gnu/amanda/libamanda-3.5.1.so
#3  0x55d8a8be1895 in start_index ()
#4  0x55d8a8be2c29 in ?? ()
#5  0x55d8a8bde56d in main ()
Detaching from program: /usr/lib/amanda/sendbackup, process 28364

This is using the Amanda packages included with Ubuntu 18.04, which is Amanda 
3.5.1.

Any thoughts on how to troubleshoot/fix this?

Cheers,

Ian
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

sendbackup deadlock for my direct-to-tape dump

[Ian Turner]

So I switched out my backup server to a new host running Ubuntu 18.04
(Bionic Beaver). All the Amanda stuff is working OK except for the
localhost root filesystem dump, which is done direct to tape (no holding
disk). That one hangs for a while, then fails with the error
"sendbackup: critical (fatal): index tee cannot write [Broken pipe]".

The problem appears to be that sendbackup is deadlocked. This is what I
observe:

Thu Apr 12 16:45:18 vectro@cue:~ {1}$ pstree -ap 28078
amdump,28078 /usr/sbin/amdump vectro.org
  └─driver,28084 vectro.org --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  ├─dumper,28086 vectro.org --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  │   ├─amandad,28356 -auth=local
  │   │   ├─(amandad,28363)
  │   │   └─sendbackup,28362 amandad local --shm-name 
/amanda_shm_control-28085-0
  │   │   ├─sendbackup,28364 amandad local --shm-name 
/amanda_shm_control-28085-0
  │   │   │   └─sh,28369 -c /bin/tar -tf - 2>/dev/null | sed -e 
's/^\\.//'
  │   │   │   ├─sed,28371 -e s/^\\.//
  │   │   │   └─tar,28370 -tf -
  │   │   ├─tar,28367 --create --file - --directory / --one-file-system 
--listed-incremental /var/lib/amanda/gnutar-lists/localhost__0.new ...
  │   │   │   └─(sh,28372)
  │   │   └─{sendbackup},28368
  │   ├─gzip,28365 --best
  │   └─{dumper},28286
  ├─dumper,28087 vectro.org --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  ├─dumper,28088 vectro.org --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  ├─dumper,28089 vectro.org --log-filename 
/var/amanda/vectro.org/logs/log.20180412164328.0
  └─taper,28085 /usr/lib/amanda/taper vectro.org --storage vectro.org 
--log-filename /var/amanda/vectro.org/logs/log.20180412164328.0
  ├─{taper},28351
  └─{taper},28352

The deadlocks are pipe and process based, as follows (I can make a
diagram if needed):

  * Process 28362 waits to read pipe 483972 (fd 0). The other end of
this pipe is held as fd 2 by pids 28367, 28369, and 28371.
  * Process 28367 waits to write pipe 483946 (fd 1). The other end of
this pipe is held as fd 1 by pid 28364.
  * Process 28364 waits to write pipe 483943 (fd 3). The other end of
this pipe is held as fd 4 by pid 28362.
  * Process 28369 is just waiting to reap its children (28371 and 28370).
  * Process 28371 waits to read pipe 485744 (fd 0). The other end of
this pipe is held as fd 1 by pid 28370.
  * Process 28370 waits to read pipe 485743 (fd 0). The other end of
this pipe is held as fd 5 by pid 28364.

Here are backtraces for the sendbackup processes:

Attaching to process 28362
0x7fd7116a9384 in __libc_read (fd=0, buf=0x55d8aa4c7090,
nbytes=8192) at ../sysdeps/unix/sysv/linux/read.c:27
27  ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
#0  0x7fd7116a9384 in __libc_read (fd=0, buf=0x55d8aa4c7090,
nbytes=8192) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x7fd711c073c3 in debug_areads () from
/usr/lib/x86_64-linux-gnu/amanda/libamanda-3.5.1.so
#2  0x55d8a8be12d3 in parse_backup_messages ()
#3  0x55d8a8bde591 in main ()
Detaching from program: /usr/lib/amanda/sendbackup, process 28362

Attaching to process 28364
0x7fd7116a9281 in __libc_write (fd=3, buf=0x7fff8c6b19d0,
nbytes=8192) at ../sysdeps/unix/sysv/linux/write.c:27
27  ../sysdeps/unix/sysv/linux/write.c: No such file or directory.
#0  0x7fd7116a9281 in __libc_write (fd=3, buf=0x7fff8c6b19d0,
nbytes=8192) at ../sysdeps/unix/sysv/linux/write.c:27
#1  0x7fd711c27556 in safe_write () from
/usr/lib/x86_64-linux-gnu/amanda/libamanda-3.5.1.so
#2  0x7fd711c26d7e in full_write () from
/usr/lib/x86_64-linux-gnu/amanda/libamanda-3.5.1.so
#3  0x55d8a8be1895 in start_index ()
#4  0x55d8a8be2c29 in ?? ()
#5  0x55d8a8bde56d in main ()
Detaching from program: /usr/lib/amanda/sendbackup, process 28364

This is using the Amanda packages included with Ubuntu 18.04, which is
Amanda 3.5.1.

Any thoughts on how to troubleshoot/fix this?

Cheers,

Ian

Re: Always fill every tape

[Jason L Tibbitts III]

>>>>> "JM" == Jean-Francois Malouin <jean-francois.malo...@bic.mni.mcgill.ca> 
>>>>> writes:

JM> This is not was I have experienced, at least with 3.3.x.

For the record, my server is 3.5.1.

JM> I've been using the following in some configs with the expected
JM> result:

OK, it's good to know that at some point it's worked for someone.  I was
using "flush-threshold-scheduled 70" and "flush-threshold-dumped 90" but
it would always flush if there was data to flush even if it only filled
a small portion of the tape.  This led to a two-day cycle:

Day 1: Nothing to flush, dump to the holding disk.
Day 2: Flush data from yesterday and write today's dumps to tape,
   filling about 40% of a tape.  Given the thresholds, it should not
   have written anything for one or two more days.
repeat

I've upped those limits to match yours so we'll see what happens after a
few days.

 - J<

Re: Always fill every tape

[Jean-Francois Malouin]

Hi Jason,

* Jason L Tibbitts III <ti...@math.uh.edu> [20180314 11:33]:
> And so it turns out that if you turn off autoflush, Amanda will never
> flush existing dumps to tape regardless of the flush-threshold-*
> settings.  Which I guess makes sense.  And with "autoflush yes", it
> seems to simply flush everything currently on the holding disk
> regardless of other settings.  So as far as I can tell, the
> flush-threshold-* settings are simply not working as designed.
> 
> What can I tweak to get more insight into how Amanda decides when to
> flush existing dumps?  I will continue to play with the thresholds to
> see if I can get behavior closer to I'm looking for but it would be good
> to have some insight into the decision process.
> 
>  - J<

This is not was I have experienced, at least with 3.3.x.
I've been using the following in some configs with the expected result:

flush-threshold-dumped 100
flush-threshold-scheduled 100
taperflush 100
autoflush yes 

As stated in the amanda.conf manpage, those will force amanda to start
writing to a new volume only if the data in the hold disk plus the
scheduled data are at least 100% of the volume capacity.  If anything is
left in the hold disk after a run, it will be flushed the next time
amdump runs. Those constraints only apply when a new volume is
requested.

Note that if you manually flush using amflush, you must override
those values with something like:

amflush -oflush-threshold-dumped=0 -oflush-threshold-scheduled=0 -otaperflush=0 
...

otherwise the flush constraints might not be met and the flush won't
complete entirely or at all.

Of course, I might be wrong so take this with a grain of salt!

cheers,
jf

Re: Always fill every tape

[Jason L Tibbitts III]

And so it turns out that if you turn off autoflush, Amanda will never
flush existing dumps to tape regardless of the flush-threshold-*
settings.  Which I guess makes sense.  And with "autoflush yes", it
seems to simply flush everything currently on the holding disk
regardless of other settings.  So as far as I can tell, the
flush-threshold-* settings are simply not working as designed.

What can I tweak to get more insight into how Amanda decides when to
flush existing dumps?  I will continue to play with the thresholds to
see if I can get behavior closer to I'm looking for but it would be good
to have some insight into the decision process.

 - J<

Re: Always fill every tape

[Jason L Tibbitts III]

>>>>> "DSB" == Debra S Baddorf <badd...@fnal.gov> writes:

DSB> Well, if it’s already not doing what you want, you might read
DSB> these.

Well, yeah, that's pretty much the same as I indicated that I'm using
now, except that I'm using different thresholds.  My problem is that I
asked for at least 70% usage and it's giving me just about 50%.  I also
checked the logs and it seems that it basically flushes every other day,
which is consistent with it always flushing existing dumps to tape at
startup.  So I'll turn off autoflush and see if that makes any
difference.

 - J<

Re: Always fill every tape

[Debra S Baddorf]

> On Mar 9, 2018, at 1:43 PM, Jason L Tibbitts III <ti...@math.uh.edu> wrote:
> 
> With tape sizes what they are now I'm finally at the point where I have
> way more tape than stuff to back up.  To satisfy my inherent laziness,
> I'd like to actually fill every tape (or at least come close to filling
> them).  And then just leave my library full of tapes without ever
> needing to change them out.
> 
> My understanding is that Amanda has no way to append to tapes; once it
> has started writing to a tape, that tape won't be used again until the
> next pass through the tapecycle.  That implies that you must keep
> backups on the holding disk and write them only when you'll have a
> nearly-full tape.  I have plenty of holding disk (at least until I move
> to LTO8) so that isn't an issue.
> 
> Currently I have these in amanda.conf:
> 
> tapecycle 47 tapes
> autoflush yes
> taperflush 70
> flush-threshold-dumped 70
> flush-threshold-scheduled 90
> 
> Is that basically all I'm supposed to do?  Because according to the
> reports, each tape is only getting about 50% full.  So I think I must be
> missing something about how those four flush-related parameters
> interact.
> 
> - J<

Well, if it’s already not doing what you want,  you might read these.  They’re 
from an
example  amanda.conf,   but I don’t know how current my copy is.

# You don't want to use a new tape for every run, but want to start writing
# to tape as soon as possible:
# flush-threshold-dumped0   # (or more)
# flush-threshold-scheduled 100 # (or more)
# taperflush100
# autoflush yes
# maxdumpsize   100k # amount of data to dump each run; see above.
#
# You want to keep the most recent dumps on holding disk, for faster recovery.
# Older dumps will be rotated to tape during each run.
# flush-threshold-dumped300 # (or more)
# flush-threshold-scheduled 300 # (or more)
# taperflush300
# autoflush yes

Deb Baddorf

Always fill every tape

[Jason L Tibbitts III]

With tape sizes what they are now I'm finally at the point where I have
way more tape than stuff to back up.  To satisfy my inherent laziness,
I'd like to actually fill every tape (or at least come close to filling
them).  And then just leave my library full of tapes without ever
needing to change them out.

My understanding is that Amanda has no way to append to tapes; once it
has started writing to a tape, that tape won't be used again until the
next pass through the tapecycle.  That implies that you must keep
backups on the holding disk and write them only when you'll have a
nearly-full tape.  I have plenty of holding disk (at least until I move
to LTO8) so that isn't an issue.

Currently I have these in amanda.conf:

tapecycle 47 tapes
autoflush yes
taperflush 70
flush-threshold-dumped 70
flush-threshold-scheduled 90

Is that basically all I'm supposed to do?  Because according to the
reports, each tape is only getting about 50% full.  So I think I must be
missing something about how those four flush-related parameters
interact.

 - J<

Re: amanda is skipping over available tapes -- "next tape to use" info?

[Nathan Stratton Treadway]

On Wed, Dec 27, 2017 at 17:26:04 -0500, Jean-Louis Martineau wrote:
> 'amadmin tape' list the oldest reusable label, it doesn't use the 
> taperscan algorithm and it doesn't check what is currently loaded in the 
> changer.
> 'amcheck' and 'amtape' run the taperscan algorithm and verify they are 
> in the changer.
> 

I am using vaulting send copies of my backups to vtapes located on
external USB drives, and would like to write a script to mount
the proper partition for the next run's vault volume before kicking off
the dump (or amcheck, for that matter).  

So, I'm wondering if there is any command available to show the tape
label Amanda would like to use next which _does_ use the taperscan
algorithm but _doesn't_ check the current status of the changer?

Thanks.

Nathan

Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: Amanda compatible with Fujitsu Eternus LT20 tape library

[Stefan G. Weichinger]

Am 2018-01-18 um 11:19 schrieb Volker Jahns:
> We have a customer request to implement a backup system which
> incorporates a Fujitsu eternus lt20 tape library.
> 
> Is there any experience on operating this tape library with amanda? Any
> help would be greatly appreciated.

In general: if your linux supports it, it will be usable with amanda.

If you get device-nodes like /dev/(n)stX for the drive(s) and /dev/sgX
for the robot chances are high that you will be able to configure amanda
accordingly.




some amanda-users provide professional support