Re: amanda tape algorithm

[Jon LaBadie]

On Thu, Jan 04, 2018 at 08:59:32PM -0500, Ned Danieley wrote:
> On Thu, Jan 04, 2018 at 05:16:59PM -0700, Steven Backus wrote:
> > Ned Danieley  wrote:
> > 
> > > ah, that makes sense. no, I haven't run 'amtapetype'; I just assumed that
> > > the rated capacity would be accurate. I'll give it a try; in the meantime,
> > > has anyone run 'amtapetype' on an LTO6 tape? I have an HP Ultrium 6 drive.
> > 
> > I did and got:
> > 
> > define tapetype LTO6 {
> > comment "Created by amtapetype; compression disabled"
> > length 2442954880 kbytes
> > filemark 7456397 kbytes
> > speed 154519 kps
> > blocksize 32 kbytes
> > }
> 
> thanks. that fairly well matches what I was using
> 
> define tapetype LTO6comp {
>   length 244352 kbytes
>   filemark 868 kbytes
>   speed 157129 kps
>   blocksize 2048 kbytes
>   }
> 
> except for filemark; can anyone comment on that? looking at the tapetype
> definitions on the zmanda wiki, it seems that most of the LTO entries have
> zero kbytes for filemark...
> 
It is the space (if any) left between files written to the tape.

Suppose you write a continuous stream of data to a tape
and you can write exactly 100GB.  If no space was left
between files you should be able to write 100 x 1GB files.
When you try it (as amtapetype does) and you find you can
only write 98 x 1GB files, 2GB was used by "filemarks".
Divide 2GB/98 files and you have your filemark.

I believe Amanda does consider filemarks in its determination
of what will fit on a tape.

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda tape algorithm

[Ned Danieley]

On Thu, Jan 04, 2018 at 05:16:59PM -0700, Steven Backus wrote:
> Ned Danieley  wrote:
> 
> > ah, that makes sense. no, I haven't run 'amtapetype'; I just assumed that
> > the rated capacity would be accurate. I'll give it a try; in the meantime,
> > has anyone run 'amtapetype' on an LTO6 tape? I have an HP Ultrium 6 drive.
> 
> I did and got:
> 
> define tapetype LTO6 {
> comment "Created by amtapetype; compression disabled"
> length 2442954880 kbytes
> filemark 7456397 kbytes
> speed 154519 kps
> blocksize 32 kbytes
> }

thanks. that fairly well matches what I was using

define tapetype LTO6comp {
length 244352 kbytes
filemark 868 kbytes
speed 157129 kps
blocksize 2048 kbytes
}

except for filemark; can anyone comment on that? looking at the tapetype
definitions on the zmanda wiki, it seems that most of the LTO entries have
zero kbytes for filemark...

-- 
Ned Danieley (ned.danie...@duke.edu)
Department of Biomedical Engineering
Box 90281, Duke University
Durham, NC  27708   (919) 660-5111

http://dilbert.com/strips/comic/2012-02-11/

Re: amanda tape algorithm

[Steven Backus]

Ned Danieley  wrote:

> ah, that makes sense. no, I haven't run 'amtapetype'; I just assumed that
> the rated capacity would be accurate. I'll give it a try; in the meantime,
> has anyone run 'amtapetype' on an LTO6 tape? I have an HP Ultrium 6 drive.

I did and got:

define tapetype LTO6 {
comment "Created by amtapetype; compression disabled"
length 2442954880 kbytes
filemark 7456397 kbytes
speed 154519 kps
blocksize 32 kbytes
}

Steve
-- 
Steven J. BackusComputer Systems Manager
University of Utah  E-Mail:  steven.bac...@utah.edu
Genetic EpidemiologyAlternate:  bac...@math.utah.edu
391 Chipeta Way -- Suite D  Office:  801.587.9308
Salt Lake City, UT 84108-1266   http://www.math.utah.edu/~backus

Re: Serious tape error

[Jon LaBadie]

On Thu, Jan 04, 2018 at 05:12:27PM -0500, Jon LaBadie wrote:
> I use vtapes of 100GB and runtapes == 3.  Most days only
> one tape is used.  About once every 2 or 3 weeks a second
> tape is needed.  Yesterday was one of those days.  But
> amanda chose the same vtape for the second tape and over
> wrote the data it had just written to the "first" tape.
> 
> >From the report:
> 
>   Tape Size (meg) 179621.1   171549.4 8071.8
> 
> Need multiple tapes.
> 
>   USAGE BY TAPE:
> Label Time Size  %  DLEs Parts
> DS1-215   1:45  179621M  175.42558
> 
> Only one tape listed, filled to 175%.
> 
>   Notes:
> taper: tape DS1-215 kb 104856364 fm 34 [OK]
> taper: tape DS1-215 kb 79075697 fm 24 [OK]
> 
> Tape DS1-215 selected twice.  Only the 24 files from the
> later run are present on the vtape, not the 58 total files.
> The first run was overwritten.
> 
> 
> JLM I've saved all the log files I could find from this run.
> What might you need to analyze this further?
> 
> I looked at /var/log/amanda/server/DS1/taper. and it
> appears to only contain information about the 2nd usage of
> the tape.
> 

I was wrong, there is info from both uses of the same tape in
the taper logfile.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda tape algorithm

[Jon LaBadie]

On Thu, Jan 04, 2018 at 04:58:13PM -0500, Ned Danieley wrote:
> On Thu, Jan 04, 2018 at 09:43:35PM +, Debra S Baddorf wrote:
> > 
> > > On Jan 4, 2018, at 11:59 AM, Ned Danieley  wrote:
> > > 
> > > 
> > > I'm using
> > > 
> > > taperalgo largestfit
> > > 
> > > which I assume means that amanda will write to tape the largest DLE
> > > available. I have 'runtapes' set to 2, and occasionally I'll see amanda
> > > move on to the second tape when the first tape is reporting around 80% 
> > > full.
> > > I'm dumping almost 200 DLE, so it seems like there ought to be some DLEs
> > > that would fit in the remaining 20%. is there any way to find out (after
> > > the fact) what DLEs were in the queue? I've looked at the taper debug file
> > > but it just seems to have a list of each DLE as it is written.
> > > 
> > > I'm using LTO6 tapes, so I should have about 2.5 TB to work with. I know
> > > about the ability to split DLEs, but I'm not ready to take that step.
> > > 
> > 
> > Did you actually test the size of your own tapes? 
> > 
> 
> ah, that makes sense. no, I haven't run 'amtapetype'; I just assumed that
> the rated capacity would be accurate. I'll give it a try; in the meantime,
> has anyone run 'amtapetype' on an LTO6 tape? I have an HP Ultrium 6 drive.

That 2.5TB is likely marketing-speak using decimal number.
Switching to binary TB would immediately reduce the size
by nearly 250GB (about 9% of the listed capacity).

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Serious tape error

[Jon LaBadie]

I use vtapes of 100GB and runtapes == 3.  Most days only
one tape is used.  About once every 2 or 3 weeks a second
tape is needed.  Yesterday was one of those days.  But
amanda chose the same vtape for the second tape and over
wrote the data it had just written to the "first" tape.

>From the report:

  Tape Size (meg) 179621.1   171549.4 8071.8

Need multiple tapes.

  USAGE BY TAPE:
Label Time Size  %  DLEs Parts
DS1-215   1:45  179621M  175.42558

Only one tape listed, filled to 175%.

  Notes:
taper: tape DS1-215 kb 104856364 fm 34 [OK]
taper: tape DS1-215 kb 79075697 fm 24 [OK]

Tape DS1-215 selected twice.  Only the 24 files from the
later run are present on the vtape, not the 58 total files.
The first run was overwritten.


JLM I've saved all the log files I could find from this run.
What might you need to analyze this further?

I looked at /var/log/amanda/server/DS1/taper. and it
appears to only contain information about the 2nd usage of
the tape.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: amanda tape algorithm

[Ned Danieley]

On Thu, Jan 04, 2018 at 09:43:35PM +, Debra S Baddorf wrote:
> 
> > On Jan 4, 2018, at 11:59 AM, Ned Danieley  wrote:
> > 
> > 
> > I'm using
> > 
> > taperalgo largestfit
> > 
> > which I assume means that amanda will write to tape the largest DLE
> > available. I have 'runtapes' set to 2, and occasionally I'll see amanda
> > move on to the second tape when the first tape is reporting around 80% full.
> > I'm dumping almost 200 DLE, so it seems like there ought to be some DLEs
> > that would fit in the remaining 20%. is there any way to find out (after
> > the fact) what DLEs were in the queue? I've looked at the taper debug file
> > but it just seems to have a list of each DLE as it is written.
> > 
> > I'm using LTO6 tapes, so I should have about 2.5 TB to work with. I know
> > about the ability to split DLEs, but I'm not ready to take that step.
> > 
> > -- 
> > Ned Danieley (ned.danie...@duke.edu)
> > Department of Biomedical Engineering
> > Box 90281, Duke University
> > Durham, NC  27708   (919) 660-5111
> > 
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__dilbert.com_strips_comic_2012-2D02-2D11_=DwIGaQ=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc=seIgoLdvEn-8badm2JmCmh78EJ9F9-Hw-LulwGbFPZI=DGzKNKCqh_8C-Y1Y3Smdu29gOMu_5Y82AnrH-oAP-vg=dB2_g2FDMHahFgcGA1CJLdbMwaXZJjrvHGFYP_pmwGM=
> >  
> 
> 
> Did you actually test the size of your own tapes?   (There’s a proceedure 
> that does this …. help me
> out with the name, guys?) 
> 
> Amanda, I believe, uses your number to decide how much space is left on the 
> tape.  It’ll
> pick the biggest DLE that ought to fit in that spot   (“largestfit”).   If 
> you’ve over-estimated
> the tape size,  then this DLE  will NOT fit.  At that point,  amanda can’t 
> back up and try
> a smaller DLE.  It merely stops trying with that DLE  (saves it for later) 
> and moves on to 
> the next tape.  
> At this point, the “largestfit”  is probably a bigger DLE, since the 
> whole tape is available
> for the second tape.
> 
> Rinse and repeat.
> 
> So — maybe tell amanda a smaller size for the tape?   Then it’ll try a 
> smaller  “last”  DLE,
> and it will actually fit.
> 
> At least,  this is my understanding of how she works.

ah, that makes sense. no, I haven't run 'amtapetype'; I just assumed that
the rated capacity would be accurate. I'll give it a try; in the meantime,
has anyone run 'amtapetype' on an LTO6 tape? I have an HP Ultrium 6 drive.

-- 
Ned Danieley (ned.danie...@duke.edu)
Department of Biomedical Engineering
Box 90281, Duke University
Durham, NC  27708   (919) 660-5111

http://dilbert.com/strips/comic/2012-02-11/

Re: amanda tape algorithm

[Debra S Baddorf]

> On Jan 4, 2018, at 11:59 AM, Ned Danieley  wrote:
> 
> 
> I'm using
> 
> taperalgo largestfit
> 
> which I assume means that amanda will write to tape the largest DLE
> available. I have 'runtapes' set to 2, and occasionally I'll see amanda
> move on to the second tape when the first tape is reporting around 80% full.
> I'm dumping almost 200 DLE, so it seems like there ought to be some DLEs
> that would fit in the remaining 20%. is there any way to find out (after
> the fact) what DLEs were in the queue? I've looked at the taper debug file
> but it just seems to have a list of each DLE as it is written.
> 
> I'm using LTO6 tapes, so I should have about 2.5 TB to work with. I know
> about the ability to split DLEs, but I'm not ready to take that step.
> 
> -- 
> Ned Danieley (ned.danie...@duke.edu)
> Department of Biomedical Engineering
> Box 90281, Duke University
> Durham, NC  27708   (919) 660-5111
> 
> http://dilbert.com/strips/comic/2012-02-11/


Did you actually test the size of your own tapes?   (There’s a proceedure that 
does this …. help me
out with the name, guys?) 

Amanda, I believe, uses your number to decide how much space is left on the 
tape.  It’ll
pick the biggest DLE that ought to fit in that spot   (“largestfit”).   If 
you’ve over-estimated
the tape size,  then this DLE  will NOT fit.  At that point,  amanda can’t back 
up and try
a smaller DLE.  It merely stops trying with that DLE  (saves it for later) and 
moves on to 
the next tape.  
At this point, the “largestfit”  is probably a bigger DLE, since the whole 
tape is available
for the second tape.

Rinse and repeat.

So — maybe tell amanda a smaller size for the tape?   Then it’ll try a smaller  
“last”  DLE,
and it will actually fit.

At least,  this is my understanding of how she works.

Deb Baddorf
Fermilab

amanda tape algorithm

[Ned Danieley]

I'm using

taperalgo largestfit

which I assume means that amanda will write to tape the largest DLE
available. I have 'runtapes' set to 2, and occasionally I'll see amanda
move on to the second tape when the first tape is reporting around 80% full.
I'm dumping almost 200 DLE, so it seems like there ought to be some DLEs
that would fit in the remaining 20%. is there any way to find out (after
the fact) what DLEs were in the queue? I've looked at the taper debug file
but it just seems to have a list of each DLE as it is written.

I'm using LTO6 tapes, so I should have about 2.5 TB to work with. I know
about the ability to split DLEs, but I'm not ready to take that step.

-- 
Ned Danieley (ned.danie...@duke.edu)
Department of Biomedical Engineering
Box 90281, Duke University
Durham, NC  27708   (919) 660-5111

http://dilbert.com/strips/comic/2012-02-11/