Re: working with storage definitions

[Chris Hoogendyk]

Things are beginning to run smoothly. Amanda has caught up with full backups, and last nights run 
completed by 7:57am this morning with a total of 5.2TB of dumps and no problems.


Two issues came up in the final stages of getting this working.

*1.* After adding a PSEUDO tape type with a length of ~15TB and making it the default, I was still 
only getting ~2.5TB of dumps. I don't know how Amanda was deciding on a dump size. It seemed to be 
taking one LTO6, even though that was no longer the default tapetype.


Looking through the amanda.conf man page, I came upon the parameter maxdumpsize, which, in my 
amanda.conf, had been set to -1. I changed that to ~15TB and that night got respectable filling of 
both the LTO7 and LTO6 totaling over 9TB of dumps. This seems to work, but also seems brittle in the 
sense that if I change the number of tapes I'm using, I will have to calculate a new maxdumpsize 
manually and put it in (presuming I remember).


*2.* I was seeing a substantial number of DLEs that were being written to both tapes. The ones going 
only to the LTO7 were using dumptypes that had the tag set explicitly in the dumptype. The ones 
going to both were supposed to have inherited a tag from the global dumptype. They didn't appear to, 
although they had inherited other parameters such as auth.


When I explicitly added the tag to the dumptype, they started going only to 
that storage.


Another interesting aside. On Sunday night, I was watching the progress with amstatus. Amanda had 
nearly finished, but still had two DLEs to complete just 15 minutes before the next day's run was 
supposed to begin. I did an `amcleanup -k daily` to terminate it and clean up so that the next day's 
run would be able to take off. On Monday night, I was again watching it, and had a similar 
situation. I decided to let it go, because I wanted to see how much it would fit on the tape. On 
Tuesday, I noticed that while the previous run had completed at 12:35am 9/5, the next run 
nevertheless kicked off at 11:30pm 9/4. They were both running at the same time for about an hour. 
One consequence was that for the second run, none of the LTO6 DLEs got written to tape, because that 
tape drive was unavailable when the run was initiated. The LTO7 DLEs were fine, because there are 
two drives on that tape library. Since I have a total of 8TB of holding space, nothing is lost, and 
the LTO6 dumps got flushed to tape the next day.



On 9/1/17 3:25 PM, Jean-Louis Martineau wrote:

On 01/09/17 03:16 PM, Chris Hoogendyk wrote:

hmm. That is kind of a problem. I should have more than sufficient space now, 
but the allocation of
where it goes could overflow if the planning is blind to the storage 
allocations. Maybe that is
something to be developed?

As I already wrote, It is an unimplemented features.


As to where to report the storage use, the Amanda report doesn't have much 
space for another column.
However, the Dump Summary could be broken into separate Dump Summary tables for 
each storage. I can
obviously look at my configuration files to see where things go, but it would 
be reassuring to have
the Amanda report reflect that.

The Dump Summary is for what is dumped,they could still be in holding disk and 
not on tape.

Maybe we could completetly rewrite the columnspec to be more like a printf
  "%12.0H %-11.0D %2L  %S

H is for host
D is for disk
L is for level
...
S is for storage

We can then add many fields.



As for my current configuration, I created a pseudo tapetype that has a length 
equal to the sum of
what I expect to be able to write to one LTO6 plus one LTO7 tape. I put that as 
the global tapetype,
with runtapes=1. Each storage still references it's own tpchanger and tapetype.


On 9/1/17 2:31 PM, Jean-Louis Martineau wrote:
> On 01/09/17 02:10 PM, Chris Hoogendyk wrote:
> > OK.
> >
> > So, how does the planner plan for what goes on each storage?
> It doesn't, it use the size and plan to dump less than that.
>
> > That is, if I set up a pseudo tapetype
> > for global and give it a length of, say, 12.5TB; how will it know that
> > the uncompressed DLEs are
> > targeted to the LTO7 and the compressed DLEs are targeted at the LTO6?
> > What if the proportions came
> > out wrong (say, 8TB of DLEs intended for the LTO7 and 4.5TB of DLEs
> > intened for the LTO6), but the
> > total was within the 12.5TB?
> That's the problem, some DLEs will fail unless you have enough holdingdisk.
> I never liked the idea that amanda delay some full dump because it do
> not have enough tape space, It get worse at every days
> The solution to never hit that problem is to always provide enough tape,
> you could increase the runtapes of both storage.
>
> >
> > Also, there is no commentary in the Amanda email report indicating
> > where each DLE went, just the
> > overall amount of data that was written to each tape. I presume DLEs
> > are going to the intended
> > tapes, but . . . .
> Where would you like the information to be 

Re: working with storage definitions

[Jean-Louis Martineau]

On 01/09/17 03:16 PM, Chris Hoogendyk wrote:
hmm. That is kind of a problem. I should have more than sufficient 
space now, but the allocation of
where it goes could overflow if the planning is blind to the storage 
allocations. Maybe that is

something to be developed?

As I already wrote, It is an unimplemented features.


As to where to report the storage use, the Amanda report doesn't have 
much space for another column.
However, the Dump Summary could be broken into separate Dump Summary 
tables for each storage. I can
obviously look at my configuration files to see where things go, but 
it would be reassuring to have

the Amanda report reflect that.
The Dump Summary is for what is dumped,they could still be in holding 
disk and not on tape.


Maybe we could completetly rewrite the columnspec to be more like a printf
  "%12.0H %-11.0D %2L  %S

H is for host
D is for disk
L is for level
...
S is for storage

We can then add many fields.



As for my current configuration, I created a pseudo tapetype that has 
a length equal to the sum of
what I expect to be able to write to one LTO6 plus one LTO7 tape. I 
put that as the global tapetype,
with runtapes=1. Each storage still references it's own tpchanger and 
tapetype.



On 9/1/17 2:31 PM, Jean-Louis Martineau wrote:
> On 01/09/17 02:10 PM, Chris Hoogendyk wrote:
> > OK.
> >
> > So, how does the planner plan for what goes on each storage?
> It doesn't, it use the size and plan to dump less than that.
>
> > That is, if I set up a pseudo tapetype
> > for global and give it a length of, say, 12.5TB; how will it know that
> > the uncompressed DLEs are
> > targeted to the LTO7 and the compressed DLEs are targeted at the LTO6?
> > What if the proportions came
> > out wrong (say, 8TB of DLEs intended for the LTO7 and 4.5TB of DLEs
> > intened for the LTO6), but the
> > total was within the 12.5TB?
> That's the problem, some DLEs will fail unless you have enough 
holdingdisk.

> I never liked the idea that amanda delay some full dump because it do
> not have enough tape space, It get worse at every days
> The solution to never hit that problem is to always provide enough tape,
> you could increase the runtapes of both storage.
>
> >
> > Also, there is no commentary in the Amanda email report indicating
> > where each DLE went, just the
> > overall amount of data that was written to each tape. I presume DLEs
> > are going to the intended
> > tapes, but . . . .
> Where would you like the information to be printed.
>
> You can run 'amadmin CONF find', it should list the storage for each 
DLEs.

>
> Jean-Louis
>
>
> *Disclaimer*
>
> This message is the property of *CARBONITE, INC.* 
> 
and may contain

> confidential or privileged information.
>
> If this message has been delivered to you by mistake, then do not 
copy or deliver this message to

> anyone. Instead, destroy it and notify me by reply e-mail.
>

--
---

Chris Hoogendyk

-
O__  Systems Administrator
c/ /'_ --- Biology & Geosciences Departments
(*) \(*) -- 315 Morrill Science Center
~~ - University of Massachusetts, Amherst



---

Erdös 4

This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

Re: working with storage definitions

[Chris Hoogendyk]

hmm. That is kind of a problem. I should have more than sufficient space now, but the allocation of 
where it goes could overflow if the planning is blind to the storage allocations. Maybe that is 
something to be developed?


As to where to report the storage use, the Amanda report doesn't have much space for another column. 
However, the Dump Summary could be broken into separate Dump Summary tables for each storage. I can 
obviously look at my configuration files to see where things go, but it would be reassuring to have 
the Amanda report reflect that.


As for my current configuration, I created a pseudo tapetype that has a length equal to the sum of 
what I expect to be able to write to one LTO6 plus one LTO7 tape. I put that as the global tapetype, 
with runtapes=1. Each storage still references it's own tpchanger and tapetype.



On 9/1/17 2:31 PM, Jean-Louis Martineau wrote:

On 01/09/17 02:10 PM, Chris Hoogendyk wrote:
> OK.
>
> So, how does the planner plan for what goes on each storage?
It doesn't, it use the size and plan to dump less than that.

> That is, if I set up a pseudo tapetype
> for global and give it a length of, say, 12.5TB; how will it know that
> the uncompressed DLEs are
> targeted to the LTO7 and the compressed DLEs are targeted at the LTO6?
> What if the proportions came
> out wrong (say, 8TB of DLEs intended for the LTO7 and 4.5TB of DLEs
> intened for the LTO6), but the
> total was within the 12.5TB?
That's the problem, some DLEs will fail unless you have enough holdingdisk.
I never liked the idea that amanda delay some full dump because it do
not have enough tape space, It get worse at every days
The solution to never hit that problem is to always provide enough tape,
you could increase the runtapes of both storage.

>
> Also, there is no commentary in the Amanda email report indicating
> where each DLE went, just the
> overall amount of data that was written to each tape. I presume DLEs
> are going to the intended
> tapes, but . . . .
Where would you like the information to be printed.

You can run 'amadmin CONF find', it should list the storage for each DLEs.

Jean-Louis


*Disclaimer*

This message is the property of *CARBONITE, INC.*  and may contain 
confidential or privileged information.


If this message has been delivered to you by mistake, then do not copy or deliver this message to 
anyone. Instead, destroy it and notify me by reply e-mail.




--
---

Chris Hoogendyk

-
   O__   Systems Administrator
  c/ /'_ --- Biology & Geosciences Departments
 (*) \(*) -- 315 Morrill Science Center
~~ - University of Massachusetts, Amherst



---

Erdös 4

Re: working with storage definitions

[Jean-Louis Martineau]

On 01/09/17 02:10 PM, Chris Hoogendyk wrote:

OK.

So, how does the planner plan for what goes on each storage?

It doesn't, it use the size and plan to dump less than that.


That is, if I set up a pseudo tapetype
for global and give it a length of, say, 12.5TB; how will it know that 
the uncompressed DLEs are
targeted to the LTO7 and the compressed DLEs are targeted at the LTO6? 
What if the proportions came
out wrong (say, 8TB of DLEs intended for the LTO7 and 4.5TB of DLEs 
intened for the LTO6), but the

total was within the 12.5TB?

That's the problem, some DLEs will fail unless you have enough holdingdisk.
I never liked the idea that amanda delay some full dump because it do 
not have enough tape space, It get worse at every days
The solution to never hit that problem is to always provide enough tape, 
you could increase the runtapes of both storage.




Also, there is no commentary in the Amanda email report indicating 
where each DLE went, just the
overall amount of data that was written to each tape. I presume DLEs 
are going to the intended

tapes, but . . . .

Where would you like the information to be printed.

You can run 'amadmin CONF find', it should list the storage for each DLEs.

Jean-Louis
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

Re: working with storage definitions

[Chris Hoogendyk]

OK.

So, how does the planner plan for what goes on each storage? That is, if I set up a pseudo tapetype 
for global and give it a length of, say, 12.5TB; how will it know that the uncompressed DLEs are 
targeted to the LTO7 and the compressed DLEs are targeted at the LTO6? What if the proportions came 
out wrong (say, 8TB of DLEs intended for the LTO7 and 4.5TB of DLEs intened for the LTO6), but the 
total was within the 12.5TB?


Also, there is no commentary in the Amanda email report indicating where each DLE went, just the 
overall amount of data that was written to each tape. I presume DLEs are going to the intended 
tapes, but . . . .



On 9/1/17 1:54 PM, Jean-Louis Martineau wrote:

It is an unimplemented features.
The planner do not look at the storage, it still use the global tapetype and 
runtapes.
Add a new tapetype with a length that is the sum of the two storage tapetype length * runtapes (or 
the maximum dump size you want in one run)

Set the global tapetype to that new tapetype and set the global runtapes to 1.

Jean-Louis

On 01/09/17 01:39 PM, Chris Hoogendyk wrote:

oops. Sorry about that. Attached amdump.20170831233001


On 9/1/17 1:25 PM, Jean-Louis Martineau wrote:
> I want the amdump. file, it is in the same directory as 
log.20170831233001.0
>
> On 01/09/17 01:22 PM, Chris Hoogendyk wrote:
>> attached trace log identified in the following debug file.
>>
>> amanda@marlin:~/daily$ less 
/tmp/amanda/server/daily/amdump.20170831233001.debug
>>
>> Thu Aug 31 23:30:01.546380821 2017: pid 1898: thd-0x1013000: amdump: pid 
1898 ruid 555 euid 555
>> version 3.4.5: start at Thu Aug 31 23:30:01 2017
>> Thu Aug 31 23:30:01.546931671 2017: pid 1898: thd-0x1013000: amdump: 
Arguments: daily
>> Thu Aug 31 23:30:01.547756321 2017: pid 1898: thd-0x1013000: amdump: reading 
config file
>> /usr/local/etc/amanda/daily/amanda.conf
>> Thu Aug 31 23:30:01.680034064 2017: pid 1898: thd-0x1013000: amdump: pid 
1898 ruid 555 euid 555
>> version 3.4.5: rename at Thu Aug 31 23:30:01 2017
>> Thu Aug 31 23:30:01.680717936 2017: pid 1898: thd-0x1013000: amdump: 
*beginning trace log:
>> /usr/local/etc/amanda/daily/log/log.20170831233001.0*
>> Thu Aug 31 23:30:01.680791773 2017: pid 1898: thd-0x1013000: amdump: 
beginning amdump log
>> Thu Aug 31 23:30:01.681288047 2017: pid 1898: thd-0x1013000: amdump:
>> /usr/local/share/perl/5.18.2/Amanda/Amdump.pm:97:info:203 The timestamp 
is '20170831233001'
>> Thu Aug 31 23:30:01.681713007 2017: pid 1898: thd-0x1013000: amdump:
>> /usr/local/share/perl/5.18.2/Amanda/Amdump.pm:103:info:201 The amdump 
trace file is
>> '/usr/local/etc/amanda/daily/log/amdump.20170831233001'
>> Thu Aug 31 23:30:01.682060951 2017: pid 1898: thd-0x1013000: amdump:
>> /usr/local/share/perl/5.18.2/Amanda/Amdump.pm:109:info:200 The log file 
is
>> '/usr/local/etc/amanda/daily/log/log.20170831233001.0'
>> Thu Aug 31 23:30:01.682475809 2017: pid 1898: thd-0x1013000: amdump: 
invoking planner | driver
>> Thu Aug 31 23:30:01.684324606 2017: pid 1898: thd-0x1013000: amdump:  
planner: 1899
>> Thu Aug 31 23:30:01.684758762 2017: pid 1899: thd-0x1013000: amdump: exec:
>> /usr/local/libexec/amanda/planner daily --starttime 20170831233001 
--log-filename
>> /usr/local/etc/amanda/daily/log/log.20170831233001.0
>> Thu Aug 31 23:30:01.684983591 2017: pid 1899: thd-0x1013000: amdump: exec
>> /usr/local/libexec/amanda/planner daily --starttime 20170831233001 
--log-filename
>> /usr/local/etc/amanda/daily/log/log.20170831233001.0
>> Thu Aug 31 23:30:01.685602431 2017: pid 1898: thd-0x1013000: amdump:  
driver: 1900
>> Thu Aug 31 23:30:01.685953697 2017: pid 1900: thd-0x1013000: amdump: exec:
>> /usr/local/libexec/amanda/driver daily --log-filename
>> /usr/local/etc/amanda/daily/log/log.20170831233001.0
>> Thu Aug 31 23:30:01.686152981 2017: pid 1900: thd-0x1013000: amdump: exec
>> /usr/local/libexec/amanda/driver daily --log-filename
>> /usr/local/etc/amanda/daily/log/log.20170831233001.0
>> Thu Aug 31 23:57:34.602971243 2017: pid 1898: thd-0x1013000: amdump: planner 
finished with exit
>> code 0
>> Fri Sep 01 05:47:20.955904196 2017: pid 1898: thd-0x1013000: amdump: driver 
finished with exit
>> code 0
>> Fri Sep 01 05:47:20.956138752 2017: pid 1898: thd-0x1013000: amdump: running 
amreport
>> Fri Sep 01 05:47:20.956268742 2017: pid 1898: thd-0x1013000: amdump: Running
>> /usr/local/sbin/amreport daily --from-amdump -l 
/usr/local/etc/amanda/daily/log/log.20170831233001.0

>> Fri Sep 01 05:47:22.808415301 2017: pid 1898: thd-0x1013000: amdump: 
/usr/local/sbin/amreport
>> exited with code 14
>> Fri Sep 01 05:47:22.808511674 2017: pid 1898: thd-0x1013000: amdump: 
trimming old trace logs
>> Fri Sep 01 05:47:22.808573795 2017: pid 1898: thd-0x1013000: amdump: Running
>> /usr/local/libexec/amanda/amtrmlog daily
>> Fri Sep 01 05:47:22.926108466 2017: pid 1898: thd-0x1013000: amdump:
>> /usr/local/libexec/amanda/amtrmlog exited with code 0
>> Fri Sep 01 05:47:22.926212127 2017: pid 1898: thd-0x1013000: 

Re: working with storage definitions

[Chris Hoogendyk]

Thank you!

I'll see how things run tonight. Unfortunately, I have one researcher who has been shuffling 
terabytes of data from one array to another. So, I've also had to chase that down as well, and those 
will end up being fulls.


So, for example,

   localhost   /data/vazeylab/./emr-17-videos /data/vazeylab {
    gnutar-data-local
    include "./Data/M2-2AFC/EMR-17-/Videos"
    }   -1

became

   localhost   /data/vazeylab2/./emr-17-videos /data/vazeylab2 {
    gnutar-herbarium
    include "./Data/M2-2AFC/EMR-17-/Videos"
    }   -1

after I chased down that they were moved from one array to another. Then, Amanda sees that as a new 
DLE requiring a full. They're transitioning to a new RAID5 array that is made up of 10TB HGST SAS 
Helium Drives.



On 8/30/17 2:42 PM, Jean-Louis Martineau wrote:

On 30/08/17 02:14 PM, Chris Hoogendyk wrote:


Thank you, Jean-Louis!

I made all the changes you recommended.

Then I got:

amanda@marlin:~/daily$ amcheck daily
'/usr/local/etc/amanda/daily/amanda.conf', line 68: configuration keyword 
expected
parse error: storage 'daily' is not defined
parse error: storage 'daily' is not defined
ERROR: errors processing config file

where line 68-70 of my amanda.conf is

dump-selection "daily" ALL

storage "daily" "research"

The man pages for amanda.conf list dump-selection only in the storage section and not among the 
global parameters. So, I constructed a storage section for daily analogous to the storage section 
for research.



You did the right things.


After that, I got `amcheck daily` to work:

amanda@marlin:~/daily$ amcheck daily
Amanda Tape Server Host Check
-
NOTE: Holding disk '/amanda3': 2742194176 KB disk space available, using 
2637336576 KB
NOTE: Holding disk '/amanda4': 2055245824 KB disk space available, using 
1950388224 KB

slot 9:no drives available
slot 10:no drives available
slot 11:no drives available
slot 12:no drives available
slot 13:no drives available
slot 14:no drives available
ERROR: no drives available
slot 1: volume 'Bio-Research-001'
Will write to volume 'Bio-Research-001' in slot 1.
NOTE: skipping tape-writable test
Server check took 83.762 seconds
Amanda Backup Client Hosts Check

Client check: 4 hosts checked in 5.501 seconds.  0 problems found.
(brought to you by Amanda 3.4.5)

Since Amanda is still running, it can't access the LTO6. I'm puzzled that it seems to check those 
slots (although too quickly to actually be looking at the tapes) when Amanda is running. Version 
3.3.9 would tell you that amdump seemed to be running and would skip the tape checks. Anyway, 
this is good in that it shows the LTO7 tape library and says it will write to the first tape there.



3.4 can run many programs in parallel

Since I have an explicit storage section for daily now, is it going to treat that as different 
from the previous default daily? Will the dumps continue to reference the full and incremental 
dumps that have been run previously? A similar question for the LTO7 library and the storage 
"research". Since that is new, when I move a DLE from "daily" to "research" will it automatically 
begin with a full? That would mean that I should be cautious about mass moving DLEs.



Amanda will not do full from moved DLES.
Scheduling doesn't look at which storage the backups are.

That also brings on another question. While I have a global settings for tapecycle, at the moment 
I have fewer tapes for LTO7; but, the parameter tapecycle seems to be a global and not a storage 
section parameter. If I try to put it in the storage section, amcheck gives me a complaint 
"configuration keyword expected".  If I want to tweak things for the two storages differently, am 
I limited in the ways that the man page implies? How can I tell it that there are fewer tapes in 
the cycle for the LTO7 library?


tapecycle is the old way, look at the POLICY section in the amanda.conf man page, you can have a 
different policy for each storage.


Jean-Louis



On 8/30/17 12:14 PM, Jean-Louis Martineau wrote:

On 30/08/17 11:55 AM, Chris Hoogendyk wrote:


Not quite there yet.

I think I got the tapes labeled properly. I can see that the parameters that got invoked on my 
previous attempt with the "-o tpchanger=" were not correct. Thus, an error in block size on the 
re labeling. But I think I at least understand how that happened.


amanda@marlin:~/daily$ amlabel -f -o storage=research  daily 
"Bio-Research-001" slot 1
Reading label...
Error reading volume label: Error reading Amanda header: block size too 
small.
Writing label 'Bio-Research-001'...
Checking label...
Success!

resulted in a line in the tapelist as follows:

0 

Re: working with storage definitions

[Jean-Louis Martineau]

On 30/08/17 11:55 AM, Chris Hoogendyk wrote:


Not quite there yet.

I think I got the tapes labeled properly. I can see that the 
parameters that got invoked on my previous attempt with the "-o 
tpchanger=" were not correct. Thus, an error in block size on the re 
labeling. But I think I at least understand how that happened.


amanda@marlin:~/daily$ amlabel -f -o storage=research daily
"Bio-Research-001" slot 1
Reading label...
Error reading volume label: Error reading Amanda header: block
size too small.
Writing label 'Bio-Research-001'...
Checking label...
Success!

resulted in a line in the tapelist as follows:

0 Bio-Research-001 reuse BARCODE:01L7 BLOCKSIZE:512 POOL:daily
STORAGE:research CONFIG:daily


In the research storage section, you should add:
  tapepool "research"
to tell amanda that the research storage can use any tape in the 
research pool.

Then

amanda@marlin:~/daily$ amlabel -f --pool research --assign daily
"Bio-Research-[0-9][0-9][0-9]"
Setting Bio-Research-001

changed it to

0 Bio-Research-001 reuse BARCODE:01L7 BLOCKSIZE:512
POOL:research STORAGE:research CONFIG:daily

But, `amcheck daily` did not make any mention of the research storage 
or tapes


amanda@marlin:~/daily$ amcheck daily
Amanda Tape Server Host Check
-
NOTE: Holding disk '/amanda3': 3321401344 KB disk space available,
using 3216543744 KB
NOTE: Holding disk '/amanda4': 643121152 KB disk space available,
using 538263552 KB

slot 8: volume 'Bio-Daily-010'
Will write to volume 'Bio-Daily-010' in slot 8.
NOTE: skipping tape-writable test
Server check took 58.426 seconds
Amanda Backup Client Hosts Check

Client check: 4 hosts checked in 4.930 seconds.  3 problems found.
(brought to you by Amanda 3.4.5)


You must tell amanda which storage to use, in amanda.conf:
  storage "daily" "research"
to tell it to use both storage


And, when Amanda ran last night, it appears to be running according to 
the previous normal without any effects from my changes in the 
configuration and disklist. The backups are going to the LTO6 tape 
library in the default "daily" storage.


amanda@marlin:~/daily$ amstatus daily
Using: /usr/local/etc/amanda/daily/log/amdump
From Tue Aug 29 23:30:01 EDT 2017



SUMMARY   dle   real  estimated
size   size
   -  -
disk: 158
estimated   : 144 2459174380k
flush   : 140  3043619315k
dump failed :   4  138k   ( 0.00%)
wait for dumping:  15 1661177930k   ( 0.00%)
dumping to tape :   0 0k 0k (  0.00%) ( 0.00%)
dumping :   2 0k  564952671k (  0.00%) ( 0.00%)
dumped  : 123  301938738k  233043641k (129.56%) ( 12.28%)
wait for writing: 123  301938738k  233043641k (129.56%) ( 12.28%)
wait to flush   :  19  492421514k  492421514k (100.00%) ( 20.02%)
writing to tape
dumping to tape
failed to tape  :   1  232156245k  232156245k (100.00%) ( 9.44%)
taped   : 120  2319041556k 0k (  0.00%) ( 94.30%)
tape 1  : 121 2319041504k 2319041504k ( 94.27%)
Bio-Daily-010 (120 parts)

8 dumpers idle  : no-bandwidth
daily   qlen: 143
   0: tape error: No space left on device, splitting
not enabled (localhost:/data/vazeylab/./Windows)




I've attached my amanda.conf and disklist so that you can see what 
I've done. I put the tag for the storage "research" in the DLEs. I 
started with the herbarium entries, and expect to continue through 
many others once it works.


Since it appears that on this system (with its rapidly escalating disk 
array space) compression is the bottleneck; I decided to try letting 
the LTO7 do it's own compression. Thus the herbarium dump definition 
that I'm using has no compression.
The daily storage have no dump-selection setting, so all dles (even 
those tagged Research) will go on the daily storages.

The dle tagged Research will be written to both storages.

You need to add a dump-selection for the daily storage and add a 
corresponding tag to all dles that must go in that storage, you should 
add it in a dumptype inherited from all dles.


Jean-Louis




On 8/29/17 4:21 PM, Jean-Louis Martineau wrote:

On 29/08/17 02:37 PM, Chris Hoogendyk wrote:

Thank you! I've got that set up now.

I tried labeling tapes. I thought it was working. e.g.:



You should never use the '-o tpchanger=' argument, it is always 
better to use '-ostorage='

That will set the storage and the pool correctly.


amanda@marlin:~/daily$ amlabel -f -o tpchanger=NEO200x48 -o
'labelstr="^Bio-Research-[0-9][0-9][0-9]*$"' daily 
"Bio-Research-001" slot 1

Reading label...
Found label 

Re: working with storage definitions

[Jean-Louis Martineau]

On 29/08/17 02:37 PM, Chris Hoogendyk wrote:

Thank you! I've got that set up now.

I tried labeling tapes. I thought it was working. e.g.:



You should never use the '-o tpchanger=' argument, it is always better 
to use '-ostorage='

That will set the storage and the pool correctly.


amanda@marlin:~/daily$ amlabel -f -o tpchanger=NEO200x48 -o
'labelstr="^Bio-Research-[0-9][0-9][0-9]*$"' daily "Bio-Research-001" 
slot 1

Reading label...
Found label 'amtapetype-206980288' but it is not in the tapelist file.
Writing label 'Bio-Research-001'...
Checking label...
Success!

After that (which I repeated for several tapes), I tried

amanda@marlin:~/daily$ amlabel --storage research --assign daily 
"Bio-Research-[0-9][0-9][0-9]"
Bio-Research-001: Can't assign storage without force, old storage is 
'daily'

[etc. for each of the several tapes]

So, adding -f

amanda@marlin:~/daily$ amlabel -f --storage research --assign daily 
"Bio-Research-[0-9][0-9][0-9]"

Bio-Research-001: Can't assign storage because it is a new labelled tape.
[etc. for each of the several tapes]
All labels already correctly set.

Changing the storage is useless, but we could allow amlabel to change it.


And looking at the tapelist, I see entries like:

0 Bio-Research-001 reuse BARCODE:01L7 BLOCKSIZE:2048 POOL:daily 
STORAGE:daily CONFIG:daily


A label is not bound to a storage, the listed storage is the latest 
storage that used that tape,

A tape is bound to a pool, and it can be in any storage that use that pool.



Have I done something wrong? Or is that going to work, because it is a 
blank tape. In other words,
since it is only accessible via tpchanger=NEO200x48, which is in the 
definition of the storage
research; will it be accessed by those backups with the tag research 
and then re listed in the

tapelist as belonging to storage research?

I also wasn't finding much detail in the man pages on the implications 
of pools. I don't have any

such designations in my amanda.conf.


A pool is just a name, a storage will use any tape that is in the same pool.

Jean-Louis





On 8/29/17 6:45 AM, Jean-Louis Martineau wrote:
> Chris,
>
> Everything looks good except the 'define changer' should be before the
> 'define storage'.
>
> define changer NEO200x48 {
> tpchanger
> "chg-robot:/dev/tape/by-id/scsi-1BDT_FlexStor_II_00MX64200626_LL0" #
> your changer device file (T48)
> property "tape-device" "0=tape:/dev/nst1" # your tape
> drive device file
> property append "tape-device" "1=tape:/dev/nst2" # your
> tape drive device file
> property "use-slots" "1-11" # UPDATE WHEN TAPES
> ADDED <
> }
>
> define storage research {
> tpchanger "NEO200x48"
> tapetype LTO7
> labelstr "^Bio-Research-[0-9][0-9][0-9]*$"
> amrecover_changer "NEO200x48"
> dump-selection "Research" ALL
> }
>
> Jean-Louis
>
> On 28/08/17 05:46 PM, Chris Hoogendyk wrote:
> > I haven't found any examples of Amanda configurations using storage
> > definitions. Having read through the man pages, I've come up with a
> > modification of my amanda.conf that I think should work, and I'm
> > looking for comments and feedback.
> >
> > I had temporarily set up a separate Amanda configuration to test my
> > new LTO7 tape library. That seemed to work, and I was able to generate
> > a typetype for LTO7. Now I want to fold it into my regular daily
> > amanda.conf and have that use both my current LTO6 library and the new
> > LTO7.
> >
> > I've added a section as follows:
> >
> > define storage research {
> >
> > define changer NEO200x48 {
> > tpchanger
> > "chg-robot:/dev/tape/by-id/scsi-1BDT_FlexStor_II_00MX64200626_LL0" 
# your

> > changer device file (T48)
> > property "tape-device" "0=tape:/dev/nst1" # your tape
> > drive device file
> > property append "tape-device" "1=tape:/dev/nst2" # your
> > tape drive device file
> > property "use-slots" "1-11" # UPDATE WHEN TAPES
> > ADDED <
> > }
> > tpchanger "NEO200x48"
> >
> > tapetype LTO7
> >
> > labelstr "^Bio-Research-[0-9][0-9][0-9]*$"
> >
> > amrecover_changer "NEO200x48"
> >
> > dump-selection "Research" ALL
> >
> > }
> >
> > immediately after those same settings in the daily amanda.conf.
> >
> > I'm thinking with that in the configuration, I will be able to label
> > tapes by, e.g.
> >
> > amlabel --storage "Research" daily "Bio-Research-012"
> >
> > Then I should be able to shift DLEs from the daily to the Research
> > storage by putting a
> >
> > tag "Research"
> >
> > in the dump type, or in the DLE.
> >
> > I'm assuming that the default will follow my pre-existing amanda.conf
> > with a default storage of "daily".
> >
> >
>
>
> *Disclaimer*
>
> This message is the property of *CARBONITE, INC.* 
> 
and may contain

> confidential or privileged information.
>
> If this message has been delivered to you by mistake, then do not 
copy or deliver this message to

> anyone. Instead, destroy it and notify me by reply e-mail.
>

--

Re: working with storage definitions

[Chris Hoogendyk]

Thank you! I've got that set up now.

I tried labeling tapes. I thought it was working. e.g.:

   amanda@marlin:~/daily$ amlabel -f -o tpchanger=NEO200x48 -o
   'labelstr="^Bio-Research-[0-9][0-9][0-9]*$"' daily "Bio-Research-001" slot 1
   Reading label...
   Found label 'amtapetype-206980288' but it is not in the tapelist file.
   Writing label 'Bio-Research-001'...
   Checking label...
   Success!

After that (which I repeated for several tapes), I tried

   amanda@marlin:~/daily$ amlabel --storage research --assign daily 
"Bio-Research-[0-9][0-9][0-9]"
   Bio-Research-001: Can't assign storage without force, old storage is 'daily'
   [etc. for each of the several tapes]

So, adding -f

   amanda@marlin:~/daily$ amlabel -f --storage research --assign daily 
"Bio-Research-[0-9][0-9][0-9]"
   Bio-Research-001: Can't assign storage because it is a new labelled tape.
   [etc. for each of the several tapes]
   All labels already correctly set.

And looking at the tapelist, I see entries like:

   0 Bio-Research-001 reuse BARCODE:01L7 BLOCKSIZE:2048 POOL:daily 
STORAGE:daily CONFIG:daily


Have I done something wrong? Or is that going to work, because it is a blank tape. In other words, 
since it is only accessible via tpchanger=NEO200x48, which is in the definition of the storage 
research; will it be accessed by those backups with the tag research and then re listed in the 
tapelist as belonging to storage research?


I also wasn't finding much detail in the man pages on the implications of pools. I don't have any 
such designations in my amanda.conf.




On 8/29/17 6:45 AM, Jean-Louis Martineau wrote:

Chris,

Everything looks good except the 'define changer' should be before the
'define storage'.

define changer NEO200x48 {
tpchanger
"chg-robot:/dev/tape/by-id/scsi-1BDT_FlexStor_II_00MX64200626_LL0" #
your changer device file (T48)
property "tape-device" "0=tape:/dev/nst1" # your tape
drive device file
property append "tape-device" "1=tape:/dev/nst2" # your
tape drive device file
property "use-slots" "1-11" # UPDATE WHEN TAPES
ADDED <
}

define storage research {
tpchanger "NEO200x48"
tapetype LTO7
labelstr "^Bio-Research-[0-9][0-9][0-9]*$"
amrecover_changer "NEO200x48"
dump-selection "Research" ALL
}

Jean-Louis

On 28/08/17 05:46 PM, Chris Hoogendyk wrote:
> I haven't found any examples of Amanda configurations using storage
> definitions. Having read through the man pages, I've come up with a
> modification of my amanda.conf that I think should work, and I'm
> looking for comments and feedback.
>
> I had temporarily set up a separate Amanda configuration to test my
> new LTO7 tape library. That seemed to work, and I was able to generate
> a typetype for LTO7. Now I want to fold it into my regular daily
> amanda.conf and have that use both my current LTO6 library and the new
> LTO7.
>
> I've added a section as follows:
>
> define storage research {
>
> define changer NEO200x48 {
> tpchanger
> "chg-robot:/dev/tape/by-id/scsi-1BDT_FlexStor_II_00MX64200626_LL0" # your
> changer device file (T48)
> property "tape-device" "0=tape:/dev/nst1" # your tape
> drive device file
> property append "tape-device" "1=tape:/dev/nst2" # your
> tape drive device file
> property "use-slots" "1-11" # UPDATE WHEN TAPES
> ADDED <
> }
> tpchanger "NEO200x48"
>
> tapetype LTO7
>
> labelstr "^Bio-Research-[0-9][0-9][0-9]*$"
>
> amrecover_changer "NEO200x48"
>
> dump-selection "Research" ALL
>
> }
>
> immediately after those same settings in the daily amanda.conf.
>
> I'm thinking with that in the configuration, I will be able to label
> tapes by, e.g.
>
> amlabel --storage "Research" daily "Bio-Research-012"
>
> Then I should be able to shift DLEs from the daily to the Research
> storage by putting a
>
> tag "Research"
>
> in the dump type, or in the DLE.
>
> I'm assuming that the default will follow my pre-existing amanda.conf
> with a default storage of "daily".
>
>


*Disclaimer*

This message is the property of *CARBONITE, INC.*  and may contain 
confidential or privileged information.


If this message has been delivered to you by mistake, then do not copy or deliver this message to 
anyone. Instead, destroy it and notify me by reply e-mail.




--
---

Chris Hoogendyk

-
   O__   Systems Administrator
  c/ /'_ --- Biology & Geosciences Departments
 (*) \(*) -- 315 Morrill Science Center
~~ - University of Massachusetts, Amherst



---

Erdös 4

Re: working with storage definitions

[Jean-Louis Martineau]

Chris,

Everything looks good except the 'define changer' should be before the 
'define storage'.


define changer NEO200x48 {
  tpchanger 
"chg-robot:/dev/tape/by-id/scsi-1BDT_FlexStor_II_00MX64200626_LL0" # 
your changer device file (T48)
  property "tape-device" "0=tape:/dev/nst1"# your tape 
drive device file
  property append "tape-device" "1=tape:/dev/nst2" # your 
tape drive device file
  property "use-slots" "1-11"  # UPDATE WHEN TAPES 
ADDED <

}

define storage research {
   tpchanger "NEO200x48"
   tapetype LTO7
   labelstr "^Bio-Research-[0-9][0-9][0-9]*$"
   amrecover_changer "NEO200x48"
   dump-selection "Research" ALL
}

Jean-Louis

On 28/08/17 05:46 PM, Chris Hoogendyk wrote:
I haven't found any examples of Amanda configurations using storage 
definitions. Having read through the man pages, I've come up with a 
modification of my amanda.conf that I think should work, and I'm 
looking for comments and feedback.


I had temporarily set up a separate Amanda configuration to test my 
new LTO7 tape library. That seemed to work, and I was able to generate 
a typetype for LTO7. Now I want to fold it into my regular daily 
amanda.conf and have that use both my current LTO6 library and the new 
LTO7.


I've added a section as follows:

   define storage research {

   define changer NEO200x48 {
  tpchanger 
"chg-robot:/dev/tape/by-id/scsi-1BDT_FlexStor_II_00MX64200626_LL0" # your

   changer device file (T48)
  property "tape-device" "0=tape:/dev/nst1"# your tape 
drive device file
  property append "tape-device" "1=tape:/dev/nst2" # your 
tape drive device file
  property "use-slots" "1-11"  # UPDATE WHEN TAPES 
ADDED <

   }
   tpchanger "NEO200x48"

   tapetype LTO7

   labelstr "^Bio-Research-[0-9][0-9][0-9]*$"

   amrecover_changer "NEO200x48"

   dump-selection "Research" ALL

   }

immediately after those same settings in the daily amanda.conf.

I'm thinking with that in the configuration, I will be able to label 
tapes by, e.g.


   amlabel --storage "Research" daily "Bio-Research-012"

Then I should be able to shift DLEs from the daily to the Research 
storage by putting a


   tag "Research"

in the dump type, or in the DLE.

I'm assuming that the default will follow my pre-existing amanda.conf 
with a default storage of "daily".




This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

working with storage definitions

[Chris Hoogendyk]

I haven't found any examples of Amanda configurations using storage definitions. Having read through 
the man pages, I've come up with a modification of my amanda.conf that I think should work, and I'm 
looking for comments and feedback.


I had temporarily set up a separate Amanda configuration to test my new LTO7 tape library. That 
seemed to work, and I was able to generate a typetype for LTO7. Now I want to fold it into my 
regular daily amanda.conf and have that use both my current LTO6 library and the new LTO7.


I've added a section as follows:

   define storage research {

   define changer NEO200x48 {
  tpchanger 
"chg-robot:/dev/tape/by-id/scsi-1BDT_FlexStor_II_00MX64200626_LL0" # your
   changer device file (T48)
  property "tape-device" "0=tape:/dev/nst1"# your tape drive device 
file
  property append "tape-device" "1=tape:/dev/nst2" # your tape 
drive device file
  property "use-slots" "1-11"  # UPDATE WHEN TAPES ADDED 
<
   }
   tpchanger "NEO200x48"

   tapetype LTO7

   labelstr "^Bio-Research-[0-9][0-9][0-9]*$"

   amrecover_changer "NEO200x48"

   dump-selection "Research" ALL

   }

immediately after those same settings in the daily amanda.conf.

I'm thinking with that in the configuration, I will be able to label tapes by, 
e.g.

   amlabel --storage "Research" daily "Bio-Research-012"

Then I should be able to shift DLEs from the daily to the Research storage by 
putting a

   tag "Research"

in the dump type, or in the DLE.

I'm assuming that the default will follow my pre-existing amanda.conf with a default 
storage of "daily".


--
---

Chris Hoogendyk

-
   O__   Systems Administrator
  c/ /'_ --- Biology & Geosciences Departments
 (*) \(*) -- 315 Morrill Science Center
~~ - University of Massachusetts, Amherst



---

Erdös 4