Thank you!
I'll see how things run tonight. Unfortunately, I have one researcher who has been shuffling
terabytes of data from one array to another. So, I've also had to chase that down as well, and those
will end up being fulls.
So, for example,
localhost /data/vazeylab/./emr-17-videos /data/vazeylab {
gnutar-data-local
include "./Data/M2-2AFC/EMR-17-/Videos"
} -1
became
localhost /data/vazeylab2/./emr-17-videos /data/vazeylab2 {
gnutar-herbarium
include "./Data/M2-2AFC/EMR-17-/Videos"
} -1
after I chased down that they were moved from one array to another. Then, Amanda sees that as a new
DLE requiring a full. They're transitioning to a new RAID5 array that is made up of 10TB HGST SAS
Helium Drives.
On 8/30/17 2:42 PM, Jean-Louis Martineau wrote:
On 30/08/17 02:14 PM, Chris Hoogendyk wrote:
Thank you, Jean-Louis!
I made all the changes you recommended.
Then I got:
amanda@marlin:~/daily$ amcheck daily
'/usr/local/etc/amanda/daily/amanda.conf', line 68: configuration keyword
expected
parse error: storage 'daily' is not defined
parse error: storage 'daily' is not defined
ERROR: errors processing config file
where line 68-70 of my amanda.conf is
dump-selection "daily" ALL
storage "daily" "research"
The man pages for amanda.conf list dump-selection only in the storage section and not among the
global parameters. So, I constructed a storage section for daily analogous to the storage section
for research.
You did the right things.
After that, I got `amcheck daily` to work:
amanda@marlin:~/daily$ amcheck daily
Amanda Tape Server Host Check
-----------------------------
NOTE: Holding disk '/amanda3': 2742194176 KB disk space available, using
2637336576 KB
NOTE: Holding disk '/amanda4': 2055245824 KB disk space available, using
1950388224 KB
slot 9:no drives available
slot 10:no drives available
slot 11:no drives available
slot 12:no drives available
slot 13:no drives available
slot 14:no drives available
ERROR: no drives available
slot 1: volume 'Bio-Research-001'
Will write to volume 'Bio-Research-001' in slot 1.
NOTE: skipping tape-writable test
Server check took 83.762 seconds
Amanda Backup Client Hosts Check
--------------------------------
Client check: 4 hosts checked in 5.501 seconds. 0 problems found.
(brought to you by Amanda 3.4.5)
Since Amanda is still running, it can't access the LTO6. I'm puzzled that it seems to check those
slots (although too quickly to actually be looking at the tapes) when Amanda is running. Version
3.3.9 would tell you that amdump seemed to be running and would skip the tape checks. Anyway,
this is good in that it shows the LTO7 tape library and says it will write to the first tape there.
3.4 can run many programs in parallel
Since I have an explicit storage section for daily now, is it going to treat that as different
from the previous default daily? Will the dumps continue to reference the full and incremental
dumps that have been run previously? A similar question for the LTO7 library and the storage
"research". Since that is new, when I move a DLE from "daily" to "research" will it automatically
begin with a full? That would mean that I should be cautious about mass moving DLEs.
Amanda will not do full from moved DLES.
Scheduling doesn't look at which storage the backups are.
That also brings on another question. While I have a global settings for tapecycle, at the moment
I have fewer tapes for LTO7; but, the parameter tapecycle seems to be a global and not a storage
section parameter. If I try to put it in the storage section, amcheck gives me a complaint
"configuration keyword expected". If I want to tweak things for the two storages differently, am
I limited in the ways that the man page implies? How can I tell it that there are fewer tapes in
the cycle for the LTO7 library?
tapecycle is the old way, look at the POLICY section in the amanda.conf man page, you can have a
different policy for each storage.
Jean-Louis
On 8/30/17 12:14 PM, Jean-Louis Martineau wrote:
On 30/08/17 11:55 AM, Chris Hoogendyk wrote:
Not quite there yet.
I think I got the tapes labeled properly. I can see that the parameters that got invoked on my
previous attempt with the "-o tpchanger=" were not correct. Thus, an error in block size on the
re labeling. But I think I at least understand how that happened.
amanda@marlin:~/daily$ amlabel -f -o storage=research daily
"Bio-Research-001" slot 1
Reading label...
Error reading volume label: Error reading Amanda header: block size too
small.
Writing label 'Bio-Research-001'...
Checking label...
Success!
resulted in a line in the tapelist as follows:
0 Bio-Research-001 reuse BARCODE:000001L7 BLOCKSIZE:512 POOL:daily
STORAGE:research
CONFIG:daily
In the research storage section, you should add:
tapepool "research"
to tell amanda that the research storage can use any tape in the research pool.
Then
amanda@marlin:~/daily$ amlabel -f --pool research --assign daily
"Bio-Research-[0-9][0-9][0-9]"
Setting Bio-Research-001
changed it to
0 Bio-Research-001 reuse BARCODE:000001L7 BLOCKSIZE:512 POOL:research
STORAGE:research
CONFIG:daily
But, `amcheck daily` did not make any mention of the research storage or tapes
amanda@marlin:~/daily$ amcheck daily
Amanda Tape Server Host Check
-----------------------------
NOTE: Holding disk '/amanda3': 3321401344 KB disk space available, using
3216543744 KB
NOTE: Holding disk '/amanda4': 643121152 KB disk space available, using
538263552 KB
slot 8: volume 'Bio-Daily-010'
Will write to volume 'Bio-Daily-010' in slot 8.
NOTE: skipping tape-writable test
Server check took 58.426 seconds
Amanda Backup Client Hosts Check
--------------------------------
Client check: 4 hosts checked in 4.930 seconds. 3 problems found.
(brought to you by Amanda 3.4.5)
You must tell amanda which storage to use, in amanda.conf:
storage "daily" "research"
to tell it to use both storage
And, when Amanda ran last night, it appears to be running according to the previous normal
without any effects from my changes in the configuration and disklist. The backups are going to
the LTO6 tape library in the default "daily" storage.
amanda@marlin:~/daily$ amstatus daily
Using: /usr/local/etc/amanda/daily/log/amdump
From Tue Aug 29 23:30:01 EDT 2017
<snip>
SUMMARY dle real estimated
size size
---------------- ---- --------- ---------
disk : 158
estimated : 144 2459174380k
flush : 140 3043619315k
dump failed : 4 138k ( 0.00%)
wait for dumping: 15 1661177930k ( 0.00%)
dumping to tape : 0 0k 0k ( 0.00%) ( 0.00%)
dumping : 2 0k 564952671k ( 0.00%) ( 0.00%)
dumped : 123 301938738k 233043641k (129.56%) ( 12.28%)
wait for writing: 123 301938738k 233043641k (129.56%) ( 12.28%)
wait to flush : 19 492421514k 492421514k (100.00%) ( 20.02%)
writing to tape
dumping to tape
failed to tape : 1 232156245k 232156245k (100.00%) ( 9.44%)
taped : 120 2319041556k 0k ( 0.00%) ( 94.30%)
tape 1 : 121 2319041504k 2319041504k ( 94.27%) Bio-Daily-010 (120
parts)
8 dumpers idle : no-bandwidth
daily qlen: 143
0: tape error: No space left on device, splitting not enabled
(localhost:/data/vazeylab/./Windows)
<snip>
I've attached my amanda.conf and disklist so that you can see what I've done. I put the tag for
the storage "research" in the DLEs. I started with the herbarium entries, and expect to
continue through many others once it works.
Since it appears that on this system (with its rapidly escalating disk array space) compression
is the bottleneck; I decided to try letting the LTO7 do it's own compression. Thus the
herbarium dump definition that I'm using has no compression.
The daily storage have no dump-selection setting, so all dles (even those tagged Research) will
go on the daily storages.
The dle tagged Research will be written to both storages.
You need to add a dump-selection for the daily storage and add a corresponding tag to all dles
that must go in that storage, you should add it in a dumptype inherited from all dles.
Jean-Louis
On 8/29/17 4:21 PM, Jean-Louis Martineau wrote:
On 29/08/17 02:37 PM, Chris Hoogendyk wrote:
Thank you! I've got that set up now.
I tried labeling tapes. I thought it was working. e.g.:
You should never use the '-o tpchanger=' argument, it is always better to use
'-ostorage='
That will set the storage and the pool correctly.
amanda@marlin:~/daily$ amlabel -f -o tpchanger=NEO200x48 -o
'labelstr="^Bio-Research-[0-9][0-9][0-9]*$"' daily "Bio-Research-001" slot 1
Reading label...
Found label 'amtapetype-206980288' but it is not in the tapelist file.
Writing label 'Bio-Research-001'...
Checking label...
Success!
After that (which I repeated for several tapes), I tried
amanda@marlin:~/daily$ amlabel --storage research --assign daily
"Bio-Research-[0-9][0-9][0-9]"
Bio-Research-001: Can't assign storage without force, old storage is 'daily'
[etc. for each of the several tapes]
So, adding -f
amanda@marlin:~/daily$ amlabel -f --storage research --assign daily
"Bio-Research-[0-9][0-9][0-9]"
Bio-Research-001: Can't assign storage because it is a new labelled tape.
[etc. for each of the several tapes]
All labels already correctly set.
Changing the storage is useless, but we could allow amlabel to change it.
And looking at the tapelist, I see entries like:
0 Bio-Research-001 reuse BARCODE:000001L7 BLOCKSIZE:2048 POOL:daily
STORAGE:daily CONFIG:daily
A label is not bound to a storage, the listed storage is the latest storage
that used that tape,
A tape is bound to a pool, and it can be in any storage that use that pool.
Have I done something wrong? Or is that going to work, because it is a blank tape. In other
words,
since it is only accessible via tpchanger=NEO200x48, which is in the definition
of the storage
research; will it be accessed by those backups with the tag research and then
re listed in the
tapelist as belonging to storage research?
I also wasn't finding much detail in the man pages on the implications of pools. I don't have
any
such designations in my amanda.conf.
A pool is just a name, a storage will use any tape that is in the same pool.
Jean-Louis
On 8/29/17 6:45 AM, Jean-Louis Martineau wrote:
> Chris,
>
> Everything looks good except the 'define changer' should be before the
> 'define storage'.
>
> define changer NEO200x48 {
> tpchanger
> "chg-robot:/dev/tape/by-id/scsi-1BDT_FlexStor_II_00MX64200626_LL0" #
> your changer device file (T48)
> property "tape-device" "0=tape:/dev/nst1" # your tape
> drive device file
> property append "tape-device" "1=tape:/dev/nst2" # your
> tape drive device file
> property "use-slots" "1-11" # UPDATE WHEN TAPES
> ADDED <<<<<<<<<<<<<<<<<<<<<
> }
>
> define storage research {
> tpchanger "NEO200x48"
> tapetype LTO7
> labelstr "^Bio-Research-[0-9][0-9][0-9]*$"
> amrecover_changer "NEO200x48"
> dump-selection "Research" ALL
> }
>
> Jean-Louis
>
> On 28/08/17 05:46 PM, Chris Hoogendyk wrote:
> > I haven't found any examples of Amanda configurations using storage
> > definitions. Having read through the man pages, I've come up with a
> > modification of my amanda.conf that I think should work, and I'm
> > looking for comments and feedback.
> >
> > I had temporarily set up a separate Amanda configuration to test my
> > new LTO7 tape library. That seemed to work, and I was able to generate
> > a typetype for LTO7. Now I want to fold it into my regular daily
> > amanda.conf and have that use both my current LTO6 library and the new
> > LTO7.
> >
> > I've added a section as follows:
> >
> > define storage research {
> >
> > define changer NEO200x48 {
> > tpchanger
> > "chg-robot:/dev/tape/by-id/scsi-1BDT_FlexStor_II_00MX64200626_LL0" # your
> > changer device file (T48)
> > property "tape-device" "0=tape:/dev/nst1" # your tape
> > drive device file
> > property append "tape-device" "1=tape:/dev/nst2" # your
> > tape drive device file
> > property "use-slots" "1-11" # UPDATE WHEN TAPES
> > ADDED <<<<<<<<<<<<<<<<<<<<<
> > }
> > tpchanger "NEO200x48"
> >
> > tapetype LTO7
> >
> > labelstr "^Bio-Research-[0-9][0-9][0-9]*$"
> >
> > amrecover_changer "NEO200x48"
> >
> > dump-selection "Research" ALL
> >
> > }
> >
> > immediately after those same settings in the daily amanda.conf.
> >
> > I'm thinking with that in the configuration, I will be able to label
> > tapes by, e.g.
> >
> > amlabel --storage "Research" daily "Bio-Research-012"
> >
> > Then I should be able to shift DLEs from the daily to the Research
> > storage by putting a
> >
> > tag "Research"
> >
> > in the dump type, or in the DLE.
> >
> > I'm assuming that the default will follow my pre-existing amanda.conf
> > with a default storage of "daily".
> >
> >
>
>
> *Disclaimer*
>
> This message is the property of *CARBONITE, INC.* <http://www.carbonite.com>
and may contain
> confidential or privileged information.
>
> If this message has been delivered to you by mistake, then do not copy or deliver this
message to
> anyone. Instead, destroy it and notify me by reply e-mail.
>
--
---------------
Chris Hoogendyk
-
O__ ---- Systems Administrator
c/ /'_ --- Biology & Geosciences Departments
(*) \(*) -- 315 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst
<hoogen...@bio.umass.edu>
---------------
Erdös 4
*Disclaimer*
This message is the property of *CARBONITE, INC.* <http://www.carbonite.com> and may contain
confidential or privileged information.
If this message has been delivered to you by mistake, then do not copy or deliver this message
to anyone. Instead, destroy it and notify me by reply e-mail.
--
---------------
Chris Hoogendyk
-
O__ ---- Systems Administrator
c/ /'_ --- Biology & Geosciences Departments
(*) \(*) -- 315 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst
<hoogen...@bio.umass.edu>
---------------
Erdös 4
*Disclaimer*
This message is the property of *CARBONITE, INC.* <http://www.carbonite.com> and may contain
confidential or privileged information.
If this message has been delivered to you by mistake, then do not copy or deliver this message
to anyone. Instead, destroy it and notify me by reply e-mail.
--
---------------
Chris Hoogendyk
-
O__ ---- Systems Administrator
c/ /'_ --- Biology & Geosciences Departments
(*) \(*) -- 315 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst
<hoogen...@bio.umass.edu>
---------------
Erdös 4
*Disclaimer*
This message is the property of *CARBONITE, INC.* <http://www.carbonite.com> and may contain
confidential or privileged information.
If this message has been delivered to you by mistake, then do not copy or deliver this message to
anyone. Instead, destroy it and notify me by reply e-mail.
--
---------------
Chris Hoogendyk
-
O__ ---- Systems Administrator
c/ /'_ --- Biology & Geosciences Departments
(*) \(*) -- 315 Morrill Science Center
~~~~~~~~~~ - University of Massachusetts, Amherst
<hoogen...@bio.umass.edu>
---------------
Erdös 4
