Re: tapes and external disks: how to combine all this

[sgw]

Am 2016-08-16 um 21:59 schrieb Stefan G. Weichinger:

Am 2016-08-16 um 18:23 schrieb Jean-Louis Martineau:

Stefan,

Try the attached patch


yep, works!


could someone point me at that patch again, pls? can't find it in my ML 
archives here.


(got to patch an existing 3.3.9 installation)

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-11-09 um 19:19 schrieb Uwe Menges:

>> development seems to be only done by JLM, and zmanda.com blog entries
>> end 2013
> 
> The IRC channel's topic did still show 3.3.9 as current version 13 days
> after 3.4 was released on the web page (was changed then).
> 
> So this all together indicates that there's probably not enough
> resources at amanda/zmanda for keeping up? I'd be grateful for a more
> "official" statement on the situation (best from people actively working
> on amanda/zmanda).

I didn't even *notice* amanda-3.4 until now (can't find a release
message in my ml-folder) !

The Release Notes show quite some interesting changes, I also wonder if
3.4 fixes the issues with newer samba-releases (using smbclient).

Sounds like I need to write a gentoo ebuild asap and check things.

I didn't have time yet: are the new amanda.conf sections documented
anywhere? the parameters "retention-*" sound like rather big changes.

Stefan

Re: tapes and external disks: how to combine all this

[Uwe Menges]

On 11/09/16 15:57, Stefan G. Weichinger wrote:
> Is there any "mind for involving the community" in your opinion?

I have no idea, and was hoping to get some insights from people closer
related to the amanda/zmanda people. I thought maybe you know more
because of the @amanda.org address.

There recently was a question in the IRC channel:
> [...]is amanda dying?  The chat room is nearly empty.  The mailing
> lisst is a ghost list, the forum isn't sending e-mails, and the
> official builds don't run on "supported" platforms..

This is what I replied:
> might be.. the platforms are partly very outdated, too (eg. Fedora:
> current=24, latest amanda=21; Ubuntu: current=16.04, latest
> amanda=14.10)

> also, I couldn't register a wiki account, and didn't get any reply to
> that, neither here in the chan nor on the amanda-users mailing list

> development seems to be only done by JLM, and zmanda.com blog entries
> end 2013

The IRC channel's topic did still show 3.3.9 as current version 13 days
after 3.4 was released on the web page (was changed then).

So this all together indicates that there's probably not enough
resources at amanda/zmanda for keeping up? I'd be grateful for a more
"official" statement on the situation (best from people actively working
on amanda/zmanda).

Yours, Uwe

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

(reply to list now)

Am 2016-11-09 um 15:10 schrieb Gene Heskett:
> On Wednesday 09 November 2016 08:40:13 Uwe Menges wrote:
> 
>> On 11/08/16 20:35, Stefan G. Weichinger wrote:
>>> started a small markdown file with a quick howto anyone
>>> interested to check and contribute?
>> 
>> The usual way would be to upload the markdown to a wiki, and send
>> a link to mailing lists or interested people.

Sure, just wanted to do a first ping.

I will upload it to my github-account where the markdown gets displayed
nicely. Just let me do the next step of editing first, I only started
the first draft yesterday.

>> After my last experience with the amanda wiki (not reacting on 
>> registration attempts), I'm not sure what path the remaining
>> amanda staff has in mind for collaborating, or involving community.
>> Maybe you can shed some light on that, as you have an @amanda.org
>> e-mail address?
>> 
>> Yours, Uwe
> 
> I'll second this, there is zero response to any attempts at
> registration.

I don't have any admin access to the wiki or know who the responsible
admin is right now.

Is there any "mind for involving the community" in your opinion?

stefan

ps: my amanda.org address comes from years ago when I did much work
around the man-pages (docbook XML conversion)

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-11-09 um 15:10 schrieb Gene Heskett:
> On Wednesday 09 November 2016 08:40:13 Uwe Menges wrote:
> 
>> On 11/08/16 20:35, Stefan G. Weichinger wrote:
>>> started a small markdown file with a quick howto
>>> anyone interested to check and contribute?
>>
>> The usual way would be to upload the markdown to a wiki, and send a
>> link to mailing lists or interested people.

Sure, just wanted to do a first ping.

I will upload it to my github-account where the markdown gets displayed
nicely. Just let me do the next step of editing first, I only started
the first draft yesterday.

>> After my last experience with the amanda wiki (not reacting on
>> registration attempts), I'm not sure what path the remaining amanda
>> staff has in mind for collaborating, or involving community. Maybe you
>> can shed some light on that, as you have an @amanda.org e-mail
>> address?
>>
>> Yours, Uwe
> 
> I'll second this, there is zero response to any attempts at registration.

I don't have any admin access to the wiki or know who the responsible
admin is right now.

Is there any "mind for involving the community" in your opinion?

stefan

ps: my amanda.org address comes from years ago when I did much work
around the man-pages (docbook XML conversion)

Re: tapes and external disks: how to combine all this

[Uwe Menges]

On 11/08/16 20:35, Stefan G. Weichinger wrote:
> started a small markdown file with a quick howto
> anyone interested to check and contribute?

The usual way would be to upload the markdown to a wiki, and send a link
to mailing lists or interested people.

After my last experience with the amanda wiki (not reacting on
registration attempts), I'm not sure what path the remaining amanda
staff has in mind for collaborating, or involving community. Maybe you
can shed some light on that, as you have an @amanda.org e-mail address?

Yours, Uwe

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-08-21 um 10:35 schrieb Stefan G. Weichinger:

> Maybe I find the time for a quick howto.txt
> Thanks, Stefan

started a small markdown file with a quick howto
anyone interested to check and contribute?

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-08-19 um 17:14 schrieb Jean-Louis Martineau:

On 16/08/16 03:59 PM, Stefan G. Weichinger wrote:

Am 2016-08-16 um 18:23 schrieb Jean-Louis Martineau:

Stefan,

Try the attached patch


yep, works!



Thanks, I committed the patch


So it will go into 3.3.10 or so?
Maybe I find the time for a quick howto.txt
Thanks, Stefan

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-08-16 um 18:23 schrieb Jean-Louis Martineau:

Stefan,

Try the attached patch


yep, works!

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-08-04 um 18:51 schrieb Jean-Louis Martineau:


The aggregate.pm I sent you change the behavior, which will break many
sites using it.
The new behavior should be enabled with a property.

Try the attached aggregate.pm and add the following property in the
aggregate changer section:
  property "allow-missing-changer" "yes"

The patch will go in 3.3 once it is completed and tested.
It will be helpful if you can write a howto.


I think it does not yet work as intended.

I applied that file and added the property.
With only one external disk attached I get:


$ amcheck -t vtape
Amanda Tape Server Host Check
-
mount: can't find UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff
mount: can't find UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff
ERROR: disk1: No removable disk mounted on '/mnt/externaldisk1'

it does not find or use the vtapes on disk2
even when I load a specific slot first:

$ amtape vtape slot 1:1
mount: UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff kann nicht gefunden werden
slot 1:1: time 20160714105337 label vtape-002-001
changed to slot 1:1

amanda@backup ~ $ amcheck -t vtape
Amanda Tape Server Host Check
-
mount: can't find UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff
mount: can't find UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff
ERROR: disk1: No removable disk mounted on '/mnt/externaldisk1'

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-08-04 um 18:51 schrieb Jean-Louis Martineau:
> 
> The aggregate.pm I sent you change the behavior, which will break many
> sites using it.
> The new behavior should be enabled with a property.
> 
> Try the attached aggregate.pm and add the following property in the
> aggregate changer section:
>   property "allow-missing-changer" "yes"
> 
> The patch will go in 3.3 once it is completed and tested.
> It will be helpful if you can write a howto.

been on vacation ... will test etc in upcoming week

Re: tapes and external disks: how to combine all this

[Uwe Menges]

On 07/15/16 19:43, Stefan G. Weichinger wrote:
> Thanks for providing your script and setup.
> You should write a small howto around this and add it to the amanda wiki.

I tried to get a user account for the amanda wiki on 2016-07-19.

It requires to submit a biography, which I think is not an appropriate
requirement. It also tells "Your biography must be at least be 50 words
long.". So I wrote 50+ words which don't violate my personal privacy
policy.

Up to now, I got no reply.

AFAICS, I cannot add or edit amanda wiki pages without an account.

Yours, Uwe

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-22 um 18:03 schrieb Ochressandro Rettinger:

On the subject of changing the taperscan, how do you do that?  I tried 
seeing if that would have an effect on my chg-rait problem by adding the line

taperscan oldest

to my amanda.conf, but when I ran amcheck afterwards, Amanda complained:

amcheck: "/etc/amanda/NMHPVPR/amanda.conf", line 11: Unknown taperscan named: 
oldest

but I couldn't find any documentation that explicitly said how to 
change the taperscan algorithm.



I mentioned it in an earlier posting.
Try it like this:

define taperscan "oldest" {
plugin "oldest"
}

taperscan "oldest"

RE: tapes and external disks: how to combine all this

[Ochressandro Rettinger]

On the subject of changing the taperscan, how do you do that?  I tried 
seeing if that would have an effect on my chg-rait problem by adding the line

taperscan oldest

to my amanda.conf, but when I ran amcheck afterwards, Amanda 
complained: 

amcheck: "/etc/amanda/NMHPVPR/amanda.conf", line 11: Unknown taperscan named: 
oldest

but I couldn't find any documentation that explicitly said how to 
change the taperscan algorithm.

-Sandro

-Original Message-
From: owner-amanda-us...@amanda.org [mailto:owner-amanda-us...@amanda.org] On 
Behalf Of Stefan G. Weichinger
Sent: Friday, July 22, 2016 4:52 AM
To: Jean-Louis Martineau <jmartin...@carbonite.com>; amanda-users@amanda.org
Subject: Re: tapes and external disks: how to combine all this

Am 2016-07-20 um 19:09 schrieb Jean-Louis Martineau:
> May I suggest: autolabel "$m-$3s" any
>
> Install the new aggregate.pm and apply the Changer.diff to the 
> Changer.pm file.
>
> Your problem is because the 'traditional' taperscan fail when it try 
> to load the current slot.
> The workaround is to use the 'oldest' or 'lexical' taperscan.
>
> The meta label for a disk should never change (unless the state file 
> is removed), there is a bug somewhere if it happen again.
>

Jean-Louis,

what are your further plans with this?
Will you add the mentioned property?
Will it get into amanda-3.3.10?
Should I write a small howto for the wiki or so?

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-20 um 19:09 schrieb Jean-Louis Martineau:

May I suggest: autolabel "$m-$3s" any

Install the new aggregate.pm and apply the Changer.diff to the
Changer.pm file.

Your problem is because the 'traditional' taperscan fail when it try to
load the current slot.
The workaround is to use the 'oldest' or 'lexical' taperscan.

The meta label for a disk should never change (unless the state file is
removed), there is a bug somewhere if it happen again.



Jean-Louis,

what are your further plans with this?
Will you add the mentioned property?
Will it get into amanda-3.3.10?
Should I write a small howto for the wiki or so?

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-20 um 19:09 schrieb Jean-Louis Martineau:
> May I suggest: autolabel "$m-$3s" any

ok, edited

> Install the new aggregate.pm and apply the Changer.diff to the
> Changer.pm file.

did so, amcheck looks even better now:

$ amcheck -s vtape
Amanda Tape Server Host Check
-
Holding disk /mnt/amhold/vtape: 18 GB disk space available, using 17 GB
mount: can't find UUID=fae458a3-da30-43c5-81f1-be2d49215b09
slot 0:7: contains an empty volume
Will write label 'vtape-005-007' to new volume in slot 0:7.
NOTE: skipping tape-writable test
Server check took 1.659 seconds


> Your problem is because the 'traditional' taperscan fail when it try to
> load the current slot.
> The workaround is to use the 'oldest' or 'lexical' taperscan.

Did so:

define taperscan "lexi" {
plugin "lexical"
}

taperscan "lexi"

amcheck still OK, will test with other external disk

> The meta label for a disk should never change (unless the state file is
> removed), there is a bug somewhere if it happen again.

Understood.

Thanks for your patches, looks good now!
I will now let the people there swap the external disks and see what
happens.

Stefan

Re: tapes and external disks: how to combine all this

[Debra S Baddorf]

> On Jul 20, 2016, at 8:51 AM, Stefan G. Weichinger  wrote:
> 
> Am 2016-07-19 um 22:34 schrieb Debra S Baddorf:
> 
>>> Opinions, anyone?
>> 
>> Wouldn’t that just be a change in the  RUNTAPES parameter?
> 
> Shouldn't that be the tapecycle parameter ... ?
> 


OOOPS-  yes it should be TAPECYCLE.
Sorry and thanks!
Deb Baddorf

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-20 um 16:55 schrieb Jean-Louis Martineau:
> Nest time,cut the error,so I don't have to reread all previous
> post on the threads to find it.

ok, sorry

> I see a different problem:
> 
> Will write label 'vtape-005-001' to new volume in slot 0:6.
> 
> I think the label should be 'vtape-001-006'.
> What is in the changerfile? It should have the meta label which should
> be 'vtape-001'.
> If you don't set a changerfile, it is the 'state' file in the disk
> directory.

The relevant parts of amanda.conf (I think):

autolabel "$m-%%%" any
meta-autolabel "$c-%%%"

define changer disk1 {
tpchanger "chg-disk:/mnt/externaldisk1"
property "num_slot" "20"
property "auto-create-slot" "yes"
property "removable" "yes"
property "MOUNT" "yes"
property "UMOUNT" "yes"
property "UMOUNT-LOCKFILE"
"/etc/amanda/vtape/externaldisk1.lock"
property "UMOUNT-DELAY" "1"
}

define changer disk2 {
tpchanger "chg-disk:/mnt/externaldisk2"
property "num_slot" "20"
property "auto-create-slot" "yes"
property "removable" "yes"
property "MOUNT" "yes"
property "UMOUNT" "yes"
property "UMOUNT-LOCKFILE"
"/etc/amanda/vtape/externaldisk2.lock"
property "UMOUNT-DELAY" "1"
}

define changer aggregate {
tpchanger "chg-aggregate:{disk1,disk2}"
property "state_filename"
"/etc/amanda/vtape/aggregate.stats"
}

tpchanger "aggregate"

columnspec "OrigKB=1:9,OutKB=1:9,TapeRate=1:6,DumpRate=1:6"

tapetype HARD-DISK
labelstr "vtape-[0-9]*"

--

changerfile not set so far, so it is:

$ cat /mnt/externaldisk1/state
$STATE = {
   'meta' => 'vtape-005',
   'drives' => {
 '/mnt/externaldisk1/drive0' => {}
   }
 };


Second disk not attached now, as mentioned.

--

# amtape vtape inventory

mount: UUID=fae458a3-da30-43c5-81f1-be2d49215b09 kann nicht gefunden werden
mount: UUID=fae458a3-da30-43c5-81f1-be2d49215b09 kann nicht gefunden werden

slot 0:1: label vtape-001-001 (vtape-001)
slot 0:2: label vtape-001-002 (vtape-001)
slot 0:3: label vtape-001-003 (vtape-001)
slot 0:4: label vtape-001-004 (vtape-001)
slot 0:5: label vtape-001-005 (vtape-001)
slot 0:6: label vtape-005-001 (vtape-005)
slot 0:7: blank (current)
[..]

OK?
Maybe the new aggregate.pm does things different?

The first 5 vtapes (vtape-001-*) were labelled with the aggregate.pm
from the stable 3.3.9 tree. I can relabel if needed, this is still a
testing setup.

(got to leave the office now, will check back later)

Stefan

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-20 um 16:30 schrieb Jean-Louis Martineau:
> What's wrong with amcheck? Output looks good to me.

amcheck did not find the next vtape itself, I had to load the slot manually.

This should work on its own when users there plug disks in and out of
the backup server.

I wish no manual intervention on the shell in this case: some office
workers get the job to exchange the external usb disk every friday or so
and amanda should detect that on "her" own :-)

Yes? Thanks!

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-19 um 22:34 schrieb Debra S Baddorf:

>> Opinions, anyone?
> 
> Wouldn’t that just be a change in the  RUNTAPES parameter?

Shouldn't that be the tapecycle parameter ... ?

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-20 um 15:37 schrieb Jean-Louis Martineau:
> Stefan,
> 
> Try this file.

Did so, thanks!

Unfortunately: no change at all in plain amcheck.

But I could do this:

$ amtape vtape inventory
mount: UUID=fae458a3-da30-43c5-81f1-be2d49215b09 kann nicht gefunden werden
mount: UUID=fae458a3-da30-43c5-81f1-be2d49215b09 kann nicht gefunden werden
slot 0:1: label vtape-001-001 (vtape-001)
slot 0:2: label vtape-001-002 (vtape-001)
slot 0:3: label vtape-001-003 (vtape-001)
slot 0:4: label vtape-001-004 (vtape-001)
slot 0:5: label vtape-001-005 (vtape-001)
slot 0:6: blank
slot 0:7: blank
slot 0:8: blank
slot 0:9: blank
slot 0:10: blank
slot 0:11: blank
slot 0:12: blank
slot 0:13: blank
slot 0:14: blank
slot 0:15: blank
slot 0:16: blank
slot 0:17: blank
slot 0:18: blank
slot 0:19: blank
slot 0:20: blank

amanda@backup ~ $ amtape vtape slot 0:6
mount: UUID=fae458a3-da30-43c5-81f1-be2d49215b09 kann nicht gefunden werden
slot 0:6: Could not open device: File 0 not found
changed to slot 0:6

amanda@backup ~ $ amcheck vtape
Amanda Tape Server Host Check
-
Holding disk /mnt/amhold/vtape: 53 GB disk space available, using 52 GB
mount: can't find UUID=fae458a3-da30-43c5-81f1-be2d49215b09
slot 0:6: contains an empty volume
Will write label 'vtape-005-001' to new volume in slot 0:6.
NOTE: skipping tape-writable test
Server check took 1.371 seconds

Amanda Backup Client Hosts Check

Client check: 2 hosts checked in 1.153 seconds.  0 problems found.

(brought to you by Amanda 3.3.9)


So, yes, a step ahead!
Thanks so far, Stefan

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 19.07.2016 um 22:34 schrieb Debra S Baddorf:
> 
>> On Jul 19, 2016, at 2:18 PM, Stefan G. Weichinger 
>> wrote:
>> 
>> Am 19.07.2016 um 20:56 schrieb Jean-Louis Martineau:
>>> Stefan,
>>> 
>>> chg-aggregate fail if one of the changer fail, it is the way it
>>> is designed. Both disk must be mounted to works.
>> 
>> I understood that differently from your mail then. My use-case
>> would be more of "use one of the changers", yes.
>> 
>>> We could add a property so that it use all the available child
>>> changer, eg. fail only if all child fail.
>> 
>> Yes! Sounds great to me. And as we are at wishes:
>> 
>> it would be great to have a parameter like:
>> 
>> use_each_changer_for_number_of_X_runs
>> 
>> although I have to think about that.
>> 
>> If the external disks don't get changed (the admin forgets to
>> unplug disk1 and plug in disk2) the dumps should not FAIL. It
>> should WARN in the report mail but continue to rotate through
>> vtapes on disk1.
>> 
>> Opinions, anyone?
> 
> Wouldn’t that just be a change in the  RUNTAPES parameter? It will
> start re-using only after it exceeds that parameter.  BUT  if you
> have more tapes available than that parameter,  it uses all of them
> before beginning to re-use.   (At least, with real tapes.)
> 
> So if I want it CAPABLE of re-using tapes,  I have to set the
> RUNTAPES param to the value of the minimum I insist it keeps before
> overwriting.I have always been able to have MORE than that number
> of tapes,  and it will use them all before re-writing,  if they are
> inserted.

very good point, Debra, thanks!

Re: tapes and external disks: how to combine all this

[Debra S Baddorf]

> On Jul 19, 2016, at 2:18 PM, Stefan G. Weichinger  wrote:
> 
> Am 19.07.2016 um 20:56 schrieb Jean-Louis Martineau:
>> Stefan,
>> 
>> chg-aggregate fail if one of the changer fail, it is the way it is
>> designed.
>> Both disk must be mounted to works.
> 
> I understood that differently from your mail then.
> My use-case would be more of "use one of the changers", yes.
> 
>> We could add a property so that it use all the available child changer,
>> eg. fail only if all child fail.
> 
> Yes! Sounds great to me.
> And as we are at wishes:
> 
> it would be great to have a parameter like:
> 
> use_each_changer_for_number_of_X_runs
> 
> although I have to think about that.
> 
> If the external disks don't get changed (the admin forgets to unplug
> disk1 and plug in disk2) the dumps should not FAIL. It should WARN in
> the report mail but continue to rotate through vtapes on disk1.
> 
> Opinions, anyone?

Wouldn’t that just be a change in the  RUNTAPES parameter?
It will start re-using only after it exceeds that parameter.  BUT  if you have 
more tapes available than
that parameter,  it uses all of them before beginning to re-use.   (At least, 
with real tapes.)

So if I want it CAPABLE of re-using tapes,  I have to set the RUNTAPES param to 
the value of the minimum
I insist it keeps before overwriting.I have always been able to have MORE 
than that number of tapes,  and
it will use them all before re-writing,  if they are inserted.

Deb Baddorf


> 
>> For testing, you can try to remove all
>>   if (grep { defined($_->[0]) } @$kid_results) {
>> ...
>>   }
>> block of code
> 
> ... in /usr/lib/perl5/vendor_perl/5.20.2/Amanda/Changer/aggregate.pm, right?
> 
> Thanks, I will test asap (=tomorrow)
> 
> 

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 19.07.2016 um 21:46 schrieb Jean-Louis Martineau:
> On 19/07/16 03:34 PM, Stefan G. Weichinger wrote:
>> Am 19.07.2016 um 20:56 schrieb Jean-Louis Martineau:
>>
>>> We could add a property so that it use all the available child changer,
>>> eg. fail only if all child fail.
>>>
>>> For testing, you can try to remove all
>>> if (grep { defined($_->[0]) } @$kid_results) {
>>>   ...
>>> }
>>> block of code

> Remove the complete if block, not only the if line.

ok, did that. Doesn't work as intended.

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 19.07.2016 um 20:56 schrieb Jean-Louis Martineau:

> We could add a property so that it use all the available child changer,
> eg. fail only if all child fail.
> 
> For testing, you can try to remove all
>if (grep { defined($_->[0]) } @$kid_results) {
>  ...
>}
> block of code

JL, I was curious and tried that right now.
Did not work yet, amcheck fails the same way.




--- aggregate.pm.orig   2016-07-19 21:28:05.0 +0200
+++ aggregate.pm2016-07-19 21:30:58.0 +0200
@@ -288,7 +288,6 @@
 my $check_and_report_errors = sub {
my ($kid_results) = @_;

-   if (grep { defined($_->[0]) } @$kid_results) {
# we have errors, so collect them and make a "combined" error.
my @annotated_errs;
my @err_slots;
@@ -299,7 +298,6 @@
[ $self->{'child_names'}[$i], $kr->[0] ];
push @err_slots, $kr->[0]->{'slot'}
if (defined $kr->[0] and defined $kr->[0]->{'slot'});
-   }

my @slotarg;
if (@err_slots == $self->{'num_children'}) {
@@ -413,7 +411,6 @@

my $all_kids_done_cb = sub {
my ($kid_results) = @_;
-   if (grep { defined($_->[0]) } @$kid_results) {
# we have errors, so collect them and make a "combined"
error.
my @annotated_errs;
for my $i (0 .. $self->{'num_children'}-1) {
@@ -421,7 +418,6 @@
next unless defined($kr->[0]);
push @annotated_errs,
[ $self->{'child_names'}[$i], $kr->[0] ];
-   }
$self->make_combined_error(
$params{'finished_cb'}, [ @annotated_errs ]);
return 1;
@@ -496,7 +492,6 @@

 my $all_kids_done_cb = sub {
my ($kid_results) = @_;
-   if (grep { defined($_->[0]) } @$kid_results) {
# we have errors, so collect them and make a "combined" error.
my @annotated_errs;
for my $i (0 .. $self->{'num_children'}-1) {
@@ -504,7 +499,6 @@
next unless defined($kr->[0]);
push @annotated_errs,
[ $self->{'child_names'}[$i], $kr->[0] ];
-   }
$self->make_combined_error(
$params{'finished_cb'}, [ @annotated_errs ]);
return 1;
@@ -555,7 +549,6 @@

 step all_kids_done_cb => sub {
my ($kid_results) = @_;
-   if (grep { defined($_->[0]) } @$kid_results) {
# we have errors, so collect them and make a "combined" error.
my @annotated_errs;
for my $i (0 .. $self->{'num_children'}-1) {
@@ -563,7 +556,6 @@
next unless defined($kr->[0]);
push @annotated_errs,
[ $self->{'child_names'}[$i], $kr->[0] ];
-   }
return $self->make_combined_error(
$params{'inventory_cb'}, [ @annotated_errs ]);
}

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 19.07.2016 um 20:56 schrieb Jean-Louis Martineau:
> Stefan,
> 
> chg-aggregate fail if one of the changer fail, it is the way it is
> designed.
> Both disk must be mounted to works.

I understood that differently from your mail then.
My use-case would be more of "use one of the changers", yes.

> We could add a property so that it use all the available child changer,
> eg. fail only if all child fail.

Yes! Sounds great to me.
And as we are at wishes:

it would be great to have a parameter like:

use_each_changer_for_number_of_X_runs

although I have to think about that.

If the external disks don't get changed (the admin forgets to unplug
disk1 and plug in disk2) the dumps should not FAIL. It should WARN in
the report mail but continue to rotate through vtapes on disk1.

Opinions, anyone?

> For testing, you can try to remove all
>if (grep { defined($_->[0]) } @$kid_results) {
>  ...
>}
> block of code

... in /usr/lib/perl5/vendor_perl/5.20.2/Amanda/Changer/aggregate.pm, right?

Thanks, I will test asap (=tomorrow)

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-17 um 19:03 schrieb Stefan G. Weichinger:


amanda@backup ~ $ amcheck -s vtape
Amanda Tape Server Host Check
-
Holding disk /mnt/amhold/vtape: 64 GB disk space available, using 63 GB
mount: can't find UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff
slot 1:7: contains an empty volume
Will write label 'vtape-004-003' to new volume in slot 1:7.
NOTE: skipping tape-writable test
Server check took 1.188 seconds


another observation:

if the external disk nr. 2 is not attached to the server it even fails 
to find/mount the other one.



$ amcheck vtape
Amanda Tape Server Host Check
-
Holding disk /mnt/amhold/vtape: 7 GB disk space available, using 6 GB
mount: can't find UUID=fae458a3-da30-43c5-81f1-be2d49215b09
mount: can't find UUID=fae458a3-da30-43c5-81f1-be2d49215b09
 No removable disk mounted on '/mnt/externaldisk2'
Taper scan algorithm did not find an acceptable volume.
(expecting a new volume)
ERROR: No acceptable volumes found

Maybe it doesn't step on to check for the second disk? look at the two 
identical lines looking at the same UUID.


fstab for reference:

UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff /mnt/externaldisk1 ext4 
relatime,noauto,user 0 1
UUID=fae458a3-da30-43c5-81f1-be2d49215b09 /mnt/externaldisk2 ext4 
relatime,noauto,user 0 1


/dev/disk/by-uuid/5a5a9927-995f-4f0f-98ff-d222561f84ff is attached right 
now.

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-15 um 16:42 schrieb Stefan G. Weichinger:
> Am 2016-07-15 um 16:25 schrieb Jean-Louis Martineau:
>> Looks like it can;t find the 'first' slot unless you manually load one.
>>
>> The attached path should fix this issue.
>> You can test it by removing the state file.
> 
> patch works for me: great

another issue, maybe my fault, maybe not:

(rebuilt amanda after updating to perl-5.20, still the following message):


$ amcheck vtape
Amanda Tape Server Host Check
-
Holding disk /mnt/amhold/vtape: 64 GB disk space available, using 63 GB
mount: can't find UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff
slot 1:6: volume 'vtape-004-002' is still active and cannot be overwritten
mount: can't find UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff
amcheck-device: Can't use an undefined value as an ARRAY reference at
/usr/lib/perl5/vendor_perl/5.20.2/Amanda/Changer/aggregate.pm line 200.

Server check took 9.673 seconds

This is with one of the 2 external disks mounted (if relevant, I don't
think so), it seems as if the first chg-disk-changer does not load the
next vtape.

This helps (but should not be necessary to be done manually):
amanda@backup ~ $ amtape vtape slot 1:7
mount: UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff kann nicht gefunden werden
slot 1:7: Could not open device: File 0 not found
changed to slot 1:7

amanda@backup ~ $ amcheck -s vtape
Amanda Tape Server Host Check
-
Holding disk /mnt/amhold/vtape: 64 GB disk space available, using 63 GB
mount: can't find UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff
slot 1:7: contains an empty volume
Will write label 'vtape-004-003' to new volume in slot 1:7.
NOTE: skipping tape-writable test
Server check took 1.188 seconds

(brought to you by Amanda 3.3.9)

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-15 um 11:45 schrieb Uwe Menges:

On 07/15/16 04:40, Jon LaBadie wrote:

Each was mounted wherever, but in the same place each time.
What amanda used was a single "changer" dir of "slots" that
were symbolic links to the vtape in that slot on its
corresponding disk.


This is also how my setup looks like.
I have a SATA disk bay in my PC where I can put a bare disk in (without
needing trays).
I run amanda manually every week with a wrapper script, which handles a
couple of things, like disk encryption (script attached).


Thanks for providing your script and setup.
You should write a small howto around this and add it to the amanda wiki.

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-15 um 16:25 schrieb Jean-Louis Martineau:
> Looks like it can;t find the 'first' slot unless you manually load one.
> 
> The attached path should fix this issue.
> You can test it by removing the state file.

patch works for me: great

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-15 um 16:02 schrieb Jean-Louis Martineau:
> Do the disk are mounted?

yes:

$ mount | grep externaldisk
/dev/sde1 on /mnt/externaldisk1 type ext4
(rw,nosuid,nodev,noexec,relatime,data=ordered,user)
/dev/sdf1 on /mnt/externaldisk2 type ext4
(rw,nosuid,nodev,noexec,relatime,data=ordered,user)

> What are the output of:
> 
>   * amtape vtape show -otpchanger=disk1
>   * amtape vtape show -otpchanger=disk2


amanda@backup ~ $ amtape vtape show -otpchanger=disk1
amtape: scanning all 20 slots in changer:
slot   1: date 20160713170709 label vtape-001-001
slot   2: date 20160713190924 label vtape-001-002
slot   3: date 20160713234750 label vtape-001-003
slot   4: date 20160714113228 label vtape-001-004
slot   5: date 20160714190153 label vtape-001-005
slot   6: unlabeled volume
slot   7: unlabeled volume
slot   8: unlabeled volume
slot   9: unlabeled volume
slot  10: unlabeled volume
slot  11: unlabeled volume
slot  12: unlabeled volume
slot  13: unlabeled volume
slot  14: unlabeled volume
slot  15: unlabeled volume
slot  16: unlabeled volume
slot  17: unlabeled volume
slot  18: unlabeled volume
slot  19: unlabeled volume
slot  20: unlabeled volume
amanda@backup ~ $ amtape vtape show -otpchanger=disk2
amtape: scanning all 20 slots in changer:
slot   1: date 20160714105337 label vtape-002-001
slot   2: unlabeled volume
slot   3: unlabeled volume
slot   4: unlabeled volume
slot   5: unlabeled volume
slot   6: unlabeled volume
slot   7: unlabeled volume
slot   8: unlabeled volume
slot   9: unlabeled volume
slot  10: unlabeled volume
slot  11: unlabeled volume
slot  12: unlabeled volume
slot  13: unlabeled volume
slot  14: unlabeled volume
slot  15: unlabeled volume
slot  16: unlabeled volume
slot  17: unlabeled volume
slot  18: unlabeled volume
slot  19: unlabeled volume
slot  20: unlabeled volume


> Try to load a specific slot:
> 
>   * amtape vtape slot 1:1

works:

$ amtape vtape slot 1:1
slot 1:1: time 20160714105337 label vtape-002-001
changed to slot 1:1

and after that also amcheck looks good!

->

amanda@backup ~ $ amcheck vtape
Amanda Tape Server Host Check
-
Holding disk /mnt/amhold/vtape: 113 GB disk space available, using 112 GB
slot 1:1: volume 'vtape-002-001' is still active and cannot be overwritten
slot 1:2: contains an empty volume
Will write label 'vtape-003-001' to new volume in slot 1:2.
NOTE: skipping tape-writable test
Server check took 0.208 seconds

thanks a lot @Jean-Louis!

doing first dumps now to this new setup

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

$ cat /var/spool/amanda/tmp/server/vtape/amcheck-device.20160715160525.debug
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: pid 3000 ruid
87 euid 87 version 3.3.9: start at Fri Jul 15 16:05:25 2016
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device:
config_overrides: debug_taper 9
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: pid 3000 ruid
87 euid 87 version 3.3.9: rename at Fri Jul 15 16:05:25 2016
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: chg-disk: Dir
/mnt/externaldisk1
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: chg-disk: Using
statefile '/mnt/externaldisk1/state'
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: chg-disk: Dir
/mnt/externaldisk2
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: chg-disk: Using
statefile '/mnt/externaldisk2/state'
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device:
Amanda::Taper::Scan::traditional stage 1: search for oldest reusable volume
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device:
Amanda::Taper::Scan::traditional no oldest reusable volume
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device:
Amanda::Taper::Scan::traditional stage 2: scan for any reusable volume
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: new
Amanda::Changer::Error: type='failed', reason='invalid', message='Slot
first not found'
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device:
Amanda::Taper::Scan::traditional stage 2: scan for any reusable volume
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: new
Amanda::Changer::Error: type='failed', reason='notfound', message='all
slots have been loaded'
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device:
Amanda::Taper::Scan::traditional result: scan failed
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_utime   : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_stime   : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_maxrss  : 12180
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_ixrss   : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_idrss   : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_isrss   : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_minflt  : 1876
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_majflt  : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_nswap   : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_inblock : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_oublock : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_msgsnd  : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_msgrcv  : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_nsignals: 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_nvcsw   : 1
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: ru_nivcsw  : 0
Fri Jul 15 16:05:25 2016: thd-0x8061c00: amcheck-device: pid 3000 finish
time Fri Jul 15 16:05:25 2016

$ ls -l /mnt/externaldisk1
insgesamt 100
lrwxrwxrwx 1 amanda amanda 5 15. Jul 16:01 data -> slot1
drwx-- 2 amanda disk   16384 13. Jul 16:10 lost+found
drwx-- 2 amanda amanda  4096 13. Jul 18:56 slot1
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot10
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot11
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot12
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot13
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot14
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot15
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot16
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot17
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot18
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot19
drwx-- 2 amanda amanda  4096 13. Jul 22:20 slot2
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot20
drwx-- 2 amanda amanda  4096 14. Jul 06:05 slot3
drwx-- 2 amanda amanda  4096 14. Jul 12:01 slot4
drwx-- 2 amanda amanda  4096 14. Jul 22:33 slot5
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot6
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot7
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot8
drwx-- 2 amanda amanda  4096 13. Jul 16:11 slot9
-rw--- 1 amanda amanda   164 15. Jul 16:05 state

... hmm

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-15 um 13:41 schrieb Jean-Louis Martineau:
> Stefan,
> 
> another option is to use chg-aggregate, amanda will see all disks that
> are mounted:
> 
> define changer disk1 {
>   tpchanger "chg-disk:/mnt/externaldisk1"
>   property "num_slot" "10"
>   property "auto-create-slot" "yes"
>   property "removable" "yes"
> }
> 
> define changer disk2 {
>   tpchanger "chg-disk:/mnt/externaldisk2"
>   property "num_slot" "10"
>   property "auto-create-slot" "yes"
>   property "removable" "yes"
> }
> 
> define changer aggregate {
>   tpchanger "chg-aggregate:{disk1,disk2}"
>   property "state_filename" "/etc/amanda/CONF/aggregate.stats"
> }
> 
> tpchanger "aggregate"

nice, also worth a try, thanks!

Something is missing, it doesn't find slots here:

$ amcheck vtape
Amanda Tape Server Host Check
-
Holding disk /mnt/amhold/vtape: 113 GB disk space available, using 112 GB
slot first: Slot first not found
 all slots have been loaded
Taper scan algorithm did not find an acceptable volume.
(expecting a new volume)
ERROR: No acceptable volumes found
Server check took 0.198 seconds

Amanda Backup Client Hosts Check

Client check: 2 hosts checked in 1.143 seconds.  0 problems found.

(brought to you by Amanda 3.3.9)

amanda@backup /etc/amanda/vtape $ amtape vtape show
amtape: scanning all 40 slots in changer:
slot first: Slot first not found


amanda@backup /etc/amanda/vtape $ amtape vtape inventory
slot 0:1: label vtape-001-001 (vtape-001)
slot 0:2: label vtape-001-002 (vtape-001)
slot 0:3: label vtape-001-003 (vtape-001)
slot 0:4: label vtape-001-004 (vtape-001)
slot 0:5: label vtape-001-005 (vtape-001)
slot 0:6: blank
slot 0:7: blank

[..]

Do I have to use metalabel and/or autolabel? Doesn't change a thing
here. Start over with empty disks?

thanks!

Re: tapes and external disks: how to combine all this

[Uwe Menges]

On 07/15/16 04:40, Jon LaBadie wrote:
> Each was mounted wherever, but in the same place each time.
> What amanda used was a single "changer" dir of "slots" that
> were symbolic links to the vtape in that slot on its
> corresponding disk.

This is also how my setup looks like.
I have a SATA disk bay in my PC where I can put a bare disk in (without
needing trays).
I run amanda manually every week with a wrapper script, which handles a
couple of things, like disk encryption (script attached).

Yours, Uwe

# l /backup/amanda/
total 20
drwxrwx---.  3 amandabackup disk 4096 2016-07-09 21:57 ./
drwxr-xr-x. 29 root root 4096 2016-06-21 00:45 ../
lrwxrwxrwx.  1 amandabackup disk6 2016-07-09 19:17 data -> slot16
-rwxr-xr-x.  1 root root 4631 2016-03-12 16:28 do_backup.sh*
drwxr-xr-x.  2 root root 4096 2009-05-25 22:08 hold/
lrwxrwxrwx.  1 root root   16 2015-10-08 01:04 slot1 -> ../uwe-101/slot1
lrwxrwxrwx.  1 root root   17 2015-10-10 21:20 slot10 -> 
../uwe-a02/slot10
lrwxrwxrwx.  1 root root   17 2015-10-30 11:02 slot11 -> 
../uwe-a03/slot11
lrwxrwxrwx.  1 root root   17 2015-10-30 11:02 slot12 -> 
../uwe-a04/slot12
lrwxrwxrwx.  1 root root   17 2016-06-19 19:08 slot13 -> 
../uwe-a05/slot13
lrwxrwxrwx.  1 root root   17 2016-06-19 19:09 slot14 -> 
../uwe-a06/slot14
lrwxrwxrwx.  1 root root   17 2016-06-20 21:32 slot15 -> 
../uwe-a07/slot15
lrwxrwxrwx.  1 root root   17 2016-06-20 21:49 slot16 -> 
../uwe-a08/slot16

# cat /etc/amanda/uwe/amanda.conf
org ""  # your organization name for reports
mailto "admin"  # space separated list of operators at your site
dumpuser "backup"   # the user to run dumps under
usetimestamps yes   # enable more than one run per day
inparallel 4# maximum dumpers that will run in parallel
displayunit "m" # Possible values: "k|m|g|t"
netusage 1
bumpsize1 GB# minimum savings (threshold) to bump level 1 -> 2
bumpdays1   # minimum days at each level
bumpmult2   # threshold = bumpsize * (level-1)**bumpmult
dumpcycle 8 weeks   # the number of days in the normal dump cycle
tapetype vtape-1t
tpchanger "chg-disk:/backup/amanda"
tapecycle 23 # the number of tapes that are available to Amanda
runtapes 1 # how many tapes can be used for a single exec of amdump
runspercycle 8 # how many days you plan to execute amdump during one dumpcycle
labelstr "^uwe-[0-9a-f][0-9][0-9]$" # label constraint regex: all tapes 
must match
autoflush on# also flush old dumps to the tape if there are any
reserve 20 # percent of holding disk to reserve for incremental dumps if no 
tape is inserted
infofile "/var/lib/amanda/uwe/curinfo"  # database filename
logdir  "/var/log/amanda/uwe/log"
indexdir "/var/lib/amanda/uwe/index"
define tapetype vtape-1t {
comment "Virtual tapes on disks"
length 97600 kbyte # my disk is 976243896 kbyte, leave some slack for 
amanda files
}
define application-tool app_amgtar {
plugin "amgtar"
comment "amgtar with blocksize"
property "XATTRS" "YES"
property "TAR-BLOCKSIZE" "16384"
property "GNUTAR-PATH" "/usr/local/bin/tar_nocache"
}
define dumptype normal {
comment "dump with app_amgtar"
program "APPLICATION"
application "app_amgtar"
index on
auth "bsdtcp"
compress server custom
server_custom_compress "/bin/pigz"
estimate client
exclude "/etc/amanda/uwe/exclude.gtar"
priority medium
}
define dumptype uncompressed {
comment "dump with app_amgtar"
program "APPLICATION"
application "app_amgtar"
index on
auth "bsdtcp"
compress none
estimate client
exclude "/etc/amanda/uwe/exclude.gtar"
priority medium
}



do_backup.sh
Description: application/shellscript

Re: tapes and external disks: how to combine all this

[Schlacta, Christ]

Use USB3, UASP, or esata for better performance over just USB,
especially for spinning rust.It might be preferred to get two or
more desktop drives, and a hard drive "toaster" (They come in 1, 2,
and 4 slice varieties) to pop them into to simplify changing the
drives around on the weekly.  Also make sure before or after you swap
the disks you force a level 0 for each DLE on the backup.

On Thu, Jul 14, 2016 at 7:40 PM, Jon LaBadie  wrote:
> On Thu, Jul 14, 2016 at 07:12:38PM +0200, Stefan G. Weichinger wrote:
>> Am 2016-07-13 um 22:19 schrieb Jon LaBadie:
>>
>> > I've not used it on Linux, but an Overlay FS (OFS) was introduced
>> > in the kernel about 2 years ago.  With that, your two drives
>> > could have their own fixed mount points.  Then their root dirs
>> > could be overlayed so both trees appear under a single directory.
>> > If one drive was not there, only half the vtapes would appear.
>>
>> I have to research if it is possible to have both layers writeable.
>>
>> I like the udev-rules-way better.
>> Have the 2 disks in fstab:
>>
> Ahh, I finally remembered how I did it when I had vtapes on
> 2 external usb disks plus part of one internal hd (not the
> holding disk drive).
>
> Each was mounted wherever, but in the same place each time.
> What amanda used was a single "changer" dir of "slots" that
> were symbolic links to the vtape in that slot on its
> corresponding disk.  This too did not change.
>
> I think I had about 80 slots (and vtapes), 30 ea on usb and
> 20 on hd.  It did not have to be, but the slot numbering
> also matched my vtape numbering.  If a usb drive was not
> mounted it just seemed to amanda that the slots were empty.
>
> If the usb drive were mounted, but went to sleep due to
> inactivity, I was pleasently surprised that nothing bad
> happened.  What ever call woke up the disk held and waited
> for the disk to be available, about 10-15 sec.
>
> Jon
> --
> Jon H. LaBadie j...@jgcomp.com
>  11226 South Shore Rd.  (703) 787-0688 (H)
>  Reston, VA  20190  (703) 935-6720 (C)

Re: tapes and external disks: how to combine all this

[Jon LaBadie]

On Thu, Jul 14, 2016 at 07:12:38PM +0200, Stefan G. Weichinger wrote:
> Am 2016-07-13 um 22:19 schrieb Jon LaBadie:
> 
> > I've not used it on Linux, but an Overlay FS (OFS) was introduced
> > in the kernel about 2 years ago.  With that, your two drives
> > could have their own fixed mount points.  Then their root dirs
> > could be overlayed so both trees appear under a single directory.
> > If one drive was not there, only half the vtapes would appear.
> 
> I have to research if it is possible to have both layers writeable.
> 
> I like the udev-rules-way better.
> Have the 2 disks in fstab:
> 
Ahh, I finally remembered how I did it when I had vtapes on
2 external usb disks plus part of one internal hd (not the
holding disk drive).

Each was mounted wherever, but in the same place each time.
What amanda used was a single "changer" dir of "slots" that
were symbolic links to the vtape in that slot on its
corresponding disk.  This too did not change.

I think I had about 80 slots (and vtapes), 30 ea on usb and
20 on hd.  It did not have to be, but the slot numbering
also matched my vtape numbering.  If a usb drive was not
mounted it just seemed to amanda that the slots were empty.

If the usb drive were mounted, but went to sleep due to
inactivity, I was pleasently surprised that nothing bad
happened.  What ever call woke up the disk held and waited
for the disk to be available, about 10-15 sec.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-13 um 22:19 schrieb Jon LaBadie:


I've not used it on Linux, but an Overlay FS (OFS) was introduced
in the kernel about 2 years ago.  With that, your two drives
could have their own fixed mount points.  Then their root dirs
could be overlayed so both trees appear under a single directory.
If one drive was not there, only half the vtapes would appear.


I have to research if it is possible to have both layers writeable.

I like the udev-rules-way better.
Have the 2 disks in fstab:


UUID=57584... /mnt/externaldisk1 ext4 relatime,user 0 1
UUID=fae45... /mnt/externaldisk2 ext4 relatime,user 0 1

in /etc/udev/rules.d something like:

KERNEL=="sd*", SUBSYSTEM=="usb", ATTRS{serial}=="57584..blah", 
RUN+="/bin/ln -snf /mnt/externaldisk1 /mnt/diskchanger"


(2nd line for 2nd disk ...)

maybe let the RUN+= point at some small bash script to cover the case 
when both disks are attached at the same time or so


I will test that as soon as I have physical access to the both drives 
and can plug them in and out myself (currently ~30km away from me).


udev is part of every modern distro, dunno about overlayfs (modules or so).

greets, Stefan

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-13 um 20:32 schrieb Jean-Louis Martineau:

amanda do a 'mount /mnt/diskchanger'

If you can't configure the system to do as amanda expect, then you must
do the mount yourself.

You can make a script that symlink /mnt/diskchanger to the mount point.

That feature could be added to the disk changer, it need to know the
mount point of all disks, it find one that is mounted and set the symlink.


I am testing udev-rules to mount the 2 different harddrives to 2 
mountpoints and generate the needed link dynamically.


Maybe it would be helpful to enhance amanda in this part ... still 
learning here.


For example I am not sure how amanda handles overwriting the vtapes in 
case the "wrong" disk is mounted. Do I have to set tapecycle to the 
number of slots on one single disk, or to the overall number of vtapes 
on all disks? I will see as soon as all vtapes are used once, I assume.

Re: tapes and external disks: how to combine all this

[Jon LaBadie]

On Wed, Jul 13, 2016 at 04:20:51PM +0200, Stefan G. Weichinger wrote:
> Am 2016-07-13 um 16:12 schrieb Jean-Louis Martineau:
> 
> > amanda detect the disks via the meta label.
> > The 'changerfile' must be located on the disk,it is where the meta label
> > for the disk is written.
> 
> ok, I have
> 
> 
> 
> autolabel "$m-%%%" any
> meta-autolabel "$c-%%%"
> define changer removable-disk {
> tpchanger "chg-disk:/mnt/diskchanger"
> property "NUM-SLOT" "20"
> property "AUTO-CREATE-SLOT" "yes"
> property "REMOVABLE" "yes"
> property "MOUNT" "yes"
> property "UMOUNT" "yes"
> property "UMOUNT-LOCKFILE" "/etc/amanda/vtape/disk.lock"
> property "UMOUNT-DELAY" "1"
> }
> tpchanger "removable-disk"
> changerfile "/mnt/diskchanger/changerfile"
> 
> - amcheck looks good so far
> 
> And when I have two separate external disks  (with separate UUIDs), I
> will have to take care of mounting on or the other to the same
> mountpoint "/mnt/diskchanger" ?
> 
> I can't have both in fstab:
> 
> # /etc/fstab
> UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff /mnt/diskchanger ext4
> relatime,user 0 1
> 
> I don't yet understand how to handle this.
> 
I've not used it on Linux, but an Overlay FS (OFS) was introduced
in the kernel about 2 years ago.  With that, your two drives
could have their own fixed mount points.  Then their root dirs
could be overlayed so both trees appear under a single directory.
If one drive was not there, only half the vtapes would appear.

Jon
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-13 um 20:32 schrieb Jean-Louis Martineau:

amanda do a 'mount /mnt/diskchanger'

If you can't configure the system to do as amanda expect, then you must
do the mount yourself.

You can make a script that symlink /mnt/diskchanger to the mount point.

That feature could be added to the disk changer, it need to know the
mount point of all disks, it find one that is mounted and set the symlink.


ok, I understand the general concept and will try to get that set up 
accordingly.


Maybe I can write a small howto after that to help others.

thanks, Stefan

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-13 um 16:12 schrieb Jean-Louis Martineau:

> amanda detect the disks via the meta label.
> The 'changerfile' must be located on the disk,it is where the meta label
> for the disk is written.

ok, I have



autolabel "$m-%%%" any
meta-autolabel "$c-%%%"
define changer removable-disk {
tpchanger "chg-disk:/mnt/diskchanger"
property "NUM-SLOT" "20"
property "AUTO-CREATE-SLOT" "yes"
property "REMOVABLE" "yes"
property "MOUNT" "yes"
property "UMOUNT" "yes"
property "UMOUNT-LOCKFILE" "/etc/amanda/vtape/disk.lock"
property "UMOUNT-DELAY" "1"
}
tpchanger "removable-disk"
changerfile "/mnt/diskchanger/changerfile"

- amcheck looks good so far

And when I have two separate external disks  (with separate UUIDs), I
will have to take care of mounting on or the other to the same
mountpoint "/mnt/diskchanger" ?

I can't have both in fstab:

# /etc/fstab
UUID=5a5a9927-995f-4f0f-98ff-d222561f84ff /mnt/diskchanger ext4
relatime,user 0 1

I don't yet understand how to handle this.

Thanks for your help!

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-13 um 15:56 schrieb Jean-Louis Martineau:
> On 13/07/16 08:39 AM, Stefan G. Weichinger wrote:
>> Am 2016-07-13 um 11:10 schrieb Stefan G. Weichinger:
>>
>>> Would RAIT make sense here?
>>> Should I amvault dumps from here to there?
>>> How to deal with vtapes "not attached"?
>> Found this:
>>
>> http://wiki.zmanda.com/index.php/How_To:Using_removable_hard_drive_as_Virtual_tapes
>>
>>
>> What is a removable hard disk changer?
> Anything you can mount and create a r/w filesystem on it
> 
>   usb disk
>   usb flash
>   ...

ok.

So I would have to script my own stuff around on how to mount the
various disks at /amanda/diskchanger (directory from example)? or does
amanda detect the disks via the meta label?

I have the usb disks available already and start to test ...

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-13 um 15:13 schrieb Cuttler, Brian R. (HEALTH):
> Stefan,
> 
> You want to alternate the thumb drive by weeks, but leave both installed?
> 
> A properly constructed cron?
> 
> # amdump -o tapedev=

Only one drive attached, the other off-site.

Re: tapes and external disks: how to combine all this

[Stefan G. Weichinger]

Am 2016-07-13 um 11:10 schrieb Stefan G. Weichinger:

> Would RAIT make sense here?
> Should I amvault dumps from here to there?
> How to deal with vtapes "not attached"?

Found this:

http://wiki.zmanda.com/index.php/How_To:Using_removable_hard_drive_as_Virtual_tapes

What is a removable hard disk changer?
Does that mean hardware like RDX drives?