Re: How does amanda invoke gnutar on clients?

[Jon LaBadie]

On Wed, May 29, 2019 at 09:50:50AM +0700, Olivier wrote:
> Winston Sorfleet  writes:
> 
> > I'll try that, thanks!  Out of curiosity, would using "include" get
> > around that? 
> 
> My guess is that it won't. But you can explore the manual of gtar :)
> 
I'm pretty sure that "one-file-system" tells tar that during its
recursion don't cross a mount point to a different filesystem.

But if you "include " you have
given it a second starting point, and it is not crossing a
mount point.

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)

Re: How does amanda invoke gnutar on clients?

[Nathan Stratton Treadway]

On Tue, May 28, 2019 at 22:29:19 -0400, Winston Sorfleet wrote:
> Greetings all.  I was wondering if other eyes can spot something obvious
> I'm missing, with respect to how amanda invokes GNUTAR on client
> machines (the amdump logfile doesn't show anything unusual).  It may be
> in the guts of /usr/lib/amanda/application/amgtar or runtar but short of
> looking at code I am not familiar with, I can't tell.

> > forum.romanus.ca    / {
> >     comp-root-tar
> > #   auth "bsdtcp"
> >     exclude append "./home"
> > } 
[...] 
> And the dumptype definition:
> 
> > define dumptype root-tar {
> >     global
> >     program "GNUTAR"
> >     comment "root partitions dumped with tar"
> >     compress none
> >     index
> >     priority low
> > }
> > define dumptype comp-root-tar {
> >     root-tar
> >     comment "Root partitions with compression dumped with tar"
> >     compress client fast
> > }
> 
program "GNUTAR" means that tar is being invoked by "runtar", which does
indeed add a --one-file-system to the tar command line.

You can find the full tar command line in the "runtar" log on the
Amanada client machine.  The path to the log files depends on how Amanda
was compiled, but for example on an old Debian machine where I am still
using the GNUTAR program they are found at
/var/log/amanda/client//runtar.2019* .

If you want to try doing dumps without sending --one-file-system to tar,
you can try switching your dumptype to use amgtar: look in the amgtar
man page for the ONE-FILE-SYSTEM property.

(If you make that switch, you would then find the full tar command line
used in the amgtar.2019* log files on the client.)

Nathan


Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: amanda tape header stripped, why?

[Gene Heskett]

On Tuesday 28 May 2019 11:09:43 pm Nathan Stratton Treadway wrote:

> On Tue, May 28, 2019 at 21:19:55 -0400, Gene Heskett wrote:
> > Mine, when running 3.3.7p1 on wheezy, had this good header in every
> > vtape "header"
>
> Note that the "0.*" file is simply the table label written by
> amlabel -- there is no "data" in it, the only point of the file is to
> contain the volume label.  You'll see that they are always exactly
> 32kiB (32768 bytes), and they contain just the the label info 
> followed by NUL bytes padding out to 32kiB...
>
> =
> root@tumhalad:~/rushey_etc_git# hd
> /vtapes/TestBackup/slot1/0.TESTBACKUP-01   41 4d 41 4e 44
> 41 3a 20  54 41 50 45 53 54 41 52  |AMANDA: TAPESTAR| 0010  54 20
> 44 41 54 45 20 32  30 31 38 31 31 30 38 30  |T DATE 201811080|
> 0020  37 30 31 30 33 20 54 41  50 45 20 54 45 53 54 42  |70103
> TAPE TESTB| 0030  41 43 4b 55 50 2d 30 31  0a 0c 0a 00 00 00 00 00
>  |ACKUP-01| 0040  00 00 00 00 00 00 00 00  00 00 00 00 00
> 00 00 00  || *
> 8000
> =
>
> Thus, the usual start-of-the-recovery-command-pipeline command, "dd
> if= bs=32k skip=1 |", would skip over the entire file and have
> nothing to pass to the later commands in the pipeline.
>
> (The vtape files after "0", on the other hand, all do have dump
> data following a 32kiB header block)
>
>   Nathan
>

The more I think on it, the righter you all are. So lets drop this based 
on me miss-remembering that it was the file header, not the tape header.
>
>
> --
>-- Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic
> region Ray Ontko & Co.  -  Software consulting services  -  
> http://www.ontko.com/ GPG Key:
> http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239 Key
> fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: amanda tape header stripped, why?

[Nathan Stratton Treadway]

On Tue, May 28, 2019 at 21:19:55 -0400, Gene Heskett wrote:
> Mine, when running 3.3.7p1 on wheezy, had this good header in every 
> vtape "header"

Note that the "0.*" file is simply the table label written by
amlabel -- there is no "data" in it, the only point of the file is to
contain the volume label.  You'll see that they are always exactly 32kiB
(32768 bytes), and they contain just the the label info  followed by NUL
bytes padding out to 32kiB...

=
root@tumhalad:~/rushey_etc_git# hd /vtapes/TestBackup/slot1/0.TESTBACKUP-01 
  41 4d 41 4e 44 41 3a 20  54 41 50 45 53 54 41 52  |AMANDA: TAPESTAR|
0010  54 20 44 41 54 45 20 32  30 31 38 31 31 30 38 30  |T DATE 201811080|
0020  37 30 31 30 33 20 54 41  50 45 20 54 45 53 54 42  |70103 TAPE TESTB|
0030  41 43 4b 55 50 2d 30 31  0a 0c 0a 00 00 00 00 00  |ACKUP-01|
0040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ||
*
8000
=

Thus, the usual start-of-the-recovery-command-pipeline command, "dd
if= bs=32k skip=1 |", would skip over the entire file and have
nothing to pass to the later commands in the pipeline.

(The vtape files after "0", on the other hand, all do have dump data
following a 32kiB header block)

Nathan




Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: How does amanda invoke gnutar on clients?

[Olivier]

Winston Sorfleet  writes:

> I'll try that, thanks!  Out of curiosity, would using "include" get
> around that? 

My guess is that it won't. But you can explore the manual of gtar :)

Best regards,

Olivier

>

-- 

Re: How does amanda invoke gnutar on clients?

[Winston Sorfleet]

On 2019-05-28 10:35 p.m., Olivier wrote:
> Winston Sorfleet  writes:
>> Greetings all. I was wondering if other eyes can spot something obvious I'm 
>> missing, with respect
>> to how amanda invokes GNUTAR on client machines (the amdump logfile doesn't 
>> show anything
>> unusual). It may be in the guts of /usr/lib/amanda/application/amgtar or 
>> runtar but short of looking
>> at code I am not familiar with, I can't tell.
>>
>> I have amanda running on a server, with client machine forum.romanus.ca, and 
>> it's started giving
>> me "STRANGE dump details":
> gnutar uses --one-file-system option, so it will not save what is
> mounted from a different filesystem.
>
> You must have a DEL for each of your filesystems:
> /
> /bin
> /etc
> /...
>
> Olivier
I'll try that, thanks!  Out of curiosity, would using "include" get
around that? 

Re: How does amanda invoke gnutar on clients?

[Olivier]

Winston Sorfleet  writes:
> Greetings all. I was wondering if other eyes can spot something obvious I'm 
> missing, with respect
> to how amanda invokes GNUTAR on client machines (the amdump logfile doesn't 
> show anything
> unusual). It may be in the guts of /usr/lib/amanda/application/amgtar or 
> runtar but short of looking
> at code I am not familiar with, I can't tell.
>
> I have amanda running on a server, with client machine forum.romanus.ca, and 
> it's started giving
> me "STRANGE dump details":

gnutar uses --one-file-system option, so it will not save what is
mounted from a different filesystem.

You must have a DEL for each of your filesystems:
/
/bin
/etc
/...

Olivier

>
>/-- forum.romanus.ca / lev 1 STRANGE
>   sendbackup: start [forum.romanus.ca:/ level 1]
>   sendbackup: info BACKUP=/bin/tar
>   sendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/tar -xpGf - ...
>   sendbackup: info COMPRESS_SUFFIX=.gz
>   sendbackup: info end
>   ? /bin/tar: ./bin: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./dev: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./etc: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./lib: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./opt: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./proc: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./run: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./sbin: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./sys: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./usr: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./mnt/nvme0n1: directory is on a different filesystem; not 
> dumped
>   ? /bin/tar: ./mnt/vhosts: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./var/spool/fax/incoming: directory is on a different 
> filesystem; not dumped
>
> Now recently, I moved these directories off a standard hard disk, to an nvme, 
> using btrfs.
> Because, reasons (trying to do this with a live machine, not from bare-bones 
> install) I did these as
> bind mounts, e.g. in /etc/fstab
>
>  /dev/nvme0n1 /mnt/nvme0n1 btrfs defaults 0 2
>  /mnt/nvme0n1/bin /bin none bind 
>  /mnt/nvme0n1/etc /etc none bind
>  /mnt/nvme0n1/lib /lib none bind
>  /mnt/nvme0n1/opt /opt none bind
>  /mnt/nvme0n1/sbin /sbin none bind
>  /mnt/nvme0n1/usr /usr none bind
>
> Now if I'd being doing some sort of one-file-system exclusion, I could 
> understand why tar (as
> demanded by amanda) would drop them, but as far as I can tell, I'm not: 
>
> The disklist (/home being an nfs mount from the server machine, it gets 
> backed up with the
> server-as-amanda-client).
>
>  forum.romanus.ca / {
>  comp-root-tar
>  # auth "bsdtcp"
>  exclude append "./home"
>  } 
>
> (Note that the server's amdump log file does show the intentional exclude):
>
>  driver: send-cmd time 263.809 to dumper2: SHM-DUMP 02-3 44577 NULL 1
>  forum.romanus.ca 9efefbfff3fffbf71f / N
>  ODEVICE 1 2019:5:22:6:47:2 GNUTAR "" "" "" "" "" "" "" 1 "" "" bsdtcp AMANDA
>  /amanda_shm_control-16837-0 20 |" bsdtcp\
>  n FAST\n YES\n YES\n
>  AMANDA\n \n ./home\n 
> \n"""
>
> And the dumptype definition:
>
>  define dumptype root-tar {
>  global
>  program "GNUTAR"
>  comment "root partitions dumped with tar"
>  compress none 
>  index
>  priority low
>  }
>
>  define dumptype comp-root-tar {
>  root-tar
>  comment "Root partitions with compression dumped with tar"
>  compress client fast
>  }

How does amanda invoke gnutar on clients?

[Winston Sorfleet]

Greetings all.  I was wondering if other eyes can spot something obvious
I'm missing, with respect to how amanda invokes GNUTAR on client
machines (the amdump logfile doesn't show anything unusual).  It may be
in the guts of /usr/lib/amanda/application/amgtar or runtar but short of
looking at code I am not familiar with, I can't tell.

I have amanda running on a server, with client machine forum.romanus.ca,
and it's started giving me "STRANGE dump details":

>   /-- forum.romanus.ca / lev 1 STRANGE
>   sendbackup: start [forum.romanus.ca:/ level 1]
>   sendbackup: info BACKUP=/bin/tar
>   sendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/tar -xpGf - ...
>   sendbackup: info COMPRESS_SUFFIX=.gz
>   sendbackup: info end
>   ? /bin/tar: ./bin: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./dev: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./etc: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./lib: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./opt: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./proc: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./run: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./sbin: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./sys: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./usr: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./mnt/nvme0n1: directory is on a different filesystem; not 
> dumped
>   ? /bin/tar: ./mnt/vhosts: directory is on a different filesystem; not dumped
>   ? /bin/tar: ./var/spool/fax/incoming: directory is on a different 
> filesystem; not dumped

Now recently, I moved these directories off a standard hard disk, to an
nvme, using btrfs.  Because, reasons (trying to do this with a live
machine, not from bare-bones install) I did these as bind mounts, e.g.
in /etc/fstab

> /dev/nvme0n1    /mnt/nvme0n1    btrfs   defaults    0   2
> /mnt/nvme0n1/bin    /bin    none    bind    
> /mnt/nvme0n1/etc    /etc    none    bind
> /mnt/nvme0n1/lib    /lib    none    bind
> /mnt/nvme0n1/opt    /opt    none    bind
> /mnt/nvme0n1/sbin   /sbin   none    bind
> /mnt/nvme0n1/usr    /usr    none    bind
Now if I'd being doing some sort of one-file-system exclusion, I could
understand why tar (as demanded by amanda) would drop them, but as far
as I can tell, I'm not:

The disklist (/home being an nfs mount from the server machine, it gets
backed up with the server-as-amanda-client).

> forum.romanus.ca    / {
>     comp-root-tar
> #   auth "bsdtcp"
>     exclude append "./home"
> } 

(Note that the server's amdump log file does show the /intentional/
exclude):

> driver: send-cmd time 263.809 to dumper2: SHM-DUMP 02-3 44577 NULL
> 1 forum.romanus.ca 9efefbfff3fffbf71f / N
> ODEVICE 1 2019:5:22:6:47:2 GNUTAR "" "" "" "" "" "" "" 1 "" "" bsdtcp
> AMANDA /amanda_shm_control-16837-0 20 |"  bsdtcp\
> n  FAST\n  YES\n 
> YES\n  AMANDA\n  \n
> ./home\n  \n"""

And the dumptype definition:

> define dumptype root-tar {
>     global
>     program "GNUTAR"
>     comment "root partitions dumped with tar"
>     compress none
>     index
>     priority low
> }
> define dumptype comp-root-tar {
>     root-tar
>     comment "Root partitions with compression dumped with tar"
>     compress client fast
> }

Re: amanda tape header stripped, why?

[Gene Heskett]

On Tuesday 28 May 2019 12:49:05 pm Charles Curley wrote:

> On Tue, 28 May 2019 11:40:33 -0400
>
> Gene Heskett  wrote:
> > In my build on stretch the tape header, with normally contains
> > instructions for a gzip/tar only system, the recipe for bare
> > recovery has disappeared.  It now looks like this:
> > AMANDA: TAPESTART DATE 20190527162707 TAPE Dailys-1
> >
> > There is supposed to be a 2nd line, showing how to unpack the
> > archive with nothing but gzip and tar,
> >
> >
> > Cheers, Gene Heskett
>
> I can confirm that for the first file per VTAPE on my build:
>
> root@amanda2:/var/amanda/vtapes/slot1# od -N 96 -A x -t x1z -v
> 0.DailySet1-001 00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54
> 41 52  >AMANDA: TAPESTAR< 10 54 20 44 41 54 45 20 32 30 31 39 30
> 35 32 37 31  >T DATE 201905271< 20 33 30 38 34 35 20 54 41 50 45
> 20 44 61 69 6c 79  >30845 TAPE Daily< 30 53 65 74 31 2d 30 30 31
> 0a 0c 0a 00 00 00 00 00  >Set1-001< 40 00 00 00 00 00 00
> 00 00 00 00 00 00 00 00 00 00  >< 50 00 00 00 00
> 00 00 00 00 00 00 00 00 00 00 00 00  >< 60
> root@amanda2:/var/amanda/vtapes/slot1#
>
> However it also seems to be the case for Amanda 3.3.9-5, the current
> stable version on debian:
>
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 96 -A x -t x1z
> -v 0.DailySet1_01 00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54
> 41 52  >AMANDA: TAPESTAR< 10 54 20 44 41 54 45 20 32 30 31 39 30
> 34 31 32 30  >T DATE 201904120< 20 30 30 31 30 31 20 54 41 50 45
> 20 44 61 69 6c 79  >00101 TAPE Daily< 30 53 65 74 31 5f 30 31 0a
> 0c 0a 00 00 00 00 00 00  >Set1_01.< 40 00 00 00 00 00 00
> 00 00 00 00 00 00 00 00 00 00  >< 50 00 00 00 00
> 00 00 00 00 00 00 00 00 00 00 00 00  >< 60
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1#
>
> But if you look at the next one, the first real backup file,
> starting at about 0x190 or so...
>
Mine, when running 3.3.7p1 on wheezy, had this good header in every 
vtape "header"

> root@amanda2:/var/amanda/vtapes/slot1# od -N 544 -A x -t x1z -v
> 1.localhost._root.0 00 41 4d 41 4e 44 41 3a 20 53 50 4c 49 54
> 5f 46 49  >AMANDA: SPLIT_FI< 10 4c 45 20 32 30 31 39 30 35 32 37
> 31 33 30 38 34  >LE 2019052713084< 20 35 20 6c 6f 63 61 6c 68 6f
> 73 74 20 2f 72 6f 6f  >5 localhost /roo< 30 74 20 20 70 61 72 74
> 20 31 2f 2d 31 20 20 6c 65  >t  part 1/-1  le< 40 76 20 30 20 63
> 6f 6d 70 20 2e 67 7a 20 70 72 6f  >v 0 comp .gz pro< 50 67 72 61
> 6d 20 2f 62 69 6e 2f 74 61 72 0a 4f 52  >gram /bin/tar.OR< 60 49
> 47 53 49 5a 45 3d 34 30 0a 4e 41 54 49 56 45  >IGSIZE=40.NATIVE<
> 70 2d 43 52 43 3d 32 61 61 38 62 32 65 35 3a 34 30 
> >-CRC=2aa8b2e5:40< 80 39 36 30 0a 43 4c 49 45 4e 54 2d 43 52 43 3d
> 37  >960.CLIENT-CRC=7< 90 66 36 30 33 61 32 66 3a 39 35 33 31 0a
> 53 45 52  >f603a2f:9531.SER< a0 56 45 52 2d 43 52 43 3d 37 66 36
> 30 33 61 32 66  >VER-CRC=7f603a2f< b0 3a 39 35 33 31 0a 44 4c 45
> 3d 3c 3c 45 4e 44 44  >:9531.DLE=< 3e 0a 20 20 3c 70 72 6f 67  >LE..   4e 55 54 41 52 3c 2f 70 72 6f 67  >ram>GNUTAR 3e 0a 20 20 3c 64 69 73 6b 3e 2f 72 6f  >ram>.  /ro< f0 6f
> 74 3c 2f 64 69 73 6b 3e 0a 20 20 3c 6c 65 76  >ot.   000100 65 6c 3e 30 3c 2f 6c 65 76 65 6c 3e 0a 20 20 3c  >el>0.
>  << 000110 61 75 74 68 3e 42 53 44 54 43 50 3c 2f 61 75 74 
> >auth>BSDTCP 46  >h>.  F< 000130 41 53 54 3c 2f 63 6f 6d 70 72 65 73 73
> 3e 0a 20  >AST. < 000140 20 3c 72 65 63 6f 72 64 3e 59 45
> 53 3c 2f 72 65  > YES 69 6e 64 65 78 3e 59  >cord>.  Y< 000160 45 53 3c 2f 69 6e 64
> 65 78 3e 0a 20 20 3c 64 61  >ES.   68 3e 41 4d 41 4e 44 41 3c 2f 64  >tapath>AMANDA 70 61 74 68 3e 0a 3c 2f 64 6c 65 3e 0a  >atapath>..< 000190 45
> 4e 44 44 4c 45 0a 54 6f 20 72 65 73 74 6f 72  >ENDDLE.To restor<
> 0001a0 65 2c 20 70 6f 73 69 74 69 6f 6e 20 74 61 70 65  >e, position
> tape< 0001b0 20 61 74 20 73 74 61 72 74 20 6f 66 20 66 69 6c  > at
> start of fil< 0001c0 65 20 61 6e 64 20 72 75 6e 3a 0a 09 64 64 20 69 
> >e and run:..dd i< 0001d0 66 3d 3c 74 61 70 65 3e 20 62 73 3d 33 32 6b
> 20  >f= bs=32k < 0001e0 73 6b 69 70 3d 31 20 7c 20 2f 62 69 6e
> 2f 67 7a  >skip=1 | /bin/gz< 0001f0 69 70 20 2d 64 63 20 7c 20 2f 62
> 69 6e 2f 74 61  >ip -dc | /bin/ta< 000200 72 20 2d 78 70 47 66 20 2d
> 20 2e 2e 2e 20 0a 0c  >r -xpGf - ... ..< 000210 0a 00 00 00 00 00 00
> 00 00 00 00 00 00 00 00 00  >< 000220
> root@amanda2:/var/amanda/vtapes/slot1#
>
> And similarly for 3.3.9-5:
>
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 0x270 -A x -t
> x1z -v 1.hawk.localdomain._home_charles_projects.1 00 41 4d 41
> 4e 44 41 3a 20 53 50 4c 49 54 5f 46 49  >AMANDA: SPLIT_FI< 10 4c
> 45 20 32 30 31 39 30 34 31 32 30 30 30 31 30  >LE 2019041200010<
> 20 31 20 68 61 77 6b 2e 6c 6f 63 61 6c 64 6f 6d 61  >1
> hawk.localdoma<
>
> ...
>
> 0001b0 3e 0a 20 20 3c 64 61 74 61 70 61 74 68 3e 41 4d  >>. 
> AM< 0001c0 41 

Re: amanda tape header stripped, why?

[Debra S Baddorf]

Agreed - the short form is:  the tape header needs no instructions,
and each DLE file might be done with a different mechanism  (true for me)  so
each DLE backup has the instructions in the header.

Deb Baddorf
Fermilab



> On May 28, 2019, at 11:49 AM, Charles Curley 
>  wrote:
> 
> On Tue, 28 May 2019 11:40:33 -0400
> Gene Heskett  wrote:
> 
>> In my build on stretch the tape header, with normally contains 
>> instructions for a gzip/tar only system, the recipe for bare recovery 
>> has disappeared.  It now looks like this:
>> AMANDA: TAPESTART DATE 20190527162707 TAPE Dailys-1
>> 
>> There is supposed to be a 2nd line, showing how to unpack the archive 
>> with nothing but gzip and tar,
>> 
>> 
>> Cheers, Gene Heskett
> 
> I can confirm that for the first file per VTAPE on my build:
> 
> root@amanda2:/var/amanda/vtapes/slot1# od -N 96 -A x -t x1z -v 
> 0.DailySet1-001 
> 00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54 41 52  >AMANDA: TAPESTAR<
> 10 54 20 44 41 54 45 20 32 30 31 39 30 35 32 37 31  >T DATE 201905271<
> 20 33 30 38 34 35 20 54 41 50 45 20 44 61 69 6c 79  >30845 TAPE Daily<
> 30 53 65 74 31 2d 30 30 31 0a 0c 0a 00 00 00 00 00  >Set1-001<
> 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
> 50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
> 60
> root@amanda2:/var/amanda/vtapes/slot1# 
> 
> However it also seems to be the case for Amanda 3.3.9-5, the current
> stable version on debian:
> 
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 96 -A x -t x1z -v 
> 0.DailySet1_01 
> 00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54 41 52  >AMANDA: TAPESTAR<
> 10 54 20 44 41 54 45 20 32 30 31 39 30 34 31 32 30  >T DATE 201904120<
> 20 30 30 31 30 31 20 54 41 50 45 20 44 61 69 6c 79  >00101 TAPE Daily<
> 30 53 65 74 31 5f 30 31 0a 0c 0a 00 00 00 00 00 00  >Set1_01.<
> 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
> 50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
> 60
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# 
> 
> But if you look at the next one, the first real backup file,
> starting at about 0x190 or so...
> 
> root@amanda2:/var/amanda/vtapes/slot1# od -N 544 -A x -t x1z -v 
> 1.localhost._root.0 
> 00 41 4d 41 4e 44 41 3a 20 53 50 4c 49 54 5f 46 49  >AMANDA: SPLIT_FI<
> 10 4c 45 20 32 30 31 39 30 35 32 37 31 33 30 38 34  >LE 2019052713084<
> 20 35 20 6c 6f 63 61 6c 68 6f 73 74 20 2f 72 6f 6f  >5 localhost /roo<
> 30 74 20 20 70 61 72 74 20 31 2f 2d 31 20 20 6c 65  >t  part 1/-1  le<
> 40 76 20 30 20 63 6f 6d 70 20 2e 67 7a 20 70 72 6f  >v 0 comp .gz pro<
> 50 67 72 61 6d 20 2f 62 69 6e 2f 74 61 72 0a 4f 52  >gram /bin/tar.OR<
> 60 49 47 53 49 5a 45 3d 34 30 0a 4e 41 54 49 56 45  >IGSIZE=40.NATIVE<
> 70 2d 43 52 43 3d 32 61 61 38 62 32 65 35 3a 34 30  >-CRC=2aa8b2e5:40<
> 80 39 36 30 0a 43 4c 49 45 4e 54 2d 43 52 43 3d 37  >960.CLIENT-CRC=7<
> 90 66 36 30 33 61 32 66 3a 39 35 33 31 0a 53 45 52  >f603a2f:9531.SER<
> a0 56 45 52 2d 43 52 43 3d 37 66 36 30 33 61 32 66  >VER-CRC=7f603a2f<
> b0 3a 39 35 33 31 0a 44 4c 45 3d 3c 3c 45 4e 44 44  >:9531.DLE=< c0 4c 45 0a 3c 64 6c 65 3e 0a 20 20 3c 70 72 6f 67  >LE..   d0 72 61 6d 3e 47 4e 55 54 41 52 3c 2f 70 72 6f 67  >ram>GNUTAR e0 72 61 6d 3e 0a 20 20 3c 64 69 73 6b 3e 2f 72 6f  >ram>.  /ro<
> f0 6f 74 3c 2f 64 69 73 6b 3e 0a 20 20 3c 6c 65 76  >ot.   000100 65 6c 3e 30 3c 2f 6c 65 76 65 6c 3e 0a 20 20 3c  >el>0.  <<
> 000110 61 75 74 68 3e 42 53 44 54 43 50 3c 2f 61 75 74  >auth>BSDTCP 000120 68 3e 0a 20 20 3c 63 6f 6d 70 72 65 73 73 3e 46  >h>.  F<
> 000130 41 53 54 3c 2f 63 6f 6d 70 72 65 73 73 3e 0a 20  >AST. <
> 000140 20 3c 72 65 63 6f 72 64 3e 59 45 53 3c 2f 72 65  > YES 000150 63 6f 72 64 3e 0a 20 20 3c 69 6e 64 65 78 3e 59  >cord>.  Y<
> 000160 45 53 3c 2f 69 6e 64 65 78 3e 0a 20 20 3c 64 61  >ES.   000170 74 61 70 61 74 68 3e 41 4d 41 4e 44 41 3c 2f 64  >tapath>AMANDA 000180 61 74 61 70 61 74 68 3e 0a 3c 2f 64 6c 65 3e 0a  >atapath>..<
> 000190 45 4e 44 44 4c 45 0a 54 6f 20 72 65 73 74 6f 72  >ENDDLE.To restor<
> 0001a0 65 2c 20 70 6f 73 69 74 69 6f 6e 20 74 61 70 65  >e, position tape<
> 0001b0 20 61 74 20 73 74 61 72 74 20 6f 66 20 66 69 6c  > at start of fil<
> 0001c0 65 20 61 6e 64 20 72 75 6e 3a 0a 09 64 64 20 69  >e and run:..dd i<
> 0001d0 66 3d 3c 74 61 70 65 3e 20 62 73 3d 33 32 6b 20  >f= bs=32k <
> 0001e0 73 6b 69 70 3d 31 20 7c 20 2f 62 69 6e 2f 67 7a  >skip=1 | /bin/gz<
> 0001f0 69 70 20 2d 64 63 20 7c 20 2f 62 69 6e 2f 74 61  >ip -dc | /bin/ta<
> 000200 72 20 2d 78 70 47 66 20 2d 20 2e 2e 2e 20 0a 0c  >r -xpGf - ... ..<
> 000210 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
> 000220
> root@amanda2:/var/amanda/vtapes/slot1# 
> 
> And similarly for 3.3.9-5:
> 
> root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 0x270 -A x -t x1z -v 
> 

Re: amanda tape header stripped, why?

[Charles Curley]

On Tue, 28 May 2019 11:40:33 -0400
Gene Heskett  wrote:

> In my build on stretch the tape header, with normally contains 
> instructions for a gzip/tar only system, the recipe for bare recovery 
> has disappeared.  It now looks like this:
> AMANDA: TAPESTART DATE 20190527162707 TAPE Dailys-1
> 
> There is supposed to be a 2nd line, showing how to unpack the archive 
> with nothing but gzip and tar,
> 
> 
> Cheers, Gene Heskett

I can confirm that for the first file per VTAPE on my build:

root@amanda2:/var/amanda/vtapes/slot1# od -N 96 -A x -t x1z -v 
0.DailySet1-001 
00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54 41 52  >AMANDA: TAPESTAR<
10 54 20 44 41 54 45 20 32 30 31 39 30 35 32 37 31  >T DATE 201905271<
20 33 30 38 34 35 20 54 41 50 45 20 44 61 69 6c 79  >30845 TAPE Daily<
30 53 65 74 31 2d 30 30 31 0a 0c 0a 00 00 00 00 00  >Set1-001<
40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
60
root@amanda2:/var/amanda/vtapes/slot1# 

However it also seems to be the case for Amanda 3.3.9-5, the current
stable version on debian:

root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 96 -A x -t x1z -v 
0.DailySet1_01 
00 41 4d 41 4e 44 41 3a 20 54 41 50 45 53 54 41 52  >AMANDA: TAPESTAR<
10 54 20 44 41 54 45 20 32 30 31 39 30 34 31 32 30  >T DATE 201904120<
20 30 30 31 30 31 20 54 41 50 45 20 44 61 69 6c 79  >00101 TAPE Daily<
30 53 65 74 31 5f 30 31 0a 0c 0a 00 00 00 00 00 00  >Set1_01.<
40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
50 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
60
root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# 

But if you look at the next one, the first real backup file,
starting at about 0x190 or so...

root@amanda2:/var/amanda/vtapes/slot1# od -N 544 -A x -t x1z -v 
1.localhost._root.0 
00 41 4d 41 4e 44 41 3a 20 53 50 4c 49 54 5f 46 49  >AMANDA: SPLIT_FI<
10 4c 45 20 32 30 31 39 30 35 32 37 31 33 30 38 34  >LE 2019052713084<
20 35 20 6c 6f 63 61 6c 68 6f 73 74 20 2f 72 6f 6f  >5 localhost /roo<
30 74 20 20 70 61 72 74 20 31 2f 2d 31 20 20 6c 65  >t  part 1/-1  le<
40 76 20 30 20 63 6f 6d 70 20 2e 67 7a 20 70 72 6f  >v 0 comp .gz pro<
50 67 72 61 6d 20 2f 62 69 6e 2f 74 61 72 0a 4f 52  >gram /bin/tar.OR<
60 49 47 53 49 5a 45 3d 34 30 0a 4e 41 54 49 56 45  >IGSIZE=40.NATIVE<
70 2d 43 52 43 3d 32 61 61 38 62 32 65 35 3a 34 30  >-CRC=2aa8b2e5:40<
80 39 36 30 0a 43 4c 49 45 4e 54 2d 43 52 43 3d 37  >960.CLIENT-CRC=7<
90 66 36 30 33 61 32 66 3a 39 35 33 31 0a 53 45 52  >f603a2f:9531.SER<
a0 56 45 52 2d 43 52 43 3d 37 66 36 30 33 61 32 66  >VER-CRC=7f603a2f<
b0 3a 39 35 33 31 0a 44 4c 45 3d 3c 3c 45 4e 44 44  >:9531.DLE=GNUTARram>.  /ro<
f0 6f 74 3c 2f 64 69 73 6b 3e 0a 20 20 3c 6c 65 76  >ot.  el>0.  <<
000110 61 75 74 68 3e 42 53 44 54 43 50 3c 2f 61 75 74  >auth>BSDTCPh>.  F<
000130 41 53 54 3c 2f 63 6f 6d 70 72 65 73 73 3e 0a 20  >AST. <
000140 20 3c 72 65 63 6f 72 64 3e 59 45 53 3c 2f 72 65  > YEScord>.  Y<
000160 45 53 3c 2f 69 6e 64 65 78 3e 0a 20 20 3c 64 61  >ES.  tapath>AMANDAatapath>..<
000190 45 4e 44 44 4c 45 0a 54 6f 20 72 65 73 74 6f 72  >ENDDLE.To restor<
0001a0 65 2c 20 70 6f 73 69 74 69 6f 6e 20 74 61 70 65  >e, position tape<
0001b0 20 61 74 20 73 74 61 72 74 20 6f 66 20 66 69 6c  > at start of fil<
0001c0 65 20 61 6e 64 20 72 75 6e 3a 0a 09 64 64 20 69  >e and run:..dd i<
0001d0 66 3d 3c 74 61 70 65 3e 20 62 73 3d 33 32 6b 20  >f= bs=32k <
0001e0 73 6b 69 70 3d 31 20 7c 20 2f 62 69 6e 2f 67 7a  >skip=1 | /bin/gz<
0001f0 69 70 20 2d 64 63 20 7c 20 2f 62 69 6e 2f 74 61  >ip -dc | /bin/ta<
000200 72 20 2d 78 70 47 66 20 2d 20 2e 2e 2e 20 0a 0c  >r -xpGf - ... ..<
000210 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ><
000220
root@amanda2:/var/amanda/vtapes/slot1# 

And similarly for 3.3.9-5:

root@hawk:/crc/backs/myob/amanda/DailySet1/slot1# od -N 0x270 -A x -t x1z -v 
1.hawk.localdomain._home_charles_projects.1 
00 41 4d 41 4e 44 41 3a 20 53 50 4c 49 54 5f 46 49  >AMANDA: SPLIT_FI<
10 4c 45 20 32 30 31 39 30 34 31 32 30 30 30 31 30  >LE 2019041200010<
20 31 20 68 61 77 6b 2e 6c 6f 63 61 6c 64 6f 6d 61  >1 hawk.localdoma<

...

0001b0 3e 0a 20 20 3c 64 61 74 61 70 61 74 68 3e 41 4d  >>.  AM<
0001c0 41 4e 44 41 3c 2f 64 61 74 61 70 61 74 68 3e 0a  >ANDA.<
0001d0 3c 2f 64 6c 65 3e 0a 45 4e 44 44 4c 45 0a 54 6f  >.ENDDLE.To<
0001e0 20 72 65 73 74 6f 72 65 2c 20 70 6f 73 69 74 69  > restore, positi<
0001f0 6f 6e 20 74 61 70 65 20 61 74 20 73 74 61 72 74  >on tape at start<
000200 20 6f 66 20 66 69 6c 65 20 61 6e 64 20 72 75 6e  > of file and run<
000210 3a 0a 09 64 64 20 69 66 3d 3c 74 61 70 65 3e 20  >:..dd if= <
000220 62 73 3d 33 32 6b 20 73 6b 69 70 3d 31 20 7c 20  >bs=32k skip=1 | <
000230 2f 62 69 6e 2f 67 7a 69 70 20 2d 64 20 7c 20 2f  >/bin/gzip -d | 

amanda tape header stripped, why?

[Gene Heskett]

In my build on stretch the tape header, with normally contains 
instructions for a gzip/tar only system, the recipe for bare recovery 
has disappeared.  It now looks like this:
AMANDA: TAPESTART DATE 20190527162707 TAPE Dailys-1

There is supposed to be a 2nd line, showing how to unpack the archive 
with nothing but gzip and tar,


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: building (non-Debianized) Amanda on Stretch

[Nathan Stratton Treadway]

On Tue, May 28, 2019 at 07:11:05 -0400, Gene Heskett wrote:
> I just looked at the header files of the backup just completed, and 
> 
> Houston, we have a showstopper problem, For 20 years that header has had 
> a connandline recipe in it showing how to recover the vtape to a 
> scratchpad area where one could then pick and choose what to copy back 

That header is still there on my 3.5.1 system... but the format indeed
expanded since earlier versions, and also varies depending on the dump
type of that dump, etc.

Anyway, this is no longer a "build on Stretch" question, so you should
probably start a new thread -- being sure to include the full header of
your file so we are all looking at the same thing

Nathan



Nathan Stratton Treadway  -  natha...@ontko.com  -  Mid-Atlantic region
Ray Ontko & Co.  -  Software consulting services  -   http://www.ontko.com/
 GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239
 Key fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239

Re: building (non-Debianized) Amanda on Stretch

[Gene Heskett]

On Monday 27 May 2019 08:45:58 pm Gene Heskett wrote:

> On Monday 27 May 2019 05:32:52 pm Nathan Stratton Treadway wrote:
> > On Mon, May 27, 2019 at 16:48:38 -0400, Gene Heskett wrote:
> > > Fixed that, restarted xinetd, amcheck is happy.
> > >
> > > Step into /GenesAmandaHelper-0.61 and type ./backup.sh Daily" and
> > > gkrellm acts like its going to do it,
> >
> > Yay!
> >
> > Let us know how the dump goes, in the end...
>
> getting close. gkrellm is showing a couple hundred megs a second being
> read from sda, and being stuffed into sdd. But I think its used all of
> slot1, and has not started on slot2.  No, not yet, its still making
> 2GB gzip smunched pieces out of coyote/gene/Downloads, 8 of them so
> far.
>
> I'm gonna send this, as its been munching along for over 4 hours, but
> its not finished yet, its put 47GB in each of the two vtapes it was
> allowed, and has another 25GB piled up in dumps.  So there'll need to
> be a flush to at least one more vtape by the time it is done.  Looking
> promising Nathan.  Thanks.  I'll set it up in the amanda crontab
> tomorrow if this looks ok.
>
> > > gene@coyote:~$ sudo apt install netstat
> > > Reading package lists... Done
> > > Building dependency tree
> > > Reading state information... Done
> > > E: Unable to locate package netstat
> >
> > (The netstat command is in the "net-tools" package.  At this point
> > you don't need it, but it can be a handy tool to have when you need
> > to find out whether or not xinetd has a particular port open or
> > otherwise debug network-port activity.)
> >
> > Nathan
> >
Mine finally finished. As a first run and I didn't comment out 3/4ths of 
my disklist, it took 5 47GB vtapes, and I had a runtapes=2, I had to run 
2 flushes after the backup, but it all Just Worked after all the hoo hah 
of getting it to work. I had, using the older amanda, and a new 2T 
drive, run my mkvtapes script, but the new amanda would not use them 
until I had rerun that script after installing the new amanda, so 
apparently something in amlabel has been changed which caused an 
incompatibility.

I just looked at the header files of the backup just completed, and 

Houston, we have a showstopper problem, For 20 years that header has had 
a connandline recipe in it showing how to recover the vtape to a 
scratchpad area where one could then pick and choose what to copy back 
if doing a bare metal recovery with nothing but tar and gzip to work 
with.  And thats been handier than a turn button on the outhouse door at 
a family re-union several times here.

That recovery recipe in the 32k header_must_ be restored.  ASAP. Even if 
I have to blow away these 5 vtapes with nearly 240GB of data and start 
over.

I also see a bug of mine thats going to screw things up, one that I never 
noticed before in my wrapper script, when it uses more than one tape, 
and extremely rare occurrence, it does not write over the config and 
indice files for the first tape written. Nor does it write them to the 
vtape.  That buglet is mine, and I'll have to fix it. One of my theories 
when writing that wrapper in the first place better than a decade back, 
was to have a complete set of the configs used to make that vtape, and a 
full backup of the amanda database as it existed at the _end_ of that 
run, appended to the end of that tape, so that if a bare metal recovery 
was being done, I had enough data to recover to the exact status of the 
last run, as opposed to using the 1 day stale data from that same backup 
run. I can do it by hand of course, and if it only uses one vtape, the 
housekeeping in my scripts Just Works.

Is anyone else here using my GenesAmandaHelper scripts?




> >
> > 
> >-- -- Nathan Stratton Treadway  -  natha...@ontko.com  - 
> > Mid-Atlantic region Ray Ontko & Co.  -  Software consulting services
> >  -
> > http://www.ontko.com/ GPG Key:
> > http://www.ontko.com/~nathanst/gpg_key.txt   ID: 1023D/ECFB6239 Key
> > fingerprint = 6AD8 485E 20B9 5C71 231C  0C32 15F3 ADCD ECFB 6239
>
> Copyright 2019 by Maurice E. Heskett
> Cheers, Gene Heskett



Copyright 2019 by Maurice E. Heskett
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: lev 0 FAILED [data timeout]

[Jon LaBadie]

On Tue, May 28, 2019 at 12:57:18PM +1000, Tom Robinson wrote:
> On Tue, 14 May 2019 at 22:14, Nathan Stratton Treadway 
> wrote:
> 
> Hi Nathan,
> 
> Thanks for you reply and help.
> 
> On Mon, May 13, 2019 at 09:59:13 +1000, Tom Robinson wrote:
.
> 
> I am getting a new issue now which is annoying but not a show stopper. I
> think I will have to revisit my threshold settings to fix this but maybe
> you can offer some insight.
> 
> I have a tape robot and the following settings in amanda.conf
> 
> runtapes 3
> flush-threshold-dumped 100
> flush-threshold-scheduled 100
> taperflush 100
> autoflush yes
> 
> There is enough room for all the data to go on three tapes yet after the
> amdump run is complete only two tapes are written and I am left to flush
> the remaining dumps to tape manually.
> 
> I think it's because I'm trying to get a whole tape's worth of data before
> writing to tape. Is my thinking correct?

With "taperflush 100" you are saying do not write to tape unless
the tape will be filled.  So the third tape, being a partial, is
not written.

However you should not need to manually flush the data, "autoflush yes"
will write the leftover dumps onto the first tape of the next run.
> 
> What I'd like to do is make sure there's a tape's worth of data to write to
> the first two tapes in turn and then dump all remaining backup data to tape
> three (this will not be a complete tapes worth).
> 
> Should I be setting taperflush as follows to achieve this?
> 
> taperflush 0

Yes, from the comments in the sample "amanda.conf".

# You want to improve tape performance by waiting for a complete tape
# of data before writing anything. However, all dumps will be flushed;
# none will be left on the holding disk.
#
# flush-threshold-dumped100 # (or more)
# flush-threshold-scheduled 100 # (or more)
# taperflush0

jl
-- 
Jon H. LaBadie j...@jgcomp.com
 11226 South Shore Rd.  (703) 787-0688 (H)
 Reston, VA  20190  (703) 935-6720 (C)