Re: add option --occurrence to gnutar amrecover command [Was: Re: Problems accessing my new autoloader]
I tested it, and tar doesn't exit after the first occurrence. Maybe
because it read from a pipe.
Anyway, amidxtaped will read the complete image from tape, decompress
it, send it to amrecover. amrecover will fail because tar would have exited.
Jean-Louis
Paul Bijnens wrote:
On 2007-11-23 09:35, Francis Galiegue wrote:
Le Friday 23 November 2007 09:17:01 Marc Muehlfeld, vous avez écrit :
One additional info to my last mail (sorry. I just replied to the
sender,
and not to the list, thats why I let it below):
5 Minutes after I send this mail, the silent amrecover finished it's
work
and the files where recovered. But why does it take 45 minutes to
restore
one 45kb file? On the tape were just 9.1 GB in 60 files, so it
should be
finished fast. Also its a fast LTO3 drive connected to a fast
Adaptec 29160
SCSI card
Tar will unroll ALL the archive to find your file. Not sure why it
doesn't stop if it finds the file at the beginning/middle of the
archive though.
Because a file can appear multiple times in a tar archive, and
the normal semantics says that it needs the last version, unless
specified on the command line.
In gnutar you may store files multiple times in an archive. e.g.
$ gtar -cf /tmpk/multi.tar file1 file1 file1
$ gtar -tf /tmp/multi.tar
and note the file appears 3 times.
You can even append to an existing archive:
$ date > file1
$ gtar -rf /tmp/multi.tar file1
and when you restore, you get last version only.
Or be more specific about a specific occurrence:
$ gtar --occurrence=3 -tvf /tmp/multi.tar
Amanda will *not* store multiple files in a tar archive, and
we could add indeed the option "--occurrence" (the default=1)
for extracting with amrecover and then, according to the info page,
gnutar will terminate without scanning to the end of the archive.
This could speed up the restore process considerably for the "few files"
restores, the most frequent requests.
So the default restore command would become:
gtar --occurrence --numeric-owner --xpGvf - extractlist
Patch against 2.5.2p1: file recover-src/extract_list.c:
--- extract_list.c_ORIG 2007-11-23 10:17:22.0 +0100
+++ extract_list.c 2007-11-23 10:18:36.0 +0100
@@ -1773,7 +1773,7 @@
#endif
case IS_TAR:
case IS_GNUTAR:
-extra_params = 4;
+extra_params = 5;
break;
case IS_SAMBA_TAR:
extra_params = 3;
@@ -1824,6 +1824,7 @@
case IS_TAR:
case IS_GNUTAR:
restore_args[j++] = stralloc("tar");
+ restore_args[j++] = stralloc("--occurrence"); /* do not scan
the rest of the archive if file found once */
restore_args[j++] = stralloc("--numeric-owner");
restore_args[j++] = stralloc("-xpGvf");
restore_args[j++] = stralloc("-"); /* data on stdin */
Re: Problems accessing my new autoloader
It was a broken SCSI cable that came with the autoloader. I took a different one and now it's fun: # dd if=/dev/zero bs=65536 count=81920 of=/dev/nst0 81920+0 Datensätze ein 81920+0 Datensätze aus 5368709120 Bytes (5,4 GB) kopiert, 52,5265 s, 102 MB/s :-) Regards Marc -- Marc Muehlfeld (Leitung Systemadministration) Zentrum fuer Humangenetik und Laboratoriumsmedizin Dr. Klein und Dr. Rost Lochhamer Str. 29 - D-82152 Martinsried Telefon: +49(0)89/895578-0 - Fax: +49(0)89/895578-78 http://www.medizinische-genetik.de
Re: Problems accessing my new autoloader
Francis Galiegue schrieb: Le Friday 23 November 2007 08:52:17 Marc Muehlfeld, vous avez écrit : [I think you meant to send this message to the list, so the answer goes there] Yes. Sorry. It was to early in the morning :-) * rewind: mt -f /dev/nst0 rewind Done * show header: dd if=/dev/nst0 bs=32k count=1 # dd if=/dev/nst0 bs=32k count=1 AMANDA: TAPESTART DATE 20071122213456 TAPE KAS002 014 1+0 Datensätze ein 1+0 Datensätze aus 32768 Bytes (33 kB) kopiert, 2,59912 s, 12,6 kB/s * go to next file: mt -f /dev/nst0 fsf 1 Done. * show DLE header: dd if=/dev/nst0 bs=32k count=1 # dd if=/dev/nst0 bs=32k count=1 AMANDA: SPLIT_FILE 20071122213456 protein.mr.lfmg.de /BACKUP part 1/1 lev 0 comp .gz program /bin/tar To restore, position tape at start of file and run: dd if= bs=32k skip=1 | /usr/bin/gzip -dc | /bin/tar -xpGf - ... 1+0 Datensätze ein 1+0 Datensätze aus 32768 Bytes (33 kB) kopiert, 0,0481214 s, 681 kB/s (There was a ^L or something else inside that cleared the screen after the line "dd if=..."). * IF the DLE header is correct (ie, no character substitution), then try and list what's in the archive: dd if=/dev/nst0 bs=32k|tar t(z? are your archives compressed?)f - # dd if=/dev/nst0 bs=32k|tar tzf - ./ ./IAS-Backup/ . . dd: Reading from „/dev/nst0“: Input/Output-Error 36706+0 Datensätze ein 36706+0 Datensätze aus 1202782208 Bytes (1,2 GB) kopiert, 1603,55 s, 750 kB/s gzip: stdin: unexpected end of file tar: Unerwartetes Dateiende im Archiv. tar: Nicht behebbarer Fehler: Programmabbruch. The whole dd proces took 27 Minutes. -- Marc Muehlfeld (Leitung Systemadministration) Zentrum fuer Humangenetik und Laboratoriumsmedizin Dr. Klein und Dr. Rost Lochhamer Str. 29 - D-82152 Martinsried Telefon: +49(0)89/895578-0 - Fax: +49(0)89/895578-78 http://www.medizinische-genetik.de
add option --occurrence to gnutar amrecover command [Was: Re: Problems accessing my new autoloader]
On 2007-11-23 09:35, Francis Galiegue wrote:
Le Friday 23 November 2007 09:17:01 Marc Muehlfeld, vous avez écrit :
One additional info to my last mail (sorry. I just replied to the sender,
and not to the list, thats why I let it below):
5 Minutes after I send this mail, the silent amrecover finished it's work
and the files where recovered. But why does it take 45 minutes to restore
one 45kb file? On the tape were just 9.1 GB in 60 files, so it should be
finished fast. Also its a fast LTO3 drive connected to a fast Adaptec 29160
SCSI card
Tar will unroll ALL the archive to find your file. Not sure why it doesn't
stop if it finds the file at the beginning/middle of the archive though.
Because a file can appear multiple times in a tar archive, and
the normal semantics says that it needs the last version, unless
specified on the command line.
In gnutar you may store files multiple times in an archive. e.g.
$ gtar -cf /tmpk/multi.tar file1 file1 file1
$ gtar -tf /tmp/multi.tar
and note the file appears 3 times.
You can even append to an existing archive:
$ date > file1
$ gtar -rf /tmp/multi.tar file1
and when you restore, you get last version only.
Or be more specific about a specific occurrence:
$ gtar --occurrence=3 -tvf /tmp/multi.tar
Amanda will *not* store multiple files in a tar archive, and
we could add indeed the option "--occurrence" (the default=1)
for extracting with amrecover and then, according to the info page,
gnutar will terminate without scanning to the end of the archive.
This could speed up the restore process considerably for the "few files"
restores, the most frequent requests.
So the default restore command would become:
gtar --occurrence --numeric-owner --xpGvf - extractlist
Patch against 2.5.2p1: file recover-src/extract_list.c:
--- extract_list.c_ORIG 2007-11-23 10:17:22.0 +0100
+++ extract_list.c 2007-11-23 10:18:36.0 +0100
@@ -1773,7 +1773,7 @@
#endif
case IS_TAR:
case IS_GNUTAR:
-extra_params = 4;
+extra_params = 5;
break;
case IS_SAMBA_TAR:
extra_params = 3;
@@ -1824,6 +1824,7 @@
case IS_TAR:
case IS_GNUTAR:
restore_args[j++] = stralloc("tar");
+ restore_args[j++] = stralloc("--occurrence"); /* do not scan
the rest of the archive if file found once */
restore_args[j++] = stralloc("--numeric-owner");
restore_args[j++] = stralloc("-xpGvf");
restore_args[j++] = stralloc("-"); /* data on stdin */
--
Paul Bijnens, xplanation Technology ServicesTel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512
http://www.xplanation.com/ email: [EMAIL PROTECTED]
***
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, *
* F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?" ... YES ... Phew ... I'm out *
***
Re: Problems accessing my new autoloader
Le Friday 23 November 2007 09:17:01 Marc Muehlfeld, vous avez écrit : > One additional info to my last mail (sorry. I just replied to the sender, > and not to the list, thats why I let it below): > > 5 Minutes after I send this mail, the silent amrecover finished it's work > and the files where recovered. But why does it take 45 minutes to restore > one 45kb file? On the tape were just 9.1 GB in 60 files, so it should be > finished fast. Also its a fast LTO3 drive connected to a fast Adaptec 29160 > SCSI card > Tar will unroll ALL the archive to find your file. Not sure why it doesn't stop if it finds the file at the beginning/middle of the archive though. -- Francis Galiegue, [EMAIL PROTECTED] - Ingénieur système [ATTENTION : CHANGEMENT DE COORDONNÉES !] One2team - 42 Av. Raymond Poincaré - 75116 PARIS CEDEX +33683877875, +33178945552
Re: Problems accessing my new autoloader
Le Friday 23 November 2007 08:52:17 Marc Muehlfeld, vous avez écrit : [I think you meant to send this message to the list, so the answer goes there] > Hi, > > I meanwhile tried to tar a file directly to the tapedev, rewind it and > tried to restore. What I got back was a only totally mixed part of what I > send to tape?! > > I changed the changer library emulation and control mode from "native" to > "Exabyte EZ17", reboot the server and retried. Now it was possible to label > tapes. > > Happy, that everything seams to work, I started a backup of one DLE (60 > files, 9.1 GB). The report told me, it was successfully done. Then I tried > to restore. On my first try it start loading files from tape. But after 30 > minutes of showing nothing (after amrecover restored the last file), I > canceled the operation. I retried a second time. And now after loading the > tape nothing happens. amrecover is showing since 40 minutes: > > Extracting files using tape drive chg-zd-mtx on host nucleus.mr.lfmg.de. > Load tape KAS002 now > Continue [?/Y/n/s/t]? y > > > > mt on a second console shows: > # mt -f /dev/nst0 status > mt: /dev/nst0: rmtopen failed: Device or resource busy. > > > The display on the autoloader shows that drive-1 is idle. OK, try again by hand for this tape: * rewind: mt -f /dev/nst0 rewind * show header: dd if=/dev/nst0 bs=32k count=1 * go to next file: mt -f /dev/nst0 fsf 1 * show DLE header: dd if=/dev/nst0 bs=32k count=1 * IF the DLE header is correct (ie, no character substitution), then try and list what's in the archive: dd if=/dev/nst0 bs=32k|tar t(z? are your archives compressed?)f - -- Francis Galiegue, [EMAIL PROTECTED] - Ingénieur système [ATTENTION : CHANGEMENT DE COORDONNÉES !] One2team - 42 Av. Raymond Poincaré - 75116 PARIS CEDEX +33683877875, +33178945552
Re: Problems accessing my new autoloader
Le Thursday 22 November 2007 20:44:16 Francis Galiegue, vous avez écrit :
> Le Thursday 22 November 2007 20:22:13 Paul Bijnens, vous avez écrit :
> > Marc Muehlfeld wrote:
> > > Francis Galiegue schrieb:
> > >> And what if you try to read the label by hand? Ie, try at the command
> > >> line:
> > >>
> > >> dd if=/dev/nst0 bs=32k count=1
> > >>
> > >> Does anything appear or do you have an error?
> > >
> > > # dd if=/dev/nst0 bs=32k count=1
> > > AEAFDA: TAPESTART DATE P TAPE KAS001
> >
> > AEAFDA ??
> > that should have been "AMANDA".
> > What is changing your M's into E's and N's into F's ??
>
> OK, I was wondering about this too... Also, the X seems to have been
> changed for P, if I'm not mistaken?
Heh...
[EMAIL PROTECTED] ~ $ perl -e "printf(\"%d\n\", (ord('E') - ord('M'))%13)"
5
[EMAIL PROTECTED] ~ $ perl -e "printf(\"%d\n\", (ord('F') - ord('N'))%13)"
5
[EMAIL PROTECTED] ~ $ perl -e "printf(\"%d\n\", (ord('P') - ord('X'))%13)"
5
--
Francis Galiegue, [EMAIL PROTECTED] - Ingénieur système
[ATTENTION : CHANGEMENT DE COORDONNÉES !]
One2team - 42 Av. Raymond Poincaré - 75116 PARIS CEDEX
+33683877875, +33178945552
Re: Problems accessing my new autoloader
Le Thursday 22 November 2007 20:22:13 Paul Bijnens, vous avez écrit : > Marc Muehlfeld wrote: > > Francis Galiegue schrieb: > >> And what if you try to read the label by hand? Ie, try at the command > >> line: > >> > >> dd if=/dev/nst0 bs=32k count=1 > >> > >> Does anything appear or do you have an error? > > > > # dd if=/dev/nst0 bs=32k count=1 > > AEAFDA: TAPESTART DATE P TAPE KAS001 > > AEAFDA ?? > that should have been "AMANDA". > What is changing your M's into E's and N's into F's ?? > OK, I was wondering about this too... Also, the X seems to have been changed for P, if I'm not mistaken? -- Francis Galiegue, [EMAIL PROTECTED] - Ingénieur système [ATTENTION : CHANGEMENT DE COORDONNÉES !] One2team - 42 Av. Raymond Poincaré - 75116 PARIS CEDEX +33683877875, +33178945552
Re: Problems accessing my new autoloader
Marc Muehlfeld wrote: Francis Galiegue schrieb: And what if you try to read the label by hand? Ie, try at the command line: dd if=/dev/nst0 bs=32k count=1 Does anything appear or do you have an error? # dd if=/dev/nst0 bs=32k count=1 AEAFDA: TAPESTART DATE P TAPE KAS001 AEAFDA ?? that should have been "AMANDA". What is changing your M's into E's and N's into F's ?? 014 1+0 Datensätze ein 1+0 Datensätze aus 32768 Bytes (33 kB) kopiert, 0,0182859 s, 1,8 MB/s
Re: Problems accessing my new autoloader
Francis Galiegue schrieb: > And what if you try to read the label by hand? Ie, try at the command > line: > > dd if=/dev/nst0 bs=32k count=1 > > Does anything appear or do you have an error? # dd if=/dev/nst0 bs=32k count=1 AEAFDA: TAPESTART DATE P TAPE KAS001 014 1+0 Datensätze ein 1+0 Datensätze aus 32768 Bytes (33 kB) kopiert, 0,0182859 s, 1,8 MB/s -- Marc Muehlfeld (Leitung Systemadministration) Zentrum fuer Humangenetik und Laboratoriumsmedizin Dr. Klein und Dr. Rost Lochhamer Str. 29 - D-82152 Martinsried Telefon: +49(0)89/895578-0 - Fax: +49(0)89/895578-78 http://www.medizinische-genetik.de
Re: Problems accessing my new autoloader
Le Thursday 22 November 2007 19:42:39, vous avez écrit : > FL schrieb: > > out of curiousity, I would try not using the symbolic link--use > > /dev/nst0 in /etc/amanda/KST/amanda.conf > > instead of the symbolic link dev/tape. > > Thanks, but doesn't helped: > amlabel: no label found, are you sure /dev/nst0 is non-rewinding? And what if you try to read the label by hand? Ie, try at the command line: dd if=/dev/nst0 bs=32k count=1 Does anything appear or do you have an error? -- Francis Galiegue, [EMAIL PROTECTED] - Ingénieur système [ATTENTION : CHANGEMENT DE COORDONNÉES !] One2team - 42 Av. Raymond Poincaré - 75116 PARIS CEDEX +33683877875, +33178945552
Re: Problems accessing my new autoloader
FL schrieb: > out of curiousity, I would try not using the symbolic link--use > /dev/nst0 in /etc/amanda/KST/amanda.conf > instead of the symbolic link dev/tape. Thanks, but doesn't helped: amlabel: no label found, are you sure /dev/nst0 is non-rewinding? -- Marc Muehlfeld (Leitung Systemadministration) Zentrum fuer Humangenetik und Laboratoriumsmedizin Dr. Klein und Dr. Rost Lochhamer Str. 29 - D-82152 Martinsried Telefon: +49(0)89/895578-0 - Fax: +49(0)89/895578-78 http://www.medizinische-genetik.de
