Re: to compress or not to compress ???
Michael D. Schleif wrote: Also, what is the best way to turn off compression? ... Will this persist across power cycles? Will previously hardware compressed tapes turn hardware compression back on? See also: http://groups.yahoo.com/group/amanda-users/message/44453 http://groups.yahoo.com/group/amanda-users/message/43705 (I have this strange feeling of deja-vu :-) ) -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: to compress or not to compress ???
Michael D. Schleif wrote: I am curious as to a procedure of manually -- without amanda -- viewing and restoring from amanda-written tapes . . . There is a good explanation of all the possible restore scenario's in docs/RESTORE. -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: to compress or not to compress ???
Michael D. Schleif wrote: Also sprach Joshua Baker-LePain (Thu 03 Jul 02003 at 05:41:32PM -0400): On Thu, 3 Jul 2003 at 4:20pm, Michael D. Schleif wrote Also, using only amrestore, is it possible to get at individual files/directories, or is it only a matter of restoring the entire dump/tarball? Yes, depending. If you use dump, you can usually pipe amrestore to 'restore -i', the interactive restore, which will let you pick and choose. If you use tar, you can do a 'tar t' to get a table of contents, and then 'tar x myfile ./mydir/myfile2' to get particular files. This is where I'm getting lost: # sudo mt -f /dev/nst0 rewind # sudo tar tvf /dev/nst0 tar: /dev/nst0: Cannot read: Cannot allocate memory tar: At beginning of tape, quitting now tar: Error is not recoverable: exiting now What do you think? First you have to realize that on a tape you can put any sequence of bytes. To read and write those bytes from/to tape, it's also important that you do that with a suitable blocksize. In theory you could read the tape with cat, but cat does not have a way to specify the read blocksize. That's why we use the dd command. If you read a tape, you have to do it with a blocksize equal or greater than the blocksize used when written. Amanda writes your tapes with a 32k blocksize (you can change the default blocksize in recent versions -- know your config!) A file on tape is any sequence of bytes. You can put many files on tape sequentially, each separated with a filemarker. After the last file is an end of data marker. The format of an amanda is as follows: The first file is the amanda header. Then follow the different dump images. The first block (32k in de the default setup) contains in simple ascii the description of what follows/ The header looks like: AMANDA: TAPESTART DATE 20030702 TAPE Test-01 ^L [EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED] nulls to fill up the rest of the blocksize The next files are each contain a header of 1 block, that contains the description of the next bytes: AMANDA: FILE 20030702 sunny /space lev 0 comp N program /usr/local/bin/amgtar To restore, position tape at start of file and run: dd if=tape bs=32k skip=1 | /usr/local/bin/amgtar -f... - ^L [EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@ascii nulls to fill up the rest of the block As you can see, amanda explains even what command you can use to read it: it contains a tar file, and to read it skip over the 32k header and feed those bytes to tar. What you should be doing was: mt -f /dev/nst0 rewind mt -f /dev/nst0 fsf 1 # skip over the tape label dd if=/dev/nst0 bs=32k skip=1 | tar -tvf - If you had used software compression, add the z flag to gnutar. In docs/RESTORE, you can find all the commands you need to use an amanda tape without amanda software. -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: to compress or not to compress ???
Also sprach Paul Bijnens (Fri 04 Jul 02003 at 10:51:47AM +0200): snip / In docs/RESTORE, you can find all the commands you need to use an amanda tape without amanda software. That is an excellent document! I'm sorry that I bothered the list before finding and reading it . . . -- Best Regards, mds mds resource 877.596.8237 - Dare to fix things before they break . . . - Our capacity for understanding is inversely proportional to how much we think we know. The more I know, the more I know I don't know . . . -- pgp0.pgp Description: PGP signature
Re: to compress or not to compress ???
Michael D. Schleif said: [1] Should I use hardware compression? There seem to be several schools of thought here. I want to know how Amanda works with hardware compression? What are the advantages of using software compression? What are the disadvantages of using *both* hardware and software compression? I prefer software compression personally: -Amanda can make a more accurate estimate of how much tape is needed. So if you know your tape is 20 GB, and your software-compressed dump files total 21 GB, you know they won't all fit. With hardware compression you just have to guess-timate -Less bandwidth consumed if you do your compression on the client side (eg, before it comes to the tape server) -Less disk space used on your holding disk -If you back up to disks instead of tapes, hardware compression is not even an option The only drawback to software compression that I can see is the greater amount of cpu power consumed. For me, this is not really a problem, since my backups all happen in the wee hours when no-one is on my systems anyway. -- Kurt Yoder Sport Health network administrator
RE: to compress or not to compress ???
I Have the opposite viewpoint. I prefer hardware compression. It allows me to offload processing required to the tape drive (instead of my computers) Since some of my systems (including the backup server itself) can be slow, this actually speeds things up for me. Also, with hardware compression, I know I can restore the tape without having to worry about finding the right libraries and programs to do the restore. Also (AFAIK) you can't do remote compression with samba which I use for about half of my backups. On the down side, I need more holding disk and I have to guess at my tape capacity. As for both, don't do it. That is a huge waste of cpu and tape. The first compression algorithm will work fine. The second one will waste space with its headers and such since it probably won't be able to get the data any smaller than the software compression program did. If you don't believe me, try to zip a gzip file. See if it gets any smaller. -Original Message- From: Kurt Yoder [mailto:[EMAIL PROTECTED] Sent: Thursday, July 03, 2003 11:57 AM To: Michael D. Schleif Cc: amanda mailing list Subject: Re: to compress or not to compress ??? Michael D. Schleif said: [1] Should I use hardware compression? There seem to be several schools of thought here. I want to know how Amanda works with hardware compression? What are the advantages of using software compression? What are the disadvantages of using *both* hardware and software compression? I prefer software compression personally: -Amanda can make a more accurate estimate of how much tape is needed. So if you know your tape is 20 GB, and your software-compressed dump files total 21 GB, you know they won't all fit. With hardware compression you just have to guess-timate -Less bandwidth consumed if you do your compression on the client side (eg, before it comes to the tape server) -Less disk space used on your holding disk -If you back up to disks instead of tapes, hardware compression is not even an option The only drawback to software compression that I can see is the greater amount of cpu power consumed. For me, this is not really a problem, since my backups all happen in the wee hours when no-one is on my systems anyway. -- Kurt Yoder Sport Health network administrator
RE: to compress or not to compress ???
On Thu, 3 Jul 2003 at 12:17pm, Ean Kingston wrote I Have the opposite viewpoint. I prefer hardware compression. It allows me to offload processing required to the tape drive (instead of my computers) Since some of my systems (including the backup server itself) can be slow, this actually speeds things up for me. Also, with hardware compression, I know I can restore the tape without having to worry about finding the right libraries and programs to do the restore. Also (AFAIK) you can't do remote compression with samba which I use for about half of my backups. It really depends on your data. I have filesystems which are completely uncompressable in software (and expand in hardware), and filesystems which compress to 20% of their original size. Using software compressions lets me pick and choose which DLEs get compressed or not. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: to compress or not to compress ???
Michael D. Schleif wrote: [1] Should I use hardware compression? There seem to be several schools of thought here. I want to know how Amanda works with hardware compression? What are the advantages of using software compression? What are the disadvantages of using *both* hardware and software compression? See Kurt's answer. May I add that using both hardware compression and software compression at the same time actually lowers the capacity of your tapedrive (a DAT-tape, you said). If you blindly impose the compression algorithm on an already compressed data will enlarge that data. Now much depends on the algorithm. DAT-tapes use something from the compress family. Try: compress somefile cat somefile.Z | compress -v | wc -c Gzip does much better, by implementing a way to indicate that next few bytes are not compressed. cat somefile.Z | gzip -v | wc -c The only drive I know without that problem is an LTO1 drive. see: http://groups.yahoo.com/group/amanda-users/message/44455 [2] What are the optimal dumptypes for my situation? I would go for gnutar, compress client fast, index, record, and apropriate priority (user-data = higher, system = lower). -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: to compress or not to compress ???
On Thu, Jul 03, 2003 at 11:57:22AM -0400, Kurt Yoder wrote: I prefer software compression personally: -Amanda can make a more accurate estimate of how much tape is needed. So if you know your tape is 20 GB, and your software-compressed dump files total 21 GB, you know they won't all fit. With hardware compression you just have to guess-timate Conversely, if you have a DLE full of something that compresses down to 20% of its original size, Amanda will know that there's more room on the tape for other stuff. With hardware compression, Amanda won't know that that DLE compresses better than your other ones. - Better compression, probably. Hardware compression is typically some variant of LZ, isn't it? I don't know how gzip -1 (the default compress-fast) compares with that, but gzip -9 (the default compress-best) does a lot better. Ok, here's one quickie far-from-representative test. Sorted in order of decreasing size, a largish, mostly-text file, and its compression by compress, and by the several grades of gzip. Size CPU File --- 5560320 0 amanda-2.4.4.tar 2096458 0.88amanda-2.4.4.tar.Z 1496904 0.68amanda-2.4.4.tar.gz1 1227454 1.28amanda-2.4.4.tar.gz6 1220934 2.01amanda-2.4.4.tar.gz9 This time, even gzip -1 beat LZ. I don't know whether that's typical. -- | | /\ |-_|/ Eric Siegerman, Toronto, Ont.[EMAIL PROTECTED] | | / When I came back around from the dark side, there in front of me would be the landing area where the crew was, and the Earth, all in the view of my window. I couldn't help but think that there in front of me was all of humanity, except me. - Michael Collins, Apollo 11 Command Module Pilot
Re: to compress or not to compress ???
On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote: Also, with hardware compression, I know I can restore the tape without having to worry about finding the right libraries and programs to do the restore. True. But one can work around that by backing up / uncompressed, and making sure it contains a (possibly statically linked) copy of gzip. Hmmm, something to add to my to-do list :-( -- | | /\ |-_|/ Eric Siegerman, Toronto, Ont.[EMAIL PROTECTED] | | / When I came back around from the dark side, there in front of me would be the landing area where the crew was, and the Earth, all in the view of my window. I couldn't help but think that there in front of me was all of humanity, except me. - Michael Collins, Apollo 11 Command Module Pilot
Re: to compress or not to compress ???
On Thu, Jul 03, 2003 at 02:18:26PM -0400, Eric Siegerman wrote: - Better compression, probably. Hardware compression is typically some variant of LZ, isn't it? I don't know how gzip -1 (the default compress-fast) compares with that, but gzip -9 (the default compress-best) does a lot better. Ok, here's one quickie far-from-representative test. Sorted in order of decreasing size, a largish, mostly-text file, and its compression by compress, and by the several grades of gzip. SizeCPU File --- 5560320 0 amanda-2.4.4.tar 2096458 0.88 amanda-2.4.4.tar.Z 1496904 0.68 amanda-2.4.4.tar.gz1 1227454 1.28 amanda-2.4.4.tar.gz6 1220934 2.01 amanda-2.4.4.tar.gz9 My tests have always shown similar results. I wish we could do comprss-default (no -level option), which is the same as -6, and get nearly the same compression as -9 but with far less cpu. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: to compress or not to compress ???
On Thursday 03 July 2003 11:42, Michael D. Schleif wrote: Yes, I am learning -- at the expense of many questions ; First, a brief overview: I have five (5) Linux servers, totaling ~50 Gb used diskspace, divided roughly even across all five. I have several DAT tape drives, the largest of which is an HP DDS-3. I have twelve (12) DDS-3 tapes, and twenty (20) DDS-2 tapes, as well as several cleaning tapes. So far, I have configured: dumpcycle 7 runspercycle 7 runtapes 1 You left out tapecycle, which is the number of tapes in the rotation pool, in this case it should be not less than 15. I am studying _Using Amanda_ here: http://www.backupcentral.com/amanda.html I am confused about two (2) things: [1] Should I use hardware compression? Not if you can help it, for the reasons I'll develop. There seem to be several schools of thought here. I want to know how Amanda works with hardware compression? Amanda can use hardware compression, but since the hardware compressor hides the true tape capacity from amanda, you must cheat on the tapetype size entry, often by significant amounts. What are the advantages of using software compression? Amanda can know quite well how much a tape can hold because it counts bytes of compressed data fed to the drive. The tradeoff of course is cpu horsepower to do the compression. In a client-sever world, the compression can be offloaded to the client, and several clients can be doing their compression in parallel, so its not as big a concern as it first appears. What are the disadvantages of using *both* hardware and software compression? With hardware smunching, amanda has no idea how much data has actually been written to the tape. Sparse stuff can compress to maybe 1/2.6 of its original size, but amanda doesn't have any way of knowing that. OTOH, feed a bunch of tar.gz's, and .bz2's to that hardware compressor and it will get a tummy ache and make the output data stream as much as 15% bigger than the input was. [2] What are the optimal dumptypes for my situation? Yes, I have already struggled with and overcome dump vs. GNUTAR. My first mistake was using comp-root and comp-user on localhost. Yes, I understand that Amanda can facilitate planning and scheduling full vs. incremental backups. However, I am concerned about developing a simple recovery strategy. I am currently having problems with amrecover; but, I think that is due to short vs. FQDN usage -- so, I'll save that for another time. Just make sure that your tar is at least 1.13-19, and prefereably 1.13-25. Indexes are fubared in earlier versions. I am running exclusively Debian woody on all systems. I believe that I have a good working filesystem design. I am on a fast network. Yes, I come from a traditional system administrator's backup mindset, and I do not want that to undermine Amanda's design. What do you think? One thing to be aware of is that a tape, once written in the compressed mode, remembers that, and will overwrite your choices unless you go to a rather detailed method of removing the compressed flags. -- Cheers, Gene AMD [EMAIL PROTECTED] 320M [EMAIL PROTECTED] 512M 99.26% setiathome rank, not too shabby for a WV hillbilly Yahoo.com attornies please note, additions to this message by Gene Heskett are: Copyright 2003 by Maurice Eugene Heskett, all rights reserved.
Re: to compress or not to compress ???
Also sprach Gene Heskett (Thu 03 Jul 02003 at 02:51:39PM -0400): On Thursday 03 July 2003 11:42, Michael D. Schleif wrote: Yes, I am learning -- at the expense of many questions ; First, a brief overview: I have five (5) Linux servers, totaling ~50 Gb used diskspace, divided roughly even across all five. I have several DAT tape drives, the largest of which is an HP DDS-3. I have twelve (12) DDS-3 tapes, and twenty (20) DDS-2 tapes, as well as several cleaning tapes. So far, I have configured: dumpcycle 7 runspercycle 7 runtapes 1 You left out tapecycle, which is the number of tapes in the rotation pool, in this case it should be not less than 15. Wouldn't that be eight (8)? runspercycle * runtapes + 1 snip / One thing to be aware of is that a tape, once written in the compressed mode, remembers that, and will overwrite your choices unless you go to a rather detailed method of removing the compressed flags. How do I do this? These tape drives have all used compression, and many of these tapes have been used once or twice. So, it looks like I will *not* use hardware compression, and I want to reap all of the benefits of that strategy. Also, what is the best way to turn off compression? # sudo mt-gnu -f /dev/nst0 datcompression Compression on. Compression capable. Decompression capable. # sudo mt-gnu -f /dev/nst0 datcompression 0 Compression off. Compression capable. Decompression capable. # sudo mt-gnu -f /dev/nst0 datcompression Compression off. Compression capable. Decompression capable. Will this persist across power cycles? Will previously hardware compressed tapes turn hardware compression back on? What do you think? -- Best Regards, mds mds resource 877.596.8237 - Dare to fix things before they break . . . - Our capacity for understanding is inversely proportional to how much we think we know. The more I know, the more I know I don't know . . . -- pgp0.pgp Description: PGP signature
Re: to compress or not to compress ???
True. But one can work around that by backing up / uncompressed, and making sure it contains a (possibly statically linked) copy of gzip. ... or just write a couple of copies of a CD with gzip and whatever else you might need.
Re: to compress or not to compress ???
Assuming you're running Linux, all you need is some form of Linux rescue disk. I've got a bunch of Debian installer cd's lying around and have used them for similar purposes before. Probably most Linux installer cd's can be used like this, and I'm quite certain something like Knoppix would include all the utilities anyone would need. Who knows, might even work to restore tapes made with non-Linux amanda servers. BTW, if restoring without an amanda server, doesn't one also need amrestore? The other day I tried restoring an amanda image, and had to run it through amrestore before I could untar it. Eric Siegerman said: On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote: Also, with hardware compression, I know I can restore the tape without having to worry about finding the right libraries and programs to do the restore. True. But one can work around that by backing up / uncompressed, and making sure it contains a (possibly statically linked) copy of gzip. Hmmm, something to add to my to-do list :-( -- Kurt Yoder Sport Health network administrator
Re: to compress or not to compress ???
Michael D. Schleif said: Also, what is the best way to turn off compression? # sudo mt-gnu -f /dev/nst0 datcompression Compression on. Compression capable. Decompression capable. # sudo mt-gnu -f /dev/nst0 datcompression 0 Compression off. Compression capable. Decompression capable. # sudo mt-gnu -f /dev/nst0 datcompression Compression off. Compression capable. Decompression capable. Will this persist across power cycles? Will previously hardware compressed tapes turn hardware compression back on? mt datcompression 0 will work fine for you. Just make sure you run that command every time you boot. Put it in one of your boot scripts. I'm using debian, and made a file called nocompress which contains: #!/bin/sh #turn off tape drive hardware compression /bin/mt -f /dev/nst0 datcompression 0 This file resides in /etc/rc.boot and is mode -rwxr-xr-x (755). -- Kurt Yoder Sport Health network administrator
Re: to compress or not to compress ???
restoring w/o amrestore can be done with dd, gzip, tar, and so on, way messy compared to amrecover (or amrestore) but its possible. That's one of the plus's to amanda. Kurt Yoder wrote: Assuming you're running Linux, all you need is some form of Linux rescue disk. I've got a bunch of Debian installer cd's lying around and have used them for similar purposes before. Probably most Linux installer cd's can be used like this, and I'm quite certain something like Knoppix would include all the utilities anyone would need. Who knows, might even work to restore tapes made with non-Linux amanda servers. BTW, if restoring without an amanda server, doesn't one also need amrestore? The other day I tried restoring an amanda image, and had to run it through amrestore before I could untar it. Eric Siegerman said: On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote: Also, with hardware compression, I know I can restore the tape without having to worry about finding the right libraries and programs to do the restore. True. But one can work around that by backing up / uncompressed, and making sure it contains a (possibly statically linked) copy of gzip. Hmmm, something to add to my to-do list :-( -- Kurt Yoder Sport Health network administrator -- George Kelbley System Support Group Computer Science Department University of New Mexico 505-277-6502Fax: 505-277-6927
Re: to compress or not to compress ???
On Thu, 3 Jul 2003 at 4:34pm, Kurt Yoder wrote Assuming you're running Linux, all you need is some form of Linux rescue disk. I've got a bunch of Debian installer cd's lying around and have used them for similar purposes before. Probably most Linux installer cd's can be used like this, and I'm quite certain something like Knoppix would include all the utilities anyone would need. Who knows, might even work to restore tapes made with non-Linux amanda servers. BTW, if restoring without an amanda server, doesn't one also need amrestore? The other day I tried restoring an amanda image, and had to run it through amrestore before I could untar it. You can do with mt, dd, (gzip), and tar/restore. You just need to make sure you strip off the 32KB amanda header on the front of each tape file. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: to compress or not to compress ???
How do you recover an amanda dump file without amrestore? I had dd'd one to disk and tried to untar it but got the message not a tar file. Once I ran it through amrestore I was able to untar it though. George Kelbley said: restoring w/o amrestore can be done with dd, gzip, tar, and so on, way messy compared to amrecover (or amrestore) but its possible. That's one of the plus's to amanda. Kurt Yoder wrote: Assuming you're running Linux, all you need is some form of Linux rescue disk. I've got a bunch of Debian installer cd's lying around and have used them for similar purposes before. Probably most Linux installer cd's can be used like this, and I'm quite certain something like Knoppix would include all the utilities anyone would need. Who knows, might even work to restore tapes made with non-Linux amanda servers. BTW, if restoring without an amanda server, doesn't one also need amrestore? The other day I tried restoring an amanda image, and had to run it through amrestore before I could untar it. Eric Siegerman said: On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote: Also, with hardware compression, I know I can restore the tape without having to worry about finding the right libraries and programs to do the restore. True. But one can work around that by backing up / uncompressed, and making sure it contains a (possibly statically linked) copy of gzip. Hmmm, something to add to my to-do list :-( -- Kurt Yoder Sport Health network administrator -- George KelbleySystem Support Group Computer Science Department University of New Mexico 505-277-6502 Fax: 505-277-6927 -- Kurt Yoder Sport Health network administrator
Re: to compress or not to compress ???
Also sprach George Kelbley (Thu 03 Jul 02003 at 02:41:12PM -0600): restoring w/o amrestore can be done with dd, gzip, tar, and so on, way messy compared to amrecover (or amrestore) but its possible. That's one of the plus's to amanda. Kurt Yoder wrote: Assuming you're running Linux, all you need is some form of Linux rescue disk. I've got a bunch of Debian installer cd's lying around and have used them for similar purposes before. Probably most Linux installer cd's can be used like this, and I'm quite certain something like Knoppix would include all the utilities anyone would need. Who knows, might even work to restore tapes made with non-Linux amanda servers. BTW, if restoring without an amanda server, doesn't one also need amrestore? The other day I tried restoring an amanda image, and had to run it through amrestore before I could untar it. Eric Siegerman said: On Thu, Jul 03, 2003 at 12:17:16PM -0400, Ean Kingston wrote: Also, with hardware compression, I know I can restore the tape without having to worry about finding the right libraries and programs to do the restore. True. But one can work around that by backing up / uncompressed, and making sure it contains a (possibly statically linked) copy of gzip. Hmmm, something to add to my to-do list :-( Am I right that amrecover is useless *without* an index? Also, using only amrestore, is it possible to get at individual files/directories, or is it only a matter of restoring the entire dump/tarball? I am curious as to a procedure of manually -- without amanda -- viewing and restoring from amanda-written tapes . . . -- Best Regards, mds mds resource 877.596.8237 - Dare to fix things before they break . . . - Our capacity for understanding is inversely proportional to how much we think we know. The more I know, the more I know I don't know . . . -- pgp0.pgp Description: PGP signature
Re: to compress or not to compress ???
On Thu, 3 Jul 2003 at 4:20pm, Michael D. Schleif wrote Am I right that amrecover is useless *without* an index? Yep. Also, using only amrestore, is it possible to get at individual files/directories, or is it only a matter of restoring the entire dump/tarball? Yes, depending. If you use dump, you can usually pipe amrestore to 'restore -i', the interactive restore, which will let you pick and choose. If you use tar, you can do a 'tar t' to get a table of contents, and then 'tar x myfile ./mydir/myfile2' to get particular files. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: to compress or not to compress ???
Joshua Baker-LePain wrote: On Thu, 3 Jul 2003 at 4:20pm, Michael D. Schleif wrote Am I right that amrecover is useless *without* an index? Yep. No argument about it. But, I don't keep index files along with the backup image on tape. Before running amrecover, I would run a script to extract and recreate the index files and use them to run amrecover. Are there any drawbacks to that scheme ??? Also, using only amrestore, is it possible to get at individual files/directories, or is it only a matter of restoring the entire dump/tarball? Yes, depending. If you use dump, you can usually pipe amrestore to 'restore -i', the interactive restore, which will let you pick and choose. If you use tar, you can do a 'tar t' to get a table of contents, and then 'tar x myfile ./mydir/myfile2' to get particular files.
Re: to compress or not to compress ???
On Thursday 03 July 2003 15:10, Michael D. Schleif wrote: Also sprach Gene Heskett (Thu 03 Jul 02003 at 02:51:39PM -0400): On Thursday 03 July 2003 11:42, Michael D. Schleif wrote: Yes, I am learning -- at the expense of many questions ; First, a brief overview: I have five (5) Linux servers, totaling ~50 Gb used diskspace, divided roughly even across all five. I have several DAT tape drives, the largest of which is an HP DDS-3. I have twelve (12) DDS-3 tapes, and twenty (20) DDS-2 tapes, as well as several cleaning tapes. So far, I have configured: dumpcycle 7 runspercycle 7 runtapes 1 You left out tapecycle, which is the number of tapes in the rotation pool, in this case it should be not less than 15. Wouldn't that be eight (8)? No, not if you don't want to be overwriting the only full you have with one that might just fail, leaving you with no full to anchor the recovery to. runspercycle * runtapes + 1 Most of us consider that to be 2*runspercycle*runtapes+1 snip / One thing to be aware of is that a tape, once written in the compressed mode, remembers that, and will overwrite your choices unless you go to a rather detailed method of removing the compressed flags. How do I do this? These tape drives have all used compression, and many of these tapes have been used once or twice. So, it looks like I will *not* use hardware compression, and I want to reap all of the benefits of that strategy. Clip this script and adjust to suit. -- #!/bin/sh if [ `whoami` != 'root' ]; then echo echo ! Warning !! echo fixcomp needs to be run by the user root, echo else the perms on some commands will be denied. echo exit 1 fi # blatently stolen from the bash manual NUMBERS=0 1 2 3 # Increase or decrease to match the number of slots # in your magazine, mine holds 4 DDS2 tapes for number in $NUMBERS do echo amtape DailySet1 slot $number su amanda -c 'amtape DailySet1 slot '$number mt -f /dev/nst0 rewind dd if=/dev/st0 of=./scratch bs=64K count=1 ls -l ./scratch # see how big the label is mt -f /dev/nst0 compression off mt -f /dev/nst0 defcompression -1 # write it back, with fixed 32k block in case the read # above was short dd if=./scratch of=/dev/nst0 bs=32K conv=sync mt -f /dev/nst0 tell # This seems to get rid of compressed headers echo forcing buffer flush with an 4+ meg write to tape # $number dd if=/dev/zero bs=32K count=130 of=/dev/st0 echo Now reading the label # display it on the console to verify its undamaged dd if=/dev/st0 bs=32K mt -f /dev/st0 status done exit 0 This is probably not the only way to do it, but it seems to work here just fine. This does however, assume that you have located the jumper or dipswitch that sets the drives default compression status and set it to the off position. Also, what is the best way to turn off compression? # sudo mt-gnu -f /dev/nst0 datcompression Compression on. Compression capable. Decompression capable. # sudo mt-gnu -f /dev/nst0 datcompression 0 Compression off. Compression capable. Decompression capable. # sudo mt-gnu -f /dev/nst0 datcompression Compression off. Compression capable. Decompression capable. Will this persist across power cycles? No, on powerup the drive will revert to the dipswitch settings. Will previously hardware compressed tapes turn hardware compression back on? Yes, as soon as the drive recognizes the tape. The above script should fix that. In my tests here, if the extra 4 meg write isn't done, then the tape will be uncompressed, but the header still will be. The extra write to force the buffer flush seems to fix that. -- Cheers, Gene AMD [EMAIL PROTECTED] 320M [EMAIL PROTECTED] 512M 99.26% setiathome rank, not too shabby for a WV hillbilly Yahoo.com attornies please note, additions to this message by Gene Heskett are: Copyright 2003 by Maurice Eugene Heskett, all rights reserved.
Re: to compress or not to compress ???
On Thursday 03 July 2003 18:03, bao wrote: Joshua Baker-LePain wrote: On Thu, 3 Jul 2003 at 4:20pm, Michael D. Schleif wrote Am I right that amrecover is useless *without* an index? Yep. No argument about it. But, I don't keep index files along with the backup image on tape. Before running amrecover, I would run a script to extract and recreate the index files and use them to run amrecover. Are there any drawbacks to that scheme ??? Yes. If the indices do make it into the backup tape (I've had it miss-fire here and have taken other means to assure I have an up2date backup of them on the same tape here), they will still be a day old compared to the tapes actual contents. Also, using only amrestore, is it possible to get at individual files/directories, or is it only a matter of restoring the entire dump/tarball? Yes, depending. If you use dump, you can usually pipe amrestore to 'restore -i', the interactive restore, which will let you pick and choose. If you use tar, you can do a 'tar t' to get a table of contents, and then 'tar x myfile ./mydir/myfile2' to get particular files. -- Cheers, Gene AMD [EMAIL PROTECTED] 320M [EMAIL PROTECTED] 512M 99.26% setiathome rank, not too shabby for a WV hillbilly Yahoo.com attornies please note, additions to this message by Gene Heskett are: Copyright 2003 by Maurice Eugene Heskett, all rights reserved.
