Re: AManda Virtual Tape Manager (amvtm)
On Sat, 6 Oct 2007 20:37:07 -0500, Dustin J. Mitchell [EMAIL PROTECTED] said: Dustin On 9/28/07, Ronan Keryell [EMAIL PROTECTED] wrote: I've written a small script to help us to use AMANDA on removable disks. It is in use for 6 months and is helpful, so it may be useful for someone else too. :-) Dustin We'd love to have a link to this on the wiki Dustin (http://wiki.zmanda.com). Of course you can. Dustin It looks fairly Linux- and purpose-specific, so I'm not sure Dustin it's appropriate for inclusion in the codebase, but it could Dustin certainly inspire others who are in similar situations. Yes, it is Linux specific and tested on Debian. But I've just found the right hardware to use it: http://www.engadget.com/2007/10/19/the-sata-hdd-stage-rack-at-last/ :-) Right know I need to change 3 screws every 8 days, so it could be quicker with this rack... :-) Dustin On a related note, since you're scripting around Amanda, I'd Dustin be interested to hear your thoughts on the recent proposal on Dustin amanda-hackers to move Amanda's core algorithms to a scripting Dustin language. See http://marc.info/?t=11916262923r=1w=2 That is quite interesting, but far enough from my script above, that is only used as a virtual tape management tool. Of course, with a SWIG interface, I could easily peek into AMANDA config file without having to write am AMANDA config parser or worse, hard-coding them in the bash script (what is just done now :-) ). So I'm going to follow your SWIG action... PS: I switched from Perl to Python few years ago after years of Perl programming and teaching, when I discovered I was no longer able to read code I had writtem before. :-) -- Ronan KERYELL |\/ Tel:(+33|0) 2.29.00.14.15 Département Informatique |/) Fax:(+33|0) 2.29.00.12.82 ENST Bretagne, CS 83818 KGSM:(+33|0) 6.13.14.37.66 F-29238 PLOUZANÉ CEDEX 3 |\ E-mail: [EMAIL PROTECTED] FRANCE| \ http://enstb.org/~keryell sip:[EMAIL PROTECTED]
AManda Virtual Tape Manager (amvtm)
I've written a small script to help us to use AMANDA on removable disks. It is in use for 6 months and is helpful, so it may be useful for someone else too. :-) It is on: https://info.enstb.org/projets/rire/documentation/amanda/amvtm/file_view Here is the in-line help: usage: ./amvtm [option [arguments]]+ revision of ./amvtm is amvtm,v 1.5 2007/09/28 14:01:16 keryell Exp AManda Virtual Tape Manager (amvtm) can manage a virtual tape infrastructure with tapes automatically numbered from begin to end for example. The use case is to have USB disks with few virtual tapes on them and to change a disk when it is full. The options: -b or --begin n: set the tape slot begin number. 1 by default -e or --end n: set the tape slot end number -n or --slot-number n: set the number of tape slot per disk, instead of specifying with -e -d or --disk n: set the disk number (starting at 0). The main idea is to allocate automatically slot numbers from -n and -d information -f or --format to partition and format the disk -m or --mount to mount the disk -u or --unmount to unmount the disk -t or --amtape arguments: use amtape for more specific tape control. This option must be the last one since all the remaining arguments are given to amtape --verbose: be more verbose (show issued svn commands) Examples: * To mount an old virtual tape disk: ./amvtm --mount * To unmount an old virtual tape disk before removing it: ./amvtm --unmount * To partition, format, mount and create 8 slots with 8 labeled virtual tapes on a new disk that is number 2 (note the order of the option is not important): ./amvtm --format --mount --create --slot-number 8 --disk 2 * To show the current tape content: ./amvtm -t show To display amtape short help: ./amvtm -t show -h At ENST Bretagne/RIRE we have 25 250 GB disks we have split in 8 virtual tapes each, that is 200 virtual tapes. We plug one disk in a USB2-SATA II rack and change it every 8 AMANDA runs. Our label names are of the form [EMAIL PROTECTED], that means the virtual tape 3 on disk 21 that was labeled on 28/09/2007 (to be able to have an idea of the age of the tape/disk) -- Ronan KERYELL |\/ Tel:(+33|0) 2.29.00.14.15 Département Informatique |/) Fax:(+33|0) 2.29.00.12.82 ENST Bretagne, CS 83818 KGSM:(+33|0) 6.13.14.37.66 F-29238 PLOUZANÉ CEDEX 3 |\ E-mail: [EMAIL PROTECTED] FRANCE| \ http://enstb.org/~keryell sip:[EMAIL PROTECTED]
Re: Debian packages for Amanda
On Fri, 8 Sep 2006 22:23:45 -0400, Jon LaBadie [EMAIL PROTECTED] said: Wouldn't I need to at least tell it where the backup server is? Jon The server initiates the backup session, so the client does Jon not need server knowledge for this. Yes, but often an authentication scheme is used where you need the server to be specified on the client side, for example. Jon For recovery a default server is compiled in, but that can be Jon overridden with command line arguments. Jon The system I'm typing this on was my backup server, and of course Jon a client of itself. I have now moved the amanda service to a Jon different computer. When I did, I made no changes to the current Jon system. It is just responds as a client when the new server Jon contacts it. I guess then you need to modify some entry in the DNS if you have a dedicated entry (IP) for the AMANDA service (because I'm not sure amandahost can deal with CNAME for security reasons) or you use an authentication based for example on a pure public-private key approach. -- Ronan KERYELL |\/ Tel:(+33|0) 2.29.00.14.15 Département Informatique|/) Fax:(+33|0) 2.29.00.12.82 ENST Bretagne, CS 83818 KGSM:(+33|0) 6.13.14.37.66 F-29238 PLOUZANÉ CEDEX |\ E-mail: [EMAIL PROTECTED] FRANCE | \ http://enstb.org/~keryell callto:ils.seconix.com/[EMAIL PROTECTED]
Re: 2.5.1 parsing conf problems?
On Thu, 7 Sep 2006 16:34:36 -0400, Jean-Francois Malouin [EMAIL PROTECTED] said: Jean-Francois Hello, I had my first compilation try at amanda-2.5.1 Jean-Francois today on a SGI irix-6.5 and apart from the fact that I Jean-Francois had to add '-lgen' to LDFLAGS while configuring Jean-Francois everything went smooth. I read the release notes on the Jean-Francois wiki on the new auth scheme and updated inetd.conf and Jean-Francois the amandahost file and after installing a new Jean-Francois amanda.conf and its minions (I plan to use a STK L180 Jean-Francois with LTO drives) my first attempt at checking the Jean-Francois server and client fails while parsing the conf file: Jean-Francois ...amanda.conf, line 43: configuration keyword Jean-Francois expected ...amanda.conf, line 43: end of line is Jean-Francois expected Jean-Francois ad nauseam. amanda.conf is 42 lines long... Jean-Francois I compiled using the native sgi C compiler and gcc Jean-Francois 3.3.2 with the same results. Any ideas? Same here for me. I guesse this is because the format of the conf file has changed and is less permisive about old features. So I needed to update my conf, by replacing for example option compress best by compress client best and so on. And afterwards it was fine. But with your file, we could help more... -- Ronan KERYELL |\/ Tel:(+33|0) 2.29.00.14.15 Département Informatique|/) Fax:(+33|0) 2.29.00.12.82 ENST Bretagne, CS 83818 KGSM:(+33|0) 6.13.14.37.66 F-29238 PLOUZANÉ CEDEX |\ E-mail: [EMAIL PROTECTED] FRANCE | \ http://enstb.org/~keryell callto:ils.seconix.com/[EMAIL PROTECTED]
Re: using disk instead of tape
On Tue, 5 Sep 2006 04:09:05 -0500, Phil Howard [EMAIL PROTECTED] said: Phil If tar can read from raw tape, it can read from raw disk. I've Phil already done that several times for various things. Bare metal Phil recovery will need at a minimum the tar or dump utility Phil depending on format used. I've thought about this raw partition stuff and I'm a bit afraid like some others on the list. First I would say it is possible to mkfs the disk before each new usage to have clean data structures with less overhead (no fragmentation...). Secondly you could choose a file system optimized for big files and write-ahead only. It s possible to change the parameters of the FS to push even more this behaviour (how many cylinders? block size? no logging on the data, no block reserve for fast allocation...). Third, what about bad blocks on disk? How to skip them in a raw partition if you do not have state-of-the-art disks that do block remapping for you in your back-yard (such as SCSI)? Often FS do these tricks for you on IDE disks for example. Well, IMHO, I would vote for a FS solution except if I have a real gain... :-) -- Ronan KERYELL |\/ Tel:(+33|0) 2.29.00.14.15 Département Informatique|/) Fax:(+33|0) 2.29.00.12.82 ENST Bretagne, CS 83818 KGSM:(+33|0) 6.13.14.37.66 F-29238 PLOUZANÉ CEDEX |\ E-mail: [EMAIL PROTECTED] FRANCE | \ http://enstb.org/~keryell callto:ils.seconix.com/[EMAIL PROTECTED]
Dealing more gracefully with tape overruns
We run AMANDA for many years with the same taper and the time comes with more and more users until one tape per day is sometimes not enough. :-/ We don't have any changer and often one tape is enough. So I would like a mode where AMANDA would decide it is not possible to have everything on one tape *but* would not stop dumping and would go on on the holding disk. Then the autoflush mode would flush everything the day after or a manual flush could be done on a separate tape. The problem is that when AMANDA notices that there is not enough space on tape, it stops with messages such as: minou.info.enstb.org /var lev 0 FAILED [dumps too big, 180301 KB, but no incremental estimate] minou.info.enstb.org /home lev 0 FAILED [dumps too big, 3309778 KB, but no incremental estimate] pei.info.enstb.org /mnt/bonbonne/subversion_backups lev 0 FAILED [dumps too big, 1414910 KB, but cannot incremental dump new disk] minou.info.enstb.org /export/bouteille lev 2 FAILED [dumps way too big, 4632849 KB, must skip incremental dumps] I'd like to have incremental and even level 0 dumps on the holding disk, where I have a lot of space. I've played recently with maxdumpsize -1 # Maximum number of bytes the planner will schedule # for a run (default: runtapes * tape_length). but it doesn't help since it says it can decide to dump more than a tape but it fails later with the same messages than above. So, is there an obvious way to deal with the not-so-frequent overrun we encounter? If not, what about adding a new strategy with a mode such as: dump-on-holding-disk-if-degraded-mode yes Should not be to difficult to implement. I will try also to play with the reserve keyword next night with, for example, reserve 50 % for full backups. reserve number Default: 100. The part of holding-disk space that should be reserved for incremental backups if no tape is available, expressed as a percentage of the available holding-disk space (0-100). By default, when there is no tape to write to, degraded mode (incremental) backups will be performed to the holding disk. If full backups should also be allowed in this case, the amount of holding disk space reserved for incrementals should be lowered. But if it works, why this static allocation between incremental and full backups? Another approach: what about also a mode with runtapes 1 without any changer, assuming that the user will use amflush later? Thank you, -- Ronan KERYELL |\/ Tel:(+33|0) 2.29.00.14.15 Département Informatique|/) Fax:(+33|0) 2.29.00.12.82 ENST Bretagne, CS 83818 KGSM:(+33|0) 6.13.14.37.66 F-29238 PLOUZANÉ CEDEX |\ E-mail: [EMAIL PROTECTED] FRANCE | \ http://enstb.org/~keryell callto:ils.seconix.com/[EMAIL PROTECTED]
auto-exclude mode ?
When using tar on different partitions, tar prunes the mounting points as we expect to. When we have big disks, we backup different directories defined in the disklist but if some directories are included in some other ones, data in subdirectories are backup twice if we do not build up an ad-hoc exclude list. This kind of exclude list is cumbersome and error prone... With an auto-exclude option, AMANDA could figure this up and exclude the subdirectories from the directories above. Does this make sense ? Any candidate in the audience to hack this up ? :-) Thank you ! -- Ronan KERYELL |\/ Tel:(+33|0) 2.29.00.14.15 Labo Informatique Télécom |/) Fax:(+33|0) 2.29.00.12.82 ENST Bretagne, CS 83818 KGSM:(+33|0) 6.13.14.37.66 F-29238 PLOUZANÉ CEDEX |\ E-mail: [EMAIL PROTECTED] FRANCE | \ http://www.lit.enstb.org/~keryell
How to get the last existing version of a file ?
I've written an article about amanda and I've got a question of a referee : If a user has lost a file in the past but does not know when, how to get the last version of it in a simple way ? Since amrecover needs a date to look for first, it seems hard to me and the only way I see is to sort all the index by decreasing date and to grep it for the asked file (thanks to simple file formats used in Amanda :-) ). By the way, an other stuff related with long archives with amanda (we have tapes from around 1995) is that files may move around partitions and even amanda instance (we use 3 different tapers). That means that for example you are looking for a file that was in /users/lit/keryell/TeX/ five years ago but I cannot remember on which machine and partition it was... Thus I need to add a loop on amanda instance, a loop on machines and a loop to partitions to the previous method. Well, anyway this case never happen yet, but... PS : for the article in french it is at : http://www.cri.ensmp.fr/~keryell/publications/conf/2001/JRES2001/amanda/ -- Ronan KERYELL |\/ Labo Informatique Télécom |/) Tel:(+33|0) 2.29.00.14.15 ENST Bretagne, BP832 KFax:(+33|0) 2.29.00.12.82 29285 PLOUZANE CEDEX |\ E-mail: [EMAIL PROTECTED] FRANCE | \ http://www-info.enst-bretagne.fr/~keryell
Re: Solaris file system snapshots
Jon Has anyone attempted to perform amanda backups on Solaris 8 Jon systems by first taking a static snapshot (new Sol8 feature) of Jon the file system? Jon I can see conceptually how it would be done. But as the Jon snapshots have a different device name and are mounted on a Jon different directory I am concerned about the recovery/restore. I discuss this in my administration course on Solaris 8 and in an article about amanda. Oh, hum, well, it is written in french. :-) But I think it is not an issue in amanda since the names of the saved files are relative to the partition. That is you dump the snapshot image and restore in the original partition. Well, you have to trick the default amrecover behaviour... By the way, I think that the snapshot is the *right* way for the future : 1 - you have a snapshot of the running file system during all the backup ; 2 - you can use GNU tar without changing the access time and be able to read again the data in 200 years because of the free data format of GNU tar. With restore, if you do not have the right OS, be careful... :-( Just need the new dump API, thus I'm not able to test it in the real life right now... http://www.cri.ensmp.fr/~keryell/publications/conf/2001/JRES2001/amanda http://www.cri.ensmp.fr/~keryell/cours/ENSTBr/FC/Administration_Unix -- Ronan KERYELL |\/ Labo Informatique Télécom |/) Tel:(+33|0) 2.29.00.14.15 ENST Bretagne, BP832 KFax:(+33|0) 2.29.00.12.82 29285 PLOUZANE CEDEX |\ E-mail: [EMAIL PROTECTED] FRANCE | \ http://www-info.enst-bretagne.fr/~keryell
Re: security during amrecover
On 27 Jul 2001 12:43:03 +0200, Johannes Niess [EMAIL PROTECTED] said: Johannes Tom Strickland [EMAIL PROTECTED] writes: Our system will be run largely without a competent Unix administrator on-site. The secretary and one other individual will be responsible for tape-changing, cleaning and amrecover for files and directories deleted by users. My question: Some of our files are more confidential and I would like to hide these a little: the director's files and the accounts. Is there anyway to protect these? It doesn't have to be high grade security, just security through obscurity. Johannes Tom, Johannes What about sudo? The recovered files keep owner and Johannes permissions. Let the operator's sudo to the (too powerfull Johannes in this case) Amanda user just for amcheck, amrecover and Johannes what else you like. You'll see their actions in the Johannes syslog. The good thing: no password for the Amanda user has Johannes to be given away. Johannes We have set up just our tape changing that way. I'm not sure it is enough... The main issue is once your file system is on tape, everybody can look at them if it is uncrypted. What could be nice in amanda or at least at the dumper API level is to crypt the files with the public key of their owners. The recovery would require the secret private key of their owners... -- Ronan KERYELL |\/ Labo Informatique Télécom |/) Tel:(+33|0) 2.29.00.14.15 ENST Bretagne, BP832 KFax:(+33|0) 2.29.00.12.82 29285 BREST CEDEX |\ E-mail: [EMAIL PROTECTED] FRANCE | \ http://www-info.enst-bretagne.fr/~keryell
