How do gnutar listed-incremental work?
Hello, I got somewhat curious how gnutar's listed-incremental is supposed to work. AFAICS, amanda keeps for every backup gnutar's scratchfile and its index file (basically the output of 'tar -t'). I can see how amanda can determine which files/directories were added/changed since last N-1 backup (this are the files/dirs in the incremental index). But how does amanda know which files/dirs were _removed_ before the incremental backup was done? Neither the index files nor the scratchfiles seem to be of much use to determine this information. What am I missing here?
Re: amanda over ssh
On Wed, Sep 20, 2006 at 05:41:09PM -0400, Steve Newcomb wrote: > Fabio Corazza <[EMAIL PROTECTED]> writes: > > Lee, Raymond wrote: > > I'm interested on this as well. Just a little question before starting > > the new configuration by myself: is ssh used just for the > > *authentication* or for the dumps as well? > I asked that in my first e-mail, and nobody has answered it directly, > so I'm assuming that it was a dumb question, I don't think this to be dumb question. But maybe the question is somewhat imprecise. I try to spread it into more precise questions: With ssh authentication: - Are dumps transported over a ssh tunnel? - Are indexes transported over a ssh tunnel? - Are index transfers authenticated via ssh? - How are restores authenticated/transported? Anybody dares to answer those questions? > and that *of course* the dump goes through the ssh connection. When it comes to security, _never_ make such assumptions. There's too much snake oil out there.
Re: usetimestamp question
On Wed, Sep 27, 2006 at 02:36:10PM -0400, McGraw, Robert P. wrote: > If I start using "usetimestamp yes" is there any problem using amanda > programs on older file names? I'd expect the amandatape script to be confused by multiple dumps per day unless the format of the logfiles changed. I'd expect it to be confused by the new format _if_ the format changed. Can anybody drop a note whether/how the format of the logfiles changed?
Re: Release of amanda-2.5.1
On Tue, Sep 05, 2006 at 03:34:42PM -0400, Jean-Louis Martineau wrote: >* Works with GNU tar 1.15.91 - work with new gtar state file format. Can someone please explain what this exactly means?
Re: using disk instead of tape
On Tue, Sep 05, 2006 at 05:17:31PM -0500, Phil Howard wrote: > And you could still do a "bare metal" recovery as long as the partitions > on disk were compatible (which is why, if I write such a driver in Linux > I would use the MSDOS partition table format). Some OS screw up when the partition table is modified. This is why fdisk gives a warning. Beeing paranoid, I always reboot after I modify the partition table. I don't think it is really worth the effort to implement such a functionality. I bet in most installations compression/encryption/network is the real bottlenek.
Re: [RFE] - Footer file on tape
On Sun, Aug 20, 2006 at 01:23:55PM -0400, Jon LaBadie wrote:
> On Sun, Aug 20, 2006 at 01:28:29PM +0100, Chris Lee wrote:
> > I was wondering if anyone thought it would be a good idea for Amanda to
> > write a footer file to tape after completing a dump providing a record
> > of the tapes needed to produce a full restore to the state the system
> > was in when this tape was written.
> > I don't use amanda in a way that this would be very useful to me, it is
> > only for home backups and I have a dumpcycle of 2 so I just need the
> > last 3 tapes and I am sure to have all I need. However I was thinking in
> > situations where longer dumpcycles and lots of DLEs are common it could
> > help with restores if you lose Amanda and her data.
> > The footer file would just be a list of the tapes from the last level 0
> > to now for each DLE.
> >
> > For example a file formatted like this, so anyone can read it without
> > Amanda's help:
> > /home/bob { //Tapes needed for this DLE
> > DailySet3 0 Thursday 17/08/2006
> > DailySet4 1 Friday18/08/2006
> > DailySet5 2 Monday21/08/2006
> > }
> > /home/anne { //Tapes Needed for this DLE
> > DailySet5 0 Monday21/08/2006
> > }
> > etc.
> >
> > Restore Set { //list of all tapes needed for all DLEs
> > DailySet3 Thursday 17/08/2006
> > DailySet4 Friday18/08/2006
> > DailySet5 Monday21/08/2006
> > }
> >
>
> Interesting idea. A 32KB trailer file is, I believe, already added
> to the last tape used in a dump. I'm not sure I like the idea of
> adding this data, either as a last file before the trailer, or as
> part of the trailer. In a large installation it could get quite
> large and could result in an extra tape being used if the last
> DLE nearly filled the tape.
>
> But as a separately executed report, I really like it.
> Things not included in your format are the client and the
> tape file number. With tape spanning that could be 'tapes'
> and tape file 'numbers' for a single dump level.
>
> Can Josef Wolf's amandatape program be massaged into such a format?
AFAICS, all requested information (and more) is provided by amandatape,
it's just in a different format. Would it really make sense to tweak
the output into a special format for appending to tape? Why not writing
its ASCII output to the tape?
There's a little problem, though: amandatape does its work by parsing the
logfiles, thus the logfile must be closed to ensure proper operation.
I prefer to print this information onto the tape labels. With this I have
the information right there and don't need to search the tape in order to
find out what I need.
Re: Getting list of DLE's w/o Lev 0s
On Wed, Aug 02, 2006 at 02:08:13PM -0500, C. Chan wrote: > >>Well, that's the opposite of what I want. I want to generate > >>a list of the DLEs in disklist which DO NOT have ANY Lev 0s, > >>so I can force a level 0 for those particular DLEs during a run of amanda. > > > >Please check the amandatape script from amanda's contrib directory. It > >will emit a warning in such a situation. > > Is this new in the 2.5.0 and later amanda sources? Although it exists for several years, it was added only a couple of weeks ago to amanda's cvs repository. > I am still > running 2.4.5 - will the amandatape script run with minor modification > under 2.4.5? I tried to make it perfectly backwards compatible. I don't know of any compatibility problems. But (as always with open source) I can't give any guarantee :).
Re: Getting list of DLE's w/o Lev 0s
On Fri, Jul 28, 2006 at 12:42:50PM -0500, C. Chan wrote: > >If i understand you correctly you should get what you are looking for by > >using amadmin: > >amadmin ConfigName find > Well, that's the opposite of what I want. I want to generate > a list of the DLEs in disklist which DO NOT have ANY Lev 0s, > so I can force a level 0 for those particular DLEs during a run of amanda. Please check the amandatape script from amanda's contrib directory. It will emit a warning in such a situation.
Re: New question on levels
On Wed, Jul 26, 2006 at 11:11:41AM -0400, Jon LaBadie wrote: > If talking about amanda's planning, it "shouldn't" happen. > I'm pretty sure that amanda does at most, estimates of three > dump levels; level 0, the last dump level, and the last level +1. So amanda will never fall back to lower levels once a bump was done?
Re: The Size Question
On Sat, Jul 08, 2006 at 01:50:09AM +0200, Peter Kunst wrote: > Jon LaBadie wrote: > >On Fri, Jul 07, 2006 at 05:11:50PM -0400, [EMAIL PROTECTED] wrote: > >>Dump images spanning multiple media volumes: > >And no more simple recovery of spanned dumps using > >standard unix tools when amanda is not available. > >That needs to be pointed out in any revision. > > Indeed. Just had another case last week, where the dd method tells me > "this is chunk 3 of 4"... well, on which tapes do i find the other > chunks, i asked myself. Run the amandatape program after every backup, and you will always have the answer to this question right on your tape labels. But back to the tape chunking: I don't like this static chunking. You have to choose between either wasting a lot of tape or having lots of chunks, making manual recovery a fragile task. So you trade waste against inconvenience. It would be much better if the chunking algorithm would take into account how much tape is available for the next chunk. Something like chunkfactor 3/4 # 0<=chunkfactor<=1 minsize 1GB The chunkfactor specifies how much of the rest of the tape will be allocated for the next chunk. The minsize specifies the minimum size of a chunk, to avoid a high number of chunks because they are getting too small. With the above specification and a tapesize of 100GB we would get: first chunk: 75.00GB==3/4*(100) second chunk: 18.75GB==3/4*(100-75) third chunk: 4.69GB==3/4*(100-75-18.75) fourth chunk: 1.17GB==3/4*(100-75-18.75-4.69) fifth chunk: 0.29GB==3/4*(100-75-18.75-4.69-1.17) # forced to 1GB # since the fifth chunk is less than minsize, it is foced to minsize (1GB). # If it don't fit to the tape, we start over again on the next tape: fifth chunk: 75.00GB==3/4*(100)# goes to a new tape On the new tape, the fifth chunk would again start over at 75GB because the new tape has 100GB available again. We ended up with only four chunks (the fifth is ignored since we got a write error) on the first tape and have wasted only 0.29GB (that is 0.29%) of the tape. We will never waste more than 1% (minsize/tapesize) and we will never get more than four chunks of a given dump on a single tape. (In contrast, the current algorithm will end up with 100 chunks if you don't want to waste more than 1%) With growing chunkfactor (limit is 1), you get a lower number of chunks, but you risk to waste more tape if you get an early write error. So now you trade risk against waste (instead of inconvenience against waste, as in current algorithm). With reliable tapes, you can have higher chunkfactor and end up with a low number of chunks. With unreliable tapes it is better to go lower chunkfactor because the earlier you get a write error the more tape you waste. Either way, you have bigger chunks at the start of the tape, honoring the assumption that probability of write errors is higher on the end of the tape. For vtape usage (you have no risk for write errors here), chunks can be made exactly the size that is needed to fill the vtape with: chunkfactor 1 minsize 0 So we never waste any disk space on vtapes any more! The current chunking behavior can be achieved with: chunkfactor 0 # will always be smaller than minsize minsize 10GB # thus all chunks will be 10GB I have already implemented such a system in a different project (Jon, you remember the ssh based system I mentioned a couple of months ago?) and I am pretty happy with this algorithm.
Re: LVM snapshots
On Fri, Jul 07, 2006 at 11:13:44PM +0200, Paul Bijnens wrote: > There are two approaches (maybe even many more -- I'm not the > specialist here). > > One approach is to let the snapshot mechanism understand the filesystem > and work on that level. That is how xfs and solaris ufs snapshots seem > to work. > > The other approach is a layer deeper: on the logical volume manager. > This implementation is not interested in where the inodes are located > or how directories and datablocks are layed out on disk. The LVM just > manages large blocks. The LVM2 snapshots in linux work on this layer. If the snapshot is on the driver layer, there is no way to ensure that the snapshot is consistent? Thus the snapshot would look like the power switch was turned off at that moment. Without journalling, data loss can be expected. I would rather not base my backups on such a mechanism... > Using method 1, you can probably get more optimized (quicker, using less > diskspace). Using method 2 is more general, and is is independent of > the filesystems on it. In my test setup I even managed to make a > snapshots of a vfat filesystem. I don't think vfat have journalling.
Re: determining backup set
On Thu, Jun 22, 2006 at 11:51:22AM +0200, Toralf Lund wrote: > If I have a collection of tapes from previous amanda backups, > is there a procedure for determining which set of tapes contain > a complete backup? > > >>>I'm assuming you don't have access to the logs/indexes/amrecover. > >>> > >>Actually, I probably do have access to these things, but I'm not yet > >>sure how to interpret them. Any pointers will be appreciated. (Note > >>that all recent log files represent backup failures, but the old logs > >>are still available in the archive area.) > > > >As long as the log files are available, the amandatape script that I > >posted a while ago to this list will give you the info that you are > >looking for. You can find it here: > > > Maybe I've missed something, but I can't see why a special script would > be necessary in this case. This script is not special for the situation in question. In contrast, it is meant to be run after every backup to produce tape-labels with information which files from which tapes you need to recover a given DLE. You attach those labels to the tapes and you always have an answer to OP's question, even if you loose the database. > Why not use the commands already provided as part of the amanda > distribution? I mean, like > > amadmin find > > or > > amadmin find > > or > > amadmin find Maybe because the output of "amadmin find" is no more than a plain dump of amanda's database. Please go and check the above mentioned script and compare its output with the output of "amadmin find".
Re: Amanda stopped working...
On Mon, Jun 19, 2006 at 09:50:23AM -0700, Pavel Pragin wrote: > Josef Wolf wrote: > > >Hello! > > > >Amanda-2.5.0b2 was working for about six months here. Now it stopped > >working for some reason, and I can't figure out why. It looks to me as > >if there's something wrong with getting stimates. > > > >From the amdump file it looks like estimates are scheduled, but never > >return any results. > > > >Here's the amdump file: > > I think we will need a little more information. Can you please tar up > post the debug logs from /tmp/amanda on the server? Thanks for your reply, Pavel. The debug directory is attached. amanda.tar.gz Description: application/gunzip
Amanda stopped working...
Hello! Amanda-2.5.0b2 was working for about six months here. Now it stopped working for some reason, and I can't figure out why. It looks to me as if there's something wrong with getting stimates. >From the amdump file it looks like estimates are scheduled, but never return any results. Here's the amdump file: amdump: start at Sa Jun 17 16:50:46 CEST 2006 amdump: datestamp 20060617 planner: pid 5050 executable /usr/lib/amanda/planner version 2.5.0b2 planner: build: VERSION="Amanda-2.5.0b2" planner:BUILT_DATE="Tue Feb 14 00:04:47 CET 2006" planner:BUILT_MACH="Linux host 2.6.13-15.7-default #1 Tue Nov 29 14:32:29 UTC 2005 i686 athlon i386 GNU/Linux" planner:CC="gcc" planner:CONFIGURE_COMMAND="'./configure' '--mandir=/usr/share/man' '--prefix=/usr' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--libdir=/usr/lib' '--libexecdir=/usr/lib/amanda' '--localstatedir=/var/lib' '--with-dumperdir=/usr/lib/amanda' '--with-index-server=localhost' '--with-gnutar-listdir=/var/lib/amanda/gnutar-lists' '--with-smbclient=/usr/bin/smbclient' '--with-amandahosts' '--with-user=amanda' '--with-group=disk' '--with-gnutar=/bin/tar' '--disable-libtool' '--disable-shared' '--disable-static'" planner: paths: bindir="/usr/bin" sbindir="/usr/sbin" planner:libexecdir="/usr/lib/amanda" mandir="/usr/share/man" planner:AMANDA_TMPDIR="/tmp/amanda" AMANDA_DBGDIR="/tmp/amanda" planner:CONFIG_DIR="/etc/amanda" DEV_PREFIX="/dev/" planner:RDEV_PREFIX="/dev/" DUMP="/sbin/dump" planner:RESTORE="/sbin/restore" VDUMP=UNDEF VRESTORE=UNDEF planner:XFSDUMP=UNDEF XFSRESTORE=UNDEF VXDUMP=UNDEF VXRESTORE=UNDEF planner:SAMBA_CLIENT="/usr/bin/smbclient" GNUTAR="/bin/tar" planner:COMPRESS_PATH="/usr/bin/gzip" planner:UNCOMPRESS_PATH="/usr/bin/gzip" LPRCMD="/usr/bin/lpr" planner:MAILER="/usr/bin/Mail" planner:listed_incr_dir="/var/lib/amanda/gnutar-lists" planner: defs: DEFAULT_SERVER="localhost" DEFAULT_CONFIG="DailySet1" planner:DEFAULT_TAPE_SERVER="localhost" planner:DEFAULT_TAPE_DEVICE="null:" HAVE_MMAP HAVE_SYSVSHM planner:LOCKING=POSIX_FCNTL SETPGRP_VOID DEBUG_CODE planner:AMANDA_DEBUG_DAYS=4 BSD_SECURITY RSH_SECURITY USE_AMANDAHOSTS planner:CLIENT_LOGIN="amanda" FORCE_USERID HAVE_GZIP planner:COMPRESS_SUFFIX=".gz" COMPRESS_FAST_OPT="--fast" planner:COMPRESS_BEST_OPT="--best" UNCOMPRESS_OPT="-dc" READING CONF FILES... planner: time 0.001: startup took 0.001 secs SENDING FLUSHES... FLUSH host.do.main /m/l 20060611 1 /b/amchanger/ppc/hold/20060611030501/host.do.main._m_l.1 FLUSH host.do.main /m/l 20060612 1 /b/amchanger/ppc/hold/20060612030501/host.do.main._m_l.1 ENDFLUSH SETTING UP FOR ESTIMATES... planner: time 0.002: setting up estimates for dssl-gw.do.main:/ setup_estimate: dssl-gw.do.main:/: command 1, options: nonelast_level -1 next_level0 0 level_days 0getting estimates 0 (-2) -1 (-2) -1 (-2) planner: time 0.002: setting up estimates for host.do.main:/ setup_estimate: host.do.main:/: command 1, options: nonelast_level -1 next_level0 0 level_days 0getting estimates 0 (-2) -1 (-2) -1 (-2) planner: time 0.003: setting up estimates for host.do.main:/m/b setup_estimate: host.do.main:/m/b: command 1, options: nonelast_level -1 next_level0 0 level_days 0getting estimates 0 (-2) -1 (-2) -1 (-2) planner: time 0.003: setting up estimates for host.do.main:/m/check setup_estimate: host.do.main:/m/check: command 1, options: nonelast_level -1 next_level0 0 level_days 0getting estimates 0 (-2) -1 (-2) -1 (-2) planner: time 0.004: setting up estimates for host.do.main:/m/l setup_estimate: host.do.main:/m/l: command 1, options: nonelast_level -1 next_level0 0 level_days 0getting estimates 0 (-2) -1 (-2) -1 (-2) planner: time 0.004: setting up estimates for host.do.main:/var setup_estimate: host.do.main:/var: command 1, options: nonelast_level -1 next_level0 0 level_days 0getting estimates 0 (-2) -1 (-2) -1 (-2) planner: time 0.005: setting up estimates took 0.003 secs GETTING ESTIMATES... planner: time 0.035: bind_portrange2: trying port=659 planner: time 0.035: dgram_bind: socket bound to 0.0.0.0.659 driver: pid 5051 executable /usr/lib/amanda/driver version 2.5.0b2 driver: tape size 4096000 taper: pid 5052 executable taper version 2.5.0b2 taper: page size is 4096 taper: buffer size is 32768 taper: buffer[00] at 0x401e4000 taper: buffer[01] at 0x401ec000 taper: buffer[02] at 0x401f4000 taper: buffer[03] at 0x401fc000 taper: buffer[04] at 0x40204000 taper: buffer[05] at 0x4020c000 taper: buffer[06] at 0x40214000 taper: buffer[07] at 0x4021c000 taper: buffer[08] at 0x40224000 taper: buffer[09] at 0x4022c000 taper: buffer[10] at 0x40234000 taper: buffer[11] at 0x4023c000 taper: buffer[12] at 0x40244000 taper: buffer[13] at 0x4024c000 taper: buffer[14] at 0x40254000 taper: buffer[15] at 0x4025c000 taper: buffer[16] at 0x
Re: determining backup set
On Fri, Jun 16, 2006 at 10:47:52AM -0400, Marlin Whitaker wrote: > On Jun 15, 2006, at 3:03 PM, Jon LaBadie wrote: > >On Thu, Jun 15, 2006 at 12:33:16PM -0400, Marlin Whitaker wrote: > >>If I have a collection of tapes from previous amanda backups, > >>is there a procedure for determining which set of tapes contain > >>a complete backup? > > > >I'm assuming you don't have access to the logs/indexes/amrecover. > > Actually, I probably do have access to these things, but I'm not yet > sure how to interpret them. Any pointers will be appreciated. (Note > that all recent log files represent backup failures, but the old logs > are still available in the archive area.) As long as the log files are available, the amandatape script that I posted a while ago to this list will give you the info that you are looking for. You can find it here: http://groups.yahoo.com/group/amanda-users/message/58001?l=1 But make sure you apply the patch from: http://groups.yahoo.com/group/amanda-users/message/58021?l=1
Re: Is tape spanning documented anywhere?
On Tue, Jun 13, 2006 at 05:20:13PM +0200, Toralf Lund wrote: > > And like I also said, in general, allowing "partial flush" would also > address another issue: The one of blocking the entire tape operation > when using a holding disk, and getting a dump larger that won't fit on > the tapes even though it was expected to (either because of > miscalculations during the planner phase or because it specifying the > tape size seems to be a rather inexact science.) This issue can be solved by a much simpler change: Simply assume an infinite large runtapes setting as long as no taping succeeded on the current run.
Cannot seek to 0
Hello! When estimates are running, I see lots of errors from tar: /bin/tar: ./old/src/svnrepos/db/__db.002: Warning: Cannot seek to 0: Bad file descriptor Anyone knows what could be the reason and whether I should bother? BTW: this is gnutar 1.15.1 Thanks!
Re: Tape spanning and `amadmin info`
On Fri, May 12, 2006 at 12:54:33PM -0700, jeffrey d anderson wrote: > I've begun using the new tape spanning feature of amanda 2.4.5 and it > seems to work quite well. [ ... ] > Periodically I pull a set of tapes from my changer to be set aside for > archival purposes. In the past I have relied on `amadmin info` to tell me > what tapes would need to be pulled to have a full dumpcycle. Now it seems, > with tape spanning, this information is not quite accurate. > > I'm not quite sure what the solution is, since this is a new condition for > amanda (a single DLE dump not residing on a single tape). You might check out the amandatape utility I posted to this list on Feb, 14. It reports correctly the information you need. But make sure you also apply the patch I posted on Feb, 17.
Re: Would like to get list's impression on (amanda 4TB backup)
On Thu, May 11, 2006 at 12:34:00PM -0500, Gordon J. Mills III wrote: > Hello Ronald and Ross. Just a note on other backup solutions...If you do > decide to to commercial I would recommend you take a look at IBM's Tivoli > (aka TSM) backup system if you are looking at large libraries and lots of > data. It is a very impressive system with a completely different stategy > than other backup solutions I've seen. I'm very interested about the startegy. Would you please spend a couple of words about it to us? > That said, Amanda comes the closest so far to bridging the gap (commercial > or GPL). Amanda is a great backup solution, but if I wanted to go commercial > I'd definitely look at TSM. We had it at my old job and it was fantastic. What makes it superior to amanda? > I > won't go into a detailed description unless others would like to read it. If > you want to know more about the comparisons I will answer privately. I definitely would like to know more. And I think such a discussion would be perfectly on-topic on this list. There's no reason to go privately because most people on this list are interested (IMHO). > PS: I am not trying to start a flame war here nor am I an affiliate of IBM. > Just trying to give some info. Hey, that's not fair! Please avoid statements like "they are superior" without an explanation _why_ they are superior. instead, please provide us with information about what is missing to come close to this superior system.
Re: putting error warnings into seperate mail
On Thu, May 11, 2006 at 03:41:27PM +0200, Paul Bijnens wrote: > The problem with deciding about the errors is more difficult when > you look closer. > > - Some people leave the tape drive empty on purpose, and when enough > dumps are collected in the holdingdisk, they insert the tape, and all > the dumps get autoflushed to tape. I'm doing this also, (on vtapes, via cron). > - Some people have laptops in their disklist, and "results missing" > is normal for those PC's. However, results missing on your production > file server is very important. So it might be a good idea to make "results missing" (or, maybe other errors, too?) behavior depandant on the host. > - Many people have "file changed as we read it" warnings, and most of > these are harmless. But sometimes NOT. > That's why I extended my list of reg.expressions to suppress those > warnings only for particular files only. I always wondered how to distinguish the harmful from the harmles. You dare to share your wisdom? > So, implementing this inside Amanda for general use is difficult. > Implementing a filter in perl for your own use should be easy enough. I disagree with this. Asking the (end)user to implement their filters in perl is somewhat arrogant, I think... Instead, there should be an interface to specify the severity of a specific problem. Soemthing like /host.regex.do.main/ /results missing/ IGNORE m!host.com:/var/mail/.*! /file changed/ IGNORE Forcing people to implement those things on their own invites people to introduce errors without even knowing about the fact that they have an error.
Taper problems with 2.5.0
Hello! Anyone knows what following error means? *** THE DUMPS DID NOT FINISH PROPERLY! The next 3 tapes Amanda expects to use are: PPC02, PPC03, PPC04. FAILURE AND STRANGE DUMP SUMMARY: host.do.main /etc RESULTS MISSING host.do.main /mRESULTS MISSING [ ... ] taper: FATAL cannot allocate shared memory driver: FATAL reading result from taper: Connection reset by peer [ ... ] NOTES: taper: attach_buffers: (20 tapebufs: 659696 bytes) No space left on device taper: attach_buffers: (19 tapebufs: 626916 bytes) No space left on device [ ... ] All file systems (especially holding disk) have plenty free space and inodes. Both, holding disk and tape were empty. I'd expect an empty tape when taper fails. But empty holding disk is not exactly what I'd expect in such a situation.
Re: Strugging with splits in 2.5.0
On Wed, Apr 05, 2006 at 09:08:45AM +0200, Paul Bijnens wrote:
> You can even put a little bit more on a DVD, see:
>
> http://wiki.zmanda.com/index.php/Tapetype_definitions#Virtual_tapes
>
> define tapetype DVD47 {
> comment "A 4.7 Gbyte DVD-sized vtape"
> length 4482 mbytes
> filemark 4 kbytes
> }
I find this a little bit optimistic. For one, my empirical tests gave
read errors when 47 or more were written. Thus length should
be no more than 4450 when isofs-overhead is regarded.
Besides this, in the last couple of weeks I experience something 4GB
related. I have started a thread on
http://lists.suse.com/archive/suse-linux-e/2006-Apr/0334.html
but no result so far. :-(
So I reduced the length to 4000mb
Re: how to split partitions
On Sun, Apr 02, 2006 at 09:34:59PM +0200, Paul Bijnens wrote: > That would mean that the estimate phase should be smart enough (may > be add an option "autosplit" for those DLE's where you want that) to > pick a few large subdirectories out and then split the DLE dynamically. This is not what I have meant. No autosplitting. Just automatically create appropriate exclude entries. When one sees a DLE starts becoming too big, just pich one of its subdirs and create a new DLE for this directory, that's all. No fiddling with exclude list entries any more.
Re: how to split partitions
On Sun, Apr 02, 2006 at 10:44:35AM -0500, Gene Heskett wrote: Gene, you mis-interpreted me. I am _not_ about how excluding works today (I understand how it works). I am about how to change amanda to avoid the boring and error prone procedure that is described in http://wiki.zmanda.com/index.php/Backing_up_large_amounts_of_data. I try to explain it a little more verbosely. Given following disklist: /share/ /share/financial /share/financial/company-a /share/financial/company-b /share/reports amanda should automatically create following exclude entries: DLE generated exclude entriesnote - /share/ ./financial ./reports[1] /share/financial./company-a ./company-b [2] /share/financial/company-a [3] /share/financial/company-b [3] /share/reports [3] [1] Since /share/financial and /share/reports have their own entries in the disklist, amanda should automatically add ./financial and ./reports to the exclude list of /share [2] Since /share/financial/company-[ab] have their own entries in the disklist, amanda should automatically add corresponding entries to the exclude list of /share/financial [3] No entries should be generated automatically for this DLE > > DLE exclude entry > > > > /share ./financial ./reports > Which will exclude the subdir financial and the file reports Exactly. Since /share/reports and /share/financial are in the disklist, those exclude entries should be generated automatically. > > /share/financial./company-a ./company-b > Which will exclude both of the ./company files. > You've excluded everything. What was it you wanted to get with that > disklist entry pair? Since /share/financial/company-[ab] are in the disklist, they can be automatically excluded from the /share/financial DLE.
Re: how to split partitions
On Thu, Mar 30, 2006 at 07:31:59PM +0200, Stefan G. Weichinger wrote: > > As promised, I updated the wiki docs on this topic, see > > http://wiki.zmanda.com/index.php/Backing_up_large_amounts_of_data. Am I the only one to find this setup needlessly error prone and complicated? Would amanda automatically create appropriate exclude entries from its DLE information, all this would be greatly simplified. For example, with this set of DLE entries: /share/ /share/financial /share/financial/company-a /share/financial/company-b /share/reports amanda should automatically create following exclude entries: DLE exclude entry /share ./financial ./reports /share/financial./company-a ./company-b With this, the whole described procedure would be reduced just to add a directory to the disklist. No more risk to get the "catch all" entry wrong.
Re: Whats next after Amanda 2.5
On Thu, Mar 30, 2006 at 01:23:43PM -0600, Graeme Humphries wrote: > Paddy Sreenivasan wrote: > > Yes. Backing up extended attributes is needed for many environments such > > as SE Linux. > Indeed. It'll be very helpful for us as our fileserver has a lot of > permissions in ACLs at the moment. In the mean time, something like cd $dlepath && getfacl -R --skip-base . >acl-info in a wrapper should do the job. Unfortunately, getfacl don't seem to have an --one-file-system option, so it will 1. scan too much and 2. casually error out on automounters/hald :-((
Re: encryption with 2.5.0b2
On Wed, Feb 22, 2006 at 03:34:44PM -0800, Kevin Till wrote: > >- What is the point to uuencode and encrypt (with gpg) random data to > > generate the key? Since the passphrase is stored on the same host, > > protecting the key with the passprase is not of much use (IMHO). > > It illustrates the method of using multi-key which a strong point of > aespipe. OK, I see. "multi-key" was the magic word that (after some googling) made me understand what's going on here. AFAICS, multi-keys can prevent watermark-attacks? Are there more advantages to them? > And it's a symmetric encryption and to facilitate automatic > backup, the passphrase has to be stored somewhere. This is (one) of the reasons why I'd prefer a pubkey method: You don't have the passphrase lying around on a networked box. > >- Why using aespipe at all? Is there any reason not to use gpg? > > AFAICS, aespipe introduces only an additinal layer of complexity. > Amanda users have used aespipe in the past, so it's there. Hmmm, AFAIK is aespipe part of loop-aes and loop-aes is deprecated because the kernel developers want to switch to devmapper. Please correct me and clarify if I'm wrong. > I believe aespipe gives better performance since gpg is doing more > than just encryption. AFAIK, gpg does compression in addition to encryption. But then you need to compare gzip+aespipe against gpg. Or did you mean something different? > >- Since the server says whether/which encryption is to be used, the > > server can request unencrypted backups from the client. This > > implies that the server has to be trusted. > > Use "auth ssh/krb4/krb5" to enable transport encryption. I am not about transport encryption here. I am about not trusting the amanda server. Thanks for the explanations, Kevin!
Re: Release of amanda-2.5.0b2
On Wed, Feb 22, 2006 at 05:53:52PM -0500, Jon LaBadie wrote:
> On Wed, Feb 22, 2006 at 11:23:09PM +0100, Josef Wolf wrote:
> > On Wed, Feb 22, 2006 at 12:27:32PM -0500, Jon LaBadie wrote:
> > > On Wed, Feb 22, 2006 at 08:24:00AM -0800, Kevin Till wrote:
> >
> > > INMNSO the default precision for the "*Rate" parameters and
> > > maybe for the "Compress" parameter also, should be changed
> > > to "0". As in:
> > >
> > > -{ "TapeRate", 1, 6, 1, 0, "%*.*f", "KB/s" },
> > > +{ "TapeRate", 1, 6, 0, 0, "%*.*f", "KB/s" },
> > >
> > > Does anyone really care if the rate was 1231.3 or 1231.4 KB/s?
> > > Not only does a percision of zero get rid of the useless digit,
> > > also the decimal point is eliminated. Thus two character
> > > positions are gained.
> >
> > I agree with you when the difference is between 1234.3 and 1234.4.
> > OTOH, a difference between 0.1 and 0.9 is of much more interest.
>
> Really? If you are reading a column of taping or dumping rates?
> I'd equate each to ZERO and look for what was wrong.
>
> Similarly for compressibility, I doubt I'd care if it was 0.1 or
> 0.9%. I forget which way the column reports, but one way it says
> not compressible, the other says immensely compressible. The
> fine degree of "not" or "immense" would not matter to me.
Oh, I did not recognize that you talk about those two columns _only_.
> > A better way (IMHO) would be do automatically adjust the value
> > to 2..4 digits, strip off the dot and append the unit. With this,
> > 1234567 would become 1234k, 123456 would become 123k and 12345678
> > would become 12M. Please check the kb() subroutine in the amandatape
> > program to see what I mean. I find this a good compromise because
> > the witdh is always between 3..5 characters and the error is always
> > below 10%.
>
> You are talking about the size columns.
I talk about all the columns.
> They are not reported with
> a decimal number even though the values are floats. Only the rates
> and compressibility are reported with fractional numbers. These
> don't have a big range of values in a single report.
Dumper rates can still vary very much, depending on how fast the clients
are and whether compression/encryption is enabled.
> Certainly not
> like the sizes which easily might be from 0 (or 32KB) to multi-GB
> in magnitude.
Yes. Having only one "displayunit" is not very helpful, IMHO. This is
why I made it automatically in amandatape. And while I was at it, I
did it for the rates, too.
encryption with 2.5.0b2
Hello! Now that 2.5.0b2 seems to run pretty stable, I'd like to try the new encryption functionality. I've read wiki.zmanda.com/index.php/Encryption, but have still some questions: - What is the point to uuencode and encrypt (with gpg) random data to generate the key? Since the passphrase is stored on the same host, protecting the key with the passprase is not of much use (IMHO). - Why using aespipe at all? Is there any reason not to use gpg? AFAICS, aespipe introduces only an additinal layer of complexity. - Since the server says whether/which encryption is to be used, the server can request unencrypted backups from the client. This implies that the server has to be trusted. Comments?
Re: Release of amanda-2.5.0b2
On Wed, Feb 22, 2006 at 12:27:32PM -0500, Jon LaBadie wrote:
> On Wed, Feb 22, 2006 at 08:24:00AM -0800, Kevin Till wrote:
> INMNSO the default precision for the "*Rate" parameters and
> maybe for the "Compress" parameter also, should be changed
> to "0". As in:
>
> -{ "TapeRate", 1, 6, 1, 0, "%*.*f", "KB/s" },
> +{ "TapeRate", 1, 6, 0, 0, "%*.*f", "KB/s" },
>
> Does anyone really care if the rate was 1231.3 or 1231.4 KB/s?
> Not only does a percision of zero get rid of the useless digit,
> also the decimal point is eliminated. Thus two character
> positions are gained.
I agree with you when the difference is between 1234.3 and 1234.4.
OTOH, a difference between 0.1 and 0.9 is of much more interest.
A better way (IMHO) would be do automatically adjust the value
to 2..4 digits, strip off the dot and append the unit. With this,
1234567 would become 1234k, 123456 would become 123k and 12345678
would become 12M. Please check the kb() subroutine in the amandatape
program to see what I mean. I find this a good compromise because
the witdh is always between 3..5 characters and the error is always
below 10%.
Re: Release of amanda-2.5.0b2
On Wed, Feb 22, 2006 at 09:12:56AM -0500, Jon LaBadie wrote: > On Wed, Feb 22, 2006 at 07:47:02AM +0100, Josef Wolf wrote: > > On Thu, Jan 19, 2006 at 07:31:08PM -0500, Jean-Louis Martineau wrote: > > > > > The Amanda core team is pleased to announce the release of Amanda > > > 2.5.0b2. It is stable and we hope to release 2.5.0 soon. test it and > > > report it if you find a bug in this release. > > The mail reports that are mailed by amdump have bogus numbers in the > > MMM:SS columns. For example: this line comes from the logfile: [ ... ] > > HOSTNAME DISK L ORIG-MB OUT-MB COMP% CRYPT% MMM:SS KB/s MMM:SS KB/s > > --- > > host.do.main /m/b 2 509139 27.3 1:561231.2 N/A N/A > > I think it is missing data in the new column, CRYPT% --- What is that? > > If the data are moved over one column it makes more sense, 116sec == 1:56, > 1231 == 1231. Ah, you're right! The crypt column is missing!
Re: Release of amanda-2.5.0b2
On Thu, Jan 19, 2006 at 07:31:08PM -0500, Jean-Louis Martineau wrote: > The Amanda core team is pleased to announce the release of Amanda > 2.5.0b2. It is stable and we hope to release 2.5.0 soon. test it and > report it if you find a bug in this release. The mail reports that are mailed by amdump have bogus numbers in the MMM:SS columns. For example: this line comes from the logfile: SUCCESS dumper host.do.main /m/b 20060222 2 [sec 115.740 kb 142504 kps 1231.2 orig-kb 521380] SUCCESS chunker host.do.main /m/b 20060222 2 [sec 115.822 kb 142504 kps 1230.6] And this is what the mail contains: STATISTICS: Run Time (hrs:min) 0:14 DUMPER STATSTAPER STATS HOSTNAME DISK L ORIG-MB OUT-MB COMP% CRYPT% MMM:SS KB/s MMM:SS KB/s - host.do.main /m/b 2 509139 27.3 1:561231.2 N/A N/A The real dump duration was about 115 seconds, but amdump says 1231.2 minutes. Note that the "kps" filed in the logfile contains the value that is reported my the mail.
Re: New version of amandatape script.
On Tue, Feb 14, 2006 at 10:25:07PM +0100, Josef Wolf wrote:
> On Tue, Feb 14, 2006 at 10:10:51PM +0100, Josef Wolf wrote:
> Here comes a new version of the amandatape program I posted about two years
> ago.
Here's a patch to fix following problems:
- Sort logfiles correctly when more than 10 logifles were generated in a
single day.
- Always ignore log.error.X files.
- When using the -d option, search backwards for tapecycle from given date
instead of date of newest logfile.
- Fix filemark counting when output is restricted to specific DLEs.
Please find the patch attached below. Have fun.
Index: bin/amandatape
===
--- bin/amandatape (Revision 1396)
+++ bin/amandatape (Arbeitskopie)
@@ -2,7 +2,7 @@
# amandatape -- a utility to print amanda tape labels for DAT and CD.
#
-# 2004-02-12 Josef Wolf ([EMAIL PROTECTED])
+# 2005-02-15 Josef Wolf ([EMAIL PROTECTED])
#
# Portions of this program which I authored may be used for any purpose
# so long as this notice is left intact.
@@ -295,10 +295,10 @@
sub log_bydate ($$) {
my ($a, $b) = @_;
-$a=~s!^.*?/log.([^/]+)$!$1!;
-$b=~s!^.*?/log.([^/]+)$!$1!;
+my ($ad, $anr) = $a=~m!^.*?/log\.(\d+)\.(\d+)$!;
+my ($bd, $bnr) = $b=~m!^.*?/log\.(\d+)\.(\d+)$!;
-return $a cmp $b;
+return $ad <=> $bd || $anr <=> $bnr;
}
sub add_dumping {
@@ -327,14 +327,18 @@
my %labels;# which tape labels we already have seen
my $lastvol=-1;
-my @logs=reverse sort log_bydate (<$LOGDIR/log.*>, <$LOGDIR/oldlog/log.*>);
+my @logs=reverse sort log_bydate (<$LOGDIR/log.[0-9.]*>,
+ <$LOGDIR/oldlog/log.[0-9.]*>);
splice @logs, 0, $ignore if defined $ignore;
FILE: foreach my $logfile (@logs) {
-if (defined $dignore && &log_bydate("$dignore.0", $logfile) > 0) {
- $dignore = undef;
- splice @logfiles, -$lastvol, $lastvol if $lastvol>0;
- $lastvol = -1;
+if (defined $dignore) {
+if (&log_bydate("/$dignore.0", $logfile) >= 0) {
+$dignore = undef;
+splice @logfiles, -$lastvol, $lastvol if $lastvol>0;
+$lastvol = -1;
+}
+next;
}
open (IN, $logfile) or die "$logfile: $!";
foreach my $l (reverse ) {
@@ -414,7 +418,6 @@
my ($host, $filesystem, $date, $chunk, $level, $rest) =
$2=~/(\S+) (\S+) (\S+) (\S+)? ?(\S+) (\[.*)/;
my $dle="$host:$filesystem";
-next unless &wantdle ($dle);
$dle{$dle} = 1;
$date=~s/()(..)(..)/$1-$2-$3/;
$nr++ if $#curchunk<0 || defined $chunk;
@@ -424,11 +427,13 @@
push (@curchunk, $h);
unless (defined $chunk) {
$taper{$dle}{$date}=[] unless exists $taper{$dle}{$date};
-push (@{$taper{$dle}{$date}}, [EMAIL PROTECTED]);
+push (@{$taper{$dle}{$date}}, [EMAIL PROTECTED]) if &wantdle
($dle);
@curchunk = ();
}
- $tape{$label}{"kb"} += $h->{"kb"};
- $tape{$label}{"count"}{$level}++;
+if (&wantdle ($dle)) {
+$tape{$label}{"kb"} += $h->{"kb"};
+$tape{$label}{"count"}{$level}++;
+}
}
if ($line=~/^INFO taper tape (.*)/) {
my ($t, $d2, $kb, $d3, $fm, $rest) = split (/\s+/, $1, 6);
Re: New version of amandatape script.
On Tue, Feb 14, 2006 at 10:10:51PM +0100, Josef Wolf wrote: Sorry, I wrote my mail into the script and sent an empty message ;-) Now here comes the real mail: Hello! Here comes a new version of the amandatape program I posted about two years ago. This are the changes made since the last release: - Adopt to new logfile format with chunked tapings. - Add possibility to limit output to specified DLEs. - Split output to multiple paper sheets if it doesn't fit onto one page. - Report correct tape with -l option when more than one tape was used in a single amdump/amflush run. - Output total number of DLEs and number of DLEs found on current tape. - Omit error messages on EOT when taper continues writing on next tape. Please let me know if there are bugs or if you like it. Here is the script: #! /usr/bin/perl # amandatape -- a utility to print amanda tape labels for DAT and CD. # # 2004-02-12 Josef Wolf ([EMAIL PROTECTED]) # # Portions of this program which I authored may be used for any purpose # so long as this notice is left intact. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # I wrote this program because I was dissatisfied with the original label # printing program that comes with the amanda distribution. I wanted to see # from one glance on the newest tape which tapes in which order I need to # recover a specific DLE. # # This program prints tapelabels for the amanda backup system. The output # can be in plain ASCII or in postscript. The postscript output is formatted # so that it can be folded to fit into a DAT case or into a CD jewel case. # # An example ASCII output (somewhat stripped to make it fit into 80 columns) # is attached below. Here is an explanation of the example output: # # The columns in the output have following meanings: # # date: This name seems to be intuitive, but unfortunately, it is somewhat #misleading. Actually, this is the name of the logfile that provided #the corresponding information. # label: The label of the tape. # fn:File number on the tape. # fm:Filemark # Osize: Original (that is, uncompressed) size of the dump(s). # Dsize: Size of the dump(s). This is usually pretty close to Tsize so it #is of very little interest. # Tsize: The size of dump(s) on the tape. # Dtime: Dumper time. # Ttime: Taper Time. # Dspd: Dumper speed. # Tspd: Taper speed. # DLE: Disk list entry. # lv:Dump-level. # dpl: "Dumps per level". This is a list of dump levels (starting with #level 0). # error: An error message. # # The output is split into four sections: # # The first section (if present) lists errors. In the example below we can see # that there were two taper errors and a warning that a DLE must be flushed to # tape. # # The second section contains only one line with four fields: # - date # - tape label # - total amount of data on that tape. # - number of DLEs on that tape and total number of DLEs. # # The third section lists all the tapes that are needed to recover all # DLEs. In the example below we can see that three level0, four level1 # and one level3 from tape VOL01 are needed to do a full restore of all # DLEs. In addition, three level0, one level1 and one level2 from VOL09 # are still needed. Everything else on VOL09 is obsoleted by the dumps # on VOL01. VOL08 contains two level0 and one leve1 that are not obsoleted # by newer tapes. Finally, VOL07 contain one level0 dump that is not # obsoleted by newer tapes. # Tapes that contain only obsoleted data are not mentioned at all unless you # supply the -t command line option. # # The fourth section is the main section and is itself split into multiple # sections, one for each DLE. In this section we can see which tapes we # need to recover a specific DLE. For example, we can see that in order # to recover raven:/u4 we need file 1 from VOL07, file 6 from VOL08, # file 7 from VOL09 and file 1 from VOL01, in this order. # # The Ordering of the sections can be different depending on the choosen # output format. # # Due to lack of space, there is a special handling when output is formatted # for DAT: The fourth section is printed in such a way that non-important data # (everything on the left from the Osize column) is cut off from the label. # # Here comes the example output: # # date Tsize label fm lv error # 2004-02-20.0 1499M VOL06 8 ? writing file: No space left on device # 2004-02-23.0 1499M VOL01 9 ? writing file: No space left on device # 2004-02-23.0 670M??? ? 0 raven:/m/u1
New version of amandatape script.
Re: Release of amanda-2.5.0b2
On Mon, Feb 13, 2006 at 01:58:09PM -0800, Kevin Till wrote: > Josef Wolf wrote: > >>1. VOL05:1 (this is the newest non-broken available dump) is _not_ > >> considered for retrieval at all. > >> > >>2. Instead, amfetchdump _tries_ to get the (broken) VOL04:7. > >> > >>3. But instead of VOL04:7 it gets the (older) VOL04:1. There seems to be > >> no attempt to further search for VOL04:7 > >> > >>4. The order of tapes seems to be wired. I would have expected > >> VOL05 VOL02 VOL03 VOL10 (how they were sceduled) > >> or VOL05 VOL04 VOL10 (last available for every dumpdate) > >> or VOL05 VOL10 VOL01 VOL08 VOL09 (first available for every dumpdate) > >> or some such. > >> > >>5. When trying to append the second chunk to the first one, amfetchdump > >> fails with "Bad file descriptor". The resulting dump (uncompressed) > >> is 527620009 bytes long. > >> > >>6. Next problem is with amrecover, but it seems to be closely related > >> with the "Bad file descriptor" problem. Unfortunately, I don't have a > >> transcript for this problem, because the system crashed. Here's the > >> description: > >> > >> When I tried to retrieve the above mentioned DLE mentioned in line c > >> with amrecover, the system (Athlon 1800+, 500MB RAM, 2G swap, > >> suse-10.0) freezed, but vterm switching and pinging from a different > >> host worked. This reminds me of overcommitments caused by memory-hogs. > >> > >> After reboot, I noticed following file in the slot-directory > >> of the vtape directory: > >> > >> -rw--- 1 amanda disk 527630347 Feb 7 07:52 info > >> > >> Notice that the length is almost the same as in 5. This file starts with > >> following contents: > > can you make sure you have restore-src/restore.c revision 1.19 or above? > One fix went it on r1.19 which resolved one file descriptor problem. Thanks Kevin! I have tried with newest (1.23) restore-src/restore.c. With this, bullet 5 seems to be gone. Bullet 6 don't crash the system anymore, but still don't seem to work properly: Extracting files using tape drive changer on host host.do.main. Load tape VOL01 now Continue [?/Y/n/s/t]? The following tapes are needed: VOL01 amrecover: short block 0 bytes UNKNOWN file amrecover: Can't read file header amrecover: Extractor child exited with status 1 extract_list - child returned non-zero status: 1 Continue [?/Y/n/r]? The other issues (1..4) are (of course) still present.
Re: Release of amanda-2.5.0b2
On Tue, Feb 07, 2006 at 08:58:30PM +0100, Josef Wolf wrote:
> Below, I tried to amfetchdump host.do.main:/m/b. Only full dumps were
> done of this DLE. This DLE is available seven times on the tapes:
>
>lv dumpdate chunks on tape
> a. 0 20060204 VOL08:7, VOL08:8, VOL08:9, VOL09:1, VOL09:1
> b. 0 20060204 VOL10:1, VOL10:2, VOL10:3, VOL10:4, VOL10:5
> c. 0 20060205 VOL10:10, VOL10:11, VOL01:1, VOL01:2, VOL01:3
> d. 0 20060205 VOL02:7, VOL02:8, VOL02:9, VOL03:1, VOL03:2
> e. 0 20060205 VOL04:1
> f. 0 20060207 VOL04:7, write aborted due to full tape.
> g. 0 20060207 VOL05:1
>
> Tapings a..d were done with tape_splitsize=500mb
> Tapings b, d and e were done by autoflush because of bug#1425436.
> All dumps are compressed.
>
> Below is a transcript attached. There are several problems I see here:
>
> 1. VOL05:1 (this is the newest non-broken available dump) is _not_
>considered for retrieval at all.
>
> 2. Instead, amfetchdump _tries_ to get the (broken) VOL04:7.
>
> 3. But instead of VOL04:7 it gets the (older) VOL04:1. There seems to be
>no attempt to further search for VOL04:7
>
> 4. The order of tapes seems to be wired. I would have expected
> VOL05 VOL02 VOL03 VOL10 (how they were sceduled)
>or VOL05 VOL04 VOL10 (last available for every dumpdate)
>or VOL05 VOL10 VOL01 VOL08 VOL09 (first available for every dumpdate)
>or some such.
>
> 5. When trying to append the second chunk to the first one, amfetchdump
>fails with "Bad file descriptor". The resulting dump (uncompressed)
>is 527620009 bytes long.
>
> 6. Next problem is with amrecover, but it seems to be closely related
>with the "Bad file descriptor" problem. Unfortunately, I don't have a
>transcript for this problem, because the system crashed. Here's the
>description:
>
>When I tried to retrieve the above mentioned DLE mentioned in line c
>with amrecover, the system (Athlon 1800+, 500MB RAM, 2G swap,
>suse-10.0) freezed, but vterm switching and pinging from a different
>host worked. This reminds me of overcommitments caused by memory-hogs.
>
>After reboot, I noticed following file in the slot-directory
>of the vtape directory:
>
> -rw--- 1 amanda disk 527630347 Feb 7 07:52 info
>
>Notice that the length is almost the same as in 5. This file starts with
>following contents:
>
> position 0
> AMANDA: FILE 20060205 raven.wolf.local /m/b lev 0 comp .gz program /bin/tar
> To restore, position tape at start of file and run:
> dd if= bs=32k skip=1 | /usr/bin/gzip -dc | /bin/tar -f... -
>
>
>Notice the first line "position 0" which seems to be the original
>contents of the info file. At position 32779 (that is,
>strlen("position 0\n")+32kb) starts a tar file which turns out to be
>the first chunk of the dump I tried to restore.
>
>This looks like amrecover writes the dump to the wrong file descriptor.
>The error message from amfetchdump looks as if amfetchdump has a similar
>problem.
>
>
> Here is the transcript:
>
> host:/ # amfetchdump ppc host.do.main /m/b
> 5 tape(s) needed for restoration
> changer: got exit: 0 str: 4 10 1 1
> The following tapes are needed: VOL04 VOL02 VOL10 VOL01 VOL03
> Press enter when ready
>
> Looking for tape VOL04...
> changer: got exit: 0 str: 4 10 1 1
> changer_query: changer return was 10 1 1
> changer_query: searchable = 1
> changer_find: looking for VOL04 changer is searchable = 1
> changer_search: VOL04
> changer: got exit: 0 str: 4 file:/m/amchanger/ppc
> amfetchdump: slot 4: date 20060207 label VOL04 (exact label match)
> Scanning VOL04 (slot 4)
> amfetchdump: 1: restoring FILE: date 20060205 host host.do.main disk /m/b
> lev 0 comp .gz program /bin/tar
> amfetchdump: Search of VOL04 complete
> Looking for tape VOL02...
> changer: got exit: 0 str: 4 10 1 1
> changer_query: changer return was 10 1 1
> changer_query: searchable = 1
> changer_find: looking for VOL02 changer is searchable = 1
> changer_search: VOL02
> changer: got exit: 0 str: 2 file:/m/amchanger/ppc
> amfetchdump: slot 2: date 20060206 label VOL02 (exact label match)
> Scanning VOL02 (slot 2)
> amfetchdump: 7: restoring split dumpfile: date 20060205 host host.do.main
> disk /m/b part 1/5 lev 0 comp .gz program /bin/tar
> amfetchdump: 8: restoring split dumpfile: date 20060205 host host.do.main
> disk /m/b part 2/5 lev 0 comp .gz program /bin/tar
> amfetchdump: appending to host.do.main._m_b.20060205.0.1
> restore: write error: Bad file descriptor
>
> gzip: stdin: unexpected end of file
> host:/ #
Hello!
Are there no opinions about those problems? I think at least points
5. and 6. are serious problems. Opinions?
Re: Release of amanda-2.5.0b2
On Wed, Feb 08, 2006 at 07:04:19PM +0100, Josef Wolf wrote: > On Wed, Feb 08, 2006 at 08:48:15AM -0800, Paddy Sreenivasan wrote: > > > > But I am not sure whether I correctly understand the proposal. Is the > > > following the proposed fix? > > Yes. Please add test results to the bug tracker. > I have arranged for the test tonight. I'll report tomorrow. It worked as expected. Thanks!
Re: Release of amanda-2.5.0b2
On Wed, Feb 08, 2006 at 09:28:34PM -0800, John Franks wrote: > Actually, I removed the check for a NULL pointer since str > should never be NULL. This is what confused me: removing the check without removing the cause. Smells like removing the safety belt while you are still at 300mph ;-) > str is assigned by a call to > syncpipe_getstr() which will, in practice, never return a > NULL pointer. The only way to get there in the old code is > to have the other side of the syncpipe write an invalid string length ( <= > 0). > > However, it looks as though I left out the modification to > syncpipe_getstr() which calls error() when an invalid length > is passed and guarantees a NULL is *never* returned... Hmmm, I still don't get it. If I understand your explanation and the patch correctly, the "(null)" would be generated only when a fatal protocol error occurs. But actually, this is not the case. The "(null)" strings appear in every CHUNKSUCCESS line although everything seems to be OK. Should this really be caused by a protocol error, then the cause for the protocol should be found. Could it be that there is yet another place that could generate NULL? And if so: wouldn't it be better to leave the check in place, just to be sure?
Re: Trouble to get amrecover to work...
On Mon, Feb 06, 2006 at 05:59:00PM -0500, Gene Heskett wrote: > On Monday 06 February 2006 17:29, Josef Wolf wrote: > >Any ideas why amrecover don't want to work? Looks like it tries to > > connect a server on localhost. But whom is it trying to connect to > > and why is it failing? > > Please, localhost is NOT an FQDN, please reconfigure your installation > to use Fully Qualified Domain Names or their alias's if they listed in > the /etc/hosts file. I have double checked: - All the DLEs are specified with FQDN. - "hostname -f" reports the fqdn as it should be. What might be the reason that amrecover assumes localhost is the tape server? How comes that it automatically uses "hostname -f" for the index server but fails to do the same for the tape server? I bet that on most installations the index server and the tape server are the same box. Wouldn't it make sense to default both to "hostname -f"? > localhost has a fixed address of 127.0.0.1 on any > machine and will eat your lunch when trying to recover because its > always the box the recover is running on. And that may not be the > server intended. In this case I ran amrecover on the amanda server which by accident happens to be both, the index server _and_ the tape server. > >Extracting files from holding disk on host localhost. > >The following files are needed: > > /tmp/hold/20060206030502/host.do.main._var.0 > > Holding disk? Yes. The dumps were not flushed. But this looks wired. Isn't the holding disk always supposed to be on the tape server? So how comes that amrecover can contact the index server and find the holding disk but can't find the tapes? > Possibly the 'sethost' command might bail you out, but I've no > experience with this, so I'm not positive about it. Unfortunately, the sethost command does _not_ set the tape server. Only the -t command line option seems to set it. Thanks for the reply.
Re: Release of amanda-2.5.0b2
On Wed, Feb 08, 2006 at 08:48:15AM -0800, Paddy Sreenivasan wrote: > > But I am not sure whether I correctly understand the proposal. Is the > > following the proposed fix? > > Yes. Please add test results to the bug tracker. I have arranged for the test tonight. I'll report tomorrow. Thanks! BTW: Checkin 1.111->1.112 of server-src/taper.c looks a little bit strange to me. Especially two hunks I've appended to the end of this mail. When ``str ? str : "(null)"´´ results in "(null)" the str was the NULL pointer. Will the called function newvstralloc() be able to cope with the passed NULL? @@ -1428,7 +1428,7 @@ "[sec ", walltime_str(runtime), " kb ", kb_str, " kps ", kps_str, - " ", str ? str : "(null)", + " ", str, "]", NULL); @@ -1469,7 +1467,7 @@ "[sec ", walltime_str(curdump_rt), " kb ", kb_str, " kps ", kps_str, - " ", str ? str : "(null)", + " ", str, "]", NULL); putresult(DONE, "%s %s %d %s\n", handle, label,
Re: Release of amanda-2.5.0b2
On Mon, Feb 06, 2006 at 09:38:15AM -0800, Paddy Sreenivasan wrote: > I have filed a bug in sourceforge bugtracker for this problem. > http://sourceforge.net/tracker/index.php?func=detail&aid=1425436&group_id=120&atid=100120 Ian Turner proposed a fix to this problem. I'd like to go and test it. But I am not sure whether I correctly understand the proposal. Is the following the proposed fix? --- server-src/driver.c 1 Feb 2006 11:49:37 - 1.158 +++ server-src/driver.c 8 Feb 2006 16:21:14 - @@ -1086,7 +1086,7 @@ sched(dp)->act_size -= atoi(result_argv[3]); if (sched(dp)->act_size < tape_left) tape_left -= sched(dp)->act_size; else tape_length = 0; - + break; case TAPE_ERROR: /* TAPE-ERROR */ dp = serial2disk(result_argv[2]); free_serial(result_argv[2]);
Re: Release of amanda-2.5.0b2
On Thu, Jan 19, 2006 at 07:31:08PM -0500, Jean-Louis Martineau wrote:
> The Amanda core team is pleased to announce the release of Amanda
> 2.5.0b2. It is stable and we hope to release 2.5.0 soon. test it and
> report it if you find a bug in this release.
I have one (or two?) more issue here.
Below, I tried to amfetchdump host.do.main:/m/b. Only full dumps were
done of this DLE. This DLE is available seven times on the tapes:
lv dumpdate chunks on tape
a. 0 20060204 VOL08:7, VOL08:8, VOL08:9, VOL09:1, VOL09:1
b. 0 20060204 VOL10:1, VOL10:2, VOL10:3, VOL10:4, VOL10:5
c. 0 20060205 VOL10:10, VOL10:11, VOL01:1, VOL01:2, VOL01:3
d. 0 20060205 VOL02:7, VOL02:8, VOL02:9, VOL03:1, VOL03:2
e. 0 20060205 VOL04:1
f. 0 20060207 VOL04:7, write aborted due to full tape.
g. 0 20060207 VOL05:1
Tapings a..d were done with tape_splitsize=500mb
Tapings b, d and e were done by autoflush because of bug#1425436.
All dumps are compressed.
Below is a transcript attached. There are several problems I see here:
1. VOL05:1 (this is the newest non-broken available dump) is _not_
considered for retrieval at all.
2. Instead, amfetchdump _tries_ to get the (broken) VOL04:7.
3. But instead of VOL04:7 it gets the (older) VOL04:1. There seems to be
no attempt to further search for VOL04:7
4. The order of tapes seems to be wired. I would have expected
VOL05 VOL02 VOL03 VOL10 (how they were sceduled)
or VOL05 VOL04 VOL10 (last available for every dumpdate)
or VOL05 VOL10 VOL01 VOL08 VOL09 (first available for every dumpdate)
or some such.
5. When trying to append the second chunk to the first one, amfetchdump
fails with "Bad file descriptor". The resulting dump (uncompressed)
is 527620009 bytes long.
6. Next problem is with amrecover, but it seems to be closely related
with the "Bad file descriptor" problem. Unfortunately, I don't have a
transcript for this problem, because the system crashed. Here's the
description:
When I tried to retrieve the above mentioned DLE mentioned in line c
with amrecover, the system (Athlon 1800+, 500MB RAM, 2G swap,
suse-10.0) freezed, but vterm switching and pinging from a different
host worked. This reminds me of overcommitments caused by memory-hogs.
After reboot, I noticed following file in the slot-directory
of the vtape directory:
-rw--- 1 amanda disk 527630347 Feb 7 07:52 info
Notice that the length is almost the same as in 5. This file starts with
following contents:
position 0
AMANDA: FILE 20060205 raven.wolf.local /m/b lev 0 comp .gz program /bin/tar
To restore, position tape at start of file and run:
dd if= bs=32k skip=1 | /usr/bin/gzip -dc | /bin/tar -f... -
Notice the first line "position 0" which seems to be the original
contents of the info file. At position 32779 (that is,
strlen("position 0\n")+32kb) starts a tar file which turns out to be
the first chunk of the dump I tried to restore.
This looks like amrecover writes the dump to the wrong file descriptor.
The error message from amfetchdump looks as if amfetchdump has a similar
problem.
Here is the transcript:
host:/ # amfetchdump ppc host.do.main /m/b
5 tape(s) needed for restoration
changer: got exit: 0 str: 4 10 1 1
The following tapes are needed: VOL04 VOL02 VOL10 VOL01 VOL03
Press enter when ready
Looking for tape VOL04...
changer: got exit: 0 str: 4 10 1 1
changer_query: changer return was 10 1 1
changer_query: searchable = 1
changer_find: looking for VOL04 changer is searchable = 1
changer_search: VOL04
changer: got exit: 0 str: 4 file:/m/amchanger/ppc
amfetchdump: slot 4: date 20060207 label VOL04 (exact label match)
Scanning VOL04 (slot 4)
amfetchdump: 1: restoring FILE: date 20060205 host host.do.main disk /m/b lev
0 comp .gz program /bin/tar
amfetchdump: Search of VOL04 complete
Looking for tape VOL02...
changer: got exit: 0 str: 4 10 1 1
changer_query: changer return was 10 1 1
changer_query: searchable = 1
changer_find: looking for VOL02 changer is searchable = 1
changer_search: VOL02
changer: got exit: 0 str: 2 file:/m/amchanger/ppc
amfetchdump: slot 2: date 20060206 label VOL02 (exact label match)
Scanning VOL02 (slot 2)
amfetchdump: 7: restoring split dumpfile: date 20060205 host host.do.main
disk /m/b part 1/5 lev 0 comp .gz program /bin/tar
amfetchdump: 8: restoring split dumpfile: date 20060205 host host.do.main
disk /m/b part 2/5 lev 0 comp .gz program /bin/tar
amfetchdump: appending to host.do.main._m_b.20060205.0.1
restore: write error: Bad file descriptor
gzip: stdin: unexpected end of file
host:/ #
Trouble to get amrecover to work...
Hello! Any ideas why amrecover don't want to work? Looks like it tries to connect a server on localhost. But whom is it trying to connect to and why is it failing? host:/tmp/rest # amrecover ppc AMRECOVER Version 2.5.0b2. Contacting server on localhost ... 220 host AMANDA index server (2.5.0b2) ready. 200 Access OK Setting restore date to today (2006-02-06) 200 Working date set to 2006-02-06. Scanning /tmp/hold... 20060205030502: found Amanda directory. 20060206030502: found Amanda directory. 200 Config set to ppc. 501 Host host is not in your disklist. Trying host host.do.main ... 200 Dump host set to host.do.main. Trying disk /m ... Trying disk mapper/m ... Can't determine disk and mount point from $CWD '/tmp/rest' amrecover> setdisk /var 200 Disk set to /var. amrecover> settape file:/tmp Using tape "file:/tmp" from server localhost. amrecover> add log Added dir /log at date 2006-02-06 amrecover> extract Extracting files from holding disk on host localhost. The following files are needed: /tmp/hold/20060206030502/host.do.main._var.0 Restoring files into directory /tmp/rest Continue [?/Y/n]? Extracting from file /tmp/hold/20060206030502/host.do.main._var.0 amrecover: Could not read features from control socket host:/tmp/rest #
Re: tape usage
On Mon, Feb 06, 2006 at 12:19:38PM +, Anthony Worrall wrote: > Is there an easy way to see what the usage of the tapes are? > > There seem to be was of doing this for a host or DLE but not a tape. Have you checked the script I posted a while ago to this list? Take a look at http://groups.yahoo.com/group/amanda-users/message/50164 I am currently adopting it to the new logfile format that was changed due to the new tape_splitsize dumptype option. I will repost it when it's done.
Re: gnutar error messages configurable?
On Sat, Jan 14, 2006 at 03:04:26PM -0500, Jon LaBadie wrote: > The contents of the file must always be specified as relative to the > head directory of the DLE. Since you guys are currently working hard to get 2.5.0 out, it might be a good time to put this sentence into the amanda.conf manpage?
Re: Release of amanda-2.5.0b2
On Thu, Jan 19, 2006 at 07:31:08PM -0500, Jean-Louis Martineau wrote: > The Amanda core team is pleased to announce the release of Amanda > 2.5.0b2. It is stable and we hope to release 2.5.0 soon. test it and > report it if you find a bug in this release. I've found one more problem. I reported this problem yesterday to the list, but the mail seems to be lost somehow. Sorry if this report gets through twice. The problem seems to be triggered when runtapes>1 and tape_splitsize is set. When the first tape (vtape in my case) is filled up, driver reports that it goes to degraded mode because of a tape error (how comes with runtapes>1?). Taper happily writes out all the remaining chunks to the second tape. But since driver (erronously) thinks that there is a serious tape error, the chunks that taper writes to the second tape are not removed from the holding disk. Note that taper writes the remaining chunks after driver said that it wants to quit. Here comes the interesting part of the amdump.1 file: driver: finished-cmd time 2167.114 taper wrote host.do.main:/m/b driver: state time 4979.062 free kps: 18792 space: 6766464 taper: idle idle-dumpers: 9 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers driver: interface-state time 4979.062 if : free 800 if ETH10: free 800 if ETH100: free 8000 if LOCAL: free 9192 driver: hdisk-state time 4979.062 hdisk 0: free 6766464 dumpers 1 driver: result time 4979.092 from chunker0: RQ-MORE-DISK 01-9 driver: find_diskspace: time 4979.092: want 133536 K driver: find_diskspace: time 4979.092: find diskspace: size 133536 hf 6766464 df 6766240 da 133536 ha 133568 driver: find_diskspace: time 4979.092: find diskspace: selected /m/amchanger/hold free 6766464 reserved 133568 dumpers 0 driver: assign_holdingdisk: time 4979.092: merging holding disk /m/amchanger/hold to disk host.do.main:/, add 133568 for reserved 2804288, left 0 driver: send-cmd time 4979.092 to chunker0: CONTINUE 01-9 /m/amchanger/hold/20060203030502/host.do.main._.0 1048576 133568 driver: state time 4979.362 free kps: 18792 space: 6632896 taper: idle idle-dumpers: 9 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers driver: interface-state time 4979.362 if : free 800 if ETH10: free 800 if ETH100: free 8000 if LOCAL: free 9192 driver: hdisk-state time 4979.362 hdisk 0: free 6632896 dumpers 1 driver: result time 4979.362 from dumper0: DONE 01-9 6688750 2671057 3188 [sec 3188.471 kb 2671057 kps 837.7 orig-kb 6688750] driver: finished-cmd time 4979.362 dumper0 dumped host.do.main:/ driver: send-cmd time 4979.362 to chunker0: DONE 01-9 driver: state time 4979.401 free kps: 18792 space: 6632896 taper: idle idle-dumpers: 9 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers driver: interface-state time 4979.401 if : free 800 if ETH10: free 800 if ETH100: free 8000 if LOCAL: free 9192 driver: hdisk-state time 4979.401 hdisk 0: free 6632896 dumpers 1 driver: result time 4979.401 from chunker0: DONE 01-9 2671057 [sec 3188.624 kb 2671057 kps 837.7] driver: finished-cmd time 4979.402 chunker0 chunked host.do.main:/ driver: adjust_diskspace: time 4979.499: host.do.main:/ /m/amchanger/hold/20060203030502/host.do.main._.0 driver: adjust_diskspace: time 4979.499: hdisk HD done, reserved 2804288 used 2671153 diff -133135 alloc 2671153 dumpers 0 driver: adjust_diskspace: time 4979.499: after: disk host.do.main:/ used 2671153 driver: send-cmd time 4979.499 to taper: FILE-WRITE 00-00010 /m/amchanger/hold/20060203030502/host.do.main._.0 host.do.main feff9ffe03 / 0 20060203 512000 driver: startaflush: LARGESTFIT host.do.main / 2671153 1326289 taper: r: Total dump size should be 2671057kb, chunk size is 512000kb taper: r: Expecting to split into 6 parts driver: state time 4979.500 free kps: 19600 space: 6766031 taper: writing idle-dumpers: 10 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers driver: interface-state time 4979.500 if : free 800 if ETH10: free 800 if ETH100: free 8000 if LOCAL: free 1 driver: hdisk-state time 4979.500 hdisk 0: free 6766031 dumpers 0 driver: result time 4979.500 from taper: SPLIT-CONTINUE 00-00010 VOL04 taper: r: end host.do.main./.20060203.0 part 1, splitting chunk that started at 0kb after 511968kb (next chunk will start at 511968kb) taper: reader-side: got label VOL04 filenum 10 taper: r: end host.do.main./.20060203.0 part 2, splitting chunk that started at 512000kb after 511968kb (next chunk will start at 1023968kb) taper: reader-side: got label VOL04 filenum 11 taper: r: switching to next holding chunk '/m/amchanger/hold/20060203030502/host.do.main._.0.1' taper: writing end marker. [VOL04 ERR kb 4556768 fm 12] changer: opening pipe to: /usr/local/bin/chg-disk.pl -info changer: got exit: 0 str: 4 10 1 1 changer_query: changer return was 10 1 1 changer_query: searchable = 1 changer_find: looking for VOL05 changer is searchable = 1 changer_search: VOL05 changer: opening pipe to: /usr/loca
Re: Release of amanda-2.5.0b2
On Thu, Jan 19, 2006 at 07:31:08PM -0500, Jean-Louis Martineau wrote: > The Amanda core team is pleased to announce the release of Amanda > 2.5.0b2. It is stable and we hope to release 2.5.0 soon. test it and > report it if you find a bug in this release. One more issue here: The new CHUNKSUCCESS lines in the log.mmdd.x files look like there is a NULL pointer dereference at the end of the line.
Re: Question about vtape size
On Thu, Feb 02, 2006 at 04:39:47PM +0100, Paul Bijnens wrote: > Note that when you write a dump to a vtape, and you hit EOT, the > last incomplete part of a dump is not erased, just like a real tape. Yes, this is sub-optimal. > So, many "incomplete" parts on a too small vtapes do take up real > diskspace (and time too!). 2.5.0b2 has the tape_splitsize dumptype option. I've set my config up to check this out tonight.
Re: Question about vtape size
On Thu, Feb 02, 2006 at 10:01:45AM -0500, Jon LaBadie wrote: > How are current users sizing their vtapes and what has > been their experience in disk usage? Here is what I am currently doing: size of vtapes is set to DVD size. I have written a tape-changer with the capability to be set in "offline" mode. All over the week, the changer is set to "offline", thus I circumvent amanda's balancing strategies. Incrementals cummulate in the holding area. Holding area is copied to a USB disk after backup. I have two USB disks for this which I change once a day. Thus, in case of a disaster, I loose at most one day of backups. Copies of holding-areas which are older than 2 weeks are deleted from the USB disk. At friday night, cron forces all DLEs to level 0 and sets the changer to "online". Along with autoflash, I get all the incrementals and a full dump of every DLE on the vtapes. At saturday, with the help of a script, I: 1. burn/verify all this to DVD's, 2. print the labels, 3. empty all vtapes that are already burned _and_ older than 2 weeks This setup has following advantages: 1. 2 weeks of backups are directly available from the vtape area without fiddling with DVDs. 2. Needed disk size is limited to roughly 2 times of the sum of all DLEs, no matter how many vtapes are used. 3. I have at least two copies of everything that is younger than two weeks. (exception is the level0 that was done at saturday morning. They will go to DVD only at the afternoon on the same day). 4. Since every packet of DVDs that I burn at saturday is a full backup of all my DLEs, it is pretty easy to put some of them away for long-time archival. 5. I am involved to the process only once a week. And here is the disadvantage: 1. Number and size of DLEs is limited. Burning 100 DVDs in one afternoon is not very likely to be realistic ;-) 2. I am involved to the process once a week. :-) I think the disadvantages could be resolved with a real tapechanger and some scripting. But a real tapechanger is much too expensive for me, so I prefer to accept the disaadvantages :-)
Re: Release of amanda-2.5.0b2
On Thu, Feb 02, 2006 at 12:24:24AM +0100, Paul Bijnens wrote: > Yes a documentation error. > tape_splitsize is a dumptype option. Already fixed. Thanks! BTW: I noticed some wired formatting in the documentation. But I don't know whether this problem is related to my tools here which are used to build the docu. For example amanda(8) says: //some-pc/home normalpw //another-pc/disk otheruser%otherpw.fi With clear text passwords, this file should obviously be tightly p rotected. It only needs to be readable by the Amanda-user on the Samba server. (line much too long here) or from amcheck(8): WARNING: skipping tape test because amdump or amflush seem to be run- ning, WARNING: if they are not, you must run amcleanup (warning) It looked to amcheck like either amdump or amflush [ ... ] WARNING: compress is not executable, server-compression and indexing will not work (warning) Compression program compress is not executable, so [ ... ] (second line from paragraph needs more indentation)
Re: Release of amanda-2.5.0b2
On Thu, Jan 19, 2006 at 07:31:08PM -0500, Jean-Louis Martineau wrote:
> The Amanda core team is pleased to announce the release of Amanda
> 2.5.0b2. It is stable and we hope to release 2.5.0 soon. test it and
> report it if you find a bug in this release.
I have noticed one more issue here:
The manpage says that disklist entries have the following format:
hostname diskname [diskdevice] dumptype [spindle [interface] ]
This is what older releases already had. But the description below looks
like this has been extended:
[ ... ]
interface
Default: local. The name of a network interface definition in the
amanda.conf file, used to balance network load.
tape_splitsize int
Default: none. Split dump file into pieces of a specified size. This
allows dumps to be spread across multiple tapes, and can potentially make
more efficient use of tape space. Note that if this value is too large
(more than half the size of the average dump being split), substantial
tape space can be wasted. If too small, large dumps will be split into
innumerable tiny dumpfiles, adding to restoration complexity. A good rule
of thumb, usually, is 1/10 of the size of your tape.
[ ... ]
So i tried this dumptype definition:
host.do.main / {
default
compress client best
} 1 local 500
But with this amcheck gives an error:
"/etc/amanda/ppc/disklist", line 9: end of line expected.
Is this an error with the documentation or am I missing something here?
Re: Release of amanda-2.5.0b2
On Tue, Jan 31, 2006 at 05:16:00PM -0500, Jean-Louis Martineau wrote: > This simple patch should fix the problem. Works as expected with this patch. Sorry for the late response, but I wanted to check whether it is actually true before I respond ;-) Thank you!
Re: Release of amanda-2.5.0b2
On Tue, Jan 31, 2006 at 06:53:30AM -0500, Jean-Louis Martineau wrote: > I don't see a success report? Where did you see it? I meant this line: > >taper: writing end marker. [VOL02 OK kb 0 fm 0] But you are right here. This line say just that VOL02 could be loaded successfully, 0KB written. There's nothing that says that the DLE could successfully be written. > The dump never got written to tape, that's what I see from the log below. Amanda tried to write onto VOL01, but taper reported that there's no space left: > >driver: startaflush: Using SMALLEST because nothing fit > >driver: send-cmd time 4595.439 to taper: FILE-WRITE 00-00010 > >/m/amchanger/ppc/hold/20060128030558/host.do.main._.0 host.do.main > >feff9ffe03 / 0 20060128 0 > >driver: startaflush: LARGESTFIT host.do.main / 2650817 1695391 > >taper: r: switching to next holding chunk > >'/m/amchanger/ppc/hold/20060128030558/host.do.main._.0.1' > >taper: writing end marker. [VOL01 ERR kb 4556768 fm 5] > Amanda never retried to write it to the second tape, But amanda labelled the tape and wrote the end marker, thus I assume there were no tape error. > that's a real bug, I will investigated it. Thanks! BTW: why does amanda start to write this DLE to VOL01 at all? She knows in advance that the write will fail. The tapetype says it's 4450mbytes and there were only about 1.6GB free. Why does amanda starting a 2.6gb DLE to a tape where only 1.6gb will fit? Would it be possible (at least for file: vtapes) to delete the started but failed DLEs?
Re: Release of amanda-2.5.0b2
On Thu, Jan 19, 2006 at 07:31:08PM -0500, Jean-Louis Martineau wrote: > The Amanda core team is pleased to announce the release of Amanda > 2.5.0b2. I just noticed a problem that seems to be introduced with this release. I have set runtapes=2. As the first tape filled up, the taper got "no space left on device". Then it retried the failed DLE on the second tape and reported success. But the second tape was left empty and the dump is still in the holding disk. Here are the relevant log entries: INFO taper tape VOL01 kb 4556768 fm 5 writing file: No space left on device START taper datestamp 20060128 label VOL02 tape 1 INFO taper retrying host.do.main:/.0 on new tape due to: [writing file: No space left on device] FAIL driver host.do.main / 20060128 0 [too many taper retries] INFO taper tape VOL02 kb 0 fm 0 [OK] FINISH driver date 20060128 time 4710.157 And this is from the amdump.1 file: driver: startaflush: Using SMALLEST because nothing fit driver: send-cmd time 4595.439 to taper: FILE-WRITE 00-00010 /m/amchanger/ppc/hold/20060128030558/host.do.main._.0 host.do.main feff9ffe03 / 0 20060128 0 driver: startaflush: LARGESTFIT host.do.main / 2650817 1695391 taper: r: switching to next holding chunk '/m/amchanger/ppc/hold/20060128030558/host.do.main._.0.1' taper: writing end marker. [VOL01 ERR kb 4556768 fm 5] changer: opening pipe to: /usr/local/bin/chg-disk.pl -info changer: got exit: 0 str: 1 10 1 1 changer_query: changer return was 10 1 1 changer_query: searchable = 1 changer_find: looking for VOL02 changer is searchable = 1 changer_search: VOL02 changer: opening pipe to: /usr/local/bin/chg-disk.pl -search VOL02 changer: got exit: 0 str: 2 file:/m/amchanger/ppc taper: wrote label `VOL02' date `20060128' driver: state time 4710.112 free kps: 19600 space: 6786367 taper: writing idle-dumpers: 10 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers driver: interface-state time 4710.112 if : free 800 if ETH10: free 800 if ETH100: free 8000 if LOCAL: free 1 driver: hdisk-state time 4710.112 hdisk 0: free 6786367 dumpers 0 driver: result time 4710.112 from taper: TRY-AGAIN 00-00010 [writing file: No space left on device] driver: taper-tryagain time 4710.112 disk host.do.main:/ driver: taper failed 00-00010 host.do.main /, too many taper retry driver: state time 4710.112 free kps: 19600 space: 6786367 taper: idle idle-dumpers: 10 qlen tapeq: 0 runq: 0 roomq: 0 wakeup: 0 driver-idle: no-dumpers driver: interface-state time 4710.112 if : free 800 if ETH10: free 800 if ETH100: free 8000 if LOCAL: free 1 driver: hdisk-state time 4710.112 hdisk 0: free 6786367 dumpers 0 driver: QUITTING time 4710.112 telling children to quit taper: DONE [idle wait: 4036.523 secs] taper: writing end marker. [VOL02 OK kb 0 fm 0] driver: FINISHED time 4710.157 amdump: end at Sat Jan 28 04:24:28 CET 2006 Scanning /m/amchanger/ppc/hold... 20060128030558: found Amanda directory.
Re: Why no incrementals in degraded mode?
On Fri, Jan 20, 2006 at 08:51:34AM -0500, Jean-Louis Martineau wrote: > What was your version? 2.4.5 > If you upgraded from 2.4.4p2 or earlier to a 2.4.5 then could you try > the untested attached patch. The patch seems to help. Thanks! Is this a new problem or is it already fixed in CVS?
Re: (RPMs available) Re: Release of amanda-2.5.0b2
On Mon, Jan 23, 2006 at 04:32:38PM -0800, Paddy Sreenivasan wrote: > Amanda 2.5.0b2 RPMs (including source RPM) are now available for the > following linux distributions: > > - Redhat Enterprise server 4.0, > - Suse Enterprise 9.0, > - Open Suse 10.0, > - Fedora Core 3 and > - Fedora Core 4 > > http://www.zmanda.com/downloads.html > > These rpms use "amandabackup" as the Amanda user (or dump user). This user > is created by rpm install scripts. > > Please post feedback and test results to the " Release 2.5.0 beta > discussion" category in Amanda forums > (http://forums.zmanda.com/forumdisplay.php?f=8) Is it by intent or by accident that the tarball contained in your rpm is not the same as the one that was officially released? For example, server-src/amadmin.c is at revision 1.99 instead of 1.103.
Re: Why no incrementals in degraded mode?
On Fri, Jan 20, 2006 at 02:25:41PM +0100, Paul Bijnens wrote:
> >Since I have re-installed amanda, I get no more incrementals when it runs
> >into degraded mode. From the logfiles I can't find any hint why it always
> >does full-backups. All DLEs are affected by this problem. But in the
> >description below, I list only one DLE.
>
> OK, you picked all the lines with evidence for not doing incrementals,
> and still doing full dumps. Strange indeed.
> But the other lines, which you did not find important may contain the
> hint why amanda did fulls instead of the expected incrementals.
> The complete amanda.conf + disklist file would also be interesting
> + version of amanda. Which version did you use before?
Well, I did not want to bother the list with a lengthy message which is
not interesting for most of the readers. But since you say it is needed,
I have appended it below.
> One of the obvious reasons, but difficult to tell from here, is that
> when you re-installed, you have actually two versions instead of one.
> And you update one config, but Amanda uses the other config.
This is next to impossible, since I installed the box completely from
scratch (I always do). Here is the config and the disklist:
org "PPC"
mailto "amanda"
logdir "/m/b/lib/amanda/ppc"
indexdir "/m/b/lib/amanda/ppc/index"
infofile "/m/b/lib/amanda/ppc/curinfo"
tapelist "/m/b/lib/amanda/ppc/tapelist"
tapetype dvd
labelstr "^PPC[0-9][0-9]*$"
runtapes 2
tpchanger "/usr/local/bin/chg-disk.pl"
tapedev "file:/m/amchanger/ppc"
amrecover_changer "file:/m/amchanger/ppc"
holdingdisk hd {
directory "/m/amchanger/ppc/hold"
use 9Gb
}
netusage 800 Kbps
taperalgo largestfit
dumpcycle 0
tapecycle 9 tapes
define tapetype dvd {
length 4450 mbytes
filemark 50 kbytes
speed 15000 kbytes
}
define dumptype default {
program "GNUTAR"
compress client best
exclude list optional ".amanda.exclude"
indexyes
}
define interface local {
use 1 kbps
}
define interface eth100 {
use 8000 kbps
}
define interface eth10 {
use 800 kbps
}
Disklist:
host.do.main / {
default
compress client best
} 1 local
host.do.main /m/b {
default
compress client best
} 1 local
host.do.main /m/check {
default
compress client best
} 1 local
host.do.main /m/l {
default
compress client best
} 1 local
host.do.main /var {
default
compress client best
} 1 local
> Find out with "amadmin x version" and see where Amanda gets the
> configuration files.
> It could also be a bug introduced in some later version of course.
> That's why I would like to know the current+previous version. Did
> you compile yourself, or used rpms?
This is amanda 2.4.5, that comes with the suse-10.0 distribution.
The prevoius version was 2.4.4p1 that came with suse-9.0.
Here comes the amdump.1 file:
amdump: start at Fri Jan 20 03:05:01 CET 2006
amdump: datestamp 20060120
planner: pid 736 executable /usr/lib/amanda/planner version 2.4.5
planner: build: VERSION="Amanda-2.4.5"
planner:BUILT_DATE="Tue Sep 13 01:54:22 UTC 2005"
planner:BUILT_MACH="Linux adalid 2.6.12 #1 SMP Tue Jun 7 15:57:09 UTC
2005 i686 athlon i386 GNU/Linux"
planner:CC="gcc"
planner:CONFIGURE_COMMAND="'./configure' '--mandir=/usr/share/man'
'--prefix=/usr' '--infodir=/usr/share/info' '--sysconfdir=/etc'
'--libdir=/usr/lib' '--libexecdir=/usr/lib/amanda' '--localstatedir=/var/lib'
'--with-index-server=localhost'
'--with-gnutar-listdir=/var/lib/amanda/gnutar-lists'
'--with-smbclient=/usr/bin/smbclient' '--with-amandahosts' '--with-user=amanda'
'--with-group=disk' '--with-gnutar=/bin/tar' '--disable-libtool'
'--disable-shared' '--disable-static'"
planner: paths: bindir="/usr/bin" sbindir="/usr/sbin"
planner:libexecdir="/usr/lib/amanda" mandir="/usr/share/man"
planner:AMANDA_TMPDIR="/tmp/amanda" AMANDA_DBGDIR="/tmp/amanda"
planner:CONFIG_DIR="/etc/amanda" DEV_PREFIX="/dev/"
planner:RDEV_PREFIX="/dev/" DUMP="/sbin/dump"
planner:RESTORE="/sbin/restore" VDUMP=UNDEF VRESTORE=UNDEF
planner:XFSDUMP=UNDEF XFSRESTORE=UNDEF VXDUMP=UNDEF VXRESTORE=UNDEF
planner:SAMBA_CLIENT="/usr/bin/smbclient" GNUTAR="/bin/tar"
planner:COMPRESS_PATH="/usr/bin/gzip"
planner:UNCOMPRESS_PATH="/usr/bin/gzip" LPRCMD="/usr/bin/lpr"
planner:MAILER="/usr/bin/Mail"
planner:listed_incr_dir="/var/lib/amanda/gnutar-lists"
planner: defs: DEFAULT_SERVER="localhost" DEFAULT_CONFIG="DailySet1"
planner:DEFAULT_TAPE_SERVER="localhost"
planner:DEFAULT_TAPE_DEVICE="/dev/null" HAVE_MMAP HAVE_SYSVSHM
planner:LOCKING=POSIX_FCNTL SETPGRP_VOID DEBUG_CODE
planner:AMANDA_DEBUG_DAYS=4 BSD_SECURITY USE_AMANDAHOSTS
planner:CLIENT_LOGIN="amanda" FORCE_USERID HAVE_GZIP
planner:COMPRESS_SUFF
Why no incrementals in degraded mode?
Hello! Since I have re-installed amanda, I get no more incrementals when it runs into degraded mode. From the logfiles I can't find any hint why it always does full-backups. All DLEs are affected by this problem. But in the description below, I list only one DLE. Here is what "amadmin info" says about the DLE in question: Current info for host.do.main /var: Stats: dump rates (kps), Full: 886.0, 768.0, 963.0 Incremental: -1.0, -1.0, -1.0 compressed size, Full: 42.8%, 42.9%, 42.9% Incremental: -100.0%,-100.0%,-100.0% Dumps: lev datestmp tape file origK compK secs 0 20060120 0 416110 178150 201 Here are the relevant excerpts (reformatted to improve readability) from amdump.1: reserving 9437184 out of 9437184 for degraded-mode dumps I don't use the "reserved" keyword, so no full-dumps should be done in degraded mode. That much looks OK. driver: result time 0.474 from taper: TAPE-ERROR [label PPC02 or new tape not found in rack] dump of driver schedule before start degraded mode: dump of driver schedule after start degraded mode: Since no tape was found, degraded mode should be entered? But what do the empty driver scedules mean? planner: time 433.680: got result for host host.do.main disk /var: 0 -> 428820K, 1 -> 45610K, -1 -> -2K The estimates look reasonable to me. FAILED QUEUE: empty DONE QUEUE: 0: host.do.main /var [ ... More DLEs deleted. They all are handled the same ... ] ANALYZING ESTIMATES... pondering host.do.main:/var... next_level0 -1 last_level 0 (due for level 0) (picking inclevel for degraded mode) picklev: last night 0, so tonight level 1 curr level 0 size 182652 total size 182898 total_lev0 182652 balanced-lev0size 182652 So the "pondering" line says that I should get level 1? But what does the "curr level" line say? Looks like level 0 is already picked? Why? INITIAL SCHEDULE (size 5209107): host.do.main /var pri 2 lev 0 size 182652 DELAYING DUMPS IF NEEDED, total_size 5209107, tape length 9113600 mark 50 delay: Total size now 5209107. PROMOTING DUMPS IF NEEDED, total_lev0 5208533, balanced_size 5208533... planner: time 433.726: analysis took 0.045 secs GENERATING SCHEDULE: DUMP raven.wolf.local feff9ffe7f /var 20060120 2 0 1970:1:1:0:0:0 182652 214 1 2006:1:19:2:7:50 22805 760 Here we go: full dump is choosen. Any ideas?
Re: gnutar error messages configurable?
On Sat, Jan 14, 2006 at 03:04:26PM -0500, Jon LaBadie wrote: > > In the above case the DLE in question was the root-directory. In > > filesystem semantics "/media" and "./media" is aequivalent as long as > > you are cd'd into / (as it was in this case). > > Not that it explains any of Josef's problems, but I think some are > confusing the contents of an exclude list file (the things being excluded) > and the specified name of the exclude list file. I am using amanda for almost 15 years now (IIRC), and I have fallen into this trap several times. > The contents of the file must always be specified as relative to the > head directory of the DLE. This sentence should find its way into the man page to clarify this topic. > The name of the list file can be specified absolute (full) or relative. > If relative it is relative to the head directory of the DLE. If absolute, > it is based on the client's root directory. OK, this is already in the man-page. > Thus you could have a global exclude list file affecting all DLE's on > the client plus DLE specific exclude lists. Beware of a trap here: For example assume you want to exclude /bin because you can always re-install it from your distribution CD/DVD. And then you set /usr/local on a separate DLE. Now your /usr/local/bin will be excluded, too. Therefore, with the current implementation of the exclude mechanism, I would strongly recommend a separate exclude file for every DLE. I think a better way to handle exclusions would be to do something like # in a real implementation this should be a string-match instead of # a regex, but you get the idea... # perl -pe 's!^$DLEpath/!./!g' when creating the exclude file. With this, relative and absolute paths would be possible and you could create a global exclude file without the risk to fall into the trap mentioned above.
Re: gnutar error messages configurable?
On Fri, Jan 13, 2006 at 10:44:47AM -0500, Gene Heskett wrote: > >> > and put "/media" into /.amanda.exclude on that host, but this > >> You need to put "./media" into the exclude file. > >Huh? How comes that this makes a difference on the root-filesystem? > > Perhaps the reason for that syntax needs to be repeated. > > Tar, in its traversal of a file system, looks at subdirs in the ./ > world, where the dot anchors it to the current directory. It does not > look any further down the tree than the one its currently "cd'd" to. Well, the confusing bit here is that tar does a _string_ comparison. In the above case the DLE in question was the root-directory. In filesystem semantics "/media" and "./media" is aequivalent as long as you are cd'd into / (as it was in this case). > It has no view of, or knowledge of, any file not in its currently being > addressed directory. But in this case "/media" actually _was_ in its currently addressed directory. The real problem here is two-fold: 1. when creating an archive of the root-directory, gnutar (by default, you can switch this off) prepends a dot to all path names. Gnutar has a reason to do this, but that's a different story. 2. when matching for exclusion, tar ignores the fact that it has prepended the dot, thus the match fails. All that is no big deal. But I think it would be good to put a notice into the man page that the paths should _always_ be given relative even if the DLE in question is the root directory. I just re-read the man page. There is no notice at all that they need to be given relatively.
Re: gnutar error messages configurable?
On Fri, Jan 13, 2006 at 09:52:45AM -0500, Jon LaBadie wrote: > RE the last paragraph; yes, amanda's treatment of the error messages > is configurable. You can add regex messages in sendbackup-gnutar.c. Oh. Sorry, I should have said "runtime configurable". Having to recompile is not really "configurable" (IMHO :-). > Are you sure the dump was discarded? Or are you assuming that based > on the above messages? The dump was neither in the holding disk nor on the tape. And amdump's mail marked it as "FAILED".
Re: gnutar error messages configurable?
On Fri, Jan 13, 2006 at 03:58:16PM +0100, Paul Bijnens wrote: > > ? gtar: ./media/floppy: Cannot savedir: No medium found > > ? gtar: ./media/floppy: Warning: Cannot savedir: No medium found > >The /media directory is handled by an automounter, hence the error message. > Strange it gives any error! Only for floppy and not for e.g. cdrom?? Now that you mention it, it looks strange to me too... > Just curious, what kind of automounter is that? It's hal. This is the one that is installed by default on suse-10.0. > In my environment (CentOS4) the /media is handled by hald (and > fstab updated by fstab-sync), and there is no such problem. Hmmm...
Re: gnutar error messages configurable?
On Fri, Jan 13, 2006 at 10:36:17AM +0100, Alexander Jolk wrote: > >I get from gnutar the following error messages: > > > > ? gtar: ./media/floppy: Cannot savedir: No medium found > > ? gtar: ./media/floppy: Warning: Cannot savedir: No medium found > [...] > > and put "/media" into /.amanda.exclude on that host, but this doesn't > > seem to help. > > You need to put "./media" into the exclude file. Huh? How comes that this makes a difference on the root-filesystem? Anyway, I'll try it (but I will have the result not before tomorrow). Thanks for the hint!
gnutar error messages configurable?
Hello!
I get from gnutar the following error messages:
? gtar: ./media/floppy: Cannot savedir: No medium found
? gtar: ./media/floppy: Warning: Cannot savedir: No medium found
| gtar: ./tmp/.X11-unix/X0: socket ignored
| gtar: ./tmp/ssh-cqLX19675/agent.19675: socket ignored
| Total bytes written: 6649047040 (6.2GiB, 2.4MiB/s)
? gtar: Error exit delayed from previous errors
sendbackup: error [/bin/tar returned 2]
The /media directory is handled by an automounter, hence the error message.
I have tried to
define dumptype default {
program "GNUTAR"
compress client best
exclude list optional ".amanda.exclude"
indexyes
# strategy "incronly"
}
and put "/media" into /.amanda.exclude on that host, but this doesn't seem to
help.
Now I wonder whether the expected error messages from gnutar are configurable
somehow. The "Cannot savedir: No medium found" is probably something that
should not be a fatal error and cause amanda to throw away the whole dump?
Bye.
Re: new feature: client-side, server-side encryption dumptype option
> > I think it would be helpful for you to write out your assumptions > > about threats. I am relatively unconcerned with people getting access > > to my tapes - they are locked up as well as the computers. > > They are locked up _today_. Do you know what will happen to them in a > couple of months/years? I remember at least two cases where big banks > have lost tapes with sensitive data on them and no one knows where the > tapes are or who have/had access to them. How do you know that this > will not happen to your tapes? I just stumbled over yet an other (current) case of lost backup tapes with sensitive data on them: Look at http://www.heise.de/newsticker/meldung/67824 Since this page is in german, here's a short summary: Marriot Hotels has lost backups with address and creditcard data of 206000 customers. [ ... ] A couple of weeks ago the bank ABN Amro has lost a backup tape with data of 2 million credit users. > > Really, I am trying to ask you to think about keeping transport and > > storage encryption conceptually separate, even if you have a mechanism > > that does both without any bits on the server. The above examples show that having unencrypted backups is not really a good idea. So please think once more about it. The only sane way is the client-encrypted public-key method. -- No software patents in Europe -- http://nosoftwarepatents.com -- Josef Wolf -- [EMAIL PROTECTED] --
Re: new feature: client-side, server-side encryption dumptype option
e encryption conceptually separate, even if you have a mechanism > that does both without any bits on the server. So use a tunnel for transport and GPG/PGP for storage. Both mechanisms are already available _now_ and they are already matured. Why implement yet an other mechanism and potentially introduce new bugs? I can't see what archetectual changes are needed here. The only thing I am missing is tar-command to be a runtime option. > > > It's broken from a security viewpoint to only configure this on the > > > server, particularly without authentication of the dump requests. > > You're right here. It was just an example, since amanda don't have > > client-side configs yet. But I bet you got the idea :) > Right, but if you don't fix that, you haven't really gained the > security properties you are trying to get. Why fix something that's not broken? /usr/lib/amanda/amandatar (on the client!) can (and should) take care of that. > > A completely different question (which was not touched yet in this thread) > > is to make sure that the data comes from the correct _client_. > > Sure, that's important as well. the krb4 stuff more or less handles this. I see I should take a close look on krb :) -- No software patents in Europe -- http://nosoftwarepatents.com -- Josef Wolf -- [EMAIL PROTECTED] --
Re: new feature: client-side, server-side encryption dumptype option
On Sat, Dec 24, 2005 at 03:22:09AM -0500, Jon LaBadie wrote: > Aside from your opinion that combos B,C, and D > are redundant or inferior to E, what are your objections > to allowing the amanda user to make their own flexible choice. I have no objections. It is just that - implementing redundant methods feels like waste of work. - having redundant methods makes things complicated, which in turn tend to attract bugs. - having redundant methods makes configuration complicated, which might be the cause of additional error-proneness. When it comes to security, things should be kept as simple as possible. Complicated things tend to get error-prone. Errors in security-relevant functionality is not what we really want to have. IMHO, E should be implemented anyway, so E stands for the least possible complexity. Since the functionality of the other methods can be achieved via E, adding those methods raises complexity. Oh, I forgot to mention: everything I wrote is IMHO, of course :-) -- No software patents in Europe -- http://nosoftwarepatents.com -- Josef Wolf -- [EMAIL PROTECTED] --
Re: new feature: client-side, server-side encryption dumptype option
On Wed, Dec 21, 2005 at 08:38:56AM -0500, Greg Troxel wrote:
> The fundamental point that I think you are missing is that other
> people will have different concerns and threat models that you do.
So please, can we keep this conversation going until we all have a clear
understanding? I tried to cover all of the aspects you mentioned.
But since you failed to mention _all_ aspects, I admit that I am far
from being perfect ;-)
> So I believe
> that a number of people will choose, and rationally so, not to encrypt
> the tapes.
Thus I proposed the option to write the private key onto the tape. No
key management needed for those who don't want security...
> D. What would be the benefit of this case? I see only disadvantages here:
> - high load on tapeserver.
> - more surface for attackers.
>
> I don't see much benefit, but it could work with existing krb4 support
> for transport (with old clients) and a local modification on the
> server.
Those who use krb4 today can continue to use it the way they use it today,
don't they? I don't see why they are no more able to do what they are
doing today.
> So E has no disadvantages here.
>
> With respect to confidentiality, no. But E is weaker than the cases
> that don't encrypt tapes from the backup availability perspective.
No doubt here. As long as encryption is an _option_, no one is forced
to use it. Those who actually use encryption, should be aware of the
fact that they loose data when they loose the key. IMHO, this is basic
knowledge. If an admin is not aware of such facts, he needs to be
fired (IMHO).
> define dumptype foo {
> foo-bar-dumptype
> crypt-pubkey /path/to/pubkey-%h-%d # public key is on client
> store-privkey /path/to/privkey-%h-%d # private key is on server
> }
>
> If the private key is on the server the data on the server might as
> well be in the clear.
This is to solve the case where you wanted to have the tapes to be
in clear-text. I don't think it is a big security issue to have the
keys in clear on the server when the alternative would be to have no
keys at all because the tapes are in clear. If you don't want the
priv-key on the tape (that is, if you have some sort of key-management)
then there is no need to keep the priv-key on the tape-server or to use
the store-privkey option at all.
> But, not having so makes interactive restores harder.
Security don't come for free. You have to pay a price if you want
encryption (AFAIK). Either you need a key to get the data, or everyone
can get the data without a key. If you want to be secure, you need to
store the key in a secure place.
> This really needs quite a lot of key management thought. But
> later, I see that you intend to be able to implement cleartext on
> tapes but with transport encryption via this.
I don't inted this. It was just a quick thougt in response to your
requirement list. It was just an example of how such a requirement
could be satisfied. Security don't come for free. You _need_ to do
some thoughts about key management.
> It's broken from a security viewpoint to only configure this on the
> server, particularly without authentication of the dump requests.
You're right here. It was just an example, since amanda don't have
client-side configs yet. But I bet you got the idea :)
A completely different question (which was not touched yet in this thread)
is to make sure that the data comes from the correct _client_.
--
No software patents in Europe -- http://nosoftwarepatents.com
-- Josef Wolf -- [EMAIL PROTECTED] --
Re: new feature: client-side, server-side encryption dumptype option
On Tue, Dec 20, 2005 at 09:03:50AM -0500, Greg Troxel wrote:
> Josef Wolf <[EMAIL PROTECTED]> writes:
> > On Mon, Dec 19, 2005 at 12:56:26PM -0500, Greg Troxel wrote:
> > > I think the essence is that while both are encryption, one is applied
> > > to transport and one to storage.
> > Is it really _that_ different? IMHO, a public-key method encrypted on
> > the client would be good for both, transport _and_ storage.
>
> Yes, it really is totally different.
I'm still not very convinced. IMHO, case E has (from the security/safety
perspective) only advantages and no disadvantages when compared to the
other cases. Let's take a closer look at the use-cases you listed.
> transport storage tag
> nono A
> yes no B
> noyes C
> yes yes D [separate encryption, back to
> clear on server]
> yes still E [client encrypts, server passthrough]
A. This is the trivial case. If we talk about encryption, this case
doesn't make any sense at all. Just don't activate encryption, and
you're done.
I don't see why this case deserves any discussion at all. The
end-result (if you can read the tape you have access to unencrypted
data) is identical to case B. So why not fall back to case B here?
B. If the tapes are stored in a trusted environment, it should be no
problem to store the private key along with the tapes. In addition,
you can store a copy of the key in a safe.
Technically, it would even be possible to store the key on the amanda
tapes (in a preamble). This could even be automatable by an amanda
config-option (or holding the priv-keys on an un-encrypted DLE).
On first sight, such a solution might look like a disadvantage, but
please keep reading.
C. If you are storing encrypted anyway, why not transporting encrypted
too? Most amanda-installations have lots of clients anyway, so putting
the burdon of encryption on the server will probaly slow down backups
on most installations. I don't see any benefits of this case.
D. What would be the benefit of this case? I see only disadvantages here:
- high load on tapeserver.
- more surface for attackers.
E. This is the case I would prefer. Obviosly, I don't see any
disadvantages here ;-) But I am pretty sure someone on this list will
be able to enlighten me in the near future ;-)
> In scenarios A and C, and eavesdropper on the network can observe the
> data. In B D and E an eavesdropper cannot.
So E has no disadvantages here.
> In secenarios A and B, one can retrieve the data from backup tapes
> without keying material. This is perhaps obvious, but very important
> for backups.
Should be no problem for E as long as you store the key along with the
tapes or even _on_ the tapes.
> In C, D, and E, the backup tapes are encrypted. Thus, one could store
> them someplace where you trust people not to destroy them but you
> don't trust them to read them. This can make some sense for
> particularly sensitive data.
Again, no disadvantages for E here.
> In E, the backup data doesn't appear on the server in plaintext. This
> can be useful for backing up clients with particularly sensitive
> data. To really make sense, the client should be configured so that
> it will only honor dump requests to a preconfigured set of public
> keys.
Clearly, only E can achieve that.
> Note that any of these scenarios can be implemented with symmetric
> encryption (e.g. AES) or public key techniques (e.g. OpenPGP with RSA
> used to encrypt an AES key used for the bulk data, or IKE). An
> OpenPGP approach is certainly easier to manage for storage, where one
> needs to be able to decrypt backups much later in time. For
> transport, the backups are decrypted at the server, so techniques like
> Kerberos are workable, in addition to OpenPGP or IPsec/IKE.
It's true there are many ways to implement all that. But restricting to
client-encrypted-pubkey method would greatly simplify implementation
_and_ configuration.
>From the perspective of an end-user, I would prefer to have only two
additional config-options:
define dumptype foo {
foo-bar-dumptype
crypt-pubkey /path/to/pubkey-%h-%d # public key is on client
store-privkey /path/to/privkey-%h-%d # private key is on server
}
In the above, %h (if available) would be replaced by the hostname and %d
by the DLE name. This makes it possible to have DLE-specific key-pairs,
host-specific key-pairs, or to have one key-pair for the entire amanda
configuration, depending on the security-requirements of the user.
So, to implement the use-cases you mentioned above:
case A: just omit both options
Re: Verify backup
On Mon, Dec 12, 2005 at 10:15:29AM -0500, Jon LaBadie wrote: > Giovanni, one thing to consider is that between the time amanda > backs up a file and the time you attempt to verify the accuracy > of that backup, many files are likely to have changed, been > deleted, or added. Would it be possible to calculate md5-sums on the DLEs just before they are written to the tape? This would make it quite easy to verify whether the tape munges any data. -- No software patents in Europe -- http://nosoftwarepatents.com -- Josef Wolf -- [EMAIL PROTECTED] --
Re: new feature: client-side, server-side encryption dumptype option
On Sun, Dec 11, 2005 at 04:37:12PM -0800, Paddy Sreenivasan wrote: > > [1] You need client-side encryption if you don't want your data flowing > > unencrypted over the network. > You can use ssh for server/client communication and server side data > encryption. While this is possible, I don't like it because it has two disadvantages: 1. You need to set up keys for both, ssh _and_ amanda. 2. You need to set up ssh keys either by storing the passphrase in cleartext or use ssh-agent. Using ssh-agent has security-flaws, too. In addition, with ssh-agent, you will need to type the passphrase at least once after every reboot. Not very suitable for a fully-automated backup-system, IMHO. When you use a public key on the client to encrypt, you can lock away the private key in a secure place. Both, transport _and_ storage will be encrypted with such a setup. -- No software patents in Europe -- http://nosoftwarepatents.com -- Josef Wolf -- [EMAIL PROTECTED] --
Re: new feature: client-side, server-side encryption dumptype option
On Mon, Dec 19, 2005 at 12:56:26PM -0500, Greg Troxel wrote: > I think the essence is that while both are encryption, one is applied > to transport and one to storage. Is it really _that_ different? IMHO, a public-key method encrypted on the client would be good for both, transport _and_ storage. -- No software patents in Europe -- http://nosoftwarepatents.com -- Josef Wolf -- [EMAIL PROTECTED] --
Re: new feature: client-side, server-side encryption dumptype option
On Sat, Dec 10, 2005 at 09:12:49AM -0800, Kevin Till wrote: > I have added a dumptype option, "encrypt". Great! Thanks for your afford to bring encryption into amanda's core. I have not taken a close look on it yet. From the description, I have the impression that this solution is based on symmetric encryption. I don't think this is a good idea. Having the passphrase lying around on all your clients[1] is a big weakness, IMHO. It would be better to encrypt with a public key. The private key and the passprase should be locked away to a safe place. They should be needed only for desaster-recovery. [1] You need client-side encryption if you don't want your data flowing unencrypted over the network. -- No software patents in Europe -- http://nosoftwarepatents.com -- Josef Wolf -- [EMAIL PROTECTED] --
Re: Automating the labeling of tapes
On Wed, Dec 29, 2004 at 08:30:08AM -0500, Joshua Baker-LePain wrote: > On Wed, 29 Dec 2004 at 8:00am, Kevin Alford wrote > > > It holds 15 tapes with another slot for my cleaning tape. I would like > > to setup Amanda to automatically label my tapes, so that I can have a > > totally lights out backup environment. I have not been able to install > > You only need to label tapes once. So automating it isn't really all that > great a need. Depends on what information you want to have on the labels. If you are happy with just the name of the tape, then labelling them once will be OK. But some people want more information on the label. For my part, I want to have much more information on the label. I want information like: - which DLEs (on which levels) are contained on the tape? - which tapes will I need in addition to the newest one to do a full restore for a specific DLE. - How much data is contained on a DLE and how long did it take to write it to tape (this gives a good estimation on how much disk/time I will need to restore) and much more. If you really want this information on the label, you will need to print labels after every backup. To see what I mean, please take a look on the amandatape script that I posted about one year ago to this list. -- No software patents! -- Josef Wolf -- [EMAIL PROTECTED] --
Re: Encrypted backups
On Wed, Jul 28, 2004 at 01:34:59PM -0400, Andrew Hall wrote: > http://security.uchicago.edu/tools/gpg-amanda/ There is exactly one thing I don't like with this solution: it is a compile-time configuraton. My assumption is that when the Big Bang comes, my only data is my backup and the distribution CD/DVD's. Unless my distribution's rescue-CD is compiled to support this hack, I'm out of luck: I'll need to download/compile/etc/pp amanda. In the case of desaster, I'll probably _not_ have time/resources to download/compile/etc/pp amanda. Having gpg support as a runtime option (so that you can use it when booting from a rescue disk) would be a _big_ win. IMHO. -- Please visit and sign and http://www.ffii.org -- Josef Wolf -- [EMAIL PROTECTED] --
Re: 2.6.6-rc2 and newer cause trouble with amanda
On Thu, Jun 17, 2004 at 09:07:39PM +0200, Andreas Sundstrom wrote: > Anyway here's the diff from 2.6.5-rc1 to 2.6.5-rc2: I am not a kernel hacker and I have not checked the source. But judging from the diff there is one thing that look strange to me: Old code assigned exp_amanda_info->offset before calling simple_strtoul() while new code calls simple_strtoul() before assignment. This results in exp_amanda_info->offset pointing _behind_ the parsed number instead of before the parsed number. I have no clue what this offset is good for, but if it is used for anything, I'd expect problems similar to those which you were experiencing. -- Please visit and sign http://petition-eurolinux.org and http://www.ffii.org -- Josef Wolf -- [EMAIL PROTECTED] --
Re: Printing tape labels
On Tue, Feb 24, 2004 at 11:31:43PM +0100, Josef Wolf wrote: Here comes a repost of the "amandatape" label printing script. First, a short summary of the changes since the last post: Bug fixes: * Fix dumpcycle search when more than one tape is written in a single run. * Fix dumper date for runs without taping. * Fix printing when several levels of one DLE are on a single tape. * Fix handling of runs without tape when -l option is used. * Fix bug not outputting dumps that are not taped yet. * Don't choke on "INFO taper retrying" line. * Don't choke when a dump is no longer in a logfile. * Adjust geometry of PS output. * Error out properly when no logfiles could be found or can't be read. Improvements: * Search logfiles in oldlog directory in addition to standard logfiles. * Print warning on missing backup levels. * Create real postscript instead of EPS. * Add -l option to print label for specific tape. * Add -d option to ignore logfiles not older than a specific date. * Add -p option to specify paper type. * Add -o option to specify output file. * Options parsing made consistently * Fall back to use as a directory where logfiles are located if amgetconf fails to get configuration for . Many thanks to Jon LaBadie who greatly helped to shake out bugs and gave valuable feedback for improvements. 8< 8< 8< 8< 8< 8< 8< #! /usr/bin/perl -w # amandatape -- a utility to print amanda tape labels for DAT and CD. # # 2004-02-12 Josef Wolf ([EMAIL PROTECTED]) # # Portions of this program which I authored may be used for any purpose # so long as this notice is left intact. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # I wrote this program because I was dissatisfied with the original label # printing program that comes with the amanda distribution. I wanted to see # from one glance on the newest tape which tapes in which order I need to # recover a specific DLE. # # This program prints tapelabels for the amanda backup system. The output # can be in plain ASCII or in postscript. The postscript output is formatted # so that it can be folded to fit into a DAT case or into a CD jewel case. # # An example ASCII output (somewhat stripped to make it fit into 80 columns) # is attached below. Here is an explanation of the example output: # # The columns in the output have following meanings: # # date: This name seems to be intuitive, but unfortunately, it is somewhat #misleading. Actually, this is the name of the logfile that provided #the corresponding information. # label: The label of the tape. # fn:File number on the tape. # fm:Filemark # Osize: Original (that is, uncompressed) size of the dump(s). # Dsize: Size of the dump(s). This is usually pretty close to Tsize so it #is of very little interest. # Tsize: The size of dump(s) on the tape. # Dtime: Dumper time. # Ttime: Taper Time. # Dspd: Dumper speed. # Tspd: Taper speed. # DLE: Disk list entry. # lv:Dump-level. # dpl: "Dumps per level". This is a list of dump levels (starting with #level 0). # error: An error message. # # The output is split into four sections. The first section (if present) # lists errors. In the example below we can see that there were two taper # errors and a warning that a DLE must be flushed to tape. # # The second section contains only one line with date, label and amount of # data on the tape. # # The third section lists all the tapes that are needed to recover all # DLEs. In the example below we can see that three level0, four level1 # and one level3 from tape VOL01 are needed to do a full restore of all # DLEs. In addition, three level0, one level1 and one level2 from VOL09 # are still needed. Everything else on VOL09 is obsoleted by the dumps # on VOL01. VOL08 contains two level0 and one leve1 that are not obsoleted # by newer tapes. Finally, VOL07 contain one level0 dump that is not # obsolted by newer tapes. # Tapes that contain only obsoleted data are not mentioned at all unless you # supply the -t command line option. # # The fourth section is the main section and is itself split into multiple # sections, one for each DLE. In this section we can see which tapes we # need to recover a specific DLE. For example, we can see that in order # to recover raven:/u4 we need file 1 from VOL07, file 6 from VOL08, # file 7 from VOL09 and file 1 from VOL01, in this order. # # Here is the example output: # # date Tsize
Re: Degraded mode with changer broken?
On Tue, Apr 13, 2004 at 03:10:17PM -0400, Jon LaBadie wrote: > > So what am I doing wrong? Is degraded mode in conjunction with a changer > > meant to work at all? > > I don't think degraded mode has anything to do with changer. > > Degraded mode is a much misused term. > > No tape DOES NOT mean degraded mode. > > You need the combination of no tape AND (note the important AND) > the available holding disk space is too small to accept the > level 0's the planner wants to do. Note in this case, "available" > includes the reduction specified by the "reserve" parameter. Ah, thanks for clarifying this. Seems like I had the "reserve" set too low. But I still don't understand why the logfiles got moved to the oldlog directory right after the amdump run. -- -- Josef Wolf -- [EMAIL PROTECTED] --
Degraded mode with changer broken?
Hello. I have configured a tape changer with a tapedev "file:/foo/bar" and use a script to flush the collected "slots" from disk to DVD. This worked fine for quite a while now. Now I decided to change my backup-strategy: I modified the changer to print "0 Drive was not loaded" and exit with status 1 when called from amdump. When called from amflush the changer behaves as usual. The idea here is to have full backups once in a week (I know I need _not_ to error out once in a week, but this is a different story) and let degraded mode collect incrementals. The problem now is that when the changer reports an error, planner seems to be confused: [ ... ] WARNING planner Last full dump of raven:/var on tape overwritten in 1 run. [ ... ] ERROR taper no-tape [label VOL06 or new tape not found in rack] [ ... ] Since planner assumes the last full dump is overwritten (on which tape?) it forces full dumps and fills up the holding disk. One more curiosity is that the log files of such amdump runs are moved into the oldlog directory. Citing from the manpage: At the end of each amdump run, log files for runs whose tapes have been reused are renamed into a subdirectory of the main log directory (see the logdir parameter below) named oldlog. This looks like amdump thinks it is about to reuse a tape with an empty name, but such a tape don't even exist. So what am I doing wrong? Is degraded mode in conjunction with a changer meant to work at all? -- -- Josef Wolf -- [EMAIL PROTECTED] --
Re: Printing tape labels
On Fri, Feb 13, 2004 at 01:22:16AM +0100, Josef Wolf wrote: > I have written a script to print amanda tape labels. I think this script > might be interesting to other amanda users, so I offer to include it in > the amanda distribution. OK, here it comes: -snip--- #! /usr/bin/perl -w # A utility to print amanda tape labels for DAT and CD. # # 2004-02-12 Josef Wolf ([EMAIL PROTECTED]) # # Portions of this program which I authored may be used for any purpose # so long as this notice is left intact. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # Comments, suggestions, bugfixes and improvements are welcome. # I wrote this program because I was dissatisfied with the original label # printing program that comes with the amanda distribution. I wanted to see # from one glance on the newest tape which tapes in which order I need to # recover a specific DLE. # # This program prints tapelabels for the amanda backup system. The output # can be in plain ASCII or in postscript. The postscript output is formatted # so that it can be folded to fit into a DAT case or into a CD jewel case. # # An example ASCII output (somewhat stripped to make it fit into 80 columns) # is attached below. Here is an explanation of the example output: # # The columns in the output have following meanings: # # date: This name seems to be intuitive, but unfortunately, it is somewhat #misleading. Actually, this is the name of the logfile that provided #the corresponding information. # label: The label of the tape. # fn:File number on the tape. # fm:Filemark # Osize: Original (that is, uncompressed) size of the dump(s). # Dsize: Size of the dump(s). This is usually pretty close to Tsize so it #is of very little interest. # Tsize: The size of dump(s) on the tape. # Dtime: Dumper time. # Ttime: Taper Time. # Dspd: Dumper speed. # Tspd: Taper speed. # DLE: Disk list entry. # lv:Dump-level. # dpl: "Dumps per level". This is a list of dump levels (starting with #level 0). # error: An error message. # # The output is split into four sections. The first section (if present) # lists errors. In the example below we can see that there were two taper # errors and a warning that a DLE must be flushed to tape. # # The second section contains only one line with date, label and amount of # data on the tape. # # The third section lists all the tapes that are needed to recover all # DLEs. In the example below we can see that three level0, four level1 # and one level3 from tape VOL01 are needed to do a full restore of all # DLEs. In addition, three level0, one level1 and one level2 from VOL09 # are still needed. Everything else on VOL09 is obsoleted by the dumps # on VOL01. VOL08 contains two level0 and one leve1 that are not obsoleted # by newer tapes. Finally, VOL07 contain one level0 dump that is not # obsolted by newer tapes. # Tapes that contain only obsoleted data are not mentioned at all unless you # supply the -t command line option. # # The fourth section is the main section and is itself split into multiple # sections, one for each DLE. In this section we can see which tapes we # need to recover a specific DLE. For example, we can see that in order # to recover raven:/u4 we need file 1 from VOL07, file 6 from VOL08, # file 7 from VOL09 and file 1 from VOL01, in this order. # # Here is the example output: # # date Tsize label fm lv error # 2004-02-20.0 1499M VOL06 8 ? writing file: No space left on device # 2004-02-23.0 1499M VOL01 9 ? writing file: No space left on device # 2004-02-23.0 670M??? ? 0 raven:/m/u1 not on tape yet # # 2004-02-23.0 VOL01 Total size: 834M # # date label Osize TsizeTtime dpl # 2004-02-23.0 VOL01 1616M 834M 0:01:06 3/4/0/1 # 2004-02-22.0 VOL09 2885M 1164M 0:01:12 3/1/1 # 2004-02-21.0 VOL08 3427M 1127M 0:01:12 2/1 # 2004-02-20.1 VOL07 2077M 1376M 0:01:24 1 # # Dspd Tspd Dtime Osize Tsize Ttime date label:fn lv DLE # 461k 105M 0:00:06 15M 2963k 0:00:00 2004-02-23.0 VOL01:04 1 raven:/ # 803k 11M 0:09:13 1163M 434M 0:00:37 2004-02-21.0 VOL08:08 0 raven:/ # #0k 1647k 0:00:00 10k1k 0:00:00 2004-02-23.0 VOL01:03 1 raven:/boot # 2515k 40M 0:00:01 4520k 3487k 0:00:00 2004-02-22.0 VOL09:03 0 raven:/boot # # 190k 25M 0:00:00 290k 22k 0:00:00 2004-02-22.0 VOL09:02 1 raven:/u1 # 971k 19M 0:11:46 1757M 670M 0:00:34 2004-02-21.0 VOL08:09 0 raven:/u1 # #0k 1237k 0:00:00
Printing tape labels
Hello folks. I have written a script to print amanda tape labels. I think this script might be interesting to other amanda users, so I offer to include it in the amanda distribution. I wrote this program because I was dissatisfied with the original label printing program that comes with the amanda distribution. I wanted to see from one glance on the newest tape which tapes in which order I need to recover a specific DLE. This program prints tapelabels for the amanda backup system. The output can be in plain ASCII or in postscript. The postscript output is formatted so that it can be folded to fit into a DAT case or into a CD jewel case. An example ASCII output (somewhat stripped to make it fit into 75 columns) is attached below. Here is an explanation of the example output: The output is split into four sections. The first section contains only one line with date, label and amount of data on the tape. The second section lists all the tapes that are needed to recover all DLEs. In the example below we can see that four level0, three level1 and one level2 from tape VOL09 are needed to do a full restore of all DLEs. In addition, one level0 and two level1 from VOL08 are still needed. Everything else on VOL08 is obsoleted by the dumps on VOL09. VOL07 and VOL06 each contain two level0 dumps that are not obsolted by newer tapes. Tapes that contain only obsoleted data are not mentioned at all. The third section is the main section and is itself split into multiple sections, one for each DLE. In this section we can see which tapes we need to recover a specific DLE. For example, we can see that in order to recover raven:/usr we need file 9 from VOL07, file 5 from VOL08 and file 4 from VOL09. Finally, the fourth section lists errors. In this case level1 from raven:/u3 was not written to tape, requiring an amflush run. Here is the example output: 2004-02-05 VOL09 Total size: 858M 2004-02-05.0 VOL09 2054M 858M 0:01:14 4/3/1 2004-02-04.0 VOL08 2369M 1592M 0:01:49 1/2 2004-02-03.0 VOL07 2629M 878M 0:00:47 2 2004-02-02.0 VOL06 2545M 1283M 0:02:24 2 Dspd Tspd Dtime Tsize Osize Ttime date label:fn L filesystem 301k 109M 00:09 2882k 14M 00:00 2004-02-05.0 VOL09:03 1 raven:/ 786k 14M 09:00 415M 1122M 00:27 2004-02-03.0 VOL07:08 0 raven:/ 2022k 109M 00:01 3485k 4510k 00:00 2004-02-05.0 VOL09:02 0 raven:/boot 135k 1346k 00:00 22k 290k 00:00 2004-02-05.0 VOL09:01 1 raven:/u1 899k 10M 12:27 656M 1731M 01:02 2004-02-02.0 VOL06:09 0 raven:/u1 840k 45M 02:10 107M 113M 00:02 2004-02-05.0 VOL09:07 0 raven:/u2 2776k 14M 01:17 209M 266M 00:14 2004-02-04.0 VOL08:06 1 raven:/u3 2569k 7790k 04:09 626M 813M 01:22 2004-02-02.0 VOL06:08 0 raven:/u3 369k 46M 01:10 25M 463M 00:00 2004-02-05.0 VOL09:05 1 raven:/u4 1490k 12M 15:39 1367M 2023M 01:48 2004-02-04.0 VOL08:07 0 raven:/u4 239k 115M 00:10 2601k 31M 00:00 2004-02-05.0 VOL09:04 2 raven:/usr 146k 87M 01:45 15M 80M 00:00 2004-02-04.0 VOL08:05 1 raven:/usr 553k 23M 14:16 462M 1507M 00:19 2004-02-03.0 VOL07:09 0 raven:/usr 975k 16M 06:13 355M 858M 00:22 2004-02-05.0 VOL09:09 0 raven:/opt 1877k 10M 03:18 363M 571M 00:34 2004-02-05.0 VOL09:08 0 raven:/var Dumps that are not taped yet: 1 raven:/u3 226M 2004-02-05.0 bugs: - Parses amanda's log files instead of reading its database. - Depends on assumption that only one amdump per day is run. - Nuber of output lines limited to paper size. - Output formats hardcoded. - Postscript output is not split onto multiple sheets. - Error messages should get a better place in CD/DVD postscript output. todo: - ignore obsolete log files - remove some kludges in the code. Any interest? -- -- Josef Wolf -- [EMAIL PROTECTED] --
Re: How to omit holding disk? -- SOLVED
On Wed, Feb 04, 2004 at 01:14:12AM -0500, Jon LaBadie wrote: > On Wed, Feb 04, 2004 at 12:10:17AM +0100, Josef Wolf wrote: > > top - 23:19:51 up 5 days, 12:08, 23 users, load average: 0.00, 0.08, 0.17 > > Tasks: 141 total, 1 running, 137 sleeping, 0 stopped, 3 zombie > > Cpu(s): 1.9% user, 2.2% system, 0.0% nice, 95.9% idle > > Mem:514224k total, 475768k used,38456k free,50224k buffers > > Swap:0k total,0k used,0k free, 182920k cached ^^ Argh! Me little idiot forgot to mkswap on the swap partition the last time I changed my disk configuration. Therefore the swapon command failed to add swap space to the system (which I clearly should have noticed earlier). As soon as I added swap space, the system is pretty responsive again and even throughput for the copy operation went up to 19MB/s. > My eye focused on something else. On an "idle" system you had 38MB free > of 512MB total. During copying that dropped to just 4MB free. My Solaris > system also has 512MB total, but 120MB free when "idle". Doing a copy > such as you report only dropped it by about 25MB. I saw little degradation > in interactive performance. This may be the case where the old saw "add > more memory" may be a valid recommendation. Sometimes even adding virtual memory helps :) Thanks a lot and sorry for the confusion! -- -- Josef Wolf -- [EMAIL PROTECTED] --
Re: How to omit holding disk?
On Mon, Feb 02, 2004 at 05:57:16PM -0800, Jay Lessert wrote: > On Mon, Feb 02, 2004 at 11:39:24PM +0100, Josef Wolf wrote: > Just so you are aware, this is definitely not normal behavior (extreme > system sluggishness during a single large file copy/move)! Hmm.. > I assume you're running some flavor of linux. The usual suspect > in that case is DMA not enabled on your IDE disk drive(s). Use > hdparm to check. Don't let the man page scare you (too much): > > % sudo hdparm /dev/hda /dev/hdb > > ...is a safe thing to do. :-) raven:~ # hdparm /dev/hda (this is the holding disk) /dev/hda: multcount= 16 (on) IO_support = 1 (32-bit) unmaskirq= 1 (on) using_dma= 1 (on) keepsettings = 0 (off) readonly = 0 (off) readahead= 8 (on) geometry = 7297/255/63, sectors = 117231408, start = 0 raven:~ # hdparm /dev/hdd (this is the disk for the virtual tapes) /dev/hdd: multcount= 16 (on) IO_support = 1 (32-bit) unmaskirq= 1 (on) using_dma= 1 (on) keepsettings = 0 (off) readonly = 0 (off) readahead= 8 (on) geometry = 2491/255/63, sectors = 40020624, start = 0 raven:~ # uname -a Linux raven 2.4.21-166-athlon #1 Thu Dec 18 18:24:05 UTC 2003 i686 athlon i386 GNU/Linux raven:~ # head /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 6 model name : AMD Athlon(tm) 4 Processor stepping: 2 cpu MHz : 1394.065 cache size : 256 KB fdiv_bug: no hlt_bug : no > Just a thought: one of the disks involved isn't an NFS mount, is it? I don't use NFS. > I made myself a 20GB file, and started a copy between two IDE > disks on a relatively modest box: > [ ... ] > So that's about 26MB/s, not bad. I tried with 4GB since I don't have that much free disk: raven:/m/amchanger # time dd if=/dev/zero of=t bs=1024k count=4096 4096+0 records in 4096+0 records out real3m25.584s user0m0.031s sys 0m15.144s That's only 19MB/s. System slown a little bit down, but is still good usable for interactive work. raven:/m/amchanger # time cp t /m/spool/tmp/t real4m39.285s user0m0.398s sys 0m25.867s That's 14MB/s. Changing virtual desktops in kde lasts 20..30 seconds. Pressing "UP" to access bash's history in kde's konsole lasts about 5..10 seconds. This top output is on idle system: top - 23:19:51 up 5 days, 12:08, 23 users, load average: 0.00, 0.08, 0.17 Tasks: 141 total, 1 running, 137 sleeping, 0 stopped, 3 zombie Cpu(s): 1.9% user, 2.2% system, 0.0% nice, 95.9% idle Mem:514224k total, 475768k used,38456k free,50224k buffers Swap:0k total,0k used,0k free, 182920k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 4676 root 15 0 61376 19m 712 S 1.3 4.0 0:59.98 X This one is when copying: top - 23:22:41 up 5 days, 12:11, 24 users, load average: 2.86, 1.13, 0.54 Tasks: 143 total, 3 running, 137 sleeping, 0 stopped, 3 zombie Cpu(s): 1.8% user, 22.5% system, 0.0% nice, 75.6% idle Mem:514224k total, 509680k used, 4544k free, 8456k buffers Swap:0k total,0k used,0k free, 275012k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 27744 root 16 088 88 24 R 12.5 0.0 0:14.56 cp As you can see, "cached" goes up (the contents of the huge file get cached) while "free" and "buffers" go down. Everything from "buffers" gets flushed. The 182MB from "cached" get flushed and replaced with the contents from the copied file as well. I can't see anything interesting in the vmstat/iostat outputs, but I include it here to be complete: [EMAIL PROTECTED]:~> vmstat 5 procs ---memory-- ---swap-- -io --system-- cpu r b swpd free buff cache si sobibo incs us sy id wa 2 2 0 4364 8312 27724000 2 95 8 13 3 84 0 2 0 0 5116 7844 27696400 19914 10231 1535 848 1 21 78 0 1 1 0 4708 8044 27717200 18770 18381 1618 823 1 23 75 0 1 1 0 5340 8236 27634800 19107 26817 1765 849 1 23 76 0 1 0 0 4380 8412 27713200 19479 12169 1551 827 1 21 78 0 [EMAIL PROTECTED]:~> iostat 5 Linux 2.4.21-166-athlon (raven) 03.02.2004 avg-cpu: %user %nice%sys %idle 12,080,483,29 84,15 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-01,4117,2528,40 82084494 135164232 dev8-00,00 0,00 0,00 1530 0 dev8-1
Re: How to omit holding disk?
On Mon, Feb 02, 2004 at 01:07:35AM +0100, Paul Bijnens wrote: > >Yes, I know. This is the reason that I asked whether it would be possible > >to change the copy operation into a rename operation for the file: output > >driver. > It's not as easy as that: the header on a holdingdisk file is different > from that on a tape file (even a tape file on disk). > Moreover, the holdingdisk backup can be split into chunks, while the > tape file (currently) cannot. Simply moving the file would not be enough. Ah, I see. Thanks for the clarification. > But, if you say that the holdingdisk is a different disk than your > virtual tapedisk, then you can't move without copying the data too :-) Oh, I would move holding disk to the vdisk drive if it would be possible. > PS. personnally I've never seen the behaviour you described. My > backup server has a loadavg of 10 to 15 when the backups run, but > the system still keeps very responsive for interactive commands, > e.g. doing an amstatus etc. I do not run X on it (it does not even > have a screen attached permanently). > It is a 300 MHz pentium with 128 Mbyte RAM with Linux 2.4.20, > with a 80 Gbyte IDE holding disk an 2 AIT-1 tape drives: not the > fastest machine in the world, but still capable of backing up > about 220 Gbyte in less than 8 hours in my archive run in the weekends. You don't use virtual tapedisk, so it's no wonder that you don't see this effect. Please try to copy really huge files from one disk drive to an other. -- -- Josef Wolf -- [EMAIL PROTECTED] --
Re: How to omit holding disk?
On Fri, Jan 30, 2004 at 09:05:29PM -0500, Jon LaBadie wrote: > > I set up amanda with chg-disk. By design, dumps are done to holding disk and > > copied to the disk afterwards. This copy operation makes the machine pretty > > much unusable for interactive usage (puts almost everything to swap). > > > > Now I wonder whether there is a way to omit this copy operation. First > > thought was to disable holding disk (or setting it to 0). But (AFAIK) this > > would disable amanda's ability to dump several DLEs in parallel. > > If you can design a way to simultaneously write several dump streams > to a tape (remember you are emulating a tape) but have each of them > appear on the tape as separate, contiguous files, perhaps some one > could work on coding it. But I doubt the design is possible. :( Yes, I know. This is the reason that I asked whether it would be possible to change the copy operation into a rename operation for the file: output driver. > Are you sure it is the copy operation that makes your system unusable? > I would expect disk to disk copy to be a fairly short part of the > total dump time and effort. It is short compared to the total dump time. But it still takes several minutes per DLE. This copy operation flushes all the buffer caches and makes the system almost unusable for interactive operations. > Is the holding disk the same physical drive as the virtual tapes? > That would be a poor choice. No. I have a separate disk for the virtual tapes. > Are either/both the holding disk and the virtual tape disk IDE type > drives rather than SCSI. IDE drives do take substantial cpu cycles > compared to SCSI drives. The disks are IDE. But the problem is _not_ the cpu cycles. The problem is that the copy operation fills all the buffer cache with the copied data. This leads to lots of page faults making the system almost unusable. > If you haven't tested the backup without a holding disk, perhaps you > should. It may surprise you with a sufficiently quick total dump > time despite the lack of a holding disk. This would loose the capability to dump several hosts in parallel. > Are you doing server side compression rather than client side? Nope. > Compression processes take lots of cpu cycles from interactive sessions. The problem is _not_ cpu cycles. The problem is the (unnecessary) copy operation. > You might consider setting a "nice" value on your backups. amdump _is_ running with a high nice value. But this don't really help. Again, the problem is that the copy operation fills all the buffer cache and forces all the other processes into page faults. A high nice value _can't_ help in such a situation. -- -- Josef Wolf -- [EMAIL PROTECTED] --
How to omit holding disk?
Hello! I set up amanda with chg-disk. By design, dumps are done to holding disk and copied to the disk afterwards. This copy operation makes the machine pretty much unusable for interactive usage (puts almost everything to swap). Now I wonder whether there is a way to omit this copy operation. First thought was to disable holding disk (or setting it to 0). But (AFAIK) this would disable amanda's ability to dump several DLEs in parallel. An other possibility would be to change the copy operation into a move operation. This seems not to be a very good idea because of its special-case character. Any ideas whether there are other possibilities to omit this copy operation? -- -- Josef Wolf -- [EMAIL PROTECTED] --
exclude list optional broken?
Hallo!
The amanda man page states that the exclude list can be optional. So I defined
following dumtype:
define dumptype my-global {
comp-user-tar
exclude list optional ".amanda.exclude.gtar"
include list optional ".amanda.include.gtar"
}
But unfortunately, with this definition all the backups just fail on _all_
filesystems. Even those which actually contain such an exclude file failed.
Is this a known bug or am I just doing something very stupid? Any hints?
BTW: amanda-2.4.4, gnutar-1.13.25
--
-- Josef Wolf -- [EMAIL PROTECTED] --
