Re: Can I use only one tape for backing up all the week?
Simon Mayr wrote: And hopefully the holding disk is not crashing at the same time as the backup client crashes. Maybe this is not exactly what you meant, but I will report it anyway, just to make clear what can happen in this world... I *never* put a tape (or the right tape for that matter ;-)) in the drive when AMANDA starts. This way I force all images to be copied to the holding disk and remain there till next day. After AMANDA finishes, a trivial script copies the files from the holding disk to an MO disk. I use as many MO disks as tapes for this purpose. They have roughly the same capacity too, which comes handy. When I run amflush next day, I will have the images on both the tape and the MO disk :-) Some days ago I started the usual daily amflush operation. In the middle of this, a power outage occured :-(((. What happens in this case is that some of the files in the holding disk have already been transfered to tape, so they are not on the holding disk any more. The tape, on the other side, did not have its headers updated (not the AMANDA headers, but its low level headers - on my system this is done at the end of the flushing, after the rewind command is issued), so the drive will not find the transfered files on it. The result was that all the transfered images were lost :-( My double net strategy saved me: I copied the images from the MO disk back to the holding disk and started amflush again :-) -- Regards Chris Karakas Dont waste your cpu time - crack rc5: http://www.distributed.net
Re: Can I use only one tape for backing up all the week?
On Mon, 12 Feb 2001, Chris Karakas wrote: Some days ago I started the usual daily amflush operation. In the middle of this, a power outage occured :-(((. What happens in this case is that some of the files in the holding disk have already been transfered to tape, so they are not on the holding disk any more. The tape, on the other side, did not have its headers updated (not the AMANDA headers, but its low level headers - on my system this is done at the end of the flushing, after the rewind command is issued), so the drive will not find the transfered files on it. The result was that all the transfered images were lost :-( Huh?! Upon reading this I had two distinct reactions: 1) Whatever this device is, it sounds like it no longer conforms to the commonly accepted definition of a tape drive. Please tell us what it is so we can avoid buying one. I hope the manufacturer is honest enough to market it with a large warning label: "WARNING: This is not your grandfather's tape drive. Data loss may occur! Buyer beware, etc, etc" and 2) Are you sure about this? I.e. have you tried a dd against such a tape and really not been able to pull anything off it? Also, how exactly does your system update these low level headers? My double net strategy saved me: I copied the images from the MO disk back to the holding disk and started amflush again :-) Given you description of this "tape drive"'s operation I would say you are completely justified in not trusting it. YIKES!! -Mitch
Re: Can I use only one tape for backing up all the week?
* John R. Jackson [EMAIL PROTECTED] (Wed, Feb 07, 2001 at 09:14:26PM -0500) Now, am not 100% certain on how taper does write the files to tape ... Very quick review: Rewind Read the label and verify Rewind Write a new label and tapemark Write a header, image and tapemark Write a header, image and tapemark ... Write the trailing label, tapemark and quit (leaving the tape where it is) ... Why not [save] the last one in tapelist, and/or amount of tape used? That's the plan. Why not go one step further, and write this information in the tapelabel. Of course, this breaks if fsf is broken ... so, now you are effectively doing an 'mt fsf last field' to get to end of tape ... True. The problem is that the drive may screw up. You may tell it to skip 37 files and it skips 36 (or 38, or 10, or 100 ...). So you have to Is this still the case with newer tape drives ? I cannot remeber ever having this probelm with exabyte , mammoth or LTO tapes. Come to think of it, I cannot even remeber it ahppening in the huge reel-to-reel tapedrive we used way back when, and that was a pretty bad piece of work ;) Gerhard, (@jasongeo.com) == The Acoustic Motorbiker == -- __0 Oh my God, the bomb has just dropped =`\, And everybody climbed right on top (=)/(=) Singing,"What a beautifull country
Re: Can I use only one tape for backing up all the week?
* Simon Mayr [EMAIL PROTECTED] (Thu, Feb 08, 2001 at 01:20:52PM +0100)
What would it take to have some sort of config option that did exactly
that? ...
Nothing. It's already there. Just leave the tape out of the drive and
make as many runs as you want into the holding disk.
eg
crontab:
15 0 * * * /path/to/smartscript
smartscript:
if holdingdisk space is to small
/path/to/amdump to_tape_config \
mt offline ( send extra big mail to the guy who changes the tape:)
else
/path/to/amdump to_holding_config
How about
As abovem 2 configs, each identical exept for allowed tapestring;
if(holdingdisk space is full){
amflush totape mt rewoffl alert-person-to-change-tape
}
amdump todiskconfig
also, finetune this so that if a holdingdisk is more full than fits a
single tape, you amflush oldest partitions first until tapefull
(e.g. by moving the latest dumps to a different dir, amflush nd move back).
Again:
See the note above and have in mind that this can only be usefull for
poeple whose backup device capacity is much more higher than the backup
load from a couple of days.
And for people who can afford to loose a couple of days worth of backups if
the holdingdisk crashes.
Q: what to do if "all" is to big, you have more than 26 areas and the
one you really want to flush is to new to be listed?
Whats the deal with the 26 areas ?
Cannot amflush flush more than 26 areas to tape ?
What do you mean with area ?
Gerhard, [@jasongeo.com] == The Acoustic Motorbiker ==
--
__O I spoke about wings ... You just flew
=`\, I wondered, I guessed and I tried ... You just knew
(=)/(=) I sighed ... And you swooned
I saw the crescent ... You saw the whole of the moon
Re: Can I use only one tape for backing up all the week?
Le 07/02/2001 a 15:22 -0500 , John R. Jackson ecrivait : Second is that positioning sounds nice and logical to a programmer or someone only used to working with disk, but when you throw in the reality of a physical tape device, bad stuff starts to happen. On WinNt an Macs, that's the way Dantz's Retrospect works for years. I never, never, lost a tape. Otoh, mt(1) blows me a tape so badly that I had to physically open the DAT drive to eject it :-( -- Xavier HUMBERT - Systemes et Reseaux | [EMAIL PROTECTED] INJEP | [EMAIL PROTECTED]
Re: Can I use only one tape for backing up all the week?
Gerhard den Hollander wrote:
How about
As abovem 2 configs, each identical exept for allowed tapestring;
if(holdingdisk space is full){
amflush totape mt rewoffl alert-person-to-change-tape
}
amdump todiskconfig
i think this should work.
btw:
how do you non-interactivly amflush ?
at my site, we do something like that, but instead of flushing (as it
should be) we leave the incrementals on disk and delete them together
with their index files, if the holding disk space is running low
also, finetune this so that if a holdingdisk is more full than fits a
single tape, you amflush oldest partitions first until tapefull
(e.g. by moving the latest dumps to a different dir, amflush nd move back).
to call it by name:
include some errorchecking and foolproofness
Again:
See the note above and have in mind that this can only be usefull for
poeple whose backup device capacity is much more higher than the backup
load from a couple of days.
And for people who can afford to loose a couple of days worth of backups if
the holdingdisk crashes.
That's true (!!). But at least the data is on two different machines.
And hopefully the holding disk is not crashing at the same time as the
backup client crashes.
Q: what to do if "all" is to big, you have more than 26 areas and the
one you really want to flush is to new to be listed?
What do you mean with area ?
John R. Jackson said:
At your convenience, run "amflush" and select "all" instead of a specific holding area.
and I picked up the word area :-)
it means the place where all the backup images of a run of amanda
reside. namely the directory in the holding disk named after the
corresponding date.
Cannot amflush flush more than 26 areas to tape ?
As amflush presents you a list with the directorys in the holding disk
all with a indicating letter (A-Z) there is a maximum of 26 visible
holding "areas". they are sorted by age (or alphabetically whats the
same in this case). So cou can't *see* the 27th area and therefore can
not flush it. (I DONT know if it WOULD be flushed when you select all)
Whats the deal with the 26 areas ?
Well, ... uh ... more than 26 unflushed backups ... ?!
As said above, we keep as many incrementals as possible in the holding
disk for no specific reason other than lazyness. As Incrementals are not
big, a large holding disk can hold a lot of them.
Gerhard, [@jasongeo.com] == The Acoustic Motorbiker ==
Simon
Re: Can I use only one tape for backing up all the week?
* Simon Mayr [EMAIL PROTECTED] (Thu, Feb 08, 2001 at 03:28:07PM +0100) how do you non-interactivly amflush ? echo "all" | amflush -f whatever-your-config-is (I think , I haven't tried it ) at my site, we do something like that, but instead of flushing (as it should be) we leave the incrementals on disk and delete them together with their index files, if the holding disk space is running low B ;) Cannot amflush flush more than 26 areas to tape ? As amflush presents you a list with the directorys in the holding disk all with a indicating letter (A-Z) there is a maximum of 26 visible holding "areas". they are sorted by age (or alphabetically whats the same in this case). So cou can't *see* the 27th area and therefore can not flush it. (I DONT know if it WOULD be flushed when you select all) OK, I never realized that . BTW I checked the source, and it looks like ALL will indeed dump all, so something like echo "all" | amflush -f whatever-your-config-is should do the trick (well actually, you have to type Y as well I think., Maybe you need to do something with expect ;) Gerhard, @jasongeo.com == The Acoustic Motorbiker == -- __O Some say the end is near. =`\, Some say we'll see armageddon soon (=)/(=) I certainly hope we will I could use a vacation
Re: Can I use only one tape for backing up all the week?
* Simon Mayr [EMAIL PROTECTED] (Thu, Feb 08, 2001 at 04:14:39PM +0100) something like echo -en "A\nY\n" | /path/to/amflush -f config is working. ( Arrrgh again! ;) Allright ... What about the amanda database if i set up a "flush to nirvana" config with /dev/null. Would it do any better or other than erase the images, the index files and the log files. dunno , according to the manpages it will ``update the databases, and send email similar to amreport'' It also makes a ncie printout if you have a label specified (though it's rather silly to stick a printout label to /dev/null ) I think I read somewhere that the behavior of amanda has changed about the meaning of /dev/null. Amanda didn't let me use the /dev/null device, ioctl error or something, cannot rewind, wrong tape etc., so I came up with the simply-delete-the-stuff solution ... Dunno, Solaris gives no tape loaded errors if you try to do mt commands on /dev/null , which is reasonable ;) (well actually, you have to type Y as well I think., Maybe you need to do something with expect ;) what's "expect" ? expect is a ltittle tool that lets you automate interactive programs expects sits there, reads the utput from the program, and feeds it the required input. e.g. UUCP uses it to talk to the dialout device Gerhard, @jasongeo.com == The Acoustic Motorbiker == -- __O If your watch is wound, wound to run, it will =`\, If your time is due, due to come, it will (=)/(=) Living this life, is like trying to learn latin in a chines firedrill
Re: Can I use only one tape for backing up all the week?
Christoph Scheeder has told me that I can't use only one tape for backing up all the days of the week. Can anybody suggest me a solution? By the moment, I cant pay more tapes. Thanks You should really (!) buy at least a second one. As stated in my copy of the FAQ: Q: What if my tape unit uses expensive tapes, and I don't want to use one tape per day? Can't Amanda append to tapes? A: It can't, and this is good. Tape drives and OS drivers are (in)famous for rewinding tapes at unexpected times, without telling the program that's writing to them. If you have a month's worth of backups in that tape, you really don't want them to be overwritten, so Amanda has taken the safe approach of requiring tapes to be written from the beginning on every run. This can be wasteful, specially if you have a small amount of data to back up, but expensive large-capacity tapes. One possible approach is to run amdump with tapes only, say once a week, to perform full backups, and run it without tape on the other days, so that it performs incremental backups and stores them in the holding disk. Once or twice a week, you flush all backups in the holding disk to a single tape. And you could even run the level 0 without tape and always flushing your backups, assuming you have enough space for a full week of backup. Try this ( or similar) dumpcycle 7 runspercycle 5 tapecycle what-you-have # with reserve 100 she will always degrade the level 0's reserve 30 Simon
Re: Can I use only one tape for backing up all the week?
What would it take to have some sort of config option that did exactly
that? I have a 15/30DLT ... I have a 15gig holding disk ... when holding
disk is full, dump it all to tape and then backup to the holding disk
again ... then, your tapes would always be full and you don't have to
worry about whether you have enough holding disk for a weeks worth of
data, or what not ...
You also then have the benefit of getting the tape "up to speed", dumping
15gig (or whatever) to her and getting max use of the drive ...
On Wed, 7 Feb 2001, Simon Mayr wrote:
Christoph Scheeder has told me that I can't use only one tape for backing up
all the days of the week. Can anybody suggest me a solution? By the moment,
I can´t pay more tapes.
Thanks
You should really (!) buy at least a second one.
As stated in my copy of the FAQ:
Q: What if my tape unit uses expensive tapes, and I don't want to use
one tape per day? Can't Amanda append to tapes?
A: It can't, and this is good. Tape drives and OS drivers are
(in)famous for rewinding tapes at unexpected times, without telling
the program that's writing to them. If you have a month's worth of
backups in that tape, you really don't want them to be overwritten, so
Amanda has taken the safe approach of requiring tapes to be written
from the beginning on every run.
This can be wasteful, specially if you have a small amount of data
to back up, but expensive large-capacity tapes. One possible approach
is to run amdump with tapes only, say once a week, to perform full
backups, and run it without tape on the other days, so that it
performs incremental backups and stores them in the holding disk.
Once or twice a week, you flush all backups in the holding disk to a
single tape.
And you could even run the level 0 without tape and always flushing your
backups, assuming you have enough space for a full week of backup.
Try this ( or similar)
dumpcycle 7
runspercycle 5
tapecycle what-you-have
# with reserve 100 she will always degrade the level 0's
reserve 30
Simon
Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: [EMAIL PROTECTED] secondary: scrappy@{freebsd|postgresql}.org
Re: Can I use only one tape for backing up all the week?
What would it take to have some sort of config option that did exactly that? ... Nothing. It's already there. Just leave the tape out of the drive and make as many runs as you want into the holding disk. At your convenience, run "amflush" and select "all" instead of a specific holding area to flush. Marc G. Fournier John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: Can I use only one tape for backing up all the week?
Okay, but now we're back to requiring manual intervention here to load that tape and do the amflush ... what about an automatic feature for this? ... don't write to tape until we can fill the tape. ... I proposed something called "autoflush" a few weeks ago. Maybe this can be folded in. I'll add it to the notes. I suspect the tape I/O would be triggered by either the holding disk or tape length being met so it would work either way (holding disk bigger than tape or vice versa). What is the difference between what happens now (you write a file system to tape, when the next one is ready, you spin up the tape and write the next one to tape) and appending to tape? ... The difference is that taper has the tape device open the whole time it is running. If a SCSI reset happens, taper will get an I/O error. What people usually mean by "append to tape" is that the next run starts up where the last left off. But during that time, **anything** could have happened to the tape or drive without any part of Amanda knowing about it, and that's harder to deal with when the next run starts. John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: Can I use only one tape for backing up all the week?
This part always got me ... the first thing anyone is supposed to do is do an 'amcheck' to check for the existiance of a tape, right? ... Taper basically does the same thing w.r.t. the "right" tape, so amcheck is not really an issue (in fact it's a problem because Amanda looses control again between when it runs and when taper starts). If current tape is a known label *and* is active (not full), why not just issue a 'rew' on the device and the fast-forward to the "end" and continue from there? ... Three reasons. First is that this is not what some people think of as "appending'. The fsf can be an expensive (time-wise) operation and they literally want to trust the drive to still be positioned at the right place and just start writing. Note that you cannot even check the label in this case to be sure it's the right tape. I don't think this will ever fly. Second is that positioning sounds nice and logical to a programmer or someone only used to working with disk, but when you throw in the reality of a physical tape device, bad stuff starts to happen. I've seen drives skip tapemarks, think they saw extra ones, write a few in the middle for no good reason and just about anything else you can imagine. So the "fsf to the end" operation has to be very careful that it ended up where it wants to be (and don't talk to me about "eom" -- that's nothing more than an "infinite" fsf and ignore the error when you bang into EOT). On the other hand, it wants to be as efficient as possible on devices that do fast forward spacing, in other words, it would be a bad idea to read a label, skip a file, read a label, skip a file, and so on. Third, you never, **ever** want to change directions on a tape, except to rewind clear back to the beginning. So when we find the Amanda end marker (yes, there is one), we do **not** want to back up over it and start writing there. Instead, we'll want to write the next image after that. That, in turn, will affect amrestore, etc. John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: Can I use only one tape for backing up all the week?
On Wed, 7 Feb 2001, John R. Jackson wrote:
This part always got me ... the first thing anyone is supposed to do is do
an 'amcheck' to check for the existiance of a tape, right? ...
Taper basically does the same thing w.r.t. the "right" tape, so amcheck
is not really an issue (in fact it's a problem because Amanda looses
control again between when it runs and when taper starts).
If current
tape is a known label *and* is active (not full), why not just issue a
'rew' on the device and the fast-forward to the "end" and continue from
there? ...
Three reasons. First is that this is not what some people think of
as "appending'. The fsf can be an expensive (time-wise) operation and
they literally want to trust the drive to still be positioned at the
right place and just start writing. Note that you cannot even check
the label in this case to be sure it's the right tape. I don't think
this will ever fly.
That depends on how "smart" you want to make it ... why can't tape
operations happen while the dump to the holding disk is happening? We are
all working with multi-process systems ... if the code went the route that
the holding disk, or tape, has to be full before dumping to tape, then
there should be alot of time in there for taper to find the first tape it
can use/append to ...
Now, am not 100% certain on how taper does write the files to tape, but,
if I'm thinking right, each file system/dump creates a new 'fsf tag',
right? Why not sure the last one in tapelist, and/or amount of tape used?
so, now you are effectively doing an 'mt fsf last field' to get to end
of tape ... if you store the amount of tape saved to that tape, you would
be able to pre-determine whether or not there is even any room left on
that tape to be worth the effort (ie. say if estimated tape left is 10%,
just start a new tape) ...
Second is that positioning sounds nice and logical to a programmer or
someone only used to working with disk, but when you throw in the reality
of a physical tape device, bad stuff starts to happen. I've seen drives
skip tapemarks, think they saw extra ones, write a few in the middle for
no good reason and just about anything else you can imagine. So the "fsf
to the end" operation has to be very careful that it ended up where it
wants to be (and don't talk to me about "eom" -- that's nothing more
than an "infinite" fsf and ignore the error when you bang into EOT).
On the other hand, it wants to be as efficient as possible on devices
that do fast forward spacing, in other words, it would be a bad idea to
read a label, skip a file, read a label, skip a file, and so on.
Third, you never, **ever** want to change directions on a tape, except to
rewind clear back to the beginning. So when we find the Amanda end marker
(yes, there is one), we do **not** want to back up over it and start
writing there. Instead, we'll want to write the next image after that.
That, in turn, will affect amrestore, etc.
Okay, not quite sure what the 'end marker' would be used for ... if I do a
dump to tape, using dump, onto the no-rewind device, and put n file
systems onto that tape, there is no 'end marker' put onto the tape, is
there? so, instead of an 'end marker', why not create a 'dummy dump' at
the end of a tape ... so that the last file system dump'd is "fsf n", the
dummy dump is at "fsf n+1", and you can do a 'rew;fsf n+1' to start the
next dump?
Of course, all of this is moot if the concept of an autoflush on full
holding disk/full tape is possible, since it eliminates the requirement to
append :)
Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: [EMAIL PROTECTED] secondary: scrappy@{freebsd|postgresql}.org
Re: Can I use only one tape for backing up all the week?
... why can't tape operations happen while the dump to the holding disk is happening? ... They already do. What does that have to do with not repositioning the tape at all? Now, am not 100% certain on how taper does write the files to tape ... Very quick review: Rewind Read the label and verify Rewind Write a new label and tapemark Write a header, image and tapemark Write a header, image and tapemark ... Write the trailing label, tapemark and quit (leaving the tape where it is) (for the purists among you this is not strictly correct as the tapemark is actually deferred until the start of the next writing, but this is close enough for the purposes of this discussion). ... Why not [save] the last one in tapelist, and/or amount of tape used? That's the plan. so, now you are effectively doing an 'mt fsf last field' to get to end of tape ... True. The problem is that the drive may screw up. You may tell it to skip 37 files and it skips 36 (or 38, or 10, or 100 ...). So you have to go somewhat short on the skip count, read the label to find out where you really are, then (possibly) skip some more, or even rewind and start over. if you store the amount of tape saved to that tape, you would be able to pre-determine whether or not there is even any room left on that tape to be worth the effort ... Agreed. Okay, not quite sure what the 'end marker' would be used for ... If you're trying to skip to the end of a tape, you need to read something good after that last fsf plea to the tape gods. If the device uses double tape marks as EOM and you go too far, an evil reverse motion has to happen. Or you might get an I/O error and not know whether it's a real problem or just because you ended up where you wanted. if I do a dump to tape, using dump, onto the no-rewind device, and put n file systems onto that tape, there is no 'end marker' put onto the tape, is there? ... No, but I don't understand the point. so, instead of an 'end marker', why not create a 'dummy dump' at the end of a tape ... That's effectively what we have. After the last image is a tapemark and then a single 32 KByte Amanda label. It's that label (end marker) we will be hunting for to then start writing after. Marc G. Fournier John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
Re: Can I use only one tape for backing up all the week?
* Adolfo Pachn [EMAIL PROTECTED] (Tue, Feb 06, 2001 at 03:07:35PM +0100) Christoph Scheeder has told me that I can't use only one tape for backing up all the days of the week. Can anybody suggest me a solution? By the moment, I cant pay more tapes. if you're data is important enough to back it up, it's also important enough to buy a few more tapes. With 1 single tape, you have a single point of failure (that tape). Kind regards, -- Gerhard den Hollander Phone +31-10.280.1515 Technical Support Jason Geosystems BV Fax +31-10.280.1511 (When calling please note: we are in GMT+1) [EMAIL PROTECTED] POBox 1573 visit us at http://www.jasongeo.com 3000 BN Rotterdam JASON...#1 in Reservoir CharacterizationThe Netherlands This e-mail and any attachment is/are intended solely for the named addressee(s) and may contain information that is confidential and privileged. If you are not the intended recipient, we request that you do not disseminate, forward, distribute or copy this e-mail message. If you have received this e-mail message in error, please notify us immediately by telephone and destroy the original message.
Re: Can I use only one tape for backing up all the week?
On Tue, 6 Feb 2001, Gerhard den Hollander wrote: if you're data is important enough to back it up, it's also important enough to buy a few more tapes. agreed, but... With 1 single tape, you have a single point of failure (that tape). No, not really. He'd have to lose both the disk and the tape in order to lose the data. Not that it can't happen, but it's not a single point of failure. -Mitch
Re: Can I use only one tape for backing up all the week?
With 1 single tape, you have a single point of failure (that tape). No, not really. He'd have to lose both the disk and the tape in order to lose the data. Not that it can't happen, but it's not a single point of failure. Actually, all you'd have to lose is the one file you really, really need, which is much more likely than the whole disk. The solution, as everyone agrees, is more tapes. -Mitch John R. Jackson, Technical Software Specialist, [EMAIL PROTECTED]
