Status of the amanda-win32 project?
Saw it on Sourceforge: http://sourceforge.net/projects/amanda-win32/ Anyone tried it? -- |Colin Smith: [EMAIL PROTECTED] |
Re: Estimates taking a long time.
On 21 Jul 2001, Marc SCHAEFER wrote: Colin Smith [EMAIL PROTECTED] wrote: I'm running backups on 3 Linux systems, one of the systems is a Cobalt Qube. All the backups are done using GNU tar. It works OK but the try remounting those fs noatime during estimates: mount / -o remount,noatime Already mounted noatime. It's a news server. I've run a trace on tar as it scans the filesystem. The problem is readdir reading the dirent structures on the directories holding large groups. regards, Colin Smith.
Re: Estimates taking a long time.
On Fri, 20 Jul 2001, John R. Jackson wrote: My question is this. Why run a separate estimate at all? Why not just monitor the last couple of backups and extrapolate? That's been suggested before, but nobody has had the time to work on it. It should probably be a dumptype option. I'm perfectly happy with the way estimates are done on my machines, and some are volatile enough that guesses based on past history would be, at best, humorous :-). It'd maybe need a low pass digital filter to handle guesses on volatile filesystems. On the other hand, Amanda's the only backup system I'm aware of that performs an estimation prior to backing up the systems. It uses it to schedule the level 0,1,2,3 dumps and when in the order a particular filesystem will be backed up? regards, Colin Smith.
Re: Estimates taking a long time.
On Fri, 20 Jul 2001, John R. Jackson wrote: ... I noticed that the estimates are basically tar operations with all of the ouput going to /dev/null. That is what's taking the time. Not to get the file's size, but to move the date through that system pipe into /dev/null. Nope. Tar knows it is writing to /dev/null and optimizes out the open(), read() and write() calls so there is no data motion. I think the time is being spent traversing the directory tree and stat()ing the things it runs across. In other words, it's doing pretty much what your find would do. Yes, the problem I'm seeing is related to readdir on large directories, which have thousands of files. A feature of that version of the ext2 filesystem code I guess. regards, Colin Smith.
Estimates taking a long time.
I'm running backups on 3 Linux systems, one of the systems is a Cobalt Qube. All the backups are done using GNU tar. It works OK but the estimation time on the backups is nasty. I think I'll turn off the estimation and just run full dumps every day. The Qube is the slow system with the problem being related to the Linux filesystem code which reads directory entries; The system is a news server and has a couple of directories which have a lot of files. My question is this. Why run a separate estimate at all? Why not just monitor the last couple of backups and extrapolate? i.e. Day 1 incremental /home 100mb Day 2 incremental /home 110mb Day 3 incremental /home 120mb Day 4 incremental /home ???mb -- Colin Smith
Re: FW: Backup up oracle database, part II
On Sat, 2 Dec 2000, John R. Jackson wrote: I'm not totally familiar with Amanda so off the top of my head (and assuming we're using gnu tar) we could rename the gnu tar binary and write a script which wraps it ... I should have mentioned that at the moment, this is how it would have to be done, and that the real solution to this is the Amanda DUMPER-API, yet to be implemented. There's an X/OPEN backup API which is already implemented by many of the DB vendors. Certainly Oracle's RMAN and Informix's onbar use it to back up tablespaces/dbspaces and log files to backup management systems. I can't for the life of me remember what it's called at the moment. Another couple of thoughts from this morning. My Oracle person pointed out another advantage to the big backup disk approach is that if they need to do a recovery (which they do fairly often on the test systems), it can be done immediately from there rather than bugging me to reload tapes. I consider this a feature :-). A daily database export to a filesystem FIFO with compress on the other side of it. Much much easier than reloading entire databases but on a test system I think i'd agree with you. Now, I also fully realize the PUCC Oracle databases are probably not really all that large. The backup images are a few GBytes. So the script we're using may not be appropriate for larger sites. One other advantage to the script we use is that it auto-detects new (and removed) tablespaces. Again, because of all the testing we do, that probably happens here more than in some other sites. It could be handled by hand like any other disk, but it's kind of handy at the moment. The binaries are SUID root. It couldn't do the backup otherwise. That is just not true. You do not need to be root to do backups. I certainly do not, most Amanda users do not, and it's actually recommended that you do not configure it this way. I have the raw disk partitions in a group that the Amanda user is a member of and gave that group read permission. Oh, OK, I was thinking of tar only. I don't use dump for compatibility reasons. It's true some of the Amanda binaries are setuid root, but for the most part that's so they can get a privileged port at startup. They then drop their permissions. One exception to this is runtar, which is setuid to get access to run tar as root since it goes through the standard file system. In the Oracle backup case, the GNU tar wrapper (or DUMPER-API script) could probably be written setuid root (or use an external program) to do whatever magic it has to to Oracle. But that's opening up a door that would have to be very, very carefully coded. Rather than specifying the actual backup utility, Amanda could specify a backup *type* which is the type of data to expect back from a user specified backup script.
Re: FW: Backup up oracle database, part II
On Sun, 3 Dec 2000, John R. Jackson wrote: There's an X/OPEN backup API ... I can't for the life of me remember what it's called at the moment. Do you mean NDMP? If so, that comes up here every once in a while, I don't think so, the name doesn't ring a bell. I'll look it up at work. but nobody has ever been able to explain it well enough to me (including their web site). Rather than specifying the actual backup utility, Amanda could specify a backup *type* which is the type of data to expect back from a user specified backup script. I didn't follow this. Amanda very deliberately stays away from knowing what the images it is managing look like. It starts some program (which will be more generic with DUMPER-API) and then just deals with the byte stream that comes out of it. What would it do with a "type"? No idea, just a thought. Is it easy to configure the current Amanda with an arbitrary backup script then, rather than /usr/bin/gtar or /sbin/dump? regards, Colin.
Re: Backup up oracle database, part II
On Fri, 1 Dec 2000, John R. Jackson wrote: The space required for archivelogging during a hot backup shouldn't be anywhere near the size required to take a second copy of the database. It wasn't a space issue, at least not directly. It's my understanding that if we **did** run out of archive log space, we were SOL. Again, I'm not an Oracle person, but I trust completely they guy here who is. Oracle has a set of files called redo logs. They hold all the transactional information as the transactions are happening. Normally they are a few megabytes in size and you'd have at least 3 of them. Oracle uses them in a round robin fashion one after another. If the database crashes then when it comes back up the automatic recovery system goes through the redo logs and commits any transactions which were fully successful to the database and rolls back the ones which were not successful. If the database is in archivelog mode then these redo logs are automatically copied somewhere once the database has moved on to the next redo log file. Once the copy has been done successfully then the redo log file is marked as available to the database for reuse. If the copy is not successful then he redo log file will not be marked as available and when the database gets back round to the file, it will stop dead. Running out of space in the archivelog destination is the most common way of causing this to happen. When you're backing up an Oracle data file while Oracle is live it's very likely that the file will be updated and will change as it's being backed up. Oracle call it a fuzzy backup. On recovery, the files are going to be useless. So, what Oracle do is put the datafiles into hot backup mode. It freezes the files header and instead of writing individual transactions to the redo log file on an update, it writes entire database blocks. This has two implications. 1: In order to recover the database, you need all of the archivelog files starting from before when the backup started and the database was put into hot backup mode to when the backup is finished and the database is taken out of hot backup mode. 2: The database is going to go through the redo log files faster and generate many more archivelog files because entire database blocks are being written rather than individual transactions. So you're more likely to run out of archivelog space while the database is in hot backup mode. Also, the last thing you want to backup is the archivelog destination. If you backup the controlfile last then *you may not have all the archivelogs* to get the database up to date. How does the guarantee the disks in the mirror are up to date, i.e. logically consistent from Oracle's point of view? It doesn't. The broken mirror is the equivalent of a crashed system. OK, I get it. Though I don't particularly like it, I have to say that Oracle is very very good at recovering from a crash. This technique comes up once in a while for normal backups, too, and it's never made any sense to me. It won't work any better than what dump (or tar, in some cases) does now. Except that your mirror will be on a second set of disks and SCSI channels ... By "any better" I meant the data is no more consistent, not that there weren't other issues (like performance). When it's come up in my hearing in the past, it's usually been "look, there's this neat snapshot of the system we can get for free" with the implication it's just as good as a freshly fsck'd and sync'd disk, which it isn't. That's all I was trying to say. Nah, whoever says that is wrong. To use a broken mirror backup you need to have a logged filesystem which will recover very quickly and if you're using a database it has to have similar qualities. regards, Colin.
Re: amanda to blame for NT crashes?
On Fri, 1 Dec 2000, Eric Wadsworth wrote: I'm just getting everything going with amanda, and I'm suddenly in trouble. One of the NT boxes, being backed up via samba, got a blue-screen crash while it was going. My co-workers are saying that it is amanda's fault (that computer wasn't even being used at the time, it was idle). Um. Nope. If Windows blue screens then the only people to blame are Microsoft. Nothing you can do to the system via a shared drive should crash the system. It doesn't matter if you throw invalid packets at it, the system still shouldn't crash. I take it that the system is up to date on it's service packs and patches. [snip] regards, Colin.
Re: Backup up oracle database, part II
On Thu, 30 Nov 2000, John R. Jackson wrote: Just make sure you have plenty of archivelog space for transactions which hit the database during the backup. As I understand it, this was a major reason we don't do this but use the "big backup area" approach (which I proposed in "part I"). ? The space required for archivelogging during a hot backup shouldn't be anywhere near the size required to take a second copy of the database. I'm not an Oracle type, but the guy who put this together here is very good. And we've "tested" (well, OK, the thing crashed and we had to recover :-) the result -- several times. Now, if you want some redundancy *and* faster backups then you could put the database on a raid 1 mirror. When it comes to do the backup, break the mirror and mount the broken mirror somewhere. ... How does the guarantee the disks in the mirror are up to date, i.e. logically consistent from Oracle's point of view? It doesn't. The broken mirror is the equivalent of a crashed system. Oracle will have to go through the normal roll forward through the archivelogs to bring the data files to a consistent state. This technique comes up once in a while for normal backups, too, and it's never made any sense to me. It won't work any better than what dump (or tar, in some cases) does now. Except that your mirror will be on a second set of disks and SCSI channels so the disk I/O generated by the backup won't effect the live database. And since you also have a dedicated backup LAN, users won't be affected by the network I/O either. It's most useful in 24x7 environments where you can't have downtime for backups or even degradation in performance while the backup is running. You also have plenty of time to do the backup though you do have to remember that you're exposed while the backup's running. I know of one site who have two mirrors (They really just need a faster backup system). regards, Colin.
Re: Backup up oracle database, part II
On Wed, 29 Nov 2000, Fredrik Persson P (QRA) wrote: Hi all! Thanks to those of you who helped me getting a backup for an oracle database working. Now, the solution to copy the db files to a temp storage and then have amanda back up from there was good - it works! There is a problem however, Ouch. You'll need a lot of space which will be wasted while not holding the copied data files. I haven't seen part 1 but why not simply put all of the tablespaces into hot backup mode. Backup the data files and archivelogs normally and then backup the control files last or alternatively back the control files to trace? Just make sure you have plenty of archivelog space for transactions which hit the database during the backup. In addition you should do a database export; You're more likely to have a developer drop a table by mistake than have a disk fail and it's much easier to get data back from an export. Now, if you want some redundancy *and* faster backups then you could put the database on a raid 1 mirror. When it comes to do the backup, break the mirror and mount the broken mirror somewhere. Backup the broken mirror rather than the live database. Once the backup is done, unmount and resync the mirror. Make use of all the wasted temp space. I don't think the linux md driver is up to this but some of the newer stuff might be. and that is that the copy command consumes a LOT of system resources (mainly disk of course) and my server almost look dead from time to time. I've tried nice -n 15 to make the cp command a bit more nice, but it doesn't seem to work... Any suggestions? This is a RedHat Linux 6.2 system. Thanks! /Fredrik Persson --
