Certainly, planning is always better than trying to figure-out a good
reaction anywhere, and especially with system administration. (When you
inherit someone else's server room, you benefit from their good decisions,
like Dirvish, SCSI hard drives, ReiserFS and SlackWare but you suffer from
their omissions, like log rotation and separate database backup.)
I greatly appreciate your suggestions and I plan to implement log rotation
on a couple servers where it had been neglected. It's not too late for me to
implement branches; I've got a couple of servers that I built with a common
installation script. I can move the current vault onto another storage
device for a while and make a new vault with "dirvish --init"
My vault is on a ReiserFS volume, so I don't have to worry about my inode
count. My Email data are all in the Exchange server on the other ("dark")
side of the shop, so I don't have to worry about the mbox format. My VMWare
machines are backed-up as if they were physical boxen. Don't talk to be
about .ISO images; I'm too busy blushing and moving gigabytes! Backing-up
databases? I'll have to think about that some more.
Thanks again for the suggestions for re-organization. I'm still hoping
somebody has my expiration date tuner script out there for the short term,
but you've given me some good goals for real optimization of my backups.
-dP
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On
Behalf Of Keith Lofstrom
Sent: Saturday, May 26, 2007 4:38 PM
To: Dirvish user and developer mailing list
Subject: Re: [Dirvish] Tuning my expiration dates
On Sat, May 26, 2007 at 01:59:38PM -0700, Petcher, Daniel wrote:
>
> I *REALLY* need to buy and mount more disk space, but I don't have the
> budget or time for that project this quarter.
That is indeed what you need to do. Plan on purchasing 3x space
times what you are backing up. Any less and you are filling the
backup disks too often.
> Surely I can't be the first person to have wanted this functionality.
> Does anyone else out there have a suggestion to simplify my life?
Management of what goes INTO the backups is key. This involves both
changing how the client machines are configured, what gets backed up, and
how dirvish is set up.
It is a little late now, but when you set up the dirvish archives, you can
use branches to merge similar machines into sorta-kinda the same vaults
using branches. For example, my wife and I both have T30 laptops with RHEL5
. Those, and a couple of other RHEL5 machines, are branches off the same
basic "RHEL5boot" and "RHEL5root" vaults. This saves some space for all the
common stuff.
Also a little late, but build your backup drives with small blocks (2K?) and
lots of inodes. You are storing a huge number of small directory files
every day, typically a kbyte or so, and those can chew up 16K block ext3
partitions quickly.
Avoid making big files that have small daily changes -
Do NOT use dirvish for giant files that change often, but only
a little. Mail on my systems is stored in "maildir" format,
rather than "mbox" format. Now the addition of a single mail
does not make a big new file in the backups.
Another space saver is to use "datext" style logrotate ( originally
in SuSE, recently in Red Hat). This makes log files use date
extensions instead of rotating through .1, .2, etc. Change logs
to daily instead of weekly on big logs. Now you are not storing
a new version of a big log file every day.
Yet another space saver is to exclude big files like database
files from the dirvish backups, and use some other method.
Database files usually have some kind of dedicated backup
technique anyway.
If you run something like VMWare, with big virtual disks, those
are another classic "little changes often" file. If you can,
arrange your VMWare virtual disks to be as small as possible,
and use internal samba shares (stored as linux files) for
everything possible. The virtual disks will still change, but
slowly - exclude them from the filesystem vault and store them
some other way.
Are you storing a whole lot of downloaded ISO files to make
CDs and DVDs from? There is little point in letting dirvish
and rsync manage these, if you are short of disk space. Just
make some extra CDs and DVDs.
When you have made other arrangements for 10 or so big files,
and stop backing up other big files that are stored elsewhere,
you will find the remaining moderate sized files store very
nicely. You can point to big files from a client directory
with soft links, and that directory can be the target of a
special big-file vault. Use aggressive expires. This may be
a good application of something like rdiffbackup. This is a
PITA for restores, but it is only a few files, which if
necessary you can move and check by hand.
The bottom line is that you need enough storage to do this, and
rsync, the core of dirvish, doesn't do everything well. For the
vast bulk of your files, it is magic. For giant files, all it
can do is make new daily copies of them, chewing up disk space
rapidly.
I hope that helps ...
Keith
--
Keith Lofstrom [EMAIL PROTECTED] Voice (503)-520-1993
KLIC --- Keith Lofstrom Integrated Circuits --- "Your Ideas in Silicon"
Design Contracting in Bipolar and CMOS - Analog, Digital, and Scan ICs
_______________________________________________
Dirvish mailing list
[email protected]
http://www.dirvish.org/mailman/listinfo/dirvish
_______________________________________________
Dirvish mailing list
[email protected]
http://www.dirvish.org/mailman/listinfo/dirvish
