Re: Backup Techniques onto DVD+-RW
On Fri, Dec 23, 2005 at 01:18:07PM -0700, Whyzzi wrote: > Interesting idea, and have to admit I didn't think of it. There is a > second HD ide hard drive slaved in the mail server, as well. I could > use the likes of DD or dump/restore onto the second drive (slave). > Last time I did that (dump/restore), I screwed up though, which is why > a second backup method is preferable. I was playing CCD with the likes > of that, for the separate /home partition. But for some reason it > didn't quite feel right (likely a setup problem), so I scratched the > CCD idea. I have made good experience using 'rdiff-backup' to store data on a second hard-disk or on a remote machine. It does incremental backups and even keeps track of all changes. I have to admit that I have not used it on OpenBSD, so YMMV. http://www.nongnu.org/rdiff-backup/ Bernd [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]
Re: Backup Techniques onto DVD+-RW
Steve Shockley wrote: > Whyzzi wrote: >> Hi gang. Running a lightweight mail server here (50 users total) on >> OpenBSD, and being the cheap bastard that I am I am looking forward to >> scripting a nightly backup onto some DVD-RW media. Can I assume that >> dump/restore is out of the question because of the special commands >> burners require to begin the writing process? And if that is indeed >> the case, any recommendations or uber cool few liners that would have >> say get maximum compression of the contents in /home where all related >> mail is stored (sendmail/procmail-maildir/dovecot). BTW: Happy >> Holidays to you and yours! > > You could dump to a file piped through gzip/bzip2, then copy that to > CD/DVD. I back up several OpenBSD machines at work by dump|bzip2 to an > nfs share on Windows (SFU), then the Windows box gets backed up to tape. > (No Commvault agents for Open.) Works well and doesn't require > changing our existing corporate backup process. If compression is a consideration, I have to mention something I discovered recently for a similar issue: If your e-mail flow has a lot of binary attachments (our people are very fond of e-mailing PDF files out of scanners), gzip is not very good. bzip2 was hardly any better, but many times slower -- most certainly not worth the tiny improvement in size. A "solution" turned out to be rzip, which was slightly faster than bzip2, but delivers MUCH better compression numbers, returning better than 2:1 compression most of the time (and sometimes, a WHOLE LOT more than that), compared to bzip2 and gzip, which were doing something like 30% and 35% reductions, respectively (vs. 65% reductions for rzip on the same file). These are on big (400+MB) mbox files (I'd suspect tar files would produce similar savings, but individual files in an Maildir would probably be "different"). A few warnings: 1) you can not pipe/redirect rzip. Apparently, it needs non-sequential seeks on the file to find redundancies. However, disk space is cheap, just compress/expand from a local HD partition. 2) memory, memory, memory: it works by looking over a much larger "window" than bzip/gzip/etc. use looking for redundancy, so for big files, a gig of RAM is not very wasteful. Un-rzipping a file, however, can be done on very modest machines -- I have the system burn to DVD-R, then unpack it and verify MD5 numbers, all on a old P4/128M RAM (old RAMBUS machine, thus it probably never will get another stick of RAM unless another machine dies). The actual image is created on a amd64 system with a gig of RAM. 3) I have been unable to find a Windows port of the rzip application, which really doesn't matter a whole lot, unless you want to get to your data out on a Windows system. rzip is in OpenBSD ports, and apparently can be compiled easily enough on most Unix-like platforms. (No, I really don't care how easily it "could" be ported to Windows, unless you actually have done it.) Nick.
Re: Backup Techniques onto DVD+-RW
Whyzzi wrote: Hi gang. Running a lightweight mail server here (50 users total) on OpenBSD, and being the cheap bastard that I am I am looking forward to scripting a nightly backup onto some DVD-RW media. Can I assume that dump/restore is out of the question because of the special commands burners require to begin the writing process? And if that is indeed the case, any recommendations or uber cool few liners that would have say get maximum compression of the contents in /home where all related mail is stored (sendmail/procmail-maildir/dovecot). BTW: Happy Holidays to you and yours! You could dump to a file piped through gzip/bzip2, then copy that to CD/DVD. I back up several OpenBSD machines at work by dump|bzip2 to an nfs share on Windows (SFU), then the Windows box gets backed up to tape. (No Commvault agents for Open.) Works well and doesn't require changing our existing corporate backup process.
Re: Backup Techniques onto DVD+-RW
On Fri, Dec 23, 2005 at 02:43:29PM -0600, L. V. Lammert wrote:
| The biggest disadvantage is that you're maintaining a single remote copy;
| if you wish more than one historical version you must manage separately.
Check out the --link-dest option to rsync. Consider the following
snippet of a shell script I'm writing :
--
...
for USER in $(cat ${USERSFILE})
do
SRC=${BACKUPPREFIX}/${USER}/
if [ ! -d ${SRC} ]
then
echo Source not found \(${SRC}\) >&2
exit 3
fi
DST=${STOREPREFIX}/${NOW}/${USER}
mkdir -p ${DST}
PREVDIR=${STOREPREFIX}/${PREVIOUS}/${USER}
if [ -d ${PREVDIR} ]
then
rsync -aHx --link-dest=${PREVDIR} ${SRC} ${DST}
else
rsync -aHx ${SRC} ${DST}
fi
done
...
--
I use the above for a local disk backup (to cover users accidentally
deleting their files or one of the disks failing), a few changes will
make this an ideal candidate for a remote backup. I'm still finetuning
the full script, but I hope you get the gist of it.
Using the --link-dest option, you get 14 'full backups' taking up
about 110% of the diskspace the original data uses (depending on the
type of data, of course) :
[EMAIL PROTECTED] $ cd /backup; du -shc *
13.2G 20051210
78.0M 20051211
76.3M 20051212
103M20051213
102M20051214
85.0M 20051215
81.7M 20051216
105M20051217
81.3M 20051218
81.1M 20051219
84.1M 20051220
108M20051221
116M20051222
84.3M 20051223
14.3G total
Cheers,
Paul 'WEiRD' de Weerd
--
>[<++>-]<+++.>+++[<-->-]<.>+++[<+
+++>-]<.>++[<>-]<+.--.[-]
http://www.weirdnet.nl/
[demime 1.01d removed an attachment of type application/pgp-signature]
[OT] Backup Techniques onto DVD+-RW
[Marked OT, since not specially related to OpenBSD] On Fri, Dec 23, 2005 at 01:18:07PM -0700, Whyzzi wrote: > Interesting idea, and have to admit I didn't think of it. There is a > second HD ide hard drive slaved in the mail server, as well. I could > use the likes of DD or dump/restore onto the second drive (slave). [...] > As for the cost of DVD+/-RW media, you'd be surprised at you can find. IMHO, neither a second HD nor something as unreliable as DVD or CD make a good backup medium -- it just may save you some time compared to install and configure a new system using the official install CD set. If you have sensitive/important *data* on your system, you *want* to backup on media that are *designed* for backup and archiving *and* allow for removal (i.e. no HDs). > We bought Verbatim DVD-RW media @ 1.49$ Canadian each. I had also > found a 5pack of some unknown brand (not R-DATA) being dumped at our > local Staples for under $9cdn which I purchased for home use. Test them. Put the same data on as much as you can afford for testing purposes. Do this with media from different vendors, or with media from the *same* vendor bought at different times. Keep those backups for a few months and then check how much of them are still readable. My experience with CDs is *really* bad, and whenever I talk with friends about DVDs, they tell that DVD are even worse. Two main reasons *against* CDs (and probably DVDs): - Media are very sensible wrt environmental conditions. - If the drive used for burning the media starts to die (because of dust or just aging), the medium tends to be readable for some days or weeks before you get immediate errors during the backup or a verify-read after the backup. With tape streamers, I never had this problem, i.e. if the verify was o.k., then was the data on the tape, and I could read the very last tape written with a streamer years after that streamer died (using a backwards compatible device). And this was with DDS devices, which AFAIK are still considered cheap-ware (if not crap). Ciao, Kili -- Aus Sicht der Daten sind Menschen komische Leute.
Re: Backup Techniques onto DVD+-RW
On Fri, Dec 23, 2005 at 11:51:14AM -0600, L. V. Lammert wrote: > Better recommendation - rsync /home to an external system (especially if > you're using Maildir). WAY less overhead! You can even backup more often. > An archive machine is less costly than a bundle of DVD-RWs, and you don't > have to swap media. > > Should you want more than one archive, there are various ways to manage > multiple versions. rsnapshot in the ports tree.
Re: Backup Techniques onto DVD+-RW
At 01:18 PM 12/23/2005 -0700, Whyzzi wrote: Interesting idea, and have to admit I didn't think of it. There is a second HD ide hard drive slaved in the mail server, as well. I could use the likes of DD or dump/restore onto the second drive (slave). Last time I did that (dump/restore), I screwed up though, which is why a second backup method is preferable. I was playing CCD with the likes of that, for the separate /home partition. But for some reason it didn't quite feel right (likely a setup problem), so I scratched the CCD idea. The biggest advantage of rsync is that you are only backing up changed files; sort of like the old 'incremantal'. The biggest disadvantage is that you're maintaining a single remote copy; if you wish more than one historical version you must manage separately. As for the cost of DVD+/-RW media, you'd be surprised at you can find. We bought Verbatim DVD-RW media @ 1.49$ Canadian each. I had also found a 5pack of some unknown brand (not R-DATA) being dumped at our local Staples for under $9cdn which I purchased for home use. Well, . . that's still expensive compared to free. Backup servers are not mission critical, so just about any old h/w is sufficient (assuming big enough disk). You can also locate them at a remote site (we offer off-site backup to our clients at a nominal charge). If you like, you can daisy-chain machines - h/w is commodity now, we almost always have capable machines cycling through the shop as 'upgrade leftovers'. I've got a squid-cache proxy server local to that subnet running OpenBSD as well, and it has plenty of HD space left on the 80gig Seagate SATA. Good candidate! If you have sufficient disk space, just create one or more directories with images. For example, five (or seven) directories would allow an entire week of archives. Happy Holidays! Lee
Re: Backup Techniques onto DVD+-RW
Interesting idea, and have to admit I didn't think of it. There is a second HD ide hard drive slaved in the mail server, as well. I could use the likes of DD or dump/restore onto the second drive (slave). Last time I did that (dump/restore), I screwed up though, which is why a second backup method is preferable. I was playing CCD with the likes of that, for the separate /home partition. But for some reason it didn't quite feel right (likely a setup problem), so I scratched the CCD idea. As for the cost of DVD+/-RW media, you'd be surprised at you can find. We bought Verbatim DVD-RW media @ 1.49$ Canadian each. I had also found a 5pack of some unknown brand (not R-DATA) being dumped at our local Staples for under $9cdn which I purchased for home use. I've got a squid-cache proxy server local to that subnet running OpenBSD as well, and it has plenty of HD space left on the 80gig Seagate SATA. Muchly appriciated! On 23/12/05, L. V. Lammert <[EMAIL PROTECTED]> wrote: > At 10:17 AM 12/23/2005 -0700, Whyzzi wrote: > >Hi gang. Running a lightweight mail server here (50 users total) on > >OpenBSD, and being the cheap bastard that I am I am looking forward to > >scripting a nightly backup onto some DVD-RW media. Can I assume that > >dump/restore is out of the question because of the special commands > >burners require to begin the writing process? And if that is indeed > >the case, any recommendations or uber cool few liners that would have > >say get maximum compression of the contents in /home where all related > >mail is stored (sendmail/procmail-maildir/dovecot). BTW: Happy > >Holidays to you and yours! > > Better recommendation - rsync /home to an external system (especially if > you're using Maildir). WAY less overhead! You can even backup more often. > An archive machine is less costly than a bundle of DVD-RWs, and you don't > have to swap media. > > Should you want more than one archive, there are various ways to manage > multiple versions. > > Lee
Re: Backup Techniques onto DVD+-RW
At 10:17 AM 12/23/2005 -0700, Whyzzi wrote: Hi gang. Running a lightweight mail server here (50 users total) on OpenBSD, and being the cheap bastard that I am I am looking forward to scripting a nightly backup onto some DVD-RW media. Can I assume that dump/restore is out of the question because of the special commands burners require to begin the writing process? And if that is indeed the case, any recommendations or uber cool few liners that would have say get maximum compression of the contents in /home where all related mail is stored (sendmail/procmail-maildir/dovecot). BTW: Happy Holidays to you and yours! Better recommendation - rsync /home to an external system (especially if you're using Maildir). WAY less overhead! You can even backup more often. An archive machine is less costly than a bundle of DVD-RWs, and you don't have to swap media. Should you want more than one archive, there are various ways to manage multiple versions. Lee
Backup Techniques onto DVD+-RW
Hi gang. Running a lightweight mail server here (50 users total) on OpenBSD, and being the cheap bastard that I am I am looking forward to scripting a nightly backup onto some DVD-RW media. Can I assume that dump/restore is out of the question because of the special commands burners require to begin the writing process? And if that is indeed the case, any recommendations or uber cool few liners that would have say get maximum compression of the contents in /home where all related mail is stored (sendmail/procmail-maildir/dovecot). BTW: Happy Holidays to you and yours!
