Re: Cyrus Backup Strategy
pnelson wrote: Yes dumping to removable medium. Why not write directly to tape rather than going through an extra step and requiring extra disk space? So a cyrus backup needs to contain: /var/spool/imap /var/lib/imap/mailboxes.db to be useful as a backup, right? With these to things I can recover... I'm making sure I understand this completely. You need to backup all of /var/spool/imap and /var/lib/imap. Some parts are less critical, but they are small anyway so you shouldn't try and leave out parts of it. How do you dump mailboxes to/from text? From cron as user cyrus, ctl_mboxlist -d >/var/lib/imap/mailboxes.txt What is the restore process? That gets a bit more fun. Our policy is to do restores for users at the folder level (assuming a user error restore -- a dead disk restore is easy, just restore everything). We create a RESTORE folder under the user's mailbox, set a quota on it, then restore all the message files and cyrus.* files from the folder to be recovered into that directory. Then run reconstruct on user/joe/RESTORE, and let them do what they want with the messages there. The RESTORE folder is automatically deleted 2 weeks later. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
Re: Cyrus Backup Strategy
On Fri, 13 Jun 2003, pnelson wrote: > On Fri, 2003-06-13 at 12:11, Igor Brezac wrote: > > On Fri, 13 Jun 2003, pnelson wrote: > > > > > On Fri, 2003-06-13 at 11:29, John Alton Tamplin wrote: > > > > pnelson wrote: > > > > > > > > >My last thing to do prior to converting to production is a backup > > > > >strategy. I have been doing this with tar something like: > > > > > > > > > >tar -C /var/lib-czf lib-.tar.gzimap > > > > >tar -C /var/spool -czf spool-.tar.gz imap > > > > >tar -cf cyrus-.tar > > > > > > > > > >This is producing a pretty big file(s): > > > > > > > > > > lib-.tar.gz-> 2.2M > > > > > spool-.tar.gz -> 11.0M > > > > > > > > > >and ultimately > > > > > > > > > > cyrus-.tar -> 13.5M > > > > > > > > > >and I was thinking maybe there is a better way that someone else has > > > > >come up with. Anyone doing backups a different way? > > > > > > > > > Are you then dumping these files to tape? Otherwise, just having > > > > another copy on disk doesn't protect against many potential causes of > > > > data loss. > > > > > > Yes dumping to removable medium. > > > > > > > We just backup the mail files normally with Veritas Netbackup (be sure > > > > to disable true image restore with the millions of files), with the only > > > > custom bit being to dump a copy of the mailbox list to a text file > > > > before the backups run (this is a precaution since we are backing up > > > > structured database files without synchronization with the program > > > > writing them, so there is no guarantee the resulting backup is useful). > > > > Our full backup for all Cyrus files is 98G and 4.2M files, taking 5-8 > > > > hours depending on other data hitting the tape drives. Daily > > > > incremental backups average 1-3G and 10-30k files. > > > > > > So a cyrus backup needs to contain: > > > > > > /var/spool/imap > > > /var/lib/imap/mailboxes.db > > > > > > to be useful as a backup, right? With these to things I can recover... > > > I'm making sure I understand this completely. > > > > I am not sure what your directory structure looks like, but you need to > > make sure to backup sieve and quota directories. > > > > Your procedure may require hand recovery after restore before the mail > > store is usable. I recommend using filesystem snapshot (i assume you do > > not stop the mail server during backup) before performing backup. This > > will make your backup 'sane.' > > > > > How do you dump mailboxes to/from text? > > > What is the restore process? > > Good point. So the latest process would be: > > So a cyrus backup needs to contain: > > su cyrus -c "ctl_mboxlist -d" > mailboxes-.txt This is not neccassary, but not a bad idea. > /var/spool/imap > /var/lib/imap/quota > /var/lib/imap/user > /var/lib/imap/sieve > > Not sure how the filesystem snapshot works? On solaris do 'man fssnap'. veritas filesystem supports the snapshot feature as does Linux (at least the newer versions), I am not sure which filesystem. Basically, this feature provides a temporary stable and read-only view (snapshot) of a filesystem. The writes occur on a separate filesystem or device. -- Igor
Re: Cyrus Backup Strategy
On Fri, 2003-06-13 at 12:11, Igor Brezac wrote: > On Fri, 13 Jun 2003, pnelson wrote: > > > On Fri, 2003-06-13 at 11:29, John Alton Tamplin wrote: > > > pnelson wrote: > > > > > > >My last thing to do prior to converting to production is a backup > > > >strategy. I have been doing this with tar something like: > > > > > > > >tar -C /var/lib-czf lib-.tar.gzimap > > > >tar -C /var/spool -czf spool-.tar.gz imap > > > >tar -cf cyrus-.tar > > > > > > > >This is producing a pretty big file(s): > > > > > > > > lib-.tar.gz-> 2.2M > > > > spool-.tar.gz -> 11.0M > > > > > > > >and ultimately > > > > > > > > cyrus-.tar -> 13.5M > > > > > > > >and I was thinking maybe there is a better way that someone else has > > > >come up with. Anyone doing backups a different way? > > > > > > > Are you then dumping these files to tape? Otherwise, just having > > > another copy on disk doesn't protect against many potential causes of > > > data loss. > > > > Yes dumping to removable medium. > > > > > We just backup the mail files normally with Veritas Netbackup (be sure > > > to disable true image restore with the millions of files), with the only > > > custom bit being to dump a copy of the mailbox list to a text file > > > before the backups run (this is a precaution since we are backing up > > > structured database files without synchronization with the program > > > writing them, so there is no guarantee the resulting backup is useful). > > > Our full backup for all Cyrus files is 98G and 4.2M files, taking 5-8 > > > hours depending on other data hitting the tape drives. Daily > > > incremental backups average 1-3G and 10-30k files. > > > > So a cyrus backup needs to contain: > > > > /var/spool/imap > > /var/lib/imap/mailboxes.db > > > > to be useful as a backup, right? With these to things I can recover... > > I'm making sure I understand this completely. > > I am not sure what your directory structure looks like, but you need to > make sure to backup sieve and quota directories. > > Your procedure may require hand recovery after restore before the mail > store is usable. I recommend using filesystem snapshot (i assume you do > not stop the mail server during backup) before performing backup. This > will make your backup 'sane.' > > > How do you dump mailboxes to/from text? > > What is the restore process? Good point. So the latest process would be: So a cyrus backup needs to contain: su cyrus -c "ctl_mboxlist -d" > mailboxes-.txt /var/spool/imap /var/lib/imap/quota /var/lib/imap/user /var/lib/imap/sieve Not sure how the filesystem snapshot works? No I do not want to stop the server.
Re: Cyrus Backup Strategy
On Fri, 13 Jun 2003, pnelson wrote: > On Fri, 2003-06-13 at 11:29, John Alton Tamplin wrote: > > pnelson wrote: > > > > >My last thing to do prior to converting to production is a backup > > >strategy. I have been doing this with tar something like: > > > > > >tar -C /var/lib-czf lib-.tar.gzimap > > >tar -C /var/spool -czf spool-.tar.gz imap > > >tar -cf cyrus-.tar > > > > > >This is producing a pretty big file(s): > > > > > > lib-.tar.gz-> 2.2M > > > spool-.tar.gz -> 11.0M > > > > > >and ultimately > > > > > > cyrus-.tar -> 13.5M > > > > > >and I was thinking maybe there is a better way that someone else has > > >come up with. Anyone doing backups a different way? > > > > > Are you then dumping these files to tape? Otherwise, just having > > another copy on disk doesn't protect against many potential causes of > > data loss. > > Yes dumping to removable medium. > > > We just backup the mail files normally with Veritas Netbackup (be sure > > to disable true image restore with the millions of files), with the only > > custom bit being to dump a copy of the mailbox list to a text file > > before the backups run (this is a precaution since we are backing up > > structured database files without synchronization with the program > > writing them, so there is no guarantee the resulting backup is useful). > > Our full backup for all Cyrus files is 98G and 4.2M files, taking 5-8 > > hours depending on other data hitting the tape drives. Daily > > incremental backups average 1-3G and 10-30k files. > > So a cyrus backup needs to contain: > > /var/spool/imap > /var/lib/imap/mailboxes.db > > to be useful as a backup, right? With these to things I can recover... > I'm making sure I understand this completely. I am not sure what your directory structure looks like, but you need to make sure to backup sieve and quota directories. Your procedure may require hand recovery after restore before the mail store is usable. I recommend using filesystem snapshot (i assume you do not stop the mail server during backup) before performing backup. This will make your backup 'sane.' > How do you dump mailboxes to/from text? > What is the restore process? > > Thanks > > -- Igor
Re: Cyrus Backup Strategy
On Fri, 2003-06-13 at 11:29, John Alton Tamplin wrote: > pnelson wrote: > > >My last thing to do prior to converting to production is a backup > >strategy. I have been doing this with tar something like: > > > >tar -C /var/lib-czf lib-.tar.gzimap > >tar -C /var/spool -czf spool-.tar.gz imap > >tar -cf cyrus-.tar > > > >This is producing a pretty big file(s): > > > > lib-.tar.gz-> 2.2M > > spool-.tar.gz -> 11.0M > > > >and ultimately > > > > cyrus-.tar -> 13.5M > > > >and I was thinking maybe there is a better way that someone else has > >come up with. Anyone doing backups a different way? > > > Are you then dumping these files to tape? Otherwise, just having > another copy on disk doesn't protect against many potential causes of > data loss. Yes dumping to removable medium. > We just backup the mail files normally with Veritas Netbackup (be sure > to disable true image restore with the millions of files), with the only > custom bit being to dump a copy of the mailbox list to a text file > before the backups run (this is a precaution since we are backing up > structured database files without synchronization with the program > writing them, so there is no guarantee the resulting backup is useful). > Our full backup for all Cyrus files is 98G and 4.2M files, taking 5-8 > hours depending on other data hitting the tape drives. Daily > incremental backups average 1-3G and 10-30k files. So a cyrus backup needs to contain: /var/spool/imap /var/lib/imap/mailboxes.db to be useful as a backup, right? With these to things I can recover... I'm making sure I understand this completely. How do you dump mailboxes to/from text? What is the restore process? Thanks
Re: Cyrus Backup Strategy
pnelson wrote: My last thing to do prior to converting to production is a backup strategy. I have been doing this with tar something like: tar -C /var/lib-czf lib-.tar.gzimap tar -C /var/spool -czf spool-.tar.gz imap tar -cf cyrus-.tar This is producing a pretty big file(s): lib-.tar.gz-> 2.2M spool-.tar.gz -> 11.0M and ultimately cyrus-.tar -> 13.5M and I was thinking maybe there is a better way that someone else has come up with. Anyone doing backups a different way? Are you then dumping these files to tape? Otherwise, just having another copy on disk doesn't protect against many potential causes of data loss. We just backup the mail files normally with Veritas Netbackup (be sure to disable true image restore with the millions of files), with the only custom bit being to dump a copy of the mailbox list to a text file before the backups run (this is a precaution since we are backing up structured database files without synchronization with the program writing them, so there is no guarantee the resulting backup is useful). Our full backup for all Cyrus files is 98G and 4.2M files, taking 5-8 hours depending on other data hitting the tape drives. Daily incremental backups average 1-3G and 10-30k files. -- John A. Tamplin Unix System Administrator Emory University, School of Public Health +1 404/727-9931
