Re: [Dovecot] backup mdbox best strategy
Am 2013-12-07 10:04, schrieb Robert Schetterer: Am 07.12.2013 08:53, schrieb Claus: Thanks for your suggestion to use maildir, but i'm sure the key problem are the large Number of small files. the backup goes first to diskstorage and afterwards to LTO4 Tapes. We backup with EMC-Networker. The backuptime of large files (some GB) are normal. I wonder if there is no one who operates a large Mailsystem with mdbox and has suggestions to backup this? you are able to backup and restore mdbox with dsync ( never tested ), but its not like with maildir simply i.e rsync the mailbox or one only mail so you may find some combined strategy using both formats depending to your kind of storage, as people said i wouldnt recommand backup to tape in a direct way Thanks Robert! As i wrote the backup goes first to diskstorage and than to tape. why not simply do some testing with only one mdbox mailbox , should be easy with dsync Claus Am 06.12.2013 11:51, schrieb Nick Edwards: Claus, Robert's suggestion is good, we backup maildir, with 5.9TB in 2 hours to SATA , we also run a backup of that backup, so we have two extra copies (this is on top of NAS snapshots) tape is just too slow for anything serious these days. N ik On 12/5/13, Claus wrote: Am 02.12.2013 20:37, schrieb Robert Schetterer: Am 02.12.2013 18:47, schrieb Claus: Hello, i have to backup (tape library) a mailsystem with about 300.000 Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. The mailstore is mdbox. Is it save to do a simple filesystem backup (full and incremental) with backupsoftware? hm i wouldnt do that with mdbox, cause it does strongly depends to index, but if you use dsync you could convert to maildir at backup, maildir should be fine to store on tape, but with 2 TB this is only good for total desaster archive, i guess it would take long time to restore specially single mailboxes in acceptable time, from tape Better use cheap sata storage raids for frequent backups, and do tape snapshots sometimes. What is the prefered strategy to do a backup for desaster recovery (mailsystem crash) and restoring single usermailboxes? dsync should do fine http://wiki2.dovecot.org/Tools/Dsync but there is no universal answer to this, perhaps filesystem snapshots strategy might better, it depened on your general setup , filesystems , mailstore etc. wait for others report their solutions, and find the one fits best at your place Regards, Claus Best Regards MfG Robert Schetterer thanks for your answer, the main reason we plan to use mdbox is to reduce backuptime. Now we are using Maildir on cyrus and the (full)-backup takes about 20 hours on one backend storing 150 GB of Mails in about 3 million files. Because of that we have many backends, so we can parallelize the backup prozess. greets, Claus Best Regards MfG Robert Schetterer
Re: [Dovecot] backup mdbox best strategy
Am 2013-12-02 18:47, schrieb Claus: Hello, i have to backup (tape library) a mailsystem with about 300.000 Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. The mailstore is mdbox. Is it save to do a simple filesystem backup (full and incremental) with backupsoftware? What is the prefered strategy to do a backup for desaster recovery (mailsystem crash) and restoring single usermailboxes? Regards, Claus Regardless of the discussion about maildir and mdbox (many thanks to all) i hope Timo could give an statement and valueable suggestion to me how to backup a mdbox based mailsystem. Regards, Claus
Re: [Dovecot] backup mdbox best strategy
Am 07.12.2013 08:53, schrieb Claus: > Thanks for your suggestion to use maildir, but > i'm sure the key problem are the large Number of small files. > the backup goes first to diskstorage and afterwards to LTO4 Tapes. > We backup with EMC-Networker. The backuptime of large files (some GB) > are normal. > > I wonder if there is no one who operates a large Mailsystem with mdbox > and has suggestions to backup this? you are able to backup and restore mdbox with dsync ( never tested ), but its not like with maildir simply i.e rsync the mailbox or one only mail so you may find some combined strategy using both formats depending to your kind of storage, as people said i wouldnt recommand backup to tape in a direct way why not simply do some testing with only one mdbox mailbox , should be easy with dsync > > > Claus > > > > Am 06.12.2013 11:51, schrieb Nick Edwards: >> Claus, >> Robert's suggestion is good, we backup maildir, with 5.9TB in 2 hours >> to SATA , we also run a backup of that backup, so we have two extra >> copies (this is on top of NAS snapshots) >> tape is just too slow for anything serious these days. >> >> N ik >> >> >> On 12/5/13, Claus wrote: >>> Am 02.12.2013 20:37, schrieb Robert Schetterer: Am 02.12.2013 18:47, schrieb Claus: > Hello, > > i have to backup (tape library) a mailsystem with about 300.000 > Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. > The mailstore is mdbox. > Is it save to do a simple filesystem backup (full and incremental) > with > backupsoftware? hm i wouldnt do that with mdbox, cause it does strongly depends to index, but if you use dsync you could convert to maildir at backup, maildir should be fine to store on tape, but with 2 TB this is only good for total desaster archive, i guess it would take long time to restore specially single mailboxes in acceptable time, from tape Better use cheap sata storage raids for frequent backups, and do tape snapshots sometimes. > What is the prefered strategy to do a backup for desaster recovery > (mailsystem crash) and restoring single usermailboxes? dsync should do fine http://wiki2.dovecot.org/Tools/Dsync but there is no universal answer to this, perhaps filesystem snapshots strategy might better, it depened on your general setup , filesystems , mailstore etc. wait for others report their solutions, and find the one fits best at your place > Regards, > Claus Best Regards MfG Robert Schetterer >>> thanks for your answer, >>> the main reason we plan to use mdbox is to reduce backuptime. >>> Now we are using Maildir on cyrus and the (full)-backup takes about 20 >>> hours on one backend storing 150 GB of Mails in about 3 million files. >>> Because of that we have many backends, so we can parallelize the backup >>> prozess. >>> >>> greets, Claus >>> >>> >>> >>> >>> >>> >>> >>> >>> > Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Re: [Dovecot] backup mdbox best strategy
Thanks for your suggestion to use maildir, but i'm sure the key problem are the large Number of small files. the backup goes first to diskstorage and afterwards to LTO4 Tapes. We backup with EMC-Networker. The backuptime of large files (some GB) are normal. I wonder if there is no one who operates a large Mailsystem with mdbox and has suggestions to backup this? Claus Am 06.12.2013 11:51, schrieb Nick Edwards: Claus, Robert's suggestion is good, we backup maildir, with 5.9TB in 2 hours to SATA , we also run a backup of that backup, so we have two extra copies (this is on top of NAS snapshots) tape is just too slow for anything serious these days. N ik On 12/5/13, Claus wrote: Am 02.12.2013 20:37, schrieb Robert Schetterer: Am 02.12.2013 18:47, schrieb Claus: Hello, i have to backup (tape library) a mailsystem with about 300.000 Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. The mailstore is mdbox. Is it save to do a simple filesystem backup (full and incremental) with backupsoftware? hm i wouldnt do that with mdbox, cause it does strongly depends to index, but if you use dsync you could convert to maildir at backup, maildir should be fine to store on tape, but with 2 TB this is only good for total desaster archive, i guess it would take long time to restore specially single mailboxes in acceptable time, from tape Better use cheap sata storage raids for frequent backups, and do tape snapshots sometimes. What is the prefered strategy to do a backup for desaster recovery (mailsystem crash) and restoring single usermailboxes? dsync should do fine http://wiki2.dovecot.org/Tools/Dsync but there is no universal answer to this, perhaps filesystem snapshots strategy might better, it depened on your general setup , filesystems , mailstore etc. wait for others report their solutions, and find the one fits best at your place Regards, Claus Best Regards MfG Robert Schetterer thanks for your answer, the main reason we plan to use mdbox is to reduce backuptime. Now we are using Maildir on cyrus and the (full)-backup takes about 20 hours on one backend storing 150 GB of Mails in about 3 million files. Because of that we have many backends, so we can parallelize the backup prozess. greets, Claus
Re: [Dovecot] backup mdbox best strategy
Claus, Robert's suggestion is good, we backup maildir, with 5.9TB in 2 hours to SATA , we also run a backup of that backup, so we have two extra copies (this is on top of NAS snapshots) tape is just too slow for anything serious these days. N ik On 12/5/13, Claus wrote: > Am 02.12.2013 20:37, schrieb Robert Schetterer: >> Am 02.12.2013 18:47, schrieb Claus: >>> Hello, >>> >>> i have to backup (tape library) a mailsystem with about 300.000 >>> Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. >>> The mailstore is mdbox. >>> Is it save to do a simple filesystem backup (full and incremental) with >>> backupsoftware? >> hm i wouldnt do that with mdbox, cause it does strongly depends to >> index, but if you use dsync you could convert to maildir at backup, >> maildir should be fine to store on tape, but with 2 TB this is only good >> for total desaster archive, i guess it would take long time to restore >> specially single mailboxes in acceptable time, from tape >> Better use cheap sata storage raids for frequent backups, and do tape >> snapshots sometimes. >> >>> What is the prefered strategy to do a backup for desaster recovery >>> (mailsystem crash) and restoring single usermailboxes? >> dsync should do fine >> >> http://wiki2.dovecot.org/Tools/Dsync >> >> but there is no universal answer to this, perhaps filesystem snapshots >> strategy might better, it depened on your general setup , filesystems , >> mailstore etc. >> >> wait for others report their solutions, and find the one fits best at >> your place >> >>> >>> Regards, >>> Claus >> >> >> Best Regards >> MfG Robert Schetterer >> > thanks for your answer, > the main reason we plan to use mdbox is to reduce backuptime. > Now we are using Maildir on cyrus and the (full)-backup takes about 20 > hours on one backend storing 150 GB of Mails in about 3 million files. > Because of that we have many backends, so we can parallelize the backup > prozess. > > greets, Claus > > > > > > > > >
Re: [Dovecot] backup mdbox best strategy
On 2013-12-05 6:43 AM, Claus wrote: Now we are using Maildir on cyrus and the (full)-backup takes about 20 hours on one backend storing 150 GB of Mails in about 3 million files. !? What backup software are you using? Not to take away from the advantages of mdbox, but... I use rsnapshot, and am pushing the backups to an NFS mount on a cheaper NAS (QNAP) device, and the initial full backup of 300+GB of maildirs (not sure how many more millions of files, but a lot more than yours I imagine) only took about 4 or 5 hours. The subsequent runs now take anywhere from 1-2 hours... So, maybe it is your backup software that is broken? I am also planning on migrating to mdbox, for two reasons - much less files to backup (should reduce backup snapshot times to minutes, allowing me to add many more during the day), and I also want to enable SIS to reduce dramatically the overall storage requirements (we use a lot of large attachments, and I know for a fact that they are duplicated many times over, so the overall storage should be reduced by half or more)... -- Best regards, */Charles/*
Re: [Dovecot] backup mdbox best strategy
Am 05.12.2013 12:43, schrieb Claus: > Am 02.12.2013 20:37, schrieb Robert Schetterer: >> Am 02.12.2013 18:47, schrieb Claus: >>> Hello, >>> >>> i have to backup (tape library) a mailsystem with about 300.000 >>> Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. >>> The mailstore is mdbox. >>> Is it save to do a simple filesystem backup (full and incremental) with >>> backupsoftware? >> hm i wouldnt do that with mdbox, cause it does strongly depends to >> index, but if you use dsync you could convert to maildir at backup, >> maildir should be fine to store on tape, but with 2 TB this is only good >> for total desaster archive, i guess it would take long time to restore >> specially single mailboxes in acceptable time, from tape >> Better use cheap sata storage raids for frequent backups, and do tape >> snapshots sometimes. >> >>> What is the prefered strategy to do a backup for desaster recovery >>> (mailsystem crash) and restoring single usermailboxes? >> dsync should do fine >> >> http://wiki2.dovecot.org/Tools/Dsync >> >> but there is no universal answer to this, perhaps filesystem snapshots >> strategy might better, it depened on your general setup , filesystems , >> mailstore etc. >> >> wait for others report their solutions, and find the one fits best at >> your place >> >>> >>> Regards, >>> Claus >> >> >> Best Regards >> MfG Robert Schetterer >> > thanks for your answer, > the main reason we plan to use mdbox is to reduce backuptime. yeah but study http://wiki2.dovecot.org/MailboxFormat/dbox for diffs to maildir format, specially at restore however as i said ,dsync should play nice , beside other strategies > Now we are using Maildir on cyrus and the (full)-backup takes about 20 > hours on one backend storing 150 GB of Mails in about 3 million files. > Because of that we have many backends, so we can parallelize the backup > prozess. > > greets, Claus > > > > > > > > Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Re: [Dovecot] backup mdbox best strategy
Am 02.12.2013 20:37, schrieb Robert Schetterer: Am 02.12.2013 18:47, schrieb Claus: Hello, i have to backup (tape library) a mailsystem with about 300.000 Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. The mailstore is mdbox. Is it save to do a simple filesystem backup (full and incremental) with backupsoftware? hm i wouldnt do that with mdbox, cause it does strongly depends to index, but if you use dsync you could convert to maildir at backup, maildir should be fine to store on tape, but with 2 TB this is only good for total desaster archive, i guess it would take long time to restore specially single mailboxes in acceptable time, from tape Better use cheap sata storage raids for frequent backups, and do tape snapshots sometimes. What is the prefered strategy to do a backup for desaster recovery (mailsystem crash) and restoring single usermailboxes? dsync should do fine http://wiki2.dovecot.org/Tools/Dsync but there is no universal answer to this, perhaps filesystem snapshots strategy might better, it depened on your general setup , filesystems , mailstore etc. wait for others report their solutions, and find the one fits best at your place Regards, Claus Best Regards MfG Robert Schetterer thanks for your answer, the main reason we plan to use mdbox is to reduce backuptime. Now we are using Maildir on cyrus and the (full)-backup takes about 20 hours on one backend storing 150 GB of Mails in about 3 million files. Because of that we have many backends, so we can parallelize the backup prozess. greets, Claus
Re: [Dovecot] backup mdbox best strategy
Am 02.12.2013 18:47, schrieb Claus: > Hello, > > i have to backup (tape library) a mailsystem with about 300.000 > Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. > The mailstore is mdbox. > Is it save to do a simple filesystem backup (full and incremental) with > backupsoftware? hm i wouldnt do that with mdbox, cause it does strongly depends to index, but if you use dsync you could convert to maildir at backup, maildir should be fine to store on tape, but with 2 TB this is only good for total desaster archive, i guess it would take long time to restore specially single mailboxes in acceptable time, from tape Better use cheap sata storage raids for frequent backups, and do tape snapshots sometimes. > What is the prefered strategy to do a backup for desaster recovery > (mailsystem crash) and restoring single usermailboxes? dsync should do fine http://wiki2.dovecot.org/Tools/Dsync but there is no universal answer to this, perhaps filesystem snapshots strategy might better, it depened on your general setup , filesystems , mailstore etc. wait for others report their solutions, and find the one fits best at your place > > > Regards, > Claus Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
[Dovecot] backup mdbox best strategy
Hello, i have to backup (tape library) a mailsystem with about 300.000 Mailboxes on 2 backends. Summary of all mailboxes are 2 TByte. The mailstore is mdbox. Is it save to do a simple filesystem backup (full and incremental) with backupsoftware? What is the prefered strategy to do a backup for desaster recovery (mailsystem crash) and restoring single usermailboxes? Regards, Claus
