Backup strategy for large mailbox stores

2010-02-15 Thread ram
We have cyrus servers deployed at many places where clients have varying mail storage. We have been taking backups to help in situations of human errors ( where you get complaints like ..oops, I accidentaly deleted all my mails!! ) and in case of hardware failures Things have been working fine

Re: Backup strategy for large mailbox stores

2010-02-15 Thread Gavin McCullagh
Hi, I'm a relative newbie with cyrus, but I'm interested in this discussion... On Mon, 15 Feb 2010, ram wrote: We have cyrus servers deployed at many places where clients have varying mail storage. We have been taking backups to help in situations of human errors ( where you get complaints

Re: Backup strategy for large mailbox stores

2010-02-15 Thread Michael Menge
Hi, Quoting ram r...@netcore.co.in: We have cyrus servers deployed at many places where clients have varying mail storage. We have been taking backups to help in situations of human errors ( where you get complaints like ..oops, I accidentaly deleted all my mails!! ) and in case of hardware

Re: Backup strategy for large mailbox stores

2010-02-15 Thread Bron Gondwana
On Mon, Feb 15, 2010 at 10:28:13AM +, Gavin McCullagh wrote: Hi, I'm a relative newbie with cyrus, but I'm interested in this discussion... Hehe - you should read through the mailing list archives for how FastMail does backups for a really complex but _FAST_ solution :) Things have

Using xfer to migrate mailboxes to a new server

2010-02-15 Thread Elver Loho
Hi, We've got Cyrus running fine on our old server since 2006. Now we're setting up a new server and would like to migrate all the mailboxes over to the new one before replacing the old server. For this task the xfer command seems relevant. I've coerced Cyrus on the old server to a point where

Re: Using xfer to migrate mailboxes to a new server

2010-02-15 Thread Kevin Kobb
On 2/15/2010 8:25 AM, Elver Loho wrote: Hi, We've got Cyrus running fine on our old server since 2006. Now we're setting up a new server and would like to migrate all the mailboxes over to the new one before replacing the old server. For this task the xfer command seems relevant. I've

Re: Using xfer to migrate mailboxes to a new server

2010-02-15 Thread Elver Loho
On 15 February 2010 16:05, Simon Fraser s...@sanger.ac.uk wrote: On Mon, 2010-02-15 at 15:25 +0200, Elver Loho wrote: localhost.localdomain xfer user.elver 192.168.0.180 xfermailbox: Server(s) unavailable to complete operation There should be a bit more information in the log, too. While

Re: Using xfer to migrate mailboxes to a new server

2010-02-15 Thread Elver Loho
On 15 February 2010 16:21, Kevin Kobb kk...@skylinecorp.com wrote: The last time we moved to new hardware, I used imapsync to migrate all the mailboxes to the new hardware. We moved from a different IMAP server to Cyrus, and this worked great. I don't know if this work as well now moving from

Re: Backup strategy for large mailbox stores

2010-02-15 Thread John Madden
Is there a better strategy , probably within the cyrus framework , to take backups efficiently We're a large site (400k with 1GB quotas users and growing) and this has for years been our biggest problem too. Typical backup systems (we run NetBackup), which scan the entire filesystem looking

Re: ability to disable shared namespace?

2010-02-15 Thread John Madden
John Madden wrote: Isn't this what foolstupidclients does? I think Blackberry might meet the criteria... Not really, and already enabled in my case. According to the docs, it just converts a LIST * into LIST INBOX* and that isn't sufficient. It doesn't seem like there is a workable

Re: Backup strategy for large mailbox stores

2010-02-15 Thread Vincent Fox
I suppose replication and snapshots are out of the question for you? We run ZFS so snapshots are atomic and nearly instant. Thus we keep 14 days of daily snaps in our production pool for recovery purposes. In our setup the total of all the snaps is about a 50% overhead on the production data

Re: Backup strategy for large mailbox stores

2010-02-15 Thread Vincent Fox
John Madden wrote: That still leaves full backups as a big issue (they take days to run) and NetBackup has a solution for that: You run one full backup and store it on disk somewhere and from then on, fulls are called synthetic fulls, where the incrementals are applied periodically in

Re: Backup strategy for large mailbox stores

2010-02-15 Thread Vincent Fox
Forgot to mention we are running inline compression on our ZFS pools. With fast LZJB compression on the filesystems for metadata etc. still a savings of ~2.0. The inboxes are all in /var/cyrus/mail which is set for gzip-6 compression savings of ~ 1.7. Backups run faster so it's win-win.

Re: Backup strategy for large mailbox stores

2010-02-15 Thread John Madden
Vincent Fox wrote: I suppose replication and snapshots are out of the question for you? Replication is something we haven't gotten into yet but there's probably something there. We don't have a remote site, so there's no way to make that work in a DR/backup sense. We did quite a bit with

Re: Backup strategy for large mailbox stores

2010-02-15 Thread John Madden
Our NBU admin runs multiple streams by the way which helped a lot. So the various letters of the cyrus hashing are each broken out into their own full backup. Example out of /var/cyrus/mail/ the subdir A-K are in one stream, L-Q in another, etc. I'm not an NBU admin so I dunno why this is

Re: Backup strategy for large mailbox stores

2010-02-15 Thread Vincent Fox
John Madden wrote: We did quite a bit with snapshots (LVM) to when we were experimenting with block-level backups but there's a performance problem there -- we were saturating GbE. Snapshot doesn't really buy you anything in terms of getting the data to tape. We run the tape backup from

Re: Backup strategy for large mailbox stores

2010-02-15 Thread John Madden
Out of curiousity, how good is zfs with full fs scans when running in the 100-million file count range? What do you see in terms of aggregate MB/s throughput? -- John madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmad...@ivytech.edu On Feb 15, 2010, at 15:43,

Re: Backup strategy for large mailbox stores

2010-02-15 Thread Vincent Fox
John Madden wrote: Out of curiousity, how good is zfs with full fs scans when running in the 100-million file count range? What do you see in terms of aggregate MB/s throughput? I'm not sure what you mean by full fs scan precisely, and haven't tested anything very large. Since the