Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-28 Thread Sven Hartge
Joseph Tam jtam.h...@gmail.com wrote: Sven Hartge s...@svenhartge.de wrote: Interesting datapoint: NetApp Deduplication did only recover about 1% of storage space with mdbox-based mail storage, while on an maildir-based mail storage, the rate was about 15%. (This was tested with a copy of

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-28 Thread Joseph Tam
Just a guess, but I expect the difference is because NetApp de-dupes by checksumming blocks and mark whole blocks as duplicates if they have the same checksum. True, the start of the message is always at byte 0, but because of different header length per user for the same message (different

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-27 Thread Joseph Tam
Sven Hartge s...@svenhartge.de wrote: Interesting datapoint: NetApp Deduplication did only recover about 1% of storage space with mdbox-based mail storage, while on an maildir-based mail storage, the rate was about 15%. (This was tested with a copy of real user data, so is accurate for my

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-27 Thread Thomas Johnson
Hi Stan- (Stan gives a great run-down on the economics of using a NetApp or even homegrown NFS filer versus using an object storage backend.) Tom I'm sorry I wasted your time with my initial response. No, you absolutely didn't waste my time, and it was certainly of great advantage to

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-27 Thread Stan Hoeppner
On 1/27/2014 11:25 PM, Thomas Johnson wrote: Hi Stan- (Stan gives a great run-down on the economics of using a NetApp or even homegrown NFS filer versus using an object storage backend.) Tom I'm sorry I wasted your time with my initial response. No, you absolutely didn't waste my

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-26 Thread Tom Johnson
On Jan 24, 2014, at 7:23 PM, Stan Hoeppner s...@hardwarefreak.com wrote: On 1/24/2014 11:09 AM, Tom Johnson wrote: Is anybody using the Object Storage plugin for large-scale installations? I've not used it. We're considering it, but are thinking of an in-house S3 storage system

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-24 Thread Stan Hoeppner
Sven, why didn't you chime in? Your setup is similar scale and I think your insights would be valuable here. Or maybe you could repost your last on this topic. Or was that discussion off list? I can't recall. Anyway, I missed this post Murray. Thanks Ed for drudging this up. Maybe this will

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-24 Thread Javier de Miguel Rodríguez
Great mail, Stan Another trick: you can save storage (both space iops) using mdox and compression. CPU power is far cheaper than iops , the less data you read/write, the fewer iops. You can use gzip,bzip2 or even LZMA/xz compression for LDA. If you also use Single Instace Storage and

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-24 Thread Sven Hartge
Stan Hoeppner s...@hardwarefreak.com wrote: Sven, why didn't you chime in? Your setup is similar scale and I think your insights would be valuable here. Or maybe you could repost your last on this topic. Or was that discussion off list? I can't recall. Rather busy right now with a large

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-24 Thread Urban Loesch
Hi, and some other Dovecot mailing list threads but I am not sure how many users such a setup will handle. I have a concern about the I/O performance of NFS in the suggested architecture above. One possible option available to us is to split up the mailboxes over multiple clusters with

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-24 Thread Rick Romero
Quoting Urban Loesch b...@enas.net: Hi, and some other Dovecot mailing list threads but I am not sure how many users such a setup will handle.  I have a concern about the I/O performance of NFS in the suggested architecture above.  One possible option available to us is to split up the

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-24 Thread Urban Loesch
Am 24.01.2014 16:15, schrieb Rick Romero: - all Backends are in HA with a passive machine and DRBD with 10GBIT Cross Links How do you do backups? The underlying storage is based on lvm. So we can take a daily snapshot on the passive server, mount them readonly and have no load impact

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-24 Thread Stan Hoeppner
On 1/24/2014 6:24 AM, Javier de Miguel Rodríguez wrote: Great mail, Stan Another trick: you can save storage (both space iops) using mdox and compression. CPU power is far cheaper than iops , the less data you read/write, the fewer iops. Yeah, the cost of enterprise storage is

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-24 Thread Stan Hoeppner
This went to me only so bringing back on list. On 1/24/2014 11:09 AM, Tom Johnson wrote: Is anybody using the Object Storage plugin for large-scale installations? I've not used it. We're considering it, but are thinking of an in-house S3 storage system (riak, or ceph, or ?) Looking to

Re: [Dovecot] Architecture for large Dovecot cluster (employ an expert!)

2014-01-23 Thread Ed W
Hi and some other Dovecot mailing list threads but I am not sure how many users such a setup will handle. I have a concern about the I/O performance of NFS in the suggested architecture above. One possible option available to us is to split up the mailboxes over multiple clusters with subsets

Re: [Dovecot] Architecture for large Dovecot cluster (employ an expert!)

2014-01-23 Thread Charles Marcus
On 2014-01-23 11:57 AM, Ed W li...@wildgooses.com wrote: I'm very satisfied and have to highly recommend Timo. His prices were extremely reasonable and he offered service excellent. ...snip... Please feel encouraged to employ Timo if you use Dovecot! I will add a hearty 'seconded!' to this

[Dovecot] Architecture for large Dovecot cluster

2014-01-05 Thread Murray Trainer
Hi All, I am trying to determine whether a mail server cluster based on Dovecot will be capable of supporting 500,000+ mailboxes with about 50,000 IMAP and 5000 active POP3 connections. I have looked at the Dovecot clustering suggestions here:

Re: [Dovecot] Architecture for large Dovecot cluster

2014-01-05 Thread Robert Schetterer
Am 05.01.2014 14:06, schrieb Murray Trainer: Hi All, I am trying to determine whether a mail server cluster based on Dovecot will be capable of supporting 500,000+ mailboxes with about 50,000 IMAP and 5000 active POP3 connections. I have looked at the Dovecot clustering suggestions here: