Re: [mailop] Message Store on NFS in a high available setup
Hi, Luis, William, Steve, Noel, Kurt, Andris and all others who responded off-list, many thanks for answering my mail. Apologies for my late reply, I've been quite busy trying to pinpoint a mail server/storage problem, see below. Steve wrote: In this situation NFS is more of a delivery method than anything to do with high availability, as it has no requirement to do any more than share a single copy of the data. Maybe a clustered file system - glusterfs for example - is what you're looking for. Correct, I used the term 'NFS' but what I meant to say was NFS to access the message store, where the backend for this message store has a HA architecture. On 28-09-17 02:57, W Kern wrote: We are huge GlusterFS fans. It is easy to setup, trivial to admin and very reliable if you don't get too fancy. Biggest issue is 'growing' it as you have to be precise in how you add additional bricks. However, historically the small-file performance on Gluster has not been very good, though they are working on it. Thus a Maildir based system would not (yet) be a good fit on Gluster, especially with lots of customers who leave tens of thousands of individual messages in a particular Folder. Other OpenSource 'white box' HA options are 1) Ceph 2) MooseFS/LizardFS 3) A big NFS server with DRBD as a failover. Each of those would have its challenges as you get into the millions of busy accounts stage, but there are work arounds for each. The mail server for which I was looking for this storage solution has a maildir based format. Actually, we used GlusterFS with the FUSE client but got some serious problems with GlusterFS, where index files (and some other files) got lost due to some problem in GFS. Redhat claimed there was 'some client process' removing these files. But we were able to reproduce the problem in a test environment and we could demonstrate that the problem was solved, when replacing the FUSE client with the plain vanilla NFS client that comes with Redhat (against the NFS interface of GFS). However, GFS does not provide a HA NFS solution without Ganesha and we didn't like to build another layer of complexity on top of GFS to solve the problems in GFS. Hence my question on this list. Redhat warned us that GFS is not ideal for handling lots of small files (where they mentioned everything under 1 Mbyte as being small), but we had to use GFS as that was the only HA shared storage service available at this customer; no NAS was present nor any intention to purchase a NAS for this purpose. But things may change now. Noel Butler wrote: FFS, do NOT use virtual machines :) I think I fully agree with you :-) but as I need figures and facts I'd like to ask you: can you elaborate on why not? Of course physical hardware has major advantages re. speed etc., but as everything these days get virtualized, virtualizing shared storage (as we did with GFS) has it's advantages too (scaling, provisioning etc.) Regards, /rolf ___ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
Re: [mailop] Message Store on NFS in a high available setup
It’s not exactly NFS, we’re experimenting with a custom built mail server that uses clustered (sharded+replicated) MongoDB for mail store. Emails are stored in a custom MongoDB optmized format which allows for example to deduplicate and decode attachments to save storage space. In the end we want to migrate ~30TB of messages from our existing Courier based system to this new one we’re building but for now the largest test system deployed handles around 800GB, so it’s not too well tested yet to be suitable for actual large scale production, hopefully we get there in a year or so. For smaller installs it should be in a good enough state already. You can find the sources here: https://github.com/nodemailer/wildduck Best regards, Andris Reinman > On 28 Sep 2017, at 01:33, Rolf E. Sonneveld > wrote: > > Hi, > > is there anyone working in a larger/'more demanding' environment, using NFS > for his/her message store, where there is a business requirement of High > Availability? If so, what commercial or open source storage solution is used? > I have a similar requirement for a project and am looking for > suggestions/options for what storage to use. > > Regards, > /rolf > > ___ > mailop mailing list > mailop@mailop.org > https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop ___ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
Re: [mailop] Message Store on NFS in a high available setup
Hi! > is there anyone working in a larger/'more demanding' environment, using > NFS for his/her message store, where there is a business requirement of > High Availability? If so, what commercial or open source storage > solution is used? I have a similar requirement for a project and am > looking for suggestions/options for what storage to use. Ask the dovecot folks, their open-source imap server has a professional service team: https://www.open-xchange.com/portfolio/dovecot-pro/ Open-xchange recently pooled dovecot, powerdns and open-xchange in one company. -- p...@opsec.eu+49 171 3101372 3 years to go ! ___ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
Re: [mailop] Message Store on NFS in a high available setup
On 28/09/2017 08:33, Rolf E. Sonneveld wrote: > Hi, > > is there anyone working in a larger/'more demanding' environment, using NFS > for his/her message store, where there is a business requirement of High > Availability? If so, what commercial or open source storage solution is used? > I have a similar requirement for a project and am looking for > suggestions/options for what storage to use. > > Regards, > /rolf Most SP's have a need to HA, to avoid their support desks being flooded, which doesnt look good, NetApp and EMC have excellent options for backends, we use NFS as its perfect, so long as using maildir and not mbox, we found NFS far more overall reliable than a clustered filesystem in a uptime critical and busy environment, and its a cinch to add/remove servers multiple front ends, using, eg: postfix and dovecot, with mysql/mariadb replicated on each server using a hardware load balancer, since we NFS mount mail dir we use dovecots -lda and not lmtp, and do not use its "director" service, which forces users to a specific machine. With dovecot you can make filesystem efficient structure if you have a lot of users, the one we use is (I'll cheat - copy and paste) this format, which gives jsm...@example.com /vmail/example.com/j/s/m/jsmith mail_location = maildir:/var/vmail/%Ld/%1Ln/%1.1Ln/%2.1Ln/%Ln/Maildir on smtp/pop3 servers also add to above :INDEX=MEMORY - but dont ever use that on imap I'd suggest keeping outbound mail servers separate to inbound, and as last bit of advice, given this is HA - FFS, do NOT use virtual machines :) -- Kind Regards, Noel Butler This Email, including any attachments, may contain legally privileged information, therefore remains confidential and subject to copyright protected under international law. You may not disseminate, discuss, or reveal, any part, to anyone, without the authors express written authority to do so. If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments, immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message. Only PDF [1] and ODF [2] documents accepted, please do not send proprietary formatted documents Links: -- [1] http://www.adobe.com/ [2] http://en.wikipedia.org/wiki/OpenDocument___ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
Re: [mailop] Message Store on NFS in a high available setup
On 9/27/2017 5:14 PM, st...@greengecko.co.nz wrote: In this situation NFS is more of a delivery method than anything to do with high availability, as it has no requirement to do any more than share a single copy of the data. Maybe a clustered file system - glusterfs for example - is what you're looking for. We are huge GlusterFS fans. It is easy to setup, trivial to admin and very reliable if you don't get too fancy. Biggest issue is 'growing' it as you have to be precise in how you add additional bricks. However, historically the small-file performance on Gluster has not been very good, though they are working on it. Thus a Maildir based system would not (yet) be a good fit on Gluster, especially with lots of customers who leave tens of thousands of individual messages in a particular Folder. Other OpenSource 'white box' HA options are 1) Ceph 2) MooseFS/LizardFS 3) A big NFS server with DRBD as a failover. Each of those would have its challenges as you get into the millions of busy accounts stage, but there are work arounds for each. William Kern ___ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
Re: [mailop] Message Store on NFS in a high available setup
In this situation NFS is more of a delivery method than anything to do with high availability, as it has no requirement to do any more than share a single copy of the data. Maybe a clustered file system - glusterfs for example - is what you're looking for. Steve September 28, 2017 11:39 AM, "Rolf E. Sonneveld" wrote: > Hi, > > is there anyone working in a larger/'more demanding' environment, using NFS > for his/her message > store, where there is a business requirement of High Availability? If so, > what commercial or open > source storage solution is used? I have a similar requirement for a project > and am looking for > suggestions/options for what storage to use. > > Regards, > /rolf > > ___ > mailop mailing list > mailop@mailop.org > https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop ___ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
Re: [mailop] Message Store on NFS in a high available setup
On 27 Sep 2017, at 15:33, Rolf E. Sonneveld wrote: is there anyone working in a larger/'more demanding' environment, using NFS for his/her message store, where there is a business requirement of High Availability? If so, what commercial or open source storage solution is used? I have a similar requirement for a project and am looking for suggestions/options for what storage to use. My last large deployment used NetApp NFS storage together with a custom mailbox format based on Maildir++. Nowadays with NFSv4 and Maildir++ you should be fine. My setup handled ~7 million accounts. Ping me if you want further info. Best regards -lem ___ mailop mailing list mailop@mailop.org https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop
