Re: Dovecot cluster using GlusterFS

[Filip Pytloun]

Hello Alessio and Gordon, thank you for answers.

Dsync-based architecture looks promising, but I would preffer to stay
with GlusterFS for now as I also use it as a storage for other
components.

So director is the way to go, I don't want to setup more than two nodes
to keep this setup as simple as possible - so I will probably update to
2.2.19 and have director and backend on the same servers (and Dovecot
instance).

I asked about poolmon, because I think that Dovecot should have some
internal mechanism on how to recognize broken backend by default.
But if it works nicely, I am going to use it as well :-)

> At the moment, I cannot recognize the requirement for using lmtp over
> the directors. When using postfix for delivering e-mails to the
> backend, do this directly with an corresponding MX record.

I have two MX records of the same weight with postfix using dovecot-lmtp
for delivery. So that's why I wanted to use LMTP over directors.
Using lower weight for second MX is an option, but not truly
master-master setup :-)

Filip

On 2015/12/06 02:31, Alessio Cecchi wrote:
> Il 05.12.2015 10:42 Filip Pytloun ha scritto:
> >Hello,
> >
> >I have recently setup mailserver solution using 2-node master-master
> >setup (mainly based on MySQL M-M replication and GlusterFS with 2
> >replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
> >
> >Unfortunately even with shared-storage-aware setting:
> >
> > mail_nfs_index = yes
> > mail_nfs_storage = yes
> > mail_fsync = always
> > mmap_disable = yes
> 
> With only these setting you don't solve the problem of shared storage.
> 
> >..I have hit strange issues pretty soon especially when user was
> >manipulating same mailbox from multiple devices at the same time.
> >
> >Most issues was about corrupted indexes which was solved easily by just
> >putting them on local storage of each node:
> >
> > mail_location =
> >maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
> >
> >But I still hit issues like this one:
> >
> > dovecot: lmtp(6276, u...@example.com): Error: Broken file
> >/srv/mail/example.com/u...@example.com/dovecot-uidlist line 8529: UIDs
> >not ordered (8527 >= 8527)
> >
> >Which I am not sure how serious it is or if it's possible to solve or
> >workaround?
> 
> You need Director for POP/IMAP and also LMTP so you can solve all "Broken
> file" and "corrupted indexes" problems.
> 
> >
> >Anyway because of the above and high possibility of GlusterFS
> >split-brains, I have decided to setup Dovecot Director according to the
> >docs [1] but I have a couple of questions:
> >
> >- is custom monitoring still required? Poolmon [2] is 4 year old so I
> >  would suppose there's some progress since that?
> 
> For me poolmon works fine.
> 
> >- it's not possible to have same backends and directors in Dovecot
> >  <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is
> >  not an issue, but..
> 
> Yes is possibile (also with < 2.2.17), create two instances, like dovecot
> and director, two config directory /etc/dovecot/ and /etc/director/ and bind
> on differents IPs.
> 
> >- documentation states that it still doesn't work for LMTP [3]?
> >  Which is probably important for my setup, because both Postfix servers
> >  are using dovecot-lmtp for mail delivery so there can be still some
> >  issues (but probably less frequent?) when both servers will deliver
> >  new mails for one user at once.
> >  So do I really have to split directors from backends?
> 
> I'm running Director and backend on the same server for POP/IMAP, and in
> another configuration and Director for LMTP is on the same server (but with
> 2.2.19).
> 
> >Anyone has experience with clustered Dovecot setup?
> >Why is Dovecot behaving so bad when it pretends to be shared storage
> >friendly? Are these issues only specific for older Dovecot?
> >Or is there something wrong in my architecture design?
> 
> You need Director, Dovecot has not problems with shared storage, big
> installation are always using shared storage (like NFS).
> -- 
> Alessio Cecchi
> Postmaster AT http://www.qboxmail.it
> http://www.linkedin.com/in/alessice


signature.asc
Description: Digital signature

Re: Dovecot cluster using GlusterFS

[Timo Sirainen]

On 05 Dec 2015, at 11:42, Filip Pytloun  wrote:
> 
> Anyway because of the above and high possibility of GlusterFS
> split-brains, I have decided to setup Dovecot Director according to the
> docs [1] but I have a couple of questions:
> 
> - is custom monitoring still required? Poolmon [2] is 4 year old so I
>  would suppose there's some progress since that?

I think it's always going to be a separate script. In different environments 
people may want to do it slightly differently.

> - documentation states that it still doesn't work for LMTP [3]?
>  Which is probably important for my setup, because both Postfix servers
>  are using dovecot-lmtp for mail delivery so there can be still some
>  issues (but probably less frequent?) when both servers will deliver
>  new mails for one user at once.
>  So do I really have to split directors from backends?

You can run director and backend in the same servers, but they'd have to be in 
different config files (so two dovecot instances). Or you could also do it with 
a single instance:

> [3] "LMTP however doesn't currently support mixing recipients to both
> being proxied and store locally."

This is a problem only if there are multiple recipients in the same LMTP 
session. So if you configure your MTA to restrict the recipient limit to 1, 
this mixing can't happen and a single Dovecot instance can work.

v2.3 will solve this problem completely.

Re: Dovecot cluster using GlusterFS

[l...@airstreamcomm.net]

We ran a load test using glusterfs and were able to deliver mail (I can't 
remember specifically how much per second, maybe 100 messages per second?) 
without any issues.  We did use the glusterfs fuse client and not nfs, and used 
regular maildir.  We developed a mail bot cluster that would deliver mail, and 
simultaneously receive and delete it with pop and IMAP and we ran into zero 
issues.  We even had the replicas stretched between two datacenters.  Not sure 
what the difference here is but it can be done.



> On Dec 5, 2015, at 3:42 AM, Filip Pytloun  wrote:
> 
> Hello,
> 
> I have recently setup mailserver solution using 2-node master-master
> setup (mainly based on MySQL M-M replication and GlusterFS with 2
> replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
> 
> Unfortunately even with shared-storage-aware setting:
> 
> mail_nfs_index = yes
> mail_nfs_storage = yes
> mail_fsync = always
> mmap_disable = yes
> 
> ..I have hit strange issues pretty soon especially when user was
> manipulating same mailbox from multiple devices at the same time.
> 
> Most issues was about corrupted indexes which was solved easily by just
> putting them on local storage of each node:
> 
> mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
> 
> But I still hit issues like this one:
> 
> dovecot: lmtp(6276, u...@example.com): Error: Broken file 
> /srv/mail/example.com/u...@example.com/dovecot-uidlist line 8529: UIDs not 
> ordered (8527 >= 8527)
> 
> Which I am not sure how serious it is or if it's possible to solve or
> workaround?
> 
> Anyway because of the above and high possibility of GlusterFS
> split-brains, I have decided to setup Dovecot Director according to the
> docs [1] but I have a couple of questions:
> 
> - is custom monitoring still required? Poolmon [2] is 4 year old so I
>  would suppose there's some progress since that?
> 
> - it's not possible to have same backends and directors in Dovecot
>  <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is
>  not an issue, but..
> 
> - documentation states that it still doesn't work for LMTP [3]?
>  Which is probably important for my setup, because both Postfix servers
>  are using dovecot-lmtp for mail delivery so there can be still some
>  issues (but probably less frequent?) when both servers will deliver
>  new mails for one user at once.
>  So do I really have to split directors from backends?
> 
> 
> Anyone has experience with clustered Dovecot setup?
> Why is Dovecot behaving so bad when it pretends to be shared storage
> friendly? Are these issues only specific for older Dovecot?
> Or is there something wrong in my architecture design?
> 
> Thanks for any help,
> Filip
> 
> 
> ---
> [1] http://wiki2.dovecot.org/Director
> [2] https://github.com/brandond/poolmon/
> [3] "LMTP however doesn't currently support mixing recipients to both
> being proxied and store locally."
> 
> ---
> BTW if someone is interested in SaltStack, here are Salt formulas for
> Dovecot + Postfix + GlusterFS + Roundcube + Mailman setup which we are
> using:
> 
> https://github.com/tcpcloud/salt-formula-dovecot
> https://github.com/tcpcloud/salt-formula-postfix
> https://github.com/tcpcloud/salt-formula-roundcube
> https://github.com/tcpcloud/salt-formula-glusterfs

Re: Dovecot cluster using GlusterFS

[Gordon Grubert]

On 12/06/2015 10:16 AM, Filip Pytloun wrote:

At the moment, I cannot recognize the requirement for using lmtp over
the directors. When using postfix for delivering e-mails to the
backend, do this directly with an corresponding MX record.


I have two MX records of the same weight with postfix using dovecot-lmtp
for delivery. So that's why I wanted to use LMTP over directors.
Using lower weight for second MX is an option, but not truly
master-master setup :-)


We're using postfix for delivering to the dovecot backends via lmtp on
the base of an MX record with the same weight. Here, postfix can
directly access the dovecot backends without using the dovecot
directors. It a really master-master setup ;-)

Best regards,
Gordon

Re: Dovecot cluster using GlusterFS

[Michael Schwartzkopff]

Am Sonntag, 6. Dezember 2015, 20:33:32 schrieb l...@airstreamcomm.net:
> We ran a load test using glusterfs and were able to deliver mail (I can't
> remember specifically how much per second, maybe 100 messages per second?)
> without any issues.  We did use the glusterfs fuse client and not nfs, and
> used regular maildir.  We developed a mail bot cluster that would deliver
> mail, and simultaneously receive and delete it with pop and IMAP and we ran
> into zero issues.  We even had the replicas stretched between two
> datacenters.  Not sure what the difference here is but it can be done.
> > On Dec 5, 2015, at 3:42 AM, Filip Pytloun  wrote:
> > 
> > Hello,
> > 
> > I have recently setup mailserver solution using 2-node master-master
> > setup (mainly based on MySQL M-M replication and GlusterFS with 2
> > replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).
> > 
> > Unfortunately even with shared-storage-aware setting:
> > 
> > mail_nfs_index = yes
> > mail_nfs_storage = yes
> > mail_fsync = always
> > mmap_disable = yes
> > 
> > ..I have hit strange issues pretty soon especially when user was
> > manipulating same mailbox from multiple devices at the same time.
> > 
> > Most issues was about corrupted indexes which was solved easily by just
> > putting them on local storage of each node:
> > 
> > mail_location = maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u
> > 
> > But I still hit issues like this one:
> > 
> > dovecot: lmtp(6276, u...@example.com): Error: Broken file
> > /srv/mail/example.com/u...@example.com/dovecot-uidlist line 8529: UIDs
> > not ordered (8527 >= 8527)
> > 
> > Which I am not sure how serious it is or if it's possible to solve or
> > workaround?

hi,

I did experiments with glusterfs and dovecot about one year ago and ran into 
the same trouble. It was quite easy to corrupt users mailboxes delivering mail 
somultaniously on different nodes.

I tried a lot of different configurations (see NFS recomendations, ...) but 
nothig really solved the problem.

Also from the list I did not get any useful comments.

So I decided that plain dovecot / glusterfs is not usable.

I would be glad to hear of any other experiance and configuration tweaks.

Mit freundlichen Grüßen,

Michael Schwartzkopff

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64, +49 (162) 165 0044
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

signature.asc
Description: This is a digitally signed message part.

Re: Dovecot cluster using GlusterFS

[Gordon Grubert]

Hi Filip,

On 12/05/2015 10:42 AM, Filip Pytloun wrote:

I have recently setup mailserver solution using 2-node master-master
setup (mainly based on MySQL M-M replication and GlusterFS with 2
replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).


that's no good idea due to different reasons - see below.


Anyway because of the above and high possibility of GlusterFS
split-brains, I have decided to setup Dovecot Director according to the
docs [1] but I have a couple of questions:

- is custom monitoring still required? Poolmon [2] is 4 year old so I
   would suppose there's some progress since that?


Using the dovecot director, poolmon is strongly recommended - see the
official dovecot wiki. The tool seems to be old, but the task is the
same since a couple of years ;-)


- it's not possible to have same backends and directors in Dovecot
   <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is
   not an issue, but..


You have to use 2 instances of dovecot when running the director and
the backend on the same server. Otherwise, you have to use 2 systems.


- documentation states that it still doesn't work for LMTP [3]?
   Which is probably important for my setup, because both Postfix servers
   are using dovecot-lmtp for mail delivery so there can be still some
   issues (but probably less frequent?) when both servers will deliver
   new mails for one user at once.
   So do I really have to split directors from backends?


At the moment, I cannot recognize the requirement for using lmtp over
the directors. When using postfix for delivering e-mails to the
backend, do this directly with an corresponding MX record.


Why is Dovecot behaving so bad when it pretends to be shared storage
friendly? Are these issues only specific for older Dovecot?
Or is there something wrong in my architecture design?


IMHO, this is not a problem of dovecot. In general, using cluster
filesystems in such a setup is no good idea. You've already mentioned
all the problems of such a setup. In particular, the performance of
GlusterFS is absolutely not suitable for (bigger) mailserver cluster.

Asking any search engine for "dovecot clustering" shows a lot of
results for good cluster designs. In particular, a shared mailbox
storage should be avoided. Here, the dovecot internal replication
mechanism on the backends seems to be the best solution. The rest of
the setup (e.g., clustered directors or smtp servers) is trivial. Due
to the fact, that MUAs are able to use A records only and the behavior
using round robin A records is catastrophically, a service IP address
is recommended. Here, keepalived is your friend.

HTH.

Best regards,
Gordon
--
Technischer Leiter & stellv. Direktor
Universitätsrechenzentrum (URZ)
E.-M.-Arndt-Universität Greifswald
Felix-Hausdorff-Str. 12
17489 Greifswald
Germany

Tel. +49 3834 86 1456
Fax. +49 3834 86 1401

Re: Dovecot cluster using GlusterFS

[Alessio Cecchi]

Il 05.12.2015 10:42 Filip Pytloun ha scritto:

Hello,

I have recently setup mailserver solution using 2-node master-master
setup (mainly based on MySQL M-M replication and GlusterFS with 2
replica volume) on Ubuntu 14.04 (Dovecot 2.2.9).

Unfortunately even with shared-storage-aware setting:

 mail_nfs_index = yes
 mail_nfs_storage = yes
 mail_fsync = always
 mmap_disable = yes


With only these setting you don't solve the problem of shared storage.


..I have hit strange issues pretty soon especially when user was
manipulating same mailbox from multiple devices at the same time.

Most issues was about corrupted indexes which was solved easily by just
putting them on local storage of each node:

 mail_location = 
maildir:/srv/mail/%d/%u:INDEX=/var/lib/dovecot/index/%d/%u


But I still hit issues like this one:

 dovecot: lmtp(6276, u...@example.com): Error: Broken file
/srv/mail/example.com/u...@example.com/dovecot-uidlist line 8529: UIDs
not ordered (8527 >= 8527)

Which I am not sure how serious it is or if it's possible to solve or
workaround?


You need Director for POP/IMAP and also LMTP so you can solve all 
"Broken file" and "corrupted indexes" problems.




Anyway because of the above and high possibility of GlusterFS
split-brains, I have decided to setup Dovecot Director according to the
docs [1] but I have a couple of questions:

- is custom monitoring still required? Poolmon [2] is 4 year old so I
  would suppose there's some progress since that?


For me poolmon works fine.


- it's not possible to have same backends and directors in Dovecot
  <2.2.17. I can backport newer Dovecot for Ubuntu Trusty, so this is
  not an issue, but..


Yes is possibile (also with < 2.2.17), create two instances, like 
dovecot and director, two config directory /etc/dovecot/ and 
/etc/director/ and bind on differents IPs.



- documentation states that it still doesn't work for LMTP [3]?
  Which is probably important for my setup, because both Postfix 
servers

  are using dovecot-lmtp for mail delivery so there can be still some
  issues (but probably less frequent?) when both servers will deliver
  new mails for one user at once.
  So do I really have to split directors from backends?


I'm running Director and backend on the same server for POP/IMAP, and in 
another configuration and Director for LMTP is on the same server (but 
with 2.2.19).



Anyone has experience with clustered Dovecot setup?
Why is Dovecot behaving so bad when it pretends to be shared storage
friendly? Are these issues only specific for older Dovecot?
Or is there something wrong in my architecture design?


You need Director, Dovecot has not problems with shared storage, big 
installation are always using shared storage (like NFS).

--
Alessio Cecchi
Postmaster AT http://www.qboxmail.it
http://www.linkedin.com/in/alessice