Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
On 10/28/2011 6:35 AM, Arlin wrote: > How can we horizontally scale Dovecot across multiple servers? Do we require > to install independent instances of Dovecot on each server? The performance limiting factors for IMAP servers are disk IO, memory size, network bandwidth/latency, and processor power, usually in that order. To scale Dovecot horizontally would imply multiple servers each with independent storage handling a subset of the total concurrent user load. > We are planning to use a NAS/SAN device using ZFS or EFS for email storage. > Each logical unit will be of 10TB and similarly as the no: of user increases > we are planning to add multiple 10TB units. This does not describe horizontal maildir performance scaling. This simply describes vertical scaling of the IO, to a point, as the nodes are all attached to one controller with finite IOPS performance, which is dictated by the speed of the IO ASIC and cache memory bandwidth WRT writes, and aggregate spindle IOPS WRT reads. With this setup, you simply gain multi-node redundancy of the front end servers. However, you will *lose* performance compared to a single equivalent large host due to NFS/cluster filesystem overhead, latency, locking, etc, and your management/troubleshooting burden will be increased. If your goal is simply increased performance and storage expandability, go with a single large host system. I'd only go the cluster route if you believe you need the redundancy. -- Stan
Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
On 31/10/2011 11:28, Felipe Scarel wrote: > Quick question about the usage of DRBD: I'm thinking of a setup on my > organization here (15k+ users, 4TB of email data), but I'm holding back on > the clusterization due to the high volume of data. > > Using DRBD would implicate mirroring those 4TB of data across all cluster > nodes? If yes, I might go with a SAN-based solution, though I haven't I don't the technique with DRBD is something like having pairs of machines, each of which is a backup for the other. There were some old notes on the Dovecot website about such a setup? Roughly I seem to recall that each pair of machines ran two virtual machines, each of which ran active on one of the nodes each, but could migrate to the other if needed. Add a bunch of such paired nodes to get to the performance you require and put a dovecot proxy instance in front of the whole lot In contrast the SAN solution uses a clustered filesystem (opinion varies on which performs best) and then in theory every machine has access to every mailbox. In practice access to the SAN is relatively slow compared with local storage, so the technique seems to be to store indexes on the local machine and then using the front end proxy to be somewhat "sticky" in returning users to the same backend node so that the indexes can be re-used and not rebuilt The DRBD solution offers local disk access speed to the node and would on the surface give far faster performance (if disk were the limiting issue). However, it's likely to be more complex to maintain and manage and without buying licences you get only failover between pairs of machines. The SAN solution in theory looks like perfect scale up, big backend and just add more backend IMAP nodes as you need them, and all the clever stuff moves to the frontend load balancer to be "sticky" and obviously that's your main maintenance problem. However, based on evidence from users of big systems, IO is likely to be your main bottleneck and so just theoretically, the SAN will only scale as far as it doesn't run out of IOs... Using local disk for indexes would tend to reduce the amount of IOs needed (from the SAN) very dramatically, but you still have some limit out there and it's a question of whether you will reach it? DRBD has theoretical infinite scale out because each time you add another pair you get more IO as well as more CPU I don't have the fortune to have anything like the volume of users you have so I have no opinion to offer... However, I think the above accurately summarises your options. Others might help clarify the likely bounds on performance of each solution and maintenance headaches (eg some have had problems with maildir mounted on OCFS/GFS2 and fixed that by moving to dbox, etc) Please report on your results! Good luck Ed W
Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
There is, though not readily available unfortunately. Thanks for the input, always helpful. On Mon, Oct 31, 2011 at 10:33, Robert Schetterer wrote: > Am 31.10.2011 12:28, schrieb Felipe Scarel: > > Quick question about the usage of DRBD: I'm thinking of a setup on my > > organization here (15k+ users, 4TB of email data), but I'm holding back > > on the clusterization due to the high volume of data. > > > > Using DRBD would implicate mirroring those 4TB of data across all > > cluster nodes? If yes, I might go with a SAN-based solution, though I > > haven't studied much about that setup yet (the other sysadm > > administrates the VMs and SAN, gotta ask him a few questions). > > i wouldnt do that with drbd > as such big setup , there should be enough money using better choices > > > > On Mon, Oct 31, 2011 at 08:00, Robert Schetterer > <mailto:rob...@schetterer.org>> wrote: > > > > Am 31.10.2011 10:43, schrieb Arlin: > > > Hi Robert, > > > > > > Thanks for the reply. We are upgrading Dovecot to v2.0.15 and other > > > component's to the latest version. In that case, can we use san > > for storage > > > or are you recommending that drbd with ocfs2 is the best way to > > attain the > > > horizontal scalability for the mail storage? > > > > Hi Arlin, there is no best way, you should choose > > whatever fits best to your needs an haves > > so it depends on many stuff ( i.e at last finance, network, manpower, > > knowledge) etc > > > > So all i can say iam just using a loadbalanced cluster setup with > drbd > > ocfs2 maildir dovecot postfix mysql clamav spamassassin on ubuntu > lucid > > lts with 3000 Mailboxes without any big Problems yet > > but i can imagine that a professional SAN might be better in > performance > > but there is a lot other other questions left , i.e maildir must not > be > > the best solution for mailbox format etc > > > > cluster setups with lots of mailboxes are complex > > in many ways, if you planning a real big mailservice > > you should ask more here on this list for existing other setups and > > choose i.e > > Timo and/or others for professional and paid advice and work > > > > > > > > > > Thanks, > > > Arlin > > > > > > -Original Message----- > > > From: dovecot-boun...@dovecot.org > > <mailto:dovecot-boun...@dovecot.org> > > [mailto:dovecot-boun...@dovecot.org > > <mailto:dovecot-boun...@dovecot.org>] On > > > Behalf Of Robert Schetterer > > > Sent: 31 October 2011 14:26 > > > To: dovecot@dovecot.org <mailto:dovecot@dovecot.org> > > > Subject: Re: [Dovecot] How can we horizontally scale Dovecot > > across multiple > > > servers? > > > > > > Am 31.10.2011 09:47, schrieb Arlin: > > >> Could anyone please respond to this query. Thank you! > > > > > > you may use loadbalancers i.e (keepalived etc) and/or > > > > > > http://wiki2.dovecot.org/FeatLoginProxy > > > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy > > > > > > look the list for cluster setups etc > > > > > > reading your former post > > > you want to use many outdated prog versions dont do that a san for > > storage > > > might be a good choice some of the list use drbd with ocfs2 and > other > > > cluster filesystems > > > > > >> > > >> > > >> > > >> From: Arlin [mailto:ar...@mvs.us <mailto:ar...@mvs.us>] > > >> Sent: 28 October 2011 17:06 > > >> To: 'dovecot@dovecot.org <mailto:dovecot@dovecot.org>' > > >> Subject: How can we horizontally scale Dovecot across multiple > > servers? > > >> > > >> > > >> > > >> Hi, > > >> > > >> > > >> > > >> How can we horizontally scale Dovecot across multiple servers? Do > we > > >> require to install independent instances of Dovecot on each > server? > > >> > > &g
Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
Am 31.10.2011 12:28, schrieb Felipe Scarel: > Quick question about the usage of DRBD: I'm thinking of a setup on my > organization here (15k+ users, 4TB of email data), but I'm holding back > on the clusterization due to the high volume of data. > > Using DRBD would implicate mirroring those 4TB of data across all > cluster nodes? If yes, I might go with a SAN-based solution, though I > haven't studied much about that setup yet (the other sysadm > administrates the VMs and SAN, gotta ask him a few questions). i wouldnt do that with drbd as such big setup , there should be enough money using better choices > > On Mon, Oct 31, 2011 at 08:00, Robert Schetterer <mailto:rob...@schetterer.org>> wrote: > > Am 31.10.2011 10:43, schrieb Arlin: > > Hi Robert, > > > > Thanks for the reply. We are upgrading Dovecot to v2.0.15 and other > > component's to the latest version. In that case, can we use san > for storage > > or are you recommending that drbd with ocfs2 is the best way to > attain the > > horizontal scalability for the mail storage? > > Hi Arlin, there is no best way, you should choose > whatever fits best to your needs an haves > so it depends on many stuff ( i.e at last finance, network, manpower, > knowledge) etc > > So all i can say iam just using a loadbalanced cluster setup with drbd > ocfs2 maildir dovecot postfix mysql clamav spamassassin on ubuntu lucid > lts with 3000 Mailboxes without any big Problems yet > but i can imagine that a professional SAN might be better in performance > but there is a lot other other questions left , i.e maildir must not be > the best solution for mailbox format etc > > cluster setups with lots of mailboxes are complex > in many ways, if you planning a real big mailservice > you should ask more here on this list for existing other setups and > choose i.e > Timo and/or others for professional and paid advice and work > > > > > > Thanks, > > Arlin > > > > -Original Message- > > From: dovecot-boun...@dovecot.org > <mailto:dovecot-boun...@dovecot.org> > [mailto:dovecot-boun...@dovecot.org > <mailto:dovecot-boun...@dovecot.org>] On > > Behalf Of Robert Schetterer > > Sent: 31 October 2011 14:26 > > To: dovecot@dovecot.org <mailto:dovecot@dovecot.org> > > Subject: Re: [Dovecot] How can we horizontally scale Dovecot > across multiple > > servers? > > > > Am 31.10.2011 09:47, schrieb Arlin: > >> Could anyone please respond to this query. Thank you! > > > > you may use loadbalancers i.e (keepalived etc) and/or > > > > http://wiki2.dovecot.org/FeatLoginProxy > > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy > > > > look the list for cluster setups etc > > > > reading your former post > > you want to use many outdated prog versions dont do that a san for > storage > > might be a good choice some of the list use drbd with ocfs2 and other > > cluster filesystems > > > >> > >> > >> > >> From: Arlin [mailto:ar...@mvs.us <mailto:ar...@mvs.us>] > >> Sent: 28 October 2011 17:06 > >> To: 'dovecot@dovecot.org <mailto:dovecot@dovecot.org>' > >> Subject: How can we horizontally scale Dovecot across multiple > servers? > >> > >> > >> > >> Hi, > >> > >> > >> > >> How can we horizontally scale Dovecot across multiple servers? Do we > >> require to install independent instances of Dovecot on each server? > >> > >> > >> > >> We are planning to use a NAS/SAN device using ZFS or EFS for email > > storage. > >> Each logical unit will be of 10TB and similarly as the no: of user > >> increases we are planning to add multiple 10TB units. > >> > >> In this case how we can manage the email storage on multiple volumes > >> from Dovecot. > >> > >> > >> > >> The configuration of our existing system is:- > >> > >> > >> > >> Dovecot 1.0.15 / Maildirs > >> > >> Postfix 2.5.5 > >> > >> Debian 5.0.9 (Lenny) > >> > >> MySQL 5.0.15 > >> > >> > >> > >> Please advise. > >> > >> > >> > >> Thanks in advance. > >> > >> > >> > >> Creative Regards, > >> > >> Arlin > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > > > > > > -- > > Best Regards > > > > MfG Robert Schetterer > > > > Germany/Munich/Bavaria > > > > > -- > Best Regards > > MfG Robert Schetterer > > Germany/Munich/Bavaria > > -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
Quick question about the usage of DRBD: I'm thinking of a setup on my organization here (15k+ users, 4TB of email data), but I'm holding back on the clusterization due to the high volume of data. Using DRBD would implicate mirroring those 4TB of data across all cluster nodes? If yes, I might go with a SAN-based solution, though I haven't studied much about that setup yet (the other sysadm administrates the VMs and SAN, gotta ask him a few questions). On Mon, Oct 31, 2011 at 08:00, Robert Schetterer wrote: > Am 31.10.2011 10:43, schrieb Arlin: > > Hi Robert, > > > > Thanks for the reply. We are upgrading Dovecot to v2.0.15 and other > > component's to the latest version. In that case, can we use san for > storage > > or are you recommending that drbd with ocfs2 is the best way to attain > the > > horizontal scalability for the mail storage? > > Hi Arlin, there is no best way, you should choose > whatever fits best to your needs an haves > so it depends on many stuff ( i.e at last finance, network, manpower, > knowledge) etc > > So all i can say iam just using a loadbalanced cluster setup with drbd > ocfs2 maildir dovecot postfix mysql clamav spamassassin on ubuntu lucid > lts with 3000 Mailboxes without any big Problems yet > but i can imagine that a professional SAN might be better in performance > but there is a lot other other questions left , i.e maildir must not be > the best solution for mailbox format etc > > cluster setups with lots of mailboxes are complex > in many ways, if you planning a real big mailservice > you should ask more here on this list for existing other setups and > choose i.e > Timo and/or others for professional and paid advice and work > > > > > > Thanks, > > Arlin > > > > -Original Message- > > From: dovecot-boun...@dovecot.org [mailto:dovecot-boun...@dovecot.org] > On > > Behalf Of Robert Schetterer > > Sent: 31 October 2011 14:26 > > To: dovecot@dovecot.org > > Subject: Re: [Dovecot] How can we horizontally scale Dovecot across > multiple > > servers? > > > > Am 31.10.2011 09:47, schrieb Arlin: > >> Could anyone please respond to this query. Thank you! > > > > you may use loadbalancers i.e (keepalived etc) and/or > > > > http://wiki2.dovecot.org/FeatLoginProxy > > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy > > > > look the list for cluster setups etc > > > > reading your former post > > you want to use many outdated prog versions dont do that a san for > storage > > might be a good choice some of the list use drbd with ocfs2 and other > > cluster filesystems > > > >> > >> > >> > >> From: Arlin [mailto:ar...@mvs.us] > >> Sent: 28 October 2011 17:06 > >> To: 'dovecot@dovecot.org' > >> Subject: How can we horizontally scale Dovecot across multiple servers? > >> > >> > >> > >> Hi, > >> > >> > >> > >> How can we horizontally scale Dovecot across multiple servers? Do we > >> require to install independent instances of Dovecot on each server? > >> > >> > >> > >> We are planning to use a NAS/SAN device using ZFS or EFS for email > > storage. > >> Each logical unit will be of 10TB and similarly as the no: of user > >> increases we are planning to add multiple 10TB units. > >> > >> In this case how we can manage the email storage on multiple volumes > >> from Dovecot. > >> > >> > >> > >> The configuration of our existing system is:- > >> > >> > >> > >> Dovecot 1.0.15 / Maildirs > >> > >> Postfix 2.5.5 > >> > >> Debian 5.0.9 (Lenny) > >> > >> MySQL 5.0.15 > >> > >> > >> > >> Please advise. > >> > >> > >> > >> Thanks in advance. > >> > >> > >> > >> Creative Regards, > >> > >> Arlin > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > > > > > > -- > > Best Regards > > > > MfG Robert Schetterer > > > > Germany/Munich/Bavaria > > > > > -- > Best Regards > > MfG Robert Schetterer > > Germany/Munich/Bavaria >
Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
Am 31.10.2011 10:43, schrieb Arlin: > Hi Robert, > > Thanks for the reply. We are upgrading Dovecot to v2.0.15 and other > component's to the latest version. In that case, can we use san for storage > or are you recommending that drbd with ocfs2 is the best way to attain the > horizontal scalability for the mail storage? Hi Arlin, there is no best way, you should choose whatever fits best to your needs an haves so it depends on many stuff ( i.e at last finance, network, manpower, knowledge) etc So all i can say iam just using a loadbalanced cluster setup with drbd ocfs2 maildir dovecot postfix mysql clamav spamassassin on ubuntu lucid lts with 3000 Mailboxes without any big Problems yet but i can imagine that a professional SAN might be better in performance but there is a lot other other questions left , i.e maildir must not be the best solution for mailbox format etc cluster setups with lots of mailboxes are complex in many ways, if you planning a real big mailservice you should ask more here on this list for existing other setups and choose i.e Timo and/or others for professional and paid advice and work > > Thanks, > Arlin > > -Original Message- > From: dovecot-boun...@dovecot.org [mailto:dovecot-boun...@dovecot.org] On > Behalf Of Robert Schetterer > Sent: 31 October 2011 14:26 > To: dovecot@dovecot.org > Subject: Re: [Dovecot] How can we horizontally scale Dovecot across multiple > servers? > > Am 31.10.2011 09:47, schrieb Arlin: >> Could anyone please respond to this query. Thank you! > > you may use loadbalancers i.e (keepalived etc) and/or > > http://wiki2.dovecot.org/FeatLoginProxy > http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy > > look the list for cluster setups etc > > reading your former post > you want to use many outdated prog versions dont do that a san for storage > might be a good choice some of the list use drbd with ocfs2 and other > cluster filesystems > >> >> >> >> From: Arlin [mailto:ar...@mvs.us] >> Sent: 28 October 2011 17:06 >> To: 'dovecot@dovecot.org' >> Subject: How can we horizontally scale Dovecot across multiple servers? >> >> >> >> Hi, >> >> >> >> How can we horizontally scale Dovecot across multiple servers? Do we >> require to install independent instances of Dovecot on each server? >> >> >> >> We are planning to use a NAS/SAN device using ZFS or EFS for email > storage. >> Each logical unit will be of 10TB and similarly as the no: of user >> increases we are planning to add multiple 10TB units. >> >> In this case how we can manage the email storage on multiple volumes >> from Dovecot. >> >> >> >> The configuration of our existing system is:- >> >> >> >> Dovecot 1.0.15 / Maildirs >> >> Postfix 2.5.5 >> >> Debian 5.0.9 (Lenny) >> >> MySQL 5.0.15 >> >> >> >> Please advise. >> >> >> >> Thanks in advance. >> >> >> >> Creative Regards, >> >> Arlin >> >> >> >> >> >> >> >> >> >> >> >> > > > -- > Best Regards > > MfG Robert Schetterer > > Germany/Munich/Bavaria > -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
Hi Robert, Thanks for the reply. We are upgrading Dovecot to v2.0.15 and other component's to the latest version. In that case, can we use san for storage or are you recommending that drbd with ocfs2 is the best way to attain the horizontal scalability for the mail storage? Thanks, Arlin -Original Message- From: dovecot-boun...@dovecot.org [mailto:dovecot-boun...@dovecot.org] On Behalf Of Robert Schetterer Sent: 31 October 2011 14:26 To: dovecot@dovecot.org Subject: Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers? Am 31.10.2011 09:47, schrieb Arlin: > Could anyone please respond to this query. Thank you! you may use loadbalancers i.e (keepalived etc) and/or http://wiki2.dovecot.org/FeatLoginProxy http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy look the list for cluster setups etc reading your former post you want to use many outdated prog versions dont do that a san for storage might be a good choice some of the list use drbd with ocfs2 and other cluster filesystems > > > > From: Arlin [mailto:ar...@mvs.us] > Sent: 28 October 2011 17:06 > To: 'dovecot@dovecot.org' > Subject: How can we horizontally scale Dovecot across multiple servers? > > > > Hi, > > > > How can we horizontally scale Dovecot across multiple servers? Do we > require to install independent instances of Dovecot on each server? > > > > We are planning to use a NAS/SAN device using ZFS or EFS for email storage. > Each logical unit will be of 10TB and similarly as the no: of user > increases we are planning to add multiple 10TB units. > > In this case how we can manage the email storage on multiple volumes > from Dovecot. > > > > The configuration of our existing system is:- > > > > Dovecot 1.0.15 / Maildirs > > Postfix 2.5.5 > > Debian 5.0.9 (Lenny) > > MySQL 5.0.15 > > > > Please advise. > > > > Thanks in advance. > > > > Creative Regards, > > Arlin > > > > > > > > > > > > -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
Am 31.10.2011 09:47, schrieb Arlin: > Could anyone please respond to this query. Thank you! you may use loadbalancers i.e (keepalived etc) and/or http://wiki2.dovecot.org/FeatLoginProxy http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy look the list for cluster setups etc reading your former post you want to use many outdated prog versions dont do that a san for storage might be a good choice some of the list use drbd with ocfs2 and other cluster filesystems > > > > From: Arlin [mailto:ar...@mvs.us] > Sent: 28 October 2011 17:06 > To: 'dovecot@dovecot.org' > Subject: How can we horizontally scale Dovecot across multiple servers? > > > > Hi, > > > > How can we horizontally scale Dovecot across multiple servers? Do we require > to install independent instances of Dovecot on each server? > > > > We are planning to use a NAS/SAN device using ZFS or EFS for email storage. > Each logical unit will be of 10TB and similarly as the no: of user increases > we are planning to add multiple 10TB units. > > In this case how we can manage the email storage on multiple volumes from > Dovecot. > > > > The configuration of our existing system is:- > > > > Dovecot 1.0.15 / Maildirs > > Postfix 2.5.5 > > Debian 5.0.9 (Lenny) > > MySQL 5.0.15 > > > > Please advise. > > > > Thanks in advance. > > > > Creative Regards, > > Arlin > > > > > > > > > > > > -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] How can we horizontally scale Dovecot across multiple servers?
Could anyone please respond to this query. Thank you! From: Arlin [mailto:ar...@mvs.us] Sent: 28 October 2011 17:06 To: 'dovecot@dovecot.org' Subject: How can we horizontally scale Dovecot across multiple servers? Hi, How can we horizontally scale Dovecot across multiple servers? Do we require to install independent instances of Dovecot on each server? We are planning to use a NAS/SAN device using ZFS or EFS for email storage. Each logical unit will be of 10TB and similarly as the no: of user increases we are planning to add multiple 10TB units. In this case how we can manage the email storage on multiple volumes from Dovecot. The configuration of our existing system is:- Dovecot 1.0.15 / Maildirs Postfix 2.5.5 Debian 5.0.9 (Lenny) MySQL 5.0.15 Please advise. Thanks in advance. Creative Regards, Arlin
[Dovecot] How can we horizontally scale Dovecot across multiple servers?
Hi, How can we horizontally scale Dovecot across multiple servers? Do we require to install independent instances of Dovecot on each server? We are planning to use a NAS/SAN device using ZFS or EFS for email storage. Each logical unit will be of 10TB and similarly as the no: of user increases we are planning to add multiple 10TB units. In this case how we can manage the email storage on multiple volumes from Dovecot. The configuration of our existing system is:- Dovecot 1.0.15 / Maildirs Postfix 2.5.5 Debian 5.0.9 (Lenny) MySQL 5.0.15 Please advise. Thanks in advance. Creative Regards, Arlin
