Re: multiple messages per second to a single mailbox
On Aug 17, 2015, at 9:30 AM, Gregory Finch wrote: <..snip..> > With that many messages, it may be the index updates that are slowing > things down. Depending on your needs or how you use this folder (it > sounds automated to me), you may not need the indexes on disk at all. If > you set the indexes to RAM, it may help. > > mail_location = maildir:~/Maildir:INDEX=MEMORY > > as per http://wiki2.dovecot.org/MailLocation#Index_files > > If indexes in RAM isn't an option, maybe just put them on a separate > spindle / SSD would help. > > Please note I'm shooting from the hip with this advice, I don't have a > system that pushes dovecot that hard. > > Greg > > This particular mailbox is unique. This mailbox holds a copy of member to member communications. The mailbox also receives a lot of unfiltered spam. You're right, it is the index updates that were limiting the message insertion rate. I wrote a sieve script that divides the messages based on the first character of the message-id header. For now this appears to be enough to spread out the index updates. The performance metrics indicate I'm now seeing double the IOPS as before the sieve script. With no noticeable increase in IO wait times. If the sieve script ever stops being enough, then I'll look into moving this mailboxes index files to RAM. Thank you for the tip! Regards, -Chad
Re: multiple messages per second to a single mailbox
On Aug 14, 2015, at 1:01 PM, Andrzej A. Filip wrote: > > > Are docecot and postfix located on the same server? > Can postfix access (deliver) directly maildir file directory dovecot uses? > For the moment yes they are on the same server. I designed it to be modular, the various components can be placed on different systems with no configuration changes required, should it be necessary. I'll note that Postfix does not appear to have any problems writing the files to its queue as fast as they are being delivered to it. Postfix and Dovecot are writing to different disks, though both are on the SAN, and both have noatime set. -Chad
Re: multiple messages per second to a single mailbox
The problem happened again this morning. Removing fsync calls helped, but I'm
not sure about leaving that enabled long term.
I still believe the problem is multiple dovecot processes trying to write to a
single folder at the same time. (If I could run dtrace I might be able to
cobble together a script to prove it.)
I tried writing a sieve script to direct the messages to a set of folders, but
I'm not able to make the logic work. I was thinking something like: generate
random # ($N) between say 1-10, then file message into folder mail$N. But I
didn't find a method to do that within sieve.
My next thought was to try parsing the message-id header. If the first
character is [0-5] then fileinto mail1, etc. Then I could go so far as having
36 subfolders which the messages could be written too. This mailbox only keeps
messages for a rolling 1d window. Right now for example it has 260,186
messages in the INBOX.
The sieve script I tried (with only about 4 hours of sleep) was
require ["fileinto","regex"];
if header :regex "message-id" "^1" {
fileinto "mail1";
} else {
keep;
}
If anyone has some suggestions on how I might spread the messages out over
multiple folders I'd like to hear your thoughts. Again the servers are
configured using maildir, so each folder should have its own index and thus
file locking contention should be less, at least so goes the theory in my head.
Thank you,
Chad
Re: multiple messages per second to a single mailbox
On Aug 12, 2015, at 11:58 AM, Daniel Tröder wrote: > On 08/12/2015 17:19, Chad M Stewart wrote: >> What I'm seeing is very high load on the system (40) and queues building on >> the Postfix side. > High load means, that there are a lot of processes waiting to run. The > most likely cause for this is not CPU consumption, but I/O wait. > > Please run vmstat and iostat and post their output. I was watching iostat and avg service times, and maybe once every 30-45 seconds I'd see times of 10ms, but otherwise it was below that. I achieved the biggest impact by limiting the number of outbound connections from Postfix to Dovecot. I limited Postfix to 5 connections, which means a total of 10 inbound LMTP to Dovecot. Then I saw near 500 msgs per LMTP connection. I suspect the problem was a locking issue on the mailbox in question. Too many simultaneous delivery attempts via too many LMTP sessions. The backlog has cleared so I'm done troubleshooting for now. If this happens again I'll resume looking into it more. These are new servers so I'm tuning for the load, etc.. -Chad
Re: multiple messages per second to a single mailbox
On Aug 12, 2015, at 11:25 AM, Chad M Stewart wrote: > <..snip..> > Since I posted I've changed a couple of things: ulimit -n 8192, and disabled > fsync as in mail_fsync = never. I'm not sure if I'll put it back in the LMTP > section or not. Given all the hardware abstraction layers. I forgot to mention that on a clean start of dovecot (after making those changes), I counted 25 msgs delivered to this single mailbox (all INBOX) in a second. Then it slowed down. Maybe the answer is less LMTP connections and increase the number of messages per connection. I have Postfix configured to do outbound connection caching and I've seen entries in the logs indicating 30 messages on a given connection. I'll have to look into how to limit dovecot to a certain number of LMTP processes. -Chad
Re: multiple messages per second to a single mailbox
On Aug 12, 2015, at 11:04 AM, Andrzej A. Filip wrote: > > <..snip..> > Could you provide the following info: > a) mailbox type (maildir/mbox/dbox/...) maildir > [mail_location in dovecot's config] /srv/mail/// > b) file system type (ext2/ext3/ext4/fat32/...) > [provided by "df -T" command on my system] As I said ext4. Since I posted I've changed a couple of things: ulimit -n 8192, and disabled fsync as in mail_fsync = never. I'm not sure if I'll put it back in the LMTP section or not. Given all the hardware abstraction layers. -Chad
multiple messages per second to a single mailbox
Dovecot 2.2.18 on CentOS 6 I have a pair of servers setup with MySQL, Postfix, and Dovecot. Replication is setup and working between the two dovecot instances. The problem I'm running into is that a single mailbox receives a lot of messages, at times the rate is multiple messages per second. Delivery from Postfix to Dovecot is via tcp based LMTP. When I do 'ps -aef|grep lmtp|wc -l' I get 62 and does not appear to go higher than that. At the moment I have 4500 and 8300 messages queued on two Postfix instances waiting to deliver via LMTP to the same dovecot instance. Deliveries only happen via LMTP and only one of the two nodes actually gets the deliveries. What I'm seeing is very high load on the system (40) and queues building on the Postfix side. Replication is keeping up. Looking at the logs now I see anywhere from 4-7 messages per second delivered to this single mailbox. I would like to increase that rate a lot. These machines are VMs hosted on Xenserver 6.x. I have them setup with 8 vCPUs (2 sockets with 4 cores per socket), the dom0 machines have dual HBA connectors back to a SAN and have 128 CPUs and 256GB of RAM and are not taxed. I added a 2nd virtual disk that is used for storing mailbox data. It is ext4 and has noatime set during mount. /var is also mounted with noatime. The performance graphs in XenCenter show nearly all 8 vCPUs at about 50%, and the writes on the mailbox data disk are about 20%. iostat is showing mostly <5 for await times for the disks, though I do see a 10 now and again. I'm guessing that maybe I'm hitting a mailbox locking issue and not sure how to reduce the contention and thereby increase the delivery rate to this mailbox. -Chad
Re: dovecot proxy/director and high availability design
On 2015-07-21 02:54 PM, Laz C. Peterson wrote: The consensus seems to say no to RR DNS … I am going to take that into serious consideration. With this proxy setup you describe, what would happen if HAProxy or Dovecot Proxy were to fail? Multiple instances of each. I'll be using SmartOS as the base for my systems, so I'll be using ucarp, combined with haproxy should achieve what I'm after. Given two client facing IPs, I'm thinking two instances of HAProxy+ucarp, each handling one IP. Giving me active/active. HAProxy can hand off the connections to N+1 DP, the number of those would be based on redundancy plus load, thus my minimum would be 3. Brainstorming your situation Install HAProxy (or similar) in each data center. Then use dovecot director to route users to the store they happen to be using. In other words if the first connection from userA comes into data center 1 then it gets sent to MS1 in the same data center. Their next connection comes into data center 2 (thinking mobile device vs their desktop), the dovecot director there would route the connection over to ms1 in data center 1. Not that network efficient, but it might achieve what you want given your constraints too. -Chad
Re: dovecot proxy/director and high availability design
Round-robin DNS last I checked can be fraught with issues. While doing something else I came up with this idea: Clients --> Load Balancer(HAProxy) --> Dovecot Proxy(DP) --> Dovecot Director(DD) --> MS1 / MS2. When DP checks say user100 it'll find a host=DD-POD1 that returns two IPs, those of the two DD that sit in front of POD1. This DD pair is the only pair in the ring and only responsible for POD1. Another pair will handle POD2. When DD looks up the host value for a user it'll find the same name, but the IPs returned will be different. Instead have both IPs of the mail stores returned. I believe this will achieve what I'm after. HAProxy will do the load balancing of the DP instances. DP will balance the DDs, and DDs will do its job well and ensure that say user300 has all of their connections sent to MS1. When I need to do maintenance on MS1 I can use the DD pair for POD1 to gently move the connections to MS2, etc.. I could also make each POD a 2+1 cluster, so a silent but up-to-date and replicated store sits there waiting should it be needed, or even a 2+2 cluster. After all "two is one, and one is none". Not sure when I'll get time to implement/test this out, but in theory it sounds reasonable. I admit its a fair amount of moving parts and areas for failure but I think it maybe the balance needed to achieve the service level availability I'm after while still allowing for maintenance on the systems w/o clients noticing. -Chad On Jul 20, 2015, at 1:04 PM, Laz C. Peterson wrote: > I’m trying to do this too. But the goal would be simply for automatic > failover to the other datacenter. Everything is working if the server’s > unique hostname is entered, but I want to do something like round robin DNS > that mail clients will automatically attempt to connect to the other IP if > they cannot get to the first address. Unfortunately mail applications don’t > really do this like web browsers do … > > ~ Laz Peterson > Paravis, LLC > >> On Jul 20, 2015, at 10:29 AM, Chad M Stewart wrote: >> >> >> I'm trying to determine which dovecot components to use and how to order >> them in the network path from client to mail store. >> >> >> If I have say 1,000 users, all stored in MySQL (or LDAP) and have 4 mail >> stores, configured into 2, 2 node pods. >> >> >> MS1 and MS2 are pod1 and are configured with replication (dsync) and host >> users 0-500. MS3 and MS4 are pod2 and are configured with replication >> between them and host users 501-1000. Ideally the active connections in >> pod1 would be split 50/50 between MS1 and MS2. When maintenance is >> performed obviously all active connections/users would be moved to the other >> node in the pod and then rebalanced once maintenance is completed. >> >> I'm not sure if I need to use both the proxy and director, or just one or >> the other? If both then what is the proper path, from a network perspective? >> I like the functionality director provides, being able to add/remove >> servers on the fly and adjust connections, etc.. But from what I've read >> director needs to know about all mail servers. The problem is that not all >> servers host all users. User100 could be serviced by ms1 or ms2, but not by >> ms3 or ms4. >> >> I'm trying to design a system that should provide as close to 99.999% >> service availability as possible. >> >> >> >> Thank you, >> Chad
dovecot proxy/director and high availability design
I'm trying to determine which dovecot components to use and how to order them in the network path from client to mail store. If I have say 1,000 users, all stored in MySQL (or LDAP) and have 4 mail stores, configured into 2, 2 node pods. MS1 and MS2 are pod1 and are configured with replication (dsync) and host users 0-500. MS3 and MS4 are pod2 and are configured with replication between them and host users 501-1000. Ideally the active connections in pod1 would be split 50/50 between MS1 and MS2. When maintenance is performed obviously all active connections/users would be moved to the other node in the pod and then rebalanced once maintenance is completed. I'm not sure if I need to use both the proxy and director, or just one or the other? If both then what is the proper path, from a network perspective? I like the functionality director provides, being able to add/remove servers on the fly and adjust connections, etc.. But from what I've read director needs to know about all mail servers. The problem is that not all servers host all users. User100 could be serviced by ms1 or ms2, but not by ms3 or ms4. I'm trying to design a system that should provide as close to 99.999% service availability as possible. Thank you, Chad
