Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
Thierry,
Had a chance to test this change this morning, and in my test environment,
this does drastically improve the ability to ssh and su during heavy pop3
load (in test environment, change of 10-15sec to 1-2sec login). While this
is better, the popping is still slowing down authentication on the box
during load. Is there anything else that might be changed to increase
speed?
Just a recap, I basically have between 2-8 mailboxes on each server, and an
application that creates two connections to each mailbox, processes each
message that comes in, and then deleted each message from the pop3
mailbox. During high load times (500+ messages) with the settings that I
had, I was unable to reconnect to the server, nor su to root. Ended up
having to stop the dovecot process and revert to popa3d (luckily had a ssh
session still open to the box).
This is the config from my test environment:
root@devsmtp:~# dovecot -c /usr/local/etc/dovecot/dovecot-test.conf -n
# 2.1.4: /usr/local/etc/dovecot/dovecot-test.conf
# OS: Linux 2.6.32-33-generic-pae i686 Ubuntu 10.04.4 LTS ext4
auth_verbose = yes
base_dir = /var/run/dovecot-test/
disable_plaintext_auth = no
instance_name = dovecot-test
listen = removed
login_greeting = Dovecot-Test for removing index
mail_location = mbox:/var/empty:INBOX=/var/mail/%u
mail_privileged_group = mail
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox Sent Messages {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
driver = shadow
}
protocols = pop3
service pop3-login {
inet_listener pop3 {
port = 130
}
service_count = 0
}
ssl = no
userdb {
driver = passwd
}
protocol pop3 {
pop3_uidl_format = %08Xu%08Xv
}
And from one of my production servers that has the issue:
# 2.1.4: /usr/local/etc/dovecot/dovecot.conf-ml
# OS: Linux 2.6.18-194.17.1.el5 x86_64 Red Hat Enterprise Linux Server
release 5.6 (Tikanga) ext3
auth_verbose = yes
base_dir = /var/run/dovecotml/
disable_plaintext_auth = no
instance_name = Popper
listen = removed
login_greeting = Popper
mail_location =
mbox:/var/empty:INBOX=/opt/mailstore/spool/mail/%u:INDEX=MEMORY
mail_privileged_group = mail
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox Sent Messages {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
driver = shadow
}
protocols = pop3
ssl = no
userdb {
args = blocking=yes
driver = passwd
}
protocol pop3 {
pop3_uidl_format = %08Xu%08Xv
}
On Fri, May 25, 2012 at 2:24 PM, Thierry de Montaudry thie...@odry.netwrote:
On 25 May 2012, at 20:00, Root Kev root@gmail.com wrote:
So the best way to would be to remove it that part completely, or should
it be stored somewhere on disk?
Thanks again,
Kevin
yes, best is to remove :INDEX=MEMORY part, and it will store indexes in
your INBOX path, which is fine.
If you want to put it in a different path, you can have something like
INDEX=/path/to/indexes/%u
Regards,
Thierry
Hi,
Having a system with a third of our users on POP3 (230 of them), no
trouble with dovecot (v2.1.5, on CentOS 5).
But one thing surprises me in your config, the INDEX=MEMORY in the
location parameter. That means that for each POP3 connection, dovecot has
to read each and every mails to create the index in memory. That might be
why the machine becomes unresponsive.
Unless you have a specific reason to use memory index (and I would be
curious to know about it), I would suggest to remove this and keep standard
file indexes, and your performances should improve a lot.
Regards,
Thierry
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
On 30.5.2012, at 20.25, Root Kev wrote: mail_location = mbox:/var/empty:INBOX=/var/mail/%u Note that it's not a good idea for different users to share a single directory for indexes, which is what this does. Preferably /var/empty wouldn't be even writable to the users so this wouldn't happen accidentally. You could instead use something like: mail_location = mbox:/var/empty:INBOX=/var/mail/%u:INDEX=/var/index/%u But if the clients always just download + delete everything, the index files shouldn't make any (positive) difference. Had a chance to test this change this morning, and in my test environment, this does drastically improve the ability to ssh and su during heavy pop3 load (in test environment, change of 10-15sec to 1-2sec login). While this If you think the problem is authentication, try with passdb + userdb static (http://wiki2.dovecot.org/PasswordDatabase/Static) or passwd-file (http://wiki2.dovecot.org/AuthDatabase/PasswdFile) and see if you can reproduce the slowdowns with them.
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
On 2012-05-24 10:54 AM, Root Kev root@gmail.com wrote: Below is the majority of our configs, it is mostly basic: protocols = pop3 Pleease don't post copy/pastes from your config files... Always provide output of doveconf -n This has multiple benefits - doveconf -n output is easier to read, will identify typos, and will make it evident if you are editing the wrong config file(s)... -- Best regards, Charles
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
On 25 May 2012, at 20:00, Root Kev root@gmail.com wrote: So the best way to would be to remove it that part completely, or should it be stored somewhere on disk? Thanks again, Kevin yes, best is to remove :INDEX=MEMORY part, and it will store indexes in your INBOX path, which is fine. If you want to put it in a different path, you can have something like INDEX=/path/to/indexes/%u Regards, Thierry Hi, Having a system with a third of our users on POP3 (230 of them), no trouble with dovecot (v2.1.5, on CentOS 5). But one thing surprises me in your config, the INDEX=MEMORY in the location parameter. That means that for each POP3 connection, dovecot has to read each and every mails to create the index in memory. That might be why the machine becomes unresponsive. Unless you have a specific reason to use memory index (and I would be curious to know about it), I would suggest to remove this and keep standard file indexes, and your performances should improve a lot. Regards, Thierry
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
Currently cannot use IMAP as our application to access the mailbox
currently is only setup to access pop3 mailboxes. We are currently using
Dovecot 2.1.4. Below is the majority of our configs, it is mostly basic:
protocols = pop3
listen = ***Address here***
base_dir = /var/run/dovecot1/
instance_name = Popper
login_greeting = Popper
mail_location =
mbox:/var/empty:INBOX=/opt/mailstore/spool/mail/%u:INDEX=MEMORY
!include conf.d/*.conf
!include_try local.conf
disable_plaintext_auth = no
auth_mechanisms = plain
log_path = syslog
syslog_facility = mail
auth_verbose = yes
log_timestamp = %b %d %H:%M:%S
namespace inbox {
inbox = yes
}
mail_privileged_group = mail
lock_method = fcntl
mbox_read_locks = fcntl
mbox_write_locks = dotlock fcntl
pop3_uidl_format = %08Xu%08Xv
passdb {
driver = checkpassword
args = /usr/bin/checkpassword
}
userdb {
driver = prefetch
}
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
On 24 May 2012, at 16:54, Root Kev root@gmail.com wrote:
Currently cannot use IMAP as our application to access the mailbox
currently is only setup to access pop3 mailboxes. We are currently using
Dovecot 2.1.4. Below is the majority of our configs, it is mostly basic:
protocols = pop3
listen = ***Address here***
base_dir = /var/run/dovecot1/
instance_name = Popper
login_greeting = Popper
mail_location =
mbox:/var/empty:INBOX=/opt/mailstore/spool/mail/%u:INDEX=MEMORY
!include conf.d/*.conf
!include_try local.conf
disable_plaintext_auth = no
auth_mechanisms = plain
log_path = syslog
syslog_facility = mail
auth_verbose = yes
log_timestamp = %b %d %H:%M:%S
namespace inbox {
inbox = yes
}
mail_privileged_group = mail
lock_method = fcntl
mbox_read_locks = fcntl
mbox_write_locks = dotlock fcntl
pop3_uidl_format = %08Xu%08Xv
passdb {
driver = checkpassword
args = /usr/bin/checkpassword
}
userdb {
driver = prefetch
}
Hi,
Having a system with a third of our users on POP3 (230 of them), no trouble
with dovecot (v2.1.5, on CentOS 5).
But one thing surprises me in your config, the INDEX=MEMORY in the location
parameter. That means that for each POP3 connection, dovecot has to read each
and every mails to create the index in memory. That might be why the machine
becomes unresponsive.
Unless you have a specific reason to use memory index (and I would be curious
to know about it), I would suggest to remove this and keep standard file
indexes, and your performances should improve a lot.
Regards,
Thierry
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
Missed CCing on last reply. See below.. Also, would having Nagios checking the number of messages in a mailbox cause issues with the popping of messages? And/Or would having a user accessing the mailbox from two different applications cause issues? Ie. from Outlook and mobile device? I am under the impression that a lock file is generated which should deal with the issue of contention, deleting mail, etc. Thanks for any information. Kevin Sorry for the delay in responding, long weekend in Canada... When trying to SSH into the server, the server prompts for user name then password, then just hangs. Same if there is already a console connection over, when trying to SU to root, it just hangs after entering the password. Our passwords are in the shadow file, and because this is a server dedicated for this task, there is only the default linux users, and maybe 8 other user accounts in the shadow file. There shouldn't be high IO as all this box does is postfix, pop3ad and dovecot. This box is seeing less then 3000 emails a day, and only has 5 mailboxes on it. Thanks for the continued suggestions... Kevin On Sat, May 19, 2012 at 3:33 PM, Timo Sirainen t...@iki.fi wrote: On Fri, 2012-05-18 at 09:21 -0400, Root Kev wrote: During the last time that the load went up, it became unable to login / su to root for the entire period that dovecot was running, we had to kill dovecot and go back to Popa3d until the mailq was cleared up. We are running CentOS 5.6 server. Based on TOP running at the time the CPU usage was running under 10%. Once Dovecot was killed, we were then able to log in /su again. Like Kelsey said, a very high disk IO might explain this, although normally the login should still eventually succeed. Another thing I'm wondering is if some process limit reached. How does the login/su fail, does it just hang or immediately fail with some error? We were under the impression that checking to shadow directly should be the fastest and least amount of overhead, is any of the other ways to connect have less load on authentication to PAM? Your passwords really are in /etc/shadow file, not LDAP/something else? I don't think the problem is with authentication. Reading /etc/shadow is pretty fast (unless maybe if it's a huge file) and it anyway can't block login/su from working.
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
Am 23.05.2012 16:01, schrieb Root Kev: Missed CCing on last reply. See below.. Also, would having Nagios checking the number of messages in a mailbox cause issues with the popping of messages? And/Or would having a user accessing the mailbox from two different applications cause issues? Ie. from Outlook and mobile device? I am under the impression that a lock file is generated which should deal with the issue of contention, deleting mail, etc. Thanks for any information. Kevin i have a different setup as yours ( virtual , sql etc ) with dove 2.0.20 and extrem high pop3 traffic no problem so far same with low pop3 traffic setup dove 2.1.6 no problem reported anyway use imap where are your confs and logs ? Sorry for the delay in responding, long weekend in Canada... When trying to SSH into the server, the server prompts for user name then password, then just hangs. Same if there is already a console connection over, when trying to SU to root, it just hangs after entering the password. Our passwords are in the shadow file, and because this is a server dedicated for this task, there is only the default linux users, and maybe 8 other user accounts in the shadow file. There shouldn't be high IO as all this box does is postfix, pop3ad and dovecot. This box is seeing less then 3000 emails a day, and only has 5 mailboxes on it. Thanks for the continued suggestions... Kevin On Sat, May 19, 2012 at 3:33 PM, Timo Sirainen t...@iki.fi wrote: On Fri, 2012-05-18 at 09:21 -0400, Root Kev wrote: During the last time that the load went up, it became unable to login / su to root for the entire period that dovecot was running, we had to kill dovecot and go back to Popa3d until the mailq was cleared up. We are running CentOS 5.6 server. Based on TOP running at the time the CPU usage was running under 10%. Once Dovecot was killed, we were then able to log in /su again. Like Kelsey said, a very high disk IO might explain this, although normally the login should still eventually succeed. Another thing I'm wondering is if some process limit reached. How does the login/su fail, does it just hang or immediately fail with some error? We were under the impression that checking to shadow directly should be the fastest and least amount of overhead, is any of the other ways to connect have less load on authentication to PAM? Your passwords really are in /etc/shadow file, not LDAP/something else? I don't think the problem is with authentication. Reading /etc/shadow is pretty fast (unless maybe if it's a huge file) and it anyway can't block login/su from working. -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
On Fri, 2012-05-18 at 09:21 -0400, Root Kev wrote: During the last time that the load went up, it became unable to login / su to root for the entire period that dovecot was running, we had to kill dovecot and go back to Popa3d until the mailq was cleared up. We are running CentOS 5.6 server. Based on TOP running at the time the CPU usage was running under 10%. Once Dovecot was killed, we were then able to log in /su again. Like Kelsey said, a very high disk IO might explain this, although normally the login should still eventually succeed. Another thing I'm wondering is if some process limit reached. How does the login/su fail, does it just hang or immediately fail with some error? We were under the impression that checking to shadow directly should be the fastest and least amount of overhead, is any of the other ways to connect have less load on authentication to PAM? Your passwords really are in /etc/shadow file, not LDAP/something else? I don't think the problem is with authentication. Reading /etc/shadow is pretty fast (unless maybe if it's a huge file) and it anyway can't block login/su from working.
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
During the last time that the load went up, it became unable to login / su to root for the entire period that dovecot was running, we had to kill dovecot and go back to Popa3d until the mailq was cleared up. We are running CentOS 5.6 server. Based on TOP running at the time the CPU usage was running under 10%. Once Dovecot was killed, we were then able to log in /su again. We were under the impression that checking to shadow directly should be the fastest and least amount of overhead, is any of the other ways to connect have less load on authentication to PAM? Thanks, Kevin On Thu, May 17, 2012 at 4:57 PM, Timo Sirainen t...@iki.fi wrote: On 17.5.2012, at 18.22, Root Kev wrote: We have put Dovecot 2.1.4 on several of our production servers (CentOS, on Dell R710, with 20GB memory, dual CPU Quad-core). We have a single instance of Dovecot running and currently have several instances of Popa3d. When there are significant amount of popping from 2 mailboxes that dovecot that is popping from (500+ msgs in the mailboxes), the popping of the messages causes the boxes to become unresponsive. We use another application that connects to the Dovecot, downloads 2-10 messages, then processes them, then sends the delete command to Dovecot. Unresponsive for a long time?.. What CentOS version? When this issue occurs we are unable to become Root, or login again if we close our ssh connection. This only occures when Dovecot is doing the popping. If we only run the older Popa3d, this doesn't occur. We believe it is caused by the way dovecot is authenticating. Sounds like PAM is hanging. Is the (CPU) load in general high at this time? We are using auth_mechanisms = plain passdb drive = shadow usedb driver = passwd args = blocking=yes Using shadow/passwd directly shouldn't affect PAM at all. So this is a rather strange problem..
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
On 5/18/2012 6:21 AM, Root Kev wrote: During the last time that the load went up, it became unable to login / su to root for the entire period that dovecot was running, we had to kill This sounds more like you are getting I/O bound or swapping heavily. What does iostat -x, etc, show when this is happening? -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
On 17.5.2012, at 18.22, Root Kev wrote: We have put Dovecot 2.1.4 on several of our production servers (CentOS, on Dell R710, with 20GB memory, dual CPU Quad-core). We have a single instance of Dovecot running and currently have several instances of Popa3d. When there are significant amount of popping from 2 mailboxes that dovecot that is popping from (500+ msgs in the mailboxes), the popping of the messages causes the boxes to become unresponsive. We use another application that connects to the Dovecot, downloads 2-10 messages, then processes them, then sends the delete command to Dovecot. Unresponsive for a long time?.. What CentOS version? When this issue occurs we are unable to become Root, or login again if we close our ssh connection. This only occures when Dovecot is doing the popping. If we only run the older Popa3d, this doesn't occur. We believe it is caused by the way dovecot is authenticating. Sounds like PAM is hanging. Is the (CPU) load in general high at this time? We are using auth_mechanisms = plain passdb drive = shadow usedb driver = passwd args = blocking=yes Using shadow/passwd directly shouldn't affect PAM at all. So this is a rather strange problem..
