cannot login to imap under load
Dear list,
We moved our dovecot installation to a new vm, and ever since there are
problems logging in to our imap server during office hours. ( Evenings
and weekends are fine. ) Both the new and the old machine are dovecot
2.2.13. Symptoms:
Logging in via imap gives:
. OK Pre-login capabilities listed, post-login capabilities have more.
a login
* OK Waiting for authentication master process to respond..
closed
whereas using the same credentials with pop3:
+OK Dovecot ready.
user
+OK
pass
+OK Logged in.
Our mail.err log gives lots of:
dovecot: imap-login: Error: master(imap): Auth request timed out
(received 0/12 bytes)
dovecot: imap: Error: Login client disconnected too early
dovecot: auth: Error: Master request 24000.918 not found
dovecot: master: Error: service(imap): fork() failed: Resource
temporarily unavailable
dovecot: master: Error: service(imap): command startup failed,
throttling for 2 secs
Note thate we our users almost exclusively use imap. Normally we would
have some 7 or 800 imap processes running and only a few pop3.
Our doveconf -n output:
# 2.2.13: /etc/dovecot/dovecot.conf
# OS: Linux 4.4.38-93-default x86_64 SUSE Linux Enterprise Server 12
(x86_64)
auth_mechanisms = plain login
default_client_limit = 2000
default_process_limit = 2000
default_vsz_limit = 512 M
disable_plaintext_auth = no
imap_client_workarounds = tb-extra-mailbox-sep
import_environment = TZ DEBUG_OUTOFMEM DOVECOT_HOSTDOMAIN
mail_location = maildir:~/Maildir
mail_plugins = " quota"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
quota = maildir:User quota
quota_rule = *:storage=1G
quota_rule2 = Trash:storage=+10%%
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
}
protocols = imap pop3 lmtp sieve
service anvil {
client_limit = 8003
}
service auth {
client_limit = 1
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = dovecot
mode = 0666
}
}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_min_avail = 4
service_count = 0
}
service imap {
process_limit = 2048
}
service pop3-login {
inet_listener pop3 {
port = 110
}
inet_listener pop3s {
port = 995
ssl = yes
}
process_min_avail = 4
service_count = 0
}
service pop3 {
process_limit = 2048
}
ssl_cert =
Re: cannot login to imap under load
Hello Gerard ! On Tuesday, March 28, 2017 4:55 PM, Gerard Ranke wrote:> dovecot: master: Error: service(imap): fork() failed: Resource > >temporarily unavailable > >dovecot: master: Error: service(imap): command startup failed, > >throttling for 2 secs > > >Note thate we our users almost exclusively use imap. Normally we would > >have some 7 or 800 imap processes running and only a few pop3. Could it be an OS (or VM) limit on the number of processes you can create ? -- Yassine.
Re: cannot login to imap under load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 28 Mar 2017, Gerard Ranke wrote: dovecot: master: Error: service(imap): fork() failed: Resource temporarily unavailable dovecot: master: Error: service(imap): command startup failed, throttling for 2 secs check out the ulimits for the Dovecot process. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBWNuOn3z1H7kL/d9rAQIj1AgAxH8id+JVGJ7YBVKQkSOfb2N160UNRkNo hZ/6HLPfI3pBIzypccvvV+rHtv8pxvURjG1fbAoDBaMlmDWau0gMFJwepBunuEYx gBQGtrBvsABV2nv5kagP5V8TJjzLZplk4/vz0YGsOjlz2JhxbgHcLLA2FyQKTXgc TWGpmcfWUDTQgQeOLVJcfJUBtbdH4MV0JuDCaiVcbtDuWYpWPRWPw+7Gp4gL46X1 orzD9T4+C/80oBtnUV2fERW7ITeRJTgQ3bR1tKYFQmMDJNpQL78G5P06bJB1D8ob 43TO1Ylb/vz4B2+WnM34gKRQcorcNENuuCjLC6Cy1mQ3MK7kjjoZ8Q== =HW21 -END PGP SIGNATURE-
Re: cannot login to imap under load
Hi Steffen,
On 29-03-17 12:38, Steffen Kaiser wrote:
> On Tue, 28 Mar 2017, Gerard Ranke wrote:
>
>> dovecot: master: Error: service(imap): fork() failed: Resource
>> temporarily unavailable
>> dovecot: master: Error: service(imap): command startup failed,
>> throttling for 2 secs
>
> check out the ulimits for the Dovecot process.
>
> -- Steffen Kaiser
Here they are:
dovecot@mail:~> ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256942
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256942
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
This looks ok to me, but on startup, I still get:
dovecot[9309]: Warning: fd limit (ulimit -n) is lower than required
under max. load (1024 < 1), because of service auth { client_limit }
Strange thing is that dovecot still complains about the fd limit being
1024, while I set it to 1. And how can a ulimit be too low 'because
of service auth'? I don't get that at all. Thanks for your interest!
gerard
Re: cannot login to imap under load
Could it be that dovecot is being started from a container ?
-- Yassine
On Wednesday, March 29, 2017 12:08 PM, Gerard Ranke
wrote:
Hi Steffen,
On 29-03-17 12:38, Steffen Kaiser wrote:
> On Tue, 28 Mar 2017, Gerard Ranke wrote:
>
>> dovecot: master: Error: service(imap): fork() failed: Resource
>> temporarily unavailable
>> dovecot: master: Error: service(imap): command startup failed,
>> throttling for 2 secs
>
> check out the ulimits for the Dovecot process.
>
> -- Steffen Kaiser
Here they are:
dovecot@mail:~> ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256942
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256942
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
This looks ok to me, but on startup, I still get:
dovecot[9309]: Warning: fd limit (ulimit -n) is lower than required
under max. load (1024 < 1), because of service auth { client_limit }
Strange thing is that dovecot still complains about the fd limit being
1024, while I set it to 1. And how can a ulimit be too low 'because
of service auth'? I don't get that at all. Thanks for your interest!
gerard
Re: cannot login to imap under load
On 29-03-17 13:12, chaouche yacine wrote: > Could it be that dovecot is being started from a container ? > -- Yassine > No, it's just a service on a VM... Best, gerard
Re: cannot login to imap under load
Hi Maria,
It does indeed run from systemd, so this is what's currently in the
dovecot unit file ( /etc/systemd/system/dovecot.service ):
[Unit]
Description=Dovecot IMAP/POP3 email server
After=local-fs.target network.target
[Service]
Type=simple
ExecStart=/usr/sbin/dovecot -F
NonBlocking=yes
TasksMax=1
LIMIT_NOFILE=1
[Install]
WantedBy=multi-user.target
Unfortunately, it doesn't seem to work...
Best,
gerard
On 29-03-17 13:13, María Arrea wrote:
>
> If you are running dovecot via systemd, increase NOFILES in the
> dovecot startup script
>
> El 29/03/17 a las 13:07, Gerard Ranke escribió:
>> Hi Steffen,
>>
>> On 29-03-17 12:38, Steffen Kaiser wrote:
>>> On Tue, 28 Mar 2017, Gerard Ranke wrote:
>>>
dovecot: master: Error: service(imap): fork() failed: Resource
temporarily unavailable
dovecot: master: Error: service(imap): command startup failed,
throttling for 2 secs
>>> check out the ulimits for the Dovecot process.
>>>
>>> -- Steffen Kaiser
>> Here they are:
>>
>> dovecot@mail:~> ulimit -a
>> core file size (blocks, -c) 0
>> data seg size (kbytes, -d) unlimited
>> scheduling priority (-e) 0
>> file size (blocks, -f) unlimited
>> pending signals (-i) 256942
>> max locked memory (kbytes, -l) 64
>> max memory size (kbytes, -m) unlimited
>> open files (-n) 1
>> pipe size(512 bytes, -p) 8
>> POSIX message queues (bytes, -q) 819200
>> real-time priority (-r) 0
>> stack size (kbytes, -s) 8192
>> cpu time (seconds, -t) unlimited
>> max user processes (-u) 256942
>> virtual memory (kbytes, -v) unlimited
>> file locks (-x) unlimited
>>
>> This looks ok to me, but on startup, I still get:
>>
>> dovecot[9309]: Warning: fd limit (ulimit -n) is lower than required
>> under max. load (1024 < 1), because of service auth { client_limit }
>>
>> Strange thing is that dovecot still complains about the fd limit being
>> 1024, while I set it to 1. And how can a ulimit be too low 'because
>> of service auth'? I don't get that at all. Thanks for your interest!
>>
>> gerard
>
>
Re: cannot login to imap under load
Hello, > > It does indeed run from systemd, so this is what's currently in the > dovecot unit file ( /etc/systemd/system/dovecot.service ): > ... > [Service] > Type=simple > ExecStart=/usr/sbin/dovecot -F > NonBlocking=yes > TasksMax=1 > LIMIT_NOFILE=1 ... the parameter should be named LimitNOFile=1 (without the underscore), see http://man7.org/linux/man-pages/man7/systemd.directives.7.html and http://man7.org/linux/man-pages/man5/systemd.exec.5.html --Andreas smime.p7s Description: S/MIME Cryptographic Signature
Re: cannot login to imap under load
On 2017-03-29 13:33:52 +0200, Piper Andreas wrote: > > It does indeed run from systemd, so this is what's currently in the > > dovecot unit file ( /etc/systemd/system/dovecot.service ): > > > ... > > [Service] > > Type=simple > > ExecStart=/usr/sbin/dovecot -F > > NonBlocking=yes > > TasksMax=1 > > LIMIT_NOFILE=1 > > ... > > the parameter should be named > > LimitNOFile=1 > > (without the underscore), see > http://man7.org/linux/man-pages/man7/systemd.directives.7.html and > http://man7.org/linux/man-pages/man5/systemd.exec.5.html Also you dont have to replace the whole service file to achieve this: https://discourse.nordisch.org/t/per-service-ulimits/374 darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
Re: cannot login to imap under load - SOLVED
On 29-03-17 14:40, Marcus Rueckert wrote: > On 2017-03-29 13:33:52 +0200, Piper Andreas wrote: >>> It does indeed run from systemd, so this is what's currently in the >>> dovecot unit file ( /etc/systemd/system/dovecot.service ): >>> >> ... >>> [Service] >>> Type=simple >>> ExecStart=/usr/sbin/dovecot -F >>> NonBlocking=yes >>> TasksMax=1 >>> LIMIT_NOFILE=1 >> >> ... >> >> the parameter should be named >> >> LimitNOFile=1 >> >> (without the underscore), see >> http://man7.org/linux/man-pages/man7/systemd.directives.7.html and >> http://man7.org/linux/man-pages/man5/systemd.exec.5.html > > Also you dont have to replace the whole service file to achieve this: > > https://discourse.nordisch.org/t/per-service-ulimits/374 > >darix > Good point! I did remove the /etc/systemd/system/docevot.service file and added /etc/systemd/system/dovecot.service.d/limits.conf which reads: [Service] TasksMax=1 LimitNOFILE=1 This should survive system upgrades as well. ( The TaskMax setting is to overcome the default 512 from a cgroup controller that is new for sles12sp2. ) After that it also needed: systemctl daemon-reload systemctl restart dovecot Now dovecot starts up cleanly, and our performance problems are gone. Thank you all who took the time to answer, your remarks were very supporting and insightful! It's just what you need when you have a lot of users breathing down your neck :-) All the best! gerard
