Re: [Dovecot] Dovecot extremely slow!
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 25 Sep 2013, Patricio Rojo wrote: I can click on an email and wait for a minute or more before receiving a connection dropped or no error at all. I use many clients (thunderbird, windows 8 mail, maildroid for android, squirrelmail) and they all have similar behavior. It happens both in the inbox and on imap subfolders. Sometimes it helps changing subfolders back and forth. I have many imap folders organized in up to 3 levels of subfolders and use postfix for delivery. What about I/O load on the server? Something in kernel log? Do you use FTS? Do you get many messages at once? Then, as Lukreme and Bob already said, provide doveconf -n and check out Dovecot logs, for Error and Warning. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBUkP1Ol3r2wJMiz2NAQJ+ewf/ZcGWyYnAx50iZZ8/jkO9c9BU5WmRmMA3 AaBx8fM8IrSXnWtCUY+WcaIvn2wl9MnCFQn2Onigqv52wwdUXppuBKBqKlPKRl0b MF8MkqUh1hrM8gIqBNHNiMWhGJXKcMRF5+fk2JtgFtDzew5x2bvsd+g1WlAf5cPo 8W5gsEP8wfpYxNgsnMW4yzokJdoXUa9laqUKgOqArtXVsbRE/sJ5Kh8c71tj+YY3 J4G5aenCxunjjs6caJbKN4YuvLptI2vSw2WhAc2c5WnVtXvRsTARsAlsQAJo+kLU +aDTbaW1ChldCHzUkRSBEEH5cU8ij3yD2p0TRaYMdakeNxaf8MdYfw== =zrAV -END PGP SIGNATURE-
Re: [Dovecot] Dovecot extremely slow!
On 9/25/2013 5:05 PM, Patricio Rojo wrote: Please help, Dovecot is running extremely slow for the last couple of weeks and it seems to be getting worse (or my patience running short). Progressive degradation of mail server performance, whether an IMAP mailbox server, or an MTA, is almost always caused by the storage subsystem, usually due to filesystem free space fragmentation. If you have a parity RAID array, have lost a disk and are running in degraded mode, this can also cause large IO latency, slowing Dovecot. Another common cause is heavy swap usage. If you have a runaway process or one with a memory leak, this will eat up physical RAM, causing heavy swap usage. If swap resides on the same spindles as your mailboxes, this will degrade Dovecot performance. If your box is hosted at a colo facility, or is in fact a VPS, it's always possible a network problem or a clogged shared segment at the provider is causing packet loss, which can also cause the client delay behavior you have described. If this server resides behind consumer ADSL there could be a problem with your DSL provider's network. In other words, if you didn't change the Dovecot configuration on the day the performance first dropped, or very shortly before, then the performance problem has nothing to do with Dovecot. And this is almost always the case with performance degradation. The source of the problem lie outside Dovecot, again, usually in the storage stack. Start your troubleshooting there. -- Stan
Re: [Dovecot] Dovecot extremely slow!
Thanks all for the quick and knowledgeable replies!
More details on my system:
* Debian 7.1 server hosting many daemons which do not show any slow
behavior at all (apache, postfix, nfs, autofs, ssh, ...), nor is it slow
to run any application for test (no resource intensive application is
run routinely on this machine due to its low 4Gb RAM, in any case)
* /home partition nfs mounted from a remote firewalled QNAP NAS server
(TS-869U-RP), which also serves other machines (RAID-5 setup with
currently no bad disks). When logging in as user in any of those
machines including the dovecot server, I notice no delay (remember that
dovecot hangs for 60 or more seconds). Also, the inbox hangs as often as
the imap folders, but the former is found on local disk on /var/mail.
* user authentification using ldap with a daemon hosted on a different
server than dovecot's (and firewalled from the outside)
* the logs files give no warnings or errors other than the typical
failed connection attempts from chinese or so hackers. I do however,
find the following lines in mail.log every once in a while:
Sep 26 11:02:20 wasabi dovecot: imap(pato): Disconnected: Disconnected in IDLE
in=8017978 out=490892
Sep 26 11:02:21 wasabi dovecot: imap-login: Login: user=pato, method=PLAIN,
rip=24.58.62.118, lip=146.83.9.56, mpid=3964, TLS, session\
=lcGR0UnnugAYOj52
Sep 26 11:03:23 wasabi dovecot: imap-login: Disconnected (no auth attempts in 1
secs): user=, rip=24.58.62.118, lip=146.83.9.56, TLS, \
session=uOJE1UnnxQAYOj52
Sep 26 11:03:26 wasabi dovecot: imap-login: Login: user=pato, method=PLAIN,
rip=24.58.62.118, lip=146.83.9.56, mpid=3973, TLS, session\
=PCFr1UnnxgAYOj52
Sep 26 11:05:00 wasabi dovecot: imap(pato): Disconnected: Disconnected in IDLE
in=1205 out=28366
note how it receives a 'user=' from the same ip it received a valid
user a minute ago (and this is the timescale of my problem).
* When the problem started I did a lot of rather simultaneous changes to
my system (change the hardware of my dovecot's host, moved the ldap
daemon from the dovecot machine to a firewalled machine, installed the
QNAP NAS, updated CA certificate ...), so it is hard to pinpoint the
cause of this. Every other daemon is working as good as it was before,
though.
* 'doveconf -n' output is below.
Thank you very much!!
Patricio
--
PS: Please warn me if any of the information I have given can be used to
exploit my system. I have tried to be very careful with this
# 2.1.7: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.1
mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_privileged_group = mail
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox Sent Messages {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
driver = pam
}
protocols = imap
service auth {
inet_listener {
port = 12345
}
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0666
user = postfix
}
}
ssl_cert = /etc/dovecot/wasabi.imap.crt
ssl_key = /etc/dovecot/private/wasabi.imap.nopwd.key
userdb {
driver = passwd
}
Re: [Dovecot] Dovecot extremely slow!
hi,
Sep 26 11:03:23 wasabi dovecot: imap-login: Disconnected (no auth attempts in
1 secs): user=, rip=24.58.62.118, lip=146.83.9.56, TLS, \
session=uOJE1UnnxQAYOj52
Sep 26 11:03:26 wasabi dovecot: imap-login: Login: user=pato, method=PLAIN,
rip=24.58.62.118, lip=146.83.9.56, mpid=3973, TLS, session\
=PCFr1UnnxgAYOj52
try enabling the debug settings in your dovecot.conf, maybe you can get
more info:
#auth_debug = yes
#auth_debug_passwords = yes
#mail_debug = yes
You also mention that your auth server is on a separate machine, and 60
seconds seems a lot like a timeout threshold, maybe you are having
intermittent problems there. Maybe if you could tail the dovecot and
the ldap logs simultaneously then repeat your test, you would see a
discrepancy on the auth server when the dovecot logs show user=
ssl_cert = /etc/dovecot/wasabi.imap.crt
ssl_key = /etc/dovecot/private/wasabi.imap.nopwd.key
Hmm... a low-level guess: maybe you need to speicify your CA here? I
don't *think* that would explain your slowness, but I suppose there
could be a timeout looking for it...
userdb {
driver = passwd
}
Re: [Dovecot] Dovecot extremely slow!
On 9/26/2013 7:47 AM, Patricio Rojo wrote: * /home partition nfs mounted from a remote firewalled QNAP NAS server (TS-869U-RP), which also serves other machines (RAID-5 setup with currently no bad disks). I assume this NAS properly implements various locking services? Dovecot, like most mail MUA + MTAs, makes use of various filesystem locking primitives to maintain conherence in a multi-user access scenario. If QNAP's stack doesn't implement proper NFS locking, this is probably a cause of these odd lags. You can probably add a nolock to your /etc/fstab to resolve it, but you risk mailbox corruption. You mentioned it was firewalled... are you allowing the lockd port through to the QNAP from the Dovecot machine that's mounting it? NFS2 + 3 implement locking via communication with a lock manager that listens on port 4045, if I recall. =R=
Re: [Dovecot] Dovecot extremely slow!
On 9/26/2013 9:47 AM, Patricio Rojo wrote: You failed to mention every client device you've tested is connecting to the server from over 5000 miles away, across continents and an ocean, with your packets traversing multiple national and political boundaries. rip=24.58.62.118, lip=146.83.9.56 cpe-24-58-62-118.twcny.res.rr.com not found: 2(SERVFAIL) Time Warner Cable, New York Red Universitaria Nacional Santiago, CL Observatorio Astronomico Nacional Have you performed extensive packet tracing to eliminate the network paths as the source of the problem? From here... ~$ telnet 146.83.9.56 993 Trying 146.83.9.56... ^C ~$ telnet 146.83.9.56 143 Trying 146.83.9.56... ^C ~$ telnet 146.83.9.56 25 Trying 146.83.9.56... ^C ~$ telnet 146.83.9.56 587 Trying 146.83.9.56... ^C Given connections to the Dovecot host are apparently firewalled, either holes have been punched for 24.58.62.118, or you're going through a VPN tunnel. I'd guess your problems are network or firewall related, not Dovecot related. -- Stan
[Dovecot] Dovecot extremely slow!
Please help,
Dovecot is running extremely slow for the last couple of weeks and it
seems to be getting worse (or my patience running short).
I attach the 10-master configuration and the log file after running
strace according to: http://wiki.dovecot.org/Debugging/ProcessTracing
I can click on an email and wait for a minute or more before receiving a
connection dropped or no error at all. I use many clients (thunderbird,
windows 8 mail, maildroid for android, squirrelmail) and they all have
similar behavior. It happens both in the inbox and on imap subfolders.
Sometimes it helps changing subfolders back and forth.
I have many imap folders organized in up to 3 levels of subfolders and
use postfix for delivery.
Let me know any other info you require. Thanks!
Patricio
#default_process_limit = 100
#default_client_limit = 1000
# Default VSZ (virtual memory size) limit for service processes. This is mainly
# intended to catch and kill processes that leak memory before they eat up
# everything.
#default_vsz_limit = 256M
# Login user is internally used by login processes. This is the most untrusted
# user in Dovecot system. It shouldn't have access to anything at all.
#default_login_user = dovenull
# Internal user is used by unprivileged processes. It should be separate from
# login user, so that login processes can't disturb other processes.
#default_internal_user = dovecot
service imap-login {
inet_listener imap {
#port = 143
}
inet_listener imaps {
#port = 993
#ssl = yes
}
# Number of connections to handle before starting a new process. Typically
# the only useful values are 0 (unlimited) or 1. 1 is more secure, but 0
# is faster. doc/wiki/LoginProcess.txt
#service_count = 1
# Number of processes to always keep waiting for more connections.
#process_min_avail = 0
# If you set service_count=0, you probably need to grow this.
#vsz_limit = $default_vsz_limit
}
service pop3-login {
inet_listener pop3 {
#port = 110
}
inet_listener pop3s {
#port = 995
#ssl = yes
}
}
service lmtp {
unix_listener lmtp {
#mode = 0666
}
# Create inet listener only if you can't use the above UNIX socket
#inet_listener lmtp {
# Avoid making LMTP visible for the entire internet
#address =
#port =
#}
}
service imap {
# Most of the memory goes to mmap()ing files. You may need to increase this
# limit if you have huge mailboxes.
#vsz_limit = $default_vsz_limit
# Max. number of IMAP processes (connections)
#process_limit = 1024
}
service pop3 {
# Max. number of POP3 processes (connections)
#process_limit = 1024
}
service auth {
# auth_socket_path points to this userdb socket by default. It's typically
# used by dovecot-lda, doveadm, possibly imap process, etc. Users that have
# full permissions to this socket are able to get a list of all usernames and
# get the results of everyone's userdb lookups.
#
# The default 0666 mode allows anyone to connect to the socket, but the
# userdb lookups will succeed only if the userdb returns an uid field that
# matches the caller process's UID. Also if caller's uid or gid matches the
# socket's uid or gid the lookup succeeds. Anything else causes a failure.
#
# To give the caller full permissions to lookup all users, set the mode to
# something else than 0666 and Dovecot lets the kernel enforce the
# permissions (e.g. 0777 allows everyone full permissions).
unix_listener auth-userdb {
#mode = 0666
#user =
#group =
}
# Postfix smtp-auth
unix_listener /var/spool/postfix/private/auth {
user = postfix
group = postfix
mode = 0666
}
inet_listener {
port = 12345
}
# Auth process is run as this user.
#user = $default_internal_user
}
service auth-worker {
# Auth worker process is run as root by default, so that it can access
# /etc/shadow. If this isn't necessary, the user should be changed to
# $default_internal_user.
#user = root
}
service dict {
# If dict proxy is used, mail processes should have access to its socket.
# For example: mode=0660, group=vmail and global mail_access_groups=vmail
unix_listener dict {
#mode = 0600
#user =
#group =
}
}
18:29:13.833120 epoll_wait(9, {}, 6, 23980) = 0
18:29:37.837451 stat(/home/pato/mail/Astro/conferences,
{st_mode=S_IFREG|0644, st_size=112122083, ...}) = 0
18:29:37.838261 epoll_wait(9, {}, 6, 2) = 0
18:30:07.867721 stat(/home/pato/mail/Astro/conferences,
{st_mode=S_IFREG|0644, st_size=112122083, ...}) = 0
18:30:07.868340 epoll_wait(9, {}, 6, 28917) = 0
18:30:36.814608 write(11, * OK Still here\r\n, 17) = 17
18:30:36.832177 time(NULL) = 1380144636
18:30:36.833279 epoll_wait(9, {{EPOLLIN, {u32=1413012688,
u64=139759648824528}}}, 6, 1034) = 1
18:30:37.065994 read(11, DONE\r\n, 8030) = 6
18:30:37.066289 epoll_ctl(9, EPOLL_CTL_DEL, 11, {0, {u32=1413012688,
u64=139759648824528}}) = 0
18:30:37.066574
Re: [Dovecot] Dovecot extremely slow!
On 25 Sep 2013, at 16:05 , Patricio Rojo p...@oan.cl wrote: I attach the 10-master configuration That’s not that useful. doveconf -n is useful
Re: [Dovecot] Dovecot extremely slow!
-- Computerisms Bob Miller 867-334-7117 / 867-633-3760 http://computerisms.ca On Wed, 2013-09-25 at 16:15 -0600, LuKreme wrote: On 25 Sep 2013, at 16:05 , Patricio Rojo p...@oan.cl wrote: I attach the 10-master configuration That’s not that useful. doveconf -n is useful As are the server logs, as opposed to the strace output...
