Re: [Dovecot] service_count=0 for imap and pop3
>>> Timo,
>>> Any hints on how many POP3 and IMAP connections I'd be able to get
>>> away with in a single threads with the above setup, assuming they're
>>> relative busy?
>
> The problem is that if there is any waiting for locks, all the other
> connections hang there as well waiting for it. Same for any type of disk I/O
> waits. I don't really know how many you could get away with.. My official
> recommendation would be to keep it 1 connection = 1 process, since that's
> guaranteed to work.
>
>>> BTW, I realize process_min_avail is just a minimum, but without having
>>> service_count=1, what would cause dovecot to fork off more processes
>>> than #process_min_avail?
>
> When all the existing processes have reached client_limit, a new process is
> created.
>
>> emailuser 2625 0.0 0.1 4748 3132 ? S 17:15 0:00 \_
>> dovecot/imap [2 connections]
> ..
>> emailuser 2632 0.7 2.6 59440 54492 ? S 17:15 0:14 \_
>> dovecot/imap [209 connections]
>>
>> Is there a way to spread those out more or is there probably no need
>> to?
>
> Whichever process is fastest to grab a new connection gets it.
>
>> I imagine in the case of PID 2632, a large number of those
>> connections are just sitting in IDLE and doing nothing beyond
>> stat64()'s. Or maybe a better question would be, is there a setting
>> I'm not finding that puts an upper limit on connections for a single
>> process, so a single process will stop servicing new connections after
>> it hits a certain # and lets the other less-loaded processes handle
>> the new connection. All the various *_limit settings appear to be
>> across all processes, not per-processs like this. This btw is 2.0.7
>> (but with the proctitle patch).
>
> service imap {
> client_limit = 5
> }
>
>
Yeah, client_limit looks perfect. I'd googled "client_limit" a few
times over the past week and for some reason, I thought client_limit
was also cumulative across all procs, instead of per-proc. But it's
working perfectly now.
I see your point about any blocking operations. I'll start low and
work my way up and abandon service_limit=0 completely if I start to
get user complaints about it. Right now the possibility of cutting
context switches by 80% is just too promising to pass up. With
typically 600-700 imap procs and 200-300 pop3 procs, the time spent in
kernel scheduling was getting very high and these are only dual core
uniprocessor boxes, otherwise it'd be more of a moot point if I had 8+
cores to work with.
Thanks Timo!
Re: [Dovecot] service_count=0 for imap and pop3
On 15.11.2010, at 22.58, Mark Moseley wrote:
>> Timo,
>> Any hints on how many POP3 and IMAP connections I'd be able to get
>> away with in a single threads with the above setup, assuming they're
>> relative busy?
The problem is that if there is any waiting for locks, all the other
connections hang there as well waiting for it. Same for any type of disk I/O
waits. I don't really know how many you could get away with.. My official
recommendation would be to keep it 1 connection = 1 process, since that's
guaranteed to work.
>> BTW, I realize process_min_avail is just a minimum, but without having
>> service_count=1, what would cause dovecot to fork off more processes
>> than #process_min_avail?
When all the existing processes have reached client_limit, a new process is
created.
> emailuser 2625 0.0 0.1 4748 3132 ?S17:15 0:00 \_
> dovecot/imap [2 connections]
..
> emailuser 2632 0.7 2.6 59440 54492 ?S17:15 0:14 \_
> dovecot/imap [209 connections]
>
> Is there a way to spread those out more or is there probably no need
> to?
Whichever process is fastest to grab a new connection gets it.
> I imagine in the case of PID 2632, a large number of those
> connections are just sitting in IDLE and doing nothing beyond
> stat64()'s. Or maybe a better question would be, is there a setting
> I'm not finding that puts an upper limit on connections for a single
> process, so a single process will stop servicing new connections after
> it hits a certain # and lets the other less-loaded processes handle
> the new connection. All the various *_limit settings appear to be
> across all processes, not per-processs like this. This btw is 2.0.7
> (but with the proctitle patch).
service imap {
client_limit = 5
}
Re: [Dovecot] service_count=0 for imap and pop3
> Timo, > Any hints on how many POP3 and IMAP connections I'd be able to get > away with in a single threads with the above setup, assuming they're > relative busy? I.e. if my boxes typically have, say, 200 concurrent > POP3 connections and 600 IMAP connections, if I used > process_min_avail=50 for POP3 and process_min_avail=100 for IMAP, I'd > assume that'd be in the area of 4 POP3 concurrent connections handled > by that one POP3 thread and 6 concurrent connections handled by that > one IMAP thead. > > I was thinking something like 3-4 per thread (calculating vs max > connections) for POP3 and maybe 5-7 for IMAP (as I'm assuming that at > least a few will just be in IDLE at any given point). > > I guess it comes down to, are you aware of any internal limits for a > POP3 or IMAP thread where they're going to start to bog down and > possibly death spiral? I'm going to experiment with different > settings, but I'd love to hear any opinions you have on it. Sorry for > the vagueness. > > BTW, I realize process_min_avail is just a minimum, but without having > service_count=1, what would cause dovecot to fork off more processes > than #process_min_avail? Actually, it looks like the "spreading" of connections was in my imagination only :) It looks like it can get quite 'clumpy'. Here's a copy/paste from a small segment of ps: emailuser 2625 0.0 0.1 4748 3132 ?S17:15 0:00 \_ dovecot/imap [2 connections] emailuser 2626 0.0 0.2 7484 6124 ?S17:15 0:00 \_ dovecot/imap [16 connections] emailuser 2627 0.0 0.1 4572 2960 ?S17:15 0:00 \_ dovecot/imap [5 connections] emailuser 2628 0.0 0.5 12968 10332 ?S17:15 0:01 \_ dovecot/imap [30 connections] emailuser 2630 0.2 1.0 23316 20836 ?S17:15 0:04 \_ dovecot/imap [90 connections] emailuser 2632 0.7 2.6 59440 54492 ?S17:15 0:14 \_ dovecot/imap [209 connections] emailuser 4099 0.7 0.3 8636 7252 ?S17:34 0:06 \_ dovecot/pop3 [2 connections] emailuser 4418 0.8 0.3 9576 8196 ?S17:38 0:05 \_ dovecot/pop3 [6 connections] emailuser 4478 0.5 0.7 18664 14868 ?S17:39 0:03 \_ dovecot/imap [62 connections] emailuser 4622 1.1 0.4 10404 8780 ?S17:40 0:05 \_ dovecot/pop3 [8 connections] emailuser 4733 1.1 0.3 8576 6972 ?S17:42 0:04 \_ dovecot/pop3 [5 connections] The interesting thing is that there's quite a number of other imap/pop3 procs with just one connection and a ton of imap procs that haven't been touched (i.e. are still running as 'root', waiting to eventually setuid to 'emailuser'). Is there a way to spread those out more or is there probably no need to? I imagine in the case of PID 2632, a large number of those connections are just sitting in IDLE and doing nothing beyond stat64()'s. Or maybe a better question would be, is there a setting I'm not finding that puts an upper limit on connections for a single process, so a single process will stop servicing new connections after it hits a certain # and lets the other less-loaded processes handle the new connection. All the various *_limit settings appear to be across all processes, not per-processs like this. This btw is 2.0.7 (but with the proctitle patch).
Re: [Dovecot] service_count=0 for imap and pop3
>> Only potential problem is memory leaks that keep increasing the memory >> usage. Of course there should be no memory leaks. :) You could anyway set >> something like service_count=1000 to get it to restart after handling 1000 >> connections. > > I'll keep that one in mind. Doesn't seem like it could hurt to have > something like =1000 just to be safe. > > For anybody else reading this thread, here's vmstat from two boxes > with an almost identical amount of IMAP and POP3 connections, but the > first example has "service_count=0" for "service pop3", the second > doesn't. Check out the context switches (this will paste horribly; > it's the 5th column from the right): > > With "service_count=0": > r b swpd free buff cache si so bi bo in cs us sy id wa > 0 0 0 54056 161536 1116012 0 0 132 0 1326 7868 3 8 85 4 > 0 0 0 53568 161608 1116192 0 0 116 0 1572 5111 2 6 80 13 > 1 0 0 52320 161608 1116960 0 0 16 140 1584 6126 3 6 86 5 > 0 0 0 51940 161656 1117292 0 0 44 128 1430 5508 3 7 86 5 > 0 0 0 53316 161628 1115500 0 0 12 0 1459 5363 4 7 86 4 > 0 0 0 52176 161664 1115468 0 0 76 76 1221 6344 6 5 84 6 > 0 1 0 52300 161688 1115636 0 0 32 0 1250 3631 3 2 90 6 > 0 0 0 53580 161648 1113116 0 0 76 172 1778 6671 3 9 83 6 > > Without "service_count=0": > r b swpd free buff cache si so bi bo in cs us sy id wa > 0 2 2476 73776 147860 958728 0 0 0 0 912 17442 6 11 0 84 > 0 2 2476 61300 147860 958756 0 0 0 0 992 22130 5 19 0 76 > 21 4 2476 49484 148008 957608 0 0 140 748 2472 15922 9 24 0 68 > 30 4 2476 97928 146604 945232 0 0 404 0 7116 20700 43 58 0 0 > 4 1 2476 169280 146700 946972 0 0 172 0 7014 43727 33 67 1 0 > 7 1 2476 168684 146800 947304 0 0 168 0 5720 55485 37 62 2 1 > 9 0 2476 165956 146856 947504 0 0 80 0 5074 56371 35 65 0 0 > 6 6 2476 163032 146948 948380 0 0 92 2432 5879 63418 27 67 5 2 > > It's nice to look at 'perf top' on the first box and not see > "finish_task_switch" chewing up 15+% of kernel CPU time :) > Timo, Any hints on how many POP3 and IMAP connections I'd be able to get away with in a single threads with the above setup, assuming they're relative busy? I.e. if my boxes typically have, say, 200 concurrent POP3 connections and 600 IMAP connections, if I used process_min_avail=50 for POP3 and process_min_avail=100 for IMAP, I'd assume that'd be in the area of 4 POP3 concurrent connections handled by that one POP3 thread and 6 concurrent connections handled by that one IMAP thead. I was thinking something like 3-4 per thread (calculating vs max connections) for POP3 and maybe 5-7 for IMAP (as I'm assuming that at least a few will just be in IDLE at any given point). I guess it comes down to, are you aware of any internal limits for a POP3 or IMAP thread where they're going to start to bog down and possibly death spiral? I'm going to experiment with different settings, but I'd love to hear any opinions you have on it. Sorry for the vagueness. BTW, I realize process_min_avail is just a minimum, but without having service_count=1, what would cause dovecot to fork off more processes than #process_min_avail? Thanks!
Re: [Dovecot] service_count=0 for imap and pop3
On Tue, Nov 9, 2010 at 3:55 PM, Timo Sirainen wrote:
> On 9.11.2010, at 23.49, Mark Moseley wrote:
>
>> On Tue, Nov 9, 2010 at 3:05 PM, Timo Sirainen wrote:
>>> On 9.11.2010, at 22.14, Mark Moseley wrote:
>>>
service imap {
service_count = 0
}
>> Would the risks involved be identical to your warnings about using
>> "service_count=0" with pop3-login/imap-login, namely that if the
>> daemon gets hacked, it'd be able to access other mailboxes (presumably
>> that the imap/pop3 already had open)?
>
> If all your users share the same UID and there was a security hole in
> imap/pop3, you could access everyone's mails regardless of this setting.
> (Except if you also enabled chroots, but then you couldn't use this setting.)
>
>> Nice, it does indeed seem to burn a lot less CPU. I've also set
>> "process_min_avail=#" for 'service pop3' which appears to spread out
>> incoming POP3 connections over all # pop3 procs. Any gotchas there?
>> I've always got at least several hundred POP3 connections on a box, so
>> having them not all hitting one proc is desirable. And having, say,
>> 100 pop3 procs hanging around, possibly idle, is fine. This is pretty
>> exciting stuff.
>
> Should be fine.
>
>> Anybody running this way in production on a large scale, i.e. using
>> "service_count=0" in a "service imap" or "service pop3" ?
>
> Only potential problem is memory leaks that keep increasing the memory usage.
> Of course there should be no memory leaks. :) You could anyway set something
> like service_count=1000 to get it to restart after handling 1000 connections.
I'll keep that one in mind. Doesn't seem like it could hurt to have
something like =1000 just to be safe.
For anybody else reading this thread, here's vmstat from two boxes
with an almost identical amount of IMAP and POP3 connections, but the
first example has "service_count=0" for "service pop3", the second
doesn't. Check out the context switches (this will paste horribly;
it's the 5th column from the right):
With "service_count=0":
r b swpd free buff cache si sobibo in cs us sy id wa
0 0 0 54056 161536 111601200 132 0 1326 7868 3 8 85 4
0 0 0 53568 161608 111619200 116 0 1572 5111 2 6 80 13
1 0 0 52320 161608 11169600016 140 1584 6126 3 6 86 5
0 0 0 51940 161656 11172920044 128 1430 5508 3 7 86 5
0 0 0 53316 161628 11155000012 0 1459 5363 4 7 86 4
0 0 0 52176 161664 1115468007676 1221 6344 6 5 84 6
0 1 0 52300 161688 11156360032 0 1250 3631 3 2 90 6
0 0 0 53580 161648 11131160076 172 1778 6671 3 9 83 6
Without "service_count=0":
r b swpd free buff cache si sobibo in cs us sy id wa
0 2 2476 73776 147860 95872800 0 0 912 17442 6 11 0 84
0 2 2476 61300 147860 95875600 0 0 992 22130 5 19 0 76
21 4 2476 49484 148008 95760800 140 748 2472 15922 9 24 0 68
30 4 2476 97928 146604 94523200 404 0 7116 20700 43 58 0 0
4 1 2476 169280 146700 94697200 172 0 7014 43727 33 67 1 0
7 1 2476 168684 146800 94730400 168 0 5720 55485 37 62 2 1
9 0 2476 165956 146856 9475040080 0 5074 56371 35 65 0 0
6 6 2476 163032 146948 9483800092 2432 5879 63418 27 67 5 2
It's nice to look at 'perf top' on the first box and not see
"finish_task_switch" chewing up 15+% of kernel CPU time :)
Re: [Dovecot] service_count=0 for imap and pop3
On 9.11.2010, at 23.49, Mark Moseley wrote:
> On Tue, Nov 9, 2010 at 3:05 PM, Timo Sirainen wrote:
>> On 9.11.2010, at 22.14, Mark Moseley wrote:
>>
>>> service imap {
>>> service_count = 0
>>> }
>>>
> Would the risks involved be identical to your warnings about using
> "service_count=0" with pop3-login/imap-login, namely that if the
> daemon gets hacked, it'd be able to access other mailboxes (presumably
> that the imap/pop3 already had open)?
If all your users share the same UID and there was a security hole in
imap/pop3, you could access everyone's mails regardless of this setting.
(Except if you also enabled chroots, but then you couldn't use this setting.)
> Nice, it does indeed seem to burn a lot less CPU. I've also set
> "process_min_avail=#" for 'service pop3' which appears to spread out
> incoming POP3 connections over all # pop3 procs. Any gotchas there?
> I've always got at least several hundred POP3 connections on a box, so
> having them not all hitting one proc is desirable. And having, say,
> 100 pop3 procs hanging around, possibly idle, is fine. This is pretty
> exciting stuff.
Should be fine.
> Anybody running this way in production on a large scale, i.e. using
> "service_count=0" in a "service imap" or "service pop3" ?
Only potential problem is memory leaks that keep increasing the memory usage.
Of course there should be no memory leaks. :) You could anyway set something
like service_count=1000 to get it to restart after handling 1000 connections.
Re: [Dovecot] service_count=0 for imap and pop3
On Tue, Nov 9, 2010 at 3:05 PM, Timo Sirainen wrote:
> On 9.11.2010, at 22.14, Mark Moseley wrote:
>
>> service imap {
>> service_count = 0
>> }
>>
>> Is that safe to do in imap and/or pop3? Or at least no more insecure
>> than using service_count=0 for imap-login and pop3-login?
>
> Yep.
>
Would the risks involved be identical to your warnings about using
"service_count=0" with pop3-login/imap-login, namely that if the
daemon gets hacked, it'd be able to access other mailboxes (presumably
that the imap/pop3 already had open)?
Nice, it does indeed seem to burn a lot less CPU. I've also set
"process_min_avail=#" for 'service pop3' which appears to spread out
incoming POP3 connections over all # pop3 procs. Any gotchas there?
I've always got at least several hundred POP3 connections on a box, so
having them not all hitting one proc is desirable. And having, say,
100 pop3 procs hanging around, possibly idle, is fine. This is pretty
exciting stuff.
Anybody running this way in production on a large scale, i.e. using
"service_count=0" in a "service imap" or "service pop3" ?
Re: [Dovecot] service_count=0 for imap and pop3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Le 10.11.2010 00:05, Timo Sirainen a écrit :
> On 9.11.2010, at 22.14, Mark Moseley wrote:
>
>> service imap {
>> service_count = 0
>> }
>>
>> Is that safe to do in imap and/or pop3? Or at least no more insecure
>> than using service_count=0 for imap-login and pop3-login?
>
> Yep.
Timo, I'm quite missed us I find the answer "yep" short. we should do
what exactly. edited the wiki, Timo
- --
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x092164A7
gpg --keyserver pgp.mit.edu --recv-key 092164A7
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with CentOS - http://enigmail.mozdev.org/
iD8DBQFM2dsMtXI/OwkhZKcRAoF9AJ0SoRPEDvcg8v0MpKCWSiRjbCqILgCfZDgg
Qx5Lb230c1tNkAsL0GTC508=
=lwux
-END PGP SIGNATURE-
Re: [Dovecot] service_count=0 for imap and pop3
On 9.11.2010, at 22.14, Mark Moseley wrote:
> service imap {
> service_count = 0
> }
>
> Is that safe to do in imap and/or pop3? Or at least no more insecure
> than using service_count=0 for imap-login and pop3-login?
Yep.
