Re: [Dovecot] dovecot stats

[Robert Schetterer]

Am 08.05.2013 11:25, schrieb Jan-Frode Myklebust:
> Thanks, nice graphs. I've attached a graph over LMTP delays per minute as
> seen from the postfix side on one of our servers. This includes delays
> caused by both delivery to dovecot LMTP, and also LMTP communication
> internally on the mailservers between postfix and amavis. Unfortunately it
> says nothing about the delivery time to each individual dovecot backend,
> since these are hiding behind dovecot director, and therefor we have no way
> of knowing which of our backends are slow (if any).
> 
> 
> 
>   -jf
> 
> 
> 
>>
> 

i am not sure, i ll have to read more doku
perhaps it makes sense to use delay stuff in log for analyse i.e
after all i am not using directors, i have "real" loadbalancers

private/dovecot-lmtp], delay=1.4, delays=1.2/0/0/0.17, dsn=2.0.0,
status=sent

Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: [Dovecot] dovecot stats

[Jan-Frode Myklebust]

Thanks, nice graphs. I've attached a graph over LMTP delays per minute as
seen from the postfix side on one of our servers. This includes delays
caused by both delivery to dovecot LMTP, and also LMTP communication
internally on the mailservers between postfix and amavis. Unfortunately it
says nothing about the delivery time to each individual dovecot backend,
since these are hiding behind dovecot director, and therefor we have no way
of knowing which of our backends are slow (if any).



  -jf



>
<>

Re: [Dovecot] dovecot stats

[Robert Schetterer]

Am 08.05.2013 10:25, schrieb Jan-Frode Myklebust:
> On Mon, May 6, 2013 at 4:37 PM, Timo Sirainen  wrote:
> 
>> On 29.4.2013, at 15.30, Jan-Frode Myklebust  wrote:
>>
>>
>>> Is it possible to collect info about POP3 and LMTP commands also ?
>>
>> No. I think they would be pretty boring statistics, since with POP3 pretty
>> much everything causing disk I/O or CPU usage would be RETRs and with LMTP
>> everything would be DATAs.
>>
>>
> I think knowing the timings of writing messages to disk / reading from disk
> would be very interesting and relevant data. Especially for us with mostly
> POP3 clients, where LMTP DATAs and POP3 RETRs probably is accounting for
> major parts of the server load.

no urgent need for stats
i count pop3 logins with xymon out of syslog in realtime, also logwatch
reports lmtp/pop3/imap daily for each user by cron in daily terms

perhaps have a look at

http://sys4.de/de/blog/2013/01/10/xymon-dovecot-count-imap-pop3-logins-graph-central-rsyslog-server-ubuntu-lucid/

if you have xymon monitoring etc installed you have other stuff like cpu
, mem, disk io too

i ll plan to write some solution for xymon retrieve data out of dovecot
stats too

> 
> 
>>> Also, is "doveadm stats dump command" telling me the results of all
>>> commands that has finished the last stats_command_min_time, or will it
>>> maybe contain much more than 1 minute of activity ?
>>
>> It can contain much more. The stats process will keep as much data in
>> memory as possible until it reaches the stats_memory_limit. The doveadm
>> stats dump lists everything that the stats process knows.
>>
>>
> 
> Ok, then I guess we'll need to limit our stats dumps based on last_seen.
> 
> 
>   -jf
> 



Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

Re: [Dovecot] dovecot stats

[Jan-Frode Myklebust]

On Mon, May 6, 2013 at 4:37 PM, Timo Sirainen  wrote:

> On 29.4.2013, at 15.30, Jan-Frode Myklebust  wrote:
>
>
> > Is it possible to collect info about POP3 and LMTP commands also ?
>
> No. I think they would be pretty boring statistics, since with POP3 pretty
> much everything causing disk I/O or CPU usage would be RETRs and with LMTP
> everything would be DATAs.
>
>
I think knowing the timings of writing messages to disk / reading from disk
would be very interesting and relevant data. Especially for us with mostly
POP3 clients, where LMTP DATAs and POP3 RETRs probably is accounting for
major parts of the server load.


> > Also, is "doveadm stats dump command" telling me the results of all
> > commands that has finished the last stats_command_min_time, or will it
> > maybe contain much more than 1 minute of activity ?
>
> It can contain much more. The stats process will keep as much data in
> memory as possible until it reaches the stats_memory_limit. The doveadm
> stats dump lists everything that the stats process knows.
>
>

Ok, then I guess we'll need to limit our stats dumps based on last_seen.


  -jf

Re: [Dovecot] dovecot stats

[Timo Sirainen]

On 29.4.2013, at 15.30, Jan-Frode Myklebust  wrote:

> I just upgraded one of our servers to dovecot v2.1.16 (ee), and am looking
> into the stats feature. Am I interpreting the wiki correct in reading that
> the "doveadm stats dump command" only returns statistics about IMAP
> commands?

Right.

> Is it possible to collect info about POP3 and LMTP commands also ?

No. I think they would be pretty boring statistics, since with POP3 pretty much 
everything causing disk I/O or CPU usage would be RETRs and with LMTP 
everything would be DATAs.

> Also, is "doveadm stats dump command" telling me the results of all
> commands that has finished the last stats_command_min_time, or will it
> maybe contain much more than 1 minute of activity ?

It can contain much more. The stats process will keep as much data in memory as 
possible until it reaches the stats_memory_limit. The doveadm stats dump lists 
everything that the stats process knows.

Re: [Dovecot] dovecot stats: useful data to gather

[Kelsey Cummings]

On 06/02/12 17:10, Daniel Parthey wrote:

Patrick Ben Koetter wrote:

following our discussion on dovecot stats at the LinuxTag 2012 my team and I
sat down and put together a list of stat items we think to be useful in daily
dovecot usage.

Besides pulling together all the data we also think it would be useful to have
an SNMP interface to access the stats. Our offer to create and contribute a
standalone web interface for dovecot stats stands.


This should be done via SNMP subagent, but how could you differentiate
different dovecot instances on the same machine, different snmp ports
for the subagent, or different snmp trees?


I'd suggest some additional performance metrics like min/max/avg time to 
authenicate, establish a proxy session and perhaps include auth failure 
causes counters as well.


I personally wouldn't want to see this implemented as an SNMP subagent 
but so long as the stats would be available off a local socket directly 
I think everyone would be happy.


-K

Re: [Dovecot] dovecot stats error

[Kelsey Cummings]

On 8/25/2012 12:14 PM, Kelsey Cummings wrote:

On 6/22/2012 6:34 AM, Timo Sirainen wrote:

Which Dovecot version? I thought I fixed this already..


I'm seeing these errors running 2.1.8


Examples below, let me know if I can provide any other info Timo.

In other news, we're finally migrated to dovecot from courier.

WHOOO H


Aug 25 12:53:37 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mcache 331 < 332
Aug 25 12:53:37 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mrbytes 180435729 < 204849088
Aug 25 12:53:38 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mrbytes 50757363 < 62351358
Aug 25 12:53:38 d dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mlpath 17451 < 20067
Aug 25 12:53:41 d dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mrbytes 40483661 < 42086237
Aug 25 12:53:42 b dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mrbytes 65540465 < 67974537
Aug 25 12:53:42 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mlpath 811 < 946
Aug 25 12:53:43 b dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mrbytes 220133763 < 221888538
Aug 25 12:53:47 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mcache 13 < 14
Aug 25 12:53:48 c dovecot: stats: Error: Mail server input error: UPDATE-SESSION: 
stats shrank: mrbytes 118702153 < 121714865




--
Kelsey Cummings - k...@corp.sonic.net  sonic.net, inc.
System Architect  2260 Apollo Way
707.522.1000  Santa Rosa, CA 95407

Re: [Dovecot] dovecot stats error

[Kelsey Cummings]

On 6/22/2012 6:34 AM, Timo Sirainen wrote:

Which Dovecot version? I thought I fixed this already..


I'm seeing these errors running 2.1.8

--
Kelsey Cummings - k...@corp.sonic.net  sonic.net, inc.
System Architect  2260 Apollo Way
707.522.1000  Santa Rosa, CA 95407

Re: [Dovecot] dovecot stats error

[Timo Sirainen]

On 22.6.2012, at 15.34, Robert Schetterer wrote:

> Hi Timo,
> any idea whats this related too ?
> 
> dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats
> shrank: mrbytes 21703727 < 25193928

Which Dovecot version? I thought I fixed this already..

Re: [Dovecot] dovecot stats: useful data to gather

[Daniel Parthey]

Patrick Ben Koetter wrote:
> following our discussion on dovecot stats at the LinuxTag 2012 my team and I
> sat down and put together a list of stat items we think to be useful in daily
> dovecot usage.
> 
> Besides pulling together all the data we also think it would be useful to have
> an SNMP interface to access the stats. Our offer to create and contribute a
> standalone web interface for dovecot stats stands.

This should be done via SNMP subagent, but how could you differentiate
different dovecot instances on the same machine, different snmp ports
for the subagent, or different snmp trees?

> Here are the stats we believe to be useful:
> [...]

Here are the stats which I also consider to be useful:

Login/Logout:
- Hits/Misses for Logins via userdb cache

System resources:
- detailed memory usage of dovecot services (imap, worker, userdb cache)
- dovecot connections to mysql database
- dovecot connections to ldap
- director connections vs. backend connections

Regards,
Daniel

Re: [Dovecot] dovecot stats: useful data to gather

[Christian Rohmann]

On 01.06.2012 22:58, Patrick Ben Koetter wrote:

[...] I sat down and put together a list of stat items we think to be useful in 
daily
dovecot usage.


Quite a list. But I believe most of those values are quite useful and I 
would also love to see such a rich set of measurements being available.



Besides pulling together all the data we also think it would be useful to have
an SNMP interface to access the stats. Our offer to create and contribute a
standalone web interface for dovecot stats stands.


Yes, I second that. Otherwise quite a few installation will just hook 
the dovecot commands to netsnmp handlers, which is not a pretty solution.


Maybe dovecot could also do the SNMP for statistics that plugins 
provide? I'm thinking managesieve access, sieve processing or expire here.





Regards


Christian



smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Dovecot] dovecot stats: useful data to gather

[Patrick Ben Koetter]

* Timo Sirainen :
> On 1.6.2012, at 23.58, Patrick Ben Koetter wrote:
> 
> > Besides pulling together all the data we also think it would be useful to 
> > have
> > an SNMP interface to access the stats.
> 
> I had thought about SNMP before also, but for the current kind of stats that
> are exported I couldn't think of any reasonable way to export them.

I am not an expert on SNMP, others in my office are, but as I understand it
there's no need for Dovecot to export the data. AFAIK Dovecot would have to
offer a subagent, which could be queried by a SNMP server.

If we need more knowledge on SNMP I can ask my folks on the team to give some
guidance. For the moment I found this:


> > Here are the stats we believe to be useful:
> > 
> > Login/Logout
> > - total number login success/time
> > - total number login failure/time
> ..
> 
> I'll look at these later in more detail, but some important questions / 
> design decisions:
> 
> Currently stats process only remembers things after Dovecot was started. I
> don't think getting these kind of numbers would really work like that.
> Perhaps all of the statistics should be permanently dumped to disk every
> ~minute or so + at shutdown and loaded at startup, so the numbers would at
> least normally always just increase since the first time Dovecot was
> started?

ACK. My understanding is: Statistical data are moments in time. The
application provides these snapshots. It is up to other protocols (e.g. SNMP)
and software (e.g. RRD) to gather and create time series and also to relate
data to each other in order to come up with ratios, timelines etc.

This might be a good opportunity to check out Howard's MDB database (in order
to get around potential future law suits concerning BDB usage ...).



> > Mailbox state
> > - Inflow rate (number incoming messages/time)
> > - Deleted rate (number \Deleted flagged messages/time)
> 
> These operations/time type of things I had hoped to be able to externalize
> :) If stats process simply gives the raw stats, the reader could do this
> kind of summing up. Otherwise .. well, I guess it could maybe keep track of
> the current ops/ and the reader would then have to read the
> value about once a minute or half or something. It wouldn't give exact
> results though.

ACK. I'd externalize them too. So dump the /time aspect and only give raw data
at moment of query.


> > Performance
> > - minimum time to write a message
> > - maximum time to write a message
> > - average time to write a message
> 
> Within last .. day? hour? minute? ..

Concerning "message write time": the time the last message had to be written.

In general the stats update interval should be configurable in order to adapt
it to the overall system performance. Makes no sense to bring down the server
by gathering stats every nano second unless one likes self-induced DOS. ;)

It would probably be a useful strategy to update internal data on every event
and answer SNMP queries from memory but write the data to disc every once in a
while to have them when the server restarts. Besides that I don't see a use
case for sharing such data between processes such as exporting them to
memcache or anything alike. Do you?

p@rick

-- 
state of mind ()

http://www.state-of-mind.de

Franziskanerstraße 15  Telefon +49 89 3090 4664
81669 München  Telefax +49 89 3090 4666

Amtsgericht MünchenPartnerschaftsregister PR 563

Re: [Dovecot] dovecot stats: useful data to gather

[Timo Sirainen]

On 1.6.2012, at 23.58, Patrick Ben Koetter wrote:

> Besides pulling together all the data we also think it would be useful to have
> an SNMP interface to access the stats.

I had thought about SNMP before also, but for the current kind of stats that 
are exported I couldn't think of any reasonable way to export them.

> Here are the stats we believe to be useful:
> 
> Login/Logout
> - total number login success/time
> - total number login failure/time
..

I'll look at these later in more detail, but some important questions / design 
decisions:

Currently stats process only remembers things after Dovecot was started. I 
don't think getting these kind of numbers would really work like that. Perhaps 
all of the statistics should be permanently dumped to disk every ~minute or so 
+ at shutdown and loaded at startup, so the numbers would at least normally 
always just increase since the first time Dovecot was started?

> Mailbox state
> - Inflow rate (number incoming messages/time)
> - Deleted rate (number \Deleted flagged messages/time)

These operations/time type of things I had hoped to be able to externalize :) 
If stats process simply gives the raw stats, the reader could do this kind of 
summing up. Otherwise .. well, I guess it could maybe keep track of the current 
ops/ and the reader would then have to read the value about once 
a minute or half or something. It wouldn't give exact results though.

> Performance
> - minimum time to write a message
> - maximum time to write a message
> - average time to write a message

Within last .. day? hour? minute? ..