logging imap searches ?
Hi thanks to this list support I'm using Dovecot's MailLog plugin http://wiki2.dovecot.org/Plugins/MailLog to log the following imap events: delete undelete expunge copy mailbox_delete mailbox_rename flag_change save mailbox_create and I'm very happy. Question: is it possible to also log imap searches https://tools.ietf.org/html/rfc3501#section-6.4.4 ? Thank you, Mike
Re: [SPAM: high] imap logging ?
Works wonders thank you!
Finally I can use my accusation finger with my users!
http://cdn.scooppick.com/wp-content/uploads/2015/03/Keeps-Talking.jpg
On Thu, 26 Nov 2015 08:52:55 +0100
Urban Loesch wrote:
> Hi,
>
> perhaps this is what you need.
>
> http://wiki2.dovecot.org/Plugins/MailLog
>
> No "mail_debug enabled" neccessary.
>
> Regards
> Urban
>
> Am 26.11.2015 um 07:51 schrieb mancyb...@gmail.com:
> > Hi I'm trying to log my users imap actions, like when creating a folder,
> > moving an email or deleting an email.
> > So I've enabled 'mail_debug' and I'm checking /var/log/dovecot/debug.log
> > this is what happens when I delete an email:
> >
> > Nov 26 07:46:38 auth-worker(1555): Debug: sql(XXX,127.0.0.1): query: SELECT
> > password FROM mailbox WHERE username = 'XXX' and active = 1 and
> > restrictedAccess = 0
> > Nov 26 07:46:38 auth: Debug: client out: OK 1 user=XXX
> > Nov 26 07:46:38 auth-worker(1555): Debug: sql(XXX,127.0.0.1): SELECT
> > '/var/vmail/XXX/XXX' as home, 5000 AS uid, 5000 AS gid,
> > concat('*:storage=', quota) AS quota_rule FROM mailbox WHERE username =
> > 'XXX'
> > Nov 26 07:46:38 auth: Debug: master out: USER 374472705 XXX
> > home=/var/vmail/XXX/XXX uid=5000gid=5000
> > quota_rule=*:storage=524288
> > Nov 26 07:46:38 imap(XXX): Debug: Effective uid=5000, gid=5000,
> > home=/var/vmail/XXX/XXX
> > Nov 26 07:46:38 imap(XXX): Debug: Quota root: name=User quota
> > backend=maildir args=
> > Nov 26 07:46:38 imap(XXX): Debug: Quota rule: root=User quota mailbox=*
> > bytes=536870912 messages=0
> > Nov 26 07:46:38 imap(XXX): Debug: Quota rule: root=User quota mailbox=Trash
> > bytes=+104857600 messages=0
> > Nov 26 07:46:38 imap(XXX): Debug: maildir++:
> > root=/var/vmail/XXX/XXX/Maildir, index=/var/vmail/XXX/XXX/Maildir/indexes,
> > control=, inbox=/var/vmail/XXX/XXX/Maildir, alt=
> >
> > and when creating a folder, access an email or moving an email, the output
> > is basically the same:
> > I'm unable to find the actual IMAP command.
> >
> > So, question: is there a way to log IMAP commands to a file ?
> >
> > Thank you,
> > Mike
> >
imap logging ?
Hi I'm trying to log my users imap actions, like when creating a folder, moving
an email or deleting an email.
So I've enabled 'mail_debug' and I'm checking /var/log/dovecot/debug.log
this is what happens when I delete an email:
Nov 26 07:46:38 auth-worker(1555): Debug: sql(XXX,127.0.0.1): query: SELECT
password FROM mailbox WHERE username = 'XXX' and active = 1 and
restrictedAccess = 0
Nov 26 07:46:38 auth: Debug: client out: OK 1 user=XXX
Nov 26 07:46:38 auth-worker(1555): Debug: sql(XXX,127.0.0.1): SELECT
'/var/vmail/XXX/XXX' as home, 5000 AS uid, 5000 AS gid, concat('*:storage=',
quota) AS quota_rule FROM mailbox WHERE username = 'XXX'
Nov 26 07:46:38 auth: Debug: master out: USER 374472705 XXX
home=/var/vmail/XXX/XXX uid=5000gid=5000
quota_rule=*:storage=524288
Nov 26 07:46:38 imap(XXX): Debug: Effective uid=5000, gid=5000,
home=/var/vmail/XXX/XXX
Nov 26 07:46:38 imap(XXX): Debug: Quota root: name=User quota backend=maildir
args=
Nov 26 07:46:38 imap(XXX): Debug: Quota rule: root=User quota mailbox=*
bytes=536870912 messages=0
Nov 26 07:46:38 imap(XXX): Debug: Quota rule: root=User quota mailbox=Trash
bytes=+104857600 messages=0
Nov 26 07:46:38 imap(XXX): Debug: maildir++: root=/var/vmail/XXX/XXX/Maildir,
index=/var/vmail/XXX/XXX/Maildir/indexes, control=,
inbox=/var/vmail/XXX/XXX/Maildir, alt=
and when creating a folder, access an email or moving an email, the output is
basically the same:
I'm unable to find the actual IMAP command.
So, question: is there a way to log IMAP commands to a file ?
Thank you,
Mike
Re: OT - Re: hunting the fatty
On Wed, 11 Nov 2015 10:20:07 -0600 Rick Romero wrote: > Quoting mancyb...@gmail.com: > > > On Wed, 11 Nov 2015 09:50:29 -0600 > > Rick Romero wrote: > > > >> LOL. > >> > >> This is a horrible subject line. I've been trying to resolve a DDOS > >> issue, and ignoring a lot of email. > >> > >> Here I had been thinking this was a sex-spam, and I just got around to > >> wondering why the spam system isn't working quite right and they kept > >> coming in. :P > >> > >> Not sure if this was suggested, but use SSD's. With a ton of mail > >> combined with Maildir random reads/writes, SSD's are like night and day. > >> > >> Rick > > > > Very sorry about nasty subject ;) > > Btw is it worth the offload on i/o by having some Dovecot index or cache > > on an SSD ? > > Lately I've separated the indexes onto another SSD-based volume, just to > save read/writes, but I don't think it actually improved anything. > > Rick I'm going to try dm-cache and flashcache asap ..
Re: OT - Re: hunting the fatty
On Wed, 11 Nov 2015 09:50:29 -0600 Rick Romero wrote: > LOL. > > This is a horrible subject line. I've been trying to resolve a DDOS > issue, and ignoring a lot of email. > > Here I had been thinking this was a sex-spam, and I just got around to > wondering why the spam system isn't working quite right and they kept > coming in. :P > > Not sure if this was suggested, but use SSD's. With a ton of mail > combined with Maildir random reads/writes, SSD's are like night and day. > > Rick Very sorry about nasty subject ;) Btw is it worth the offload on i/o by having some Dovecot index or cache on an SSD ?
Re: hunting the fatty
On Wed, 11 Nov 2015 16:34:33 +0100 Christian Kivalo wrote: > On 2015-11-11 03:44, mancyb...@gmail.com wrote: > > On Tue, 10 Nov 2015 08:50:50 +0100 > > Christian Kivalo wrote: > > > >> Hi, > >> > >> On 2015-11-10 01:44, mancyb...@gmail.com wrote: > >> > Hello dear list, > >> > I've recently discovered 'doveadm stats' and I'm trying to use > >> > "doveadm stats dump user" and "doveadm stats dump session" > >> > to understand the pop/imap users that put more stress on the hard > >> > disks. > >> > > >> > My problem is that some users refuse to delete their emails from the > >> > server, > >> > so they keep 20GB of maildir files on the server, the webmail > >> > (roundcube) takes forever to open the inbox, > >> > the imap searches takes forever > >> > and meanwhile all the users wait. > >> > (already tried roundcube + memcache(d) but didn't help) > >> > >> What is forever in your context? > >> I'm using roundcube and a folder with about 78k mails opens in < 1 sec > >> unsorted. A folder with about 37k messages from a mailinglist and > >> thread > >> sort takes < 3 sec. My roundcube shows 200 messages per page by > >> default. > >> On a side note, are you using an imap proxy for roundcube? It doesn't > >> help you with your dovecot problem but it speeds up roundcube. > >> > >> To speed up imap searches i can recommend to implement fts-solr with > >> dovecot (or maybe fts-elasticsearch, am wanting to try that but solr > >> works...). That will speed up your searches after mailboxes are > >> indexed. > >> > >> > So my problem is not the storage usage itself: > >> > I don't care if the user gets tons of emails with big attachments; > >> > my problem is when the user opens / searches an imap folder with more > >> > than 10K mails > >> > and iostat util goes 100% for minutes. > >> > >> Dovecot should be very quick to open even folders with a huge amount > >> of > >> files due to its indexes. > >> > >> I'm unable to reproduce any significant numbers in iostat when > >> accessing > >> large mailfolders with roundcube. > >> > >> Whats your configuration, filesystem, ... > >> > >> > So I've enabled dovecot's stats and enjoying "doveadm stats top", > >> > "stats-top.pl" and "doveadm stats dump user/session", > >> > but talking about "doveadm stats dump user" and its output fields: > >> > > >> > user reset_timestamp last_update num_logins num_cmds > >> > user_cpusys_cpu min_faults maj_faults vol_cs invol_cs > >> >disk_input disk_output read_count read_bytes > >> > write_count write_bytes mail_lookup_pathmail_lookup_attr > >> >mail_read_count mail_read_bytes mail_cache_hits > >> > > >> > I'm not sure which of those fields can help me > >> > and I can't find any relevant documentation. > >> > > >> > So here are my questions: > >> > > >> > 1. is there a documentation for those 21 fields and for 'doveadm > >> > stats' in general ? > >> > 2. what's the difference between disk_output, read_bytes, read_count > >> > and mail_read_bytes ? > >> > 3. which field of those is, in your opinion, more representative for > >> > expressing the workload that gives me problems ? > >> > 4. which settings do I need to store 1 week worth of stats ? > >> > > >> > I'm currenty using the 'standard' values: > >> > > >> > stats_refresh = 30 secs > >> > stats_track_cmds = yes > >> > stats_memory_limit = 16 M > >> > stats_command_min_time = 1 mins > >> > stats_domain_min_time = 12 hours > >> > stats_ip_min_time = 12 hours > >> > stats_session_min_time = 15 mins > >> > stats_user_min_time = 1 hours > >> > > >> > Can you please tell me the correct parameters to store 1 week of stats > >> > ? > >> > >> For stats somebody else has to jump in, i have only enabled the plugin > >> to see what to get out of it but not made any use of
Re: hunting the fatty
On Tue, 10 Nov 2015 08:50:50 +0100 Christian Kivalo wrote: > Hi, > > On 2015-11-10 01:44, mancyb...@gmail.com wrote: > > Hello dear list, > > I've recently discovered 'doveadm stats' and I'm trying to use > > "doveadm stats dump user" and "doveadm stats dump session" > > to understand the pop/imap users that put more stress on the hard > > disks. > > > > My problem is that some users refuse to delete their emails from the > > server, > > so they keep 20GB of maildir files on the server, the webmail > > (roundcube) takes forever to open the inbox, > > the imap searches takes forever > > and meanwhile all the users wait. > > (already tried roundcube + memcache(d) but didn't help) > > What is forever in your context? > I'm using roundcube and a folder with about 78k mails opens in < 1 sec > unsorted. A folder with about 37k messages from a mailinglist and thread > sort takes < 3 sec. My roundcube shows 200 messages per page by default. > On a side note, are you using an imap proxy for roundcube? It doesn't > help you with your dovecot problem but it speeds up roundcube. > > To speed up imap searches i can recommend to implement fts-solr with > dovecot (or maybe fts-elasticsearch, am wanting to try that but solr > works...). That will speed up your searches after mailboxes are indexed. > > > So my problem is not the storage usage itself: > > I don't care if the user gets tons of emails with big attachments; > > my problem is when the user opens / searches an imap folder with more > > than 10K mails > > and iostat util goes 100% for minutes. > > Dovecot should be very quick to open even folders with a huge amount of > files due to its indexes. > > I'm unable to reproduce any significant numbers in iostat when accessing > large mailfolders with roundcube. > > Whats your configuration, filesystem, ... > > > So I've enabled dovecot's stats and enjoying "doveadm stats top", > > "stats-top.pl" and "doveadm stats dump user/session", > > but talking about "doveadm stats dump user" and its output fields: > > > > userreset_timestamp last_update num_logins num_cmds > > user_cpusys_cpu min_faults maj_faults vol_cs invol_cs > > disk_input disk_output read_count read_bytes > > write_count write_bytes mail_lookup_pathmail_lookup_attr > > mail_read_count mail_read_bytes mail_cache_hits > > > > I'm not sure which of those fields can help me > > and I can't find any relevant documentation. > > > > So here are my questions: > > > > 1. is there a documentation for those 21 fields and for 'doveadm > > stats' in general ? > > 2. what's the difference between disk_output, read_bytes, read_count > > and mail_read_bytes ? > > 3. which field of those is, in your opinion, more representative for > > expressing the workload that gives me problems ? > > 4. which settings do I need to store 1 week worth of stats ? > > > > I'm currenty using the 'standard' values: > > > > stats_refresh = 30 secs > > stats_track_cmds = yes > > stats_memory_limit = 16 M > > stats_command_min_time = 1 mins > > stats_domain_min_time = 12 hours > > stats_ip_min_time = 12 hours > > stats_session_min_time = 15 mins > > stats_user_min_time = 1 hours > > > > Can you please tell me the correct parameters to store 1 week of stats > > ? > > For stats somebody else has to jump in, i have only enabled the plugin > to see what to get out of it but not made any use of it. > > Please share your doveconf -n output > > > Thank you, > > Mike > > regards > christian By 'forever' I mean more than 1 minute. So there is no documentation / manual for 'doveadm stats' ? Do I have to read the source to know which field does what ? I mean the output fields of "doveadm stats dump user": userreset_timestamp last_update num_logins num_cmds user_cpusys_cpu min_faults maj_faults vol_cs invol_cs disk_input disk_output read_count read_bytes write_count write_bytes mail_lookup_pathmail_lookup_attrmail_read_count mail_read_bytes mail_cache_hits what's the difference between disk_output, read_bytes, read_count and mail_read_bytes ? (sorry to restate the same question, just making sure about it) Thank you, Mike
Re: hunting the fatty
On Tue, 10 Nov 2015 08:50:50 +0100 Christian Kivalo wrote: > Hi, > > On 2015-11-10 01:44, mancyb...@gmail.com wrote: > > Hello dear list, > > I've recently discovered 'doveadm stats' and I'm trying to use > > "doveadm stats dump user" and "doveadm stats dump session" > > to understand the pop/imap users that put more stress on the hard > > disks. > > > > My problem is that some users refuse to delete their emails from the > > server, > > so they keep 20GB of maildir files on the server, the webmail > > (roundcube) takes forever to open the inbox, > > the imap searches takes forever > > and meanwhile all the users wait. > > (already tried roundcube + memcache(d) but didn't help) > > What is forever in your context? > I'm using roundcube and a folder with about 78k mails opens in < 1 sec > unsorted. A folder with about 37k messages from a mailinglist and thread > sort takes < 3 sec. My roundcube shows 200 messages per page by default. > On a side note, are you using an imap proxy for roundcube? It doesn't > help you with your dovecot problem but it speeds up roundcube. > > To speed up imap searches i can recommend to implement fts-solr with > dovecot (or maybe fts-elasticsearch, am wanting to try that but solr > works...). That will speed up your searches after mailboxes are indexed. > > > So my problem is not the storage usage itself: > > I don't care if the user gets tons of emails with big attachments; > > my problem is when the user opens / searches an imap folder with more > > than 10K mails > > and iostat util goes 100% for minutes. > > Dovecot should be very quick to open even folders with a huge amount of > files due to its indexes. > > I'm unable to reproduce any significant numbers in iostat when accessing > large mailfolders with roundcube. > > Whats your configuration, filesystem, ... > > > So I've enabled dovecot's stats and enjoying "doveadm stats top", > > "stats-top.pl" and "doveadm stats dump user/session", > > but talking about "doveadm stats dump user" and its output fields: > > > > userreset_timestamp last_update num_logins num_cmds > > user_cpusys_cpu min_faults maj_faults vol_cs invol_cs > > disk_input disk_output read_count read_bytes > > write_count write_bytes mail_lookup_pathmail_lookup_attr > > mail_read_count mail_read_bytes mail_cache_hits > > > > I'm not sure which of those fields can help me > > and I can't find any relevant documentation. > > > > So here are my questions: > > > > 1. is there a documentation for those 21 fields and for 'doveadm > > stats' in general ? > > 2. what's the difference between disk_output, read_bytes, read_count > > and mail_read_bytes ? > > 3. which field of those is, in your opinion, more representative for > > expressing the workload that gives me problems ? > > 4. which settings do I need to store 1 week worth of stats ? > > > > I'm currenty using the 'standard' values: > > > > stats_refresh = 30 secs > > stats_track_cmds = yes > > stats_memory_limit = 16 M > > stats_command_min_time = 1 mins > > stats_domain_min_time = 12 hours > > stats_ip_min_time = 12 hours > > stats_session_min_time = 15 mins > > stats_user_min_time = 1 hours > > > > Can you please tell me the correct parameters to store 1 week of stats > > ? > > For stats somebody else has to jump in, i have only enabled the plugin > to see what to get out of it but not made any use of it. > > Please share your doveconf -n output > > > Thank you, > > Mike > > regards > christian By 'forever' I mean more than 1 minute. So there is no documentation / manual for 'doveadm stats' ? Do I have to read the source to know which field does what ? I mean the output fields of "doveadm stats dump user": userreset_timestamp last_update num_logins num_cmds user_cpusys_cpu min_faults maj_faults vol_cs invol_cs disk_input disk_output read_count read_bytes write_count write_bytes mail_lookup_pathmail_lookup_attrmail_read_count mail_read_bytes mail_cache_hits what's the difference between disk_output, read_bytes, read_count and mail_read_bytes ? (sorry to restate the same question, just making sure about it) Thank you, Mike
hunting the fatty
Hello dear list, I've recently discovered 'doveadm stats' and I'm trying to use "doveadm stats dump user" and "doveadm stats dump session" to understand the pop/imap users that put more stress on the hard disks. My problem is that some users refuse to delete their emails from the server, so they keep 20GB of maildir files on the server, the webmail (roundcube) takes forever to open the inbox, the imap searches takes forever and meanwhile all the users wait. (already tried roundcube + memcache(d) but didn't help) So my problem is not the storage usage itself: I don't care if the user gets tons of emails with big attachments; my problem is when the user opens / searches an imap folder with more than 10K mails and iostat util goes 100% for minutes. So I've enabled dovecot's stats and enjoying "doveadm stats top", "stats-top.pl" and "doveadm stats dump user/session", but talking about "doveadm stats dump user" and its output fields: userreset_timestamp last_update num_logins num_cmds user_cpusys_cpu min_faults maj_faults vol_cs invol_cs disk_input disk_output read_count read_bytes write_count write_bytes mail_lookup_pathmail_lookup_attrmail_read_count mail_read_bytes mail_cache_hits I'm not sure which of those fields can help me and I can't find any relevant documentation. So here are my questions: 1. is there a documentation for those 21 fields and for 'doveadm stats' in general ? 2. what's the difference between disk_output, read_bytes, read_count and mail_read_bytes ? 3. which field of those is, in your opinion, more representative for expressing the workload that gives me problems ? 4. which settings do I need to store 1 week worth of stats ? I'm currenty using the 'standard' values: stats_refresh = 30 secs stats_track_cmds = yes stats_memory_limit = 16 M stats_command_min_time = 1 mins stats_domain_min_time = 12 hours stats_ip_min_time = 12 hours stats_session_min_time = 15 mins stats_user_min_time = 1 hours Can you please tell me the correct parameters to store 1 week of stats ? Thank you, Mike
[Dovecot] dovecot and cacti (snmp ?)
Hi All, I've put online a postfix+dovecot+mysql+roundcube server that will grow till serving around 1500 accounts. I've installed the usual cacti graphs, plus the beautiful percona suite for mysql's cacti plus some graphs for iops and iostat. I was lurking around for the best solution to graph dovecot usage, the most detailed possible, in order to prevent and foresee problems. I'm finding sparse results, I'm not sure which one is the most current / complete. Do you have any suggestions ? Thank you very much for supporting, regards and have a nice day, Mike
Re: [Dovecot] dovecot dimensioning
Hi thanks to all for your pointers, very interesting. I'll do my homework about local SAS disks and SANs. Thanks and regards :) Mike On Fri, 02 Nov 2012 09:39:56 -0400 Oscar del Rio wrote: > On 11/ 1/12 07:12 PM, Gregory Finch wrote: > > I believe that Javier is talking IOPS, not throughput. You need a > > storage system that is able to provide a high random read + write IOPS. > > That single drive is going to cause you issues when things get busy. > > And a disaster when the single drive fails! >
Re: [Dovecot] dovecot dimensioning
Hi Javier, yes i see. Storage for now is just a 3tb sata2 hdd, so i guess that write performance is around 100 MB/s What do you think about that ? Thanks and regards, Mike On Thu, 1 Nov 2012 19:55:35 +0100 Javier de Miguel Rodriguez wrote: > > IMHO, fast storage is the main requirement for dovecot. > > > > El 01/11/2012, a las 16:20, "mancyb...@gmail.com" > escribió: > > > Hi All, first post here, nice to meet you :) > > > > I've been using dovecot + postfix + clamav + spamassassin for years but > > always with few users. > > > > Now i must build a server for 1500 users, > > they will use various email software (thunderbird, outlook, ..) > > the webmail (i'm not sure if squirrelmail or roundcube) > > and blackberry devices (with the BIS service). > > > > There will be around 1000 domains (virtual_domains) and postfix will read > > its users and domains from mysql. > > > > Let's say that they will send and receive around 5000 emails per hour (10K > > in total per hour). > > > > I was thinking to use karspersky server instead of clamav. > > > > I already have the hardware, it is a server with 4 physical cpu (Intel Xeon > > E5504 @ 2.00GHz) and 24 gb of ram, > > do you think it is enough ? > > > > > > Thanks for supporting, > > regards and have a nice day, > > Mike
[Dovecot] dovecot dimensioning
Hi All, first post here, nice to meet you :) I've been using dovecot + postfix + clamav + spamassassin for years but always with few users. Now i must build a server for 1500 users, they will use various email software (thunderbird, outlook, ..) the webmail (i'm not sure if squirrelmail or roundcube) and blackberry devices (with the BIS service). There will be around 1000 domains (virtual_domains) and postfix will read its users and domains from mysql. Let's say that they will send and receive around 5000 emails per hour (10K in total per hour). I was thinking to use karspersky server instead of clamav. I already have the hardware, it is a server with 4 physical cpu (Intel Xeon E5504 @ 2.00GHz) and 24 gb of ram, do you think it is enough ? Thanks for supporting, regards and have a nice day, Mike
