Benchmarking auth
Hello, I'm attempting to use imaptest to test auths/sec on my imap server, to simulate the number I'm getting now on a new server. Based on my readings of dovecot stats outputs, my current machine is doing somewhere between 11.6 and 196 logins/sec at its busiest, and I want to make sure the new server can handle that amount. With imaptest I attempt to simulate this by just simulating login/logout speed, but I'm not sure I understand the results and where things are being blocked. For testing purposes, I set: mail_max_userip_connections = 6000 mail_max_userip_connections = 6000 auth_worker_max_count = 240 If you have some additional suggestions for what I can also tweak here to improve this, I would like to know. I then ran imaptest with: imaptest clients=196 user=test pass=testpw host=127.0.0.1 port=14300 - select=0 seed=123 secs=300 It doesn't seem like I can simulate *only* authentications/logins. I have to pair them with logouts. It makes me wonder how I can simulate the amount of load on my current live system on this system. How do I read the output? I don't really understand what is being shown here: Logi Logo 100% 100% 00 100/100 [51%] does this mean that it tried 100 logins and 100 logouts and 51% succeeded? What are the first two columns? 77 104/121 [61%] 66 131/139 [70%] 33 137/148 (84 stalled >3s) [75%] 84 stalled for more than 3 seconds, how does this relate to the other values presented? 55 155/163 (79 stalled >3s) [83%] 11 166/166 (107 stalled >3s) [84%] 22 168/172 (128 stalled >3s) [87%] 65 179/190 (140 stalled >3s) [96%] 22 22 182/196 (136 stalled >3s) 43 195/196 (133 stalled >3s) 5845 129 ms/cmd avg I really wonder if these first two columns are ms/cmds? It then continues: Logi Logo 100% 100% 10 11 195/196 (131 stalled >3s) 78 194/196 (150 stalled >3s) 33 194/196 (171 stalled >3s) 109 192/196 (166 stalled >3s) 22 22 182/196 (156 stalled >3s) 89 193/196 (153 stalled >3s) - 2 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 7 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 8 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 9 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 11 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 12 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 14 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 22 stalled for 16 secs in command: 1 LOGIN "test" "testpw" - 25 stalled for 16 secs in command: 1 LOGIN "test" "testpw" ... it starts to build up like this, until I hit control-c twice: - 100 stalled for 17 secs in command: 1 LOGIN "test" "testpw" ^CInfo: Received second SIGINT - stopping immediately 11347 118 ms/cmd avg Totals: Logi Logo 100% 100% 130 134 What are these totals? Thanks! micah
Re: Apparent Maildir permission issue
On Wed, 25 Jan 2017 08:01:00 +0100 (CET) Steffen Kaiserwrote: > 1) Why does both UIDs 326 and 10001 translate back to HPRS\mark ? > What HPRS\mark translates to? > > > Permission on that folder are: > > > > $ ls -ld /home/HPRS/mark/Maildir > > drwx-- 17 HPRS\mark domusers 4096 Dec 7 23:07 /home/HPRS/mark/Maildir/ > > 2) I guess this HPRS\mark is 10001 ? (And not 326) > > > Permissions are unchanged since before the backup. > > "backup"? You've restored the Maildir's from somewhere else? What was the > _numerical_ UID within the backup and what is it now? "backup" meaning I looked at the permissions on an older routine, backup. No, I did not restore anything. BUT ... I found the problem. I upgraded Samba4 10 days ago from version 4.2.12 to 4.4.8 and, in the course of researching this problem, I found that the A/D authentication was broken: with 4.2.12 on AD/DC: $ getent passwd mark HPRS\mark:*:10001:1:Mark Foley:/home/HPRS/mark:/bin/false With 4.4.8 on AD/DC: $ getent passwd mark HPRS\mark:*:326:100:Mark Foley:/home/HPRS/mark:/bin/bash The new version of Samba is giving me this bogus UID:GID. I've no idea why. I have posted messages on the Samba List asking for help on this. Email clients authenticate with Dovecot via Kerberos/GSSAPI and Dovecot was therefore trying to use 326:100 to access Maildir files/directories created with owner 10001:1. I've done a workaround by added the correct UID, GID for this user to /etc/passwd, although one is not suppose to have AD users in /etc/passwd. However, that is working for the time being. If anyone on this list has had this experience and knows what needs to be fixed, please let me know! Thanks -- Mark
2.2.26.0 : accessing "mdbox_deleted" content destroys indexes
Accessing or listing "mdbox_deleted" contents seems to destroy MDBOX indexes. Examples of commands which triggers this problem ($home being the home directory of $user, and mail_location being mdbox:~/mdbox): doveadm -o mail="mdbox_deleted:$home/mdbox" -f table mailbox status -u "$user" 'messages vsize' INBOX doveadm -v import -s -u "$user" "mdbox_deleted:$home/mdbox" restored-mail ALL The above "doveadm mailbox status" command outputs an error: doveadm(user): Error: Log synchronization error at seq=1,offset=104908 for (in-memory index): Append with UID 1, but next_uid = 5227 doveadm(user): Warning: fscking index file (in-memory index) Subsequent "doveadm mailbox status -u $user 'messages vsize'" on the active mailbox report empty folders (null messages and vsize), whereas folders actually aren't empty. Workaround: this problem is corrected by a "doveadm search -u $user all", which obviously forces indexes to be rebuilt. Vesion: 2.2.26.0 (23d1de6) (Debian jessie-backports package) We did *not* have this problem in 2.2.24 version (previous Debian jessie-backports package). We use following mail_location with explicit DIRNAME (don't know if that matters): mail_location = mdbox:~/mdbox:DIRNAME=_@@_dbox-Mails_@@_ I tested with and without appending ":DIRNAME=_@@_dbox-Mails_@@_" to mail="mdbox_deleted:$home/mdbox" with same results. -- Benoit BRANCIARD Service InfraStructures (SIS) Direction du Système d'Information et des Usages Numériques (DSIUN) Université Paris 1 Panthéon-Sorbonne Centre Pierre Mendès France 90 rue de Tolbiac - 75634 Paris cedex 13 - France Bur. B406 - Tél +33 1 44 07 89 68 - Fax +33 1 44 07 89 66 Accueil: +33 1 44 07 89 65 - assistance-ds...@univ-paris1.fr http://dsi.univ-paris1.fr
Re: Moving to new password scheme
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 25 Jan 2017, @lbutlr wrote: On Jan 25, 2017, at 2:46 AM, Steffen Kaiserwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 25 Jan 2017, @lbutlr wrote: On Jan 25, 2017, at 1:09 AM, Alessio Cecchi wrote: Il 24/01/2017 23:29, @lbutlr ha scritto: dovecot is setup on a system with MD5-CRYPT password scheme for all users, and I would like to update this to something that is secure, probably SSHA256-CRYPT, but I want to do this seamlessly without the users having to jump through any hoops. The users are in mySQL (managed via postfixadmin) and the mailbox record simply stores the hash in the password field. Users access their accounts though IMAP MUAs or Roundcube. How would I setup my system so that if a user logs in and still has a $1$ password (MD5-CRYPT) their password will be encoded to the new SHCEME and then the SQL row updated with the $5$ password instead? Something where they are redirected after authentication to a page that forces them to renter their password (or choose a new one) is acceptable. And, while I am here, is it worthwhile to set the -r flag to a large number (like something over 100,000 which sets takes about 0.25 seconds to do on my machine)? Hi, you can convert password scheme during the login: http://wiki2.dovecot.org/HowTo/ConvertPasswordSchemes Thanks, I started to look into that and got stopped no the first step userdb { driver = prefetch } If I set that and reload dovecot users cannot login. dovecot: auth: Fatal: userdb prefetch: No args are supported: /etc/dovecot/dovecot-sql.conf.ext dovecot: master: Error: service(auth): command startup failed, throttling for 8 secs dovecot: imap-login: Disconnected: Auth process broken (disconnected before auth was ready, waited 4 secs): user=<>, I don't see no prefetch in your config. No, when I changed userdb { driver = passwd } to prefetch everything failed, so I changed it back immediately so people could login. That was the firs step on the page and I couldn’t get past it. The error may indicate that you replaced driver = sql by driver = prefetch, which is wrong. driver = sql is in the imap/sql section. The one I tried changing was the bare userdb declaration the just said driver - passwd. I guess I need to ADD another userdb declaration for the prefetch. Does the other int he file matter? I have local users stuff first and then the sql stuff later, but I’m not sure if that matters. yes, userdb's are checked in the same order as they appear in the config file(s). http://wiki2.dovecot.org/UserDatabase/Prefetch The idea described on the Wiki page is: During login, most often the same data is collected from the passdb as later from the userdb, therefore you can collect *all* information you would retrieve from userdb { } within passdb queries (that's why the home as userdb_home, \ uid as userdb_uid, gid as userdb_gid, '%w' as userdb_plain_pass entries; the prefix userdb_ indicates that data) and store it for later use by the prefetch database. That's why the prefetch userdb has to preceed the other ones, because if the passdb query filled in the values, the later userdb entries are ignored. So Place it first (or at least before all the sql stuff)? yep. You've noticed the '%w' as userdb_plain_pass ? That stores the plain password (if any) to the virtual prefetch userdb entry as field plain_pass. OK. Now, you are using two passdb's. the PAM passdb won't support this method, I guess. No, I’m not expecting it to. the local users are mostly my admin accounts and I can just change the passwords on those manually without an issue. I’ll keep at it. Thanks. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBWIiSrHz1H7kL/d9rAQJsZgf+MbLgAk7u3oUrFsVCwoU1yf013/PEUMs1 7bicH0GBotx5FNJt2KwHjjAizOzmHwdMrrIchdSVl6Fb62SOSdtwuykvmfnp2Rpu EUHBXlsyOpIytgDkyZcxnjFb4HyxbHccwoR8OWcFuknPVt/jMbwSxgAS9qjAlEnj wtvy01sn2L7ICevHRE6aaZfY3AeAEIkWPfWKoZLm5FQ6QL8ANnCj4QhKMW94It7Z EFuHx3EjvkUbyZ55fMPTYqPds8SmvW1waYVNSQD8xqZunpGhIOKSd+qIFVoStynX Nn5HG2mYnSKiBJf97UErIQJRkR4rY8DZMlS3RXncaSvl+Th9x5o0Ow== =Gpt5 -END PGP SIGNATURE-
Re: Moving to new password scheme
> On Jan 25, 2017, at 2:46 AM, Steffen Kaiser> wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Wed, 25 Jan 2017, @lbutlr wrote: >>> On Jan 25, 2017, at 1:09 AM, Alessio Cecchi wrote: >>> >>> Il 24/01/2017 23:29, @lbutlr ha scritto: dovecot is setup on a system with MD5-CRYPT password scheme for all users, and I would like to update this to something that is secure, probably SSHA256-CRYPT, but I want to do this seamlessly without the users having to jump through any hoops. The users are in mySQL (managed via postfixadmin) and the mailbox record simply stores the hash in the password field. Users access their accounts though IMAP MUAs or Roundcube. How would I setup my system so that if a user logs in and still has a $1$ password (MD5-CRYPT) their password will be encoded to the new SHCEME and then the SQL row updated with the $5$ password instead? Something where they are redirected after authentication to a page that forces them to renter their password (or choose a new one) is acceptable. And, while I am here, is it worthwhile to set the -r flag to a large number (like something over 100,000 which sets takes about 0.25 seconds to do on my machine)? >>> Hi, >>> >>> you can convert password scheme during the login: >>> >>> http://wiki2.dovecot.org/HowTo/ConvertPasswordSchemes >> >> Thanks, I started to look into that and got stopped no the first step >> >>> userdb { >>> driver = prefetch >>> } >> >> If I set that and reload dovecot users cannot login. >> >> dovecot: auth: Fatal: userdb prefetch: No args are supported: >> /etc/dovecot/dovecot-sql.conf.ext >> dovecot: master: Error: service(auth): command startup failed, throttling >> for 8 secs >> dovecot: imap-login: Disconnected: Auth process broken (disconnected before >> auth was ready, waited 4 secs): user=<>, > > I don't see no prefetch in your config. No, when I changed userdb { driver = passwd } to prefetch everything failed, so I changed it back immediately so people could login. That was the firs step on the page and I couldn’t get past it. > The error may indicate that you replaced driver = sql by driver = prefetch, > which is wrong. driver = sql is in the imap/sql section. The one I tried changing was the bare userdb declaration the just said driver - passwd. I guess I need to ADD another userdb declaration for the prefetch. Does the other int he file matter? I have local users stuff first and then the sql stuff later, but I’m not sure if that matters. > http://wiki2.dovecot.org/UserDatabase/Prefetch > > The idea described on the Wiki page is: > > During login, most often the same data is collected from the passdb as later > from the userdb, therefore you can collect *all* information you would > retrieve from userdb { } within passdb queries (that's why the home as > userdb_home, \ > uid as userdb_uid, gid as userdb_gid, '%w' as userdb_plain_pass entries; the > prefix userdb_ indicates that data) and store it for later use by the > prefetch database. > > That's why the prefetch userdb has to preceed the other ones, because if the > passdb query filled in the values, the later userdb entries are ignored. So Place it first (or at least before all the sql stuff)? > You've noticed the '%w' as userdb_plain_pass ? That stores the plain password > (if any) to the virtual prefetch userdb entry as field plain_pass. OK. > Now, you are using two passdb's. the PAM passdb won't support this method, I > guess. No, I’m not expecting it to. the local users are mostly my admin accounts and I can just change the passwords on those manually without an issue. I’ll keep at it. Thanks. -- Apple broke AppleScripting signatures in Mail.app, so no random signatures.
Timing information for passdb/userdb lookups
Hi, it would be nice to have an option to enable logging for timings without having to go with auth_verbose/auth_debug. If you try to track down if a slowdown for logging in comes from e.g. running out of login workers or if your auth backend is slow it would really help to get just a one line message per userdb/passdb with the timing for this lookup. something like: imap-login(): userdb ldap lookup time 0.4s imap-login(): passdb ldap lookup time 0.2s or even merging into one: imap-login(): userdb ldap lookup time 0.4s passdb ldap lookup time 0.2s darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
Re: Moving to new password scheme
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 25 Jan 2017, @lbutlr wrote: On Jan 25, 2017, at 1:09 AM, Alessio Cecchiwrote: Il 24/01/2017 23:29, @lbutlr ha scritto: dovecot is setup on a system with MD5-CRYPT password scheme for all users, and I would like to update this to something that is secure, probably SSHA256-CRYPT, but I want to do this seamlessly without the users having to jump through any hoops. The users are in mySQL (managed via postfixadmin) and the mailbox record simply stores the hash in the password field. Users access their accounts though IMAP MUAs or Roundcube. How would I setup my system so that if a user logs in and still has a $1$ password (MD5-CRYPT) their password will be encoded to the new SHCEME and then the SQL row updated with the $5$ password instead? Something where they are redirected after authentication to a page that forces them to renter their password (or choose a new one) is acceptable. And, while I am here, is it worthwhile to set the -r flag to a large number (like something over 100,000 which sets takes about 0.25 seconds to do on my machine)? Hi, you can convert password scheme during the login: http://wiki2.dovecot.org/HowTo/ConvertPasswordSchemes Thanks, I started to look into that and got stopped no the first step userdb { driver = prefetch } If I set that and reload dovecot users cannot login. dovecot: auth: Fatal: userdb prefetch: No args are supported: /etc/dovecot/dovecot-sql.conf.ext dovecot: master: Error: service(auth): command startup failed, throttling for 8 secs dovecot: imap-login: Disconnected: Auth process broken (disconnected before auth was ready, waited 4 secs): user=<>, I don't see no prefetch in your config. The error may indicate that you replaced driver = sql by driver = prefetch, which is wrong. http://wiki2.dovecot.org/UserDatabase/Prefetch The idea described on the Wiki page is: During login, most often the same data is collected from the passdb as later from the userdb, therefore you can collect *all* information you would retrieve from userdb { } within passdb queries (that's why the home as userdb_home, \ uid as userdb_uid, gid as userdb_gid, '%w' as userdb_plain_pass entries; the prefix userdb_ indicates that data) and store it for later use by the prefetch database. That's why the prefetch userdb has to preceed the other ones, because if the passdb query filled in the values, the later userdb entries are ignored. You've noticed the '%w' as userdb_plain_pass ? That stores the plain password (if any) to the virtual prefetch userdb entry as field plain_pass. Those fields are available later as environment variables, that's why /usr/local/etc/popafter.sh may use: 1 #!/bin/sh 2 DOVECOTPW=$(doveadm pw -s SHA512-CRYPT -p $PLAIN_PASS) to access it. Now, you are using two passdb's. the PAM passdb won't support this method, I guess. # 2.2.27 (c0f36b0): /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 10.3-RELEASE-p11 i386 auth_failure_delay = 5 secs auth_mechanisms = PLAIN LOGIN default_client_limit = 4096 default_process_limit = 1024 default_vsz_limit = 768 M disable_plaintext_auth = no first_valid_uid = 89 imap_id_log = * lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_log_format_elements = user=<%u> %r %m %c mail_location = maildir:~/Maildir mail_max_userip_connections = 90 namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox NotJunk { auto = subscribe } mailbox Sent { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocols = imap service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } } service imap-login { inet_listener imaps { port = 993 ssl = yes } } service imap-postlogin { executable = script-login /usr/local/etc/dovecot/afterlogin.sh user = $default_internal_user } ssl_cert = - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBWIhz93z1H7kL/d9rAQJUuQf/UVdPDGn0zVdyTPEG/rfiQGXONpTUArWa FI6h2amH7UwEXqfZ/Z3x3JmbFGgZ/bHavnqgsE+os19DtUIo4LYLBZvZDLA5FErt vSKlzObXzDaofKktIkpu40j3RimRJTjpgilTvh0ne+wJnaCHZTF4pD6YeR7BRHN3 CBCefg6UALG6XLZ9nTOA07l7q/GFSvCaWvqApXpjQwA4DrwwH0ZoFq+peVfblrr7 8oqIVZiWPgiqR2BRxyNpPLU5mOJIcCrWqrSBtpRxPXmqcTjXMgA7TioTs8Jm/ooE wWkKj8GBSlTZjbI1KycJlfPevSbsfxnLKdewpS7WI2Tcr4uEezWTrg== =cUh0 -END PGP SIGNATURE-
Re: Moving to new password scheme
> On Jan 25, 2017, at 1:09 AM, Alessio Cecchiwrote: > > Il 24/01/2017 23:29, @lbutlr ha scritto: >> dovecot is setup on a system with MD5-CRYPT password scheme for all users, >> and I would like to update this to something that is secure, probably >> SSHA256-CRYPT, but I want to do this seamlessly without the users having to >> jump through any hoops. >> >> The users are in mySQL (managed via postfixadmin) and the mailbox record >> simply stores the hash in the password field. Users access their accounts >> though IMAP MUAs or Roundcube. >> >> How would I setup my system so that if a user logs in and still has a $1$ >> password (MD5-CRYPT) their password will be encoded to the new SHCEME and >> then the SQL row updated with the $5$ password instead? Something where they >> are redirected after authentication to a page that forces them to renter >> their password (or choose a new one) is acceptable. >> >> And, while I am here, is it worthwhile to set the -r flag to a large number >> (like something over 100,000 which sets takes about 0.25 seconds to do on my >> machine)? >> > Hi, > > you can convert password scheme during the login: > > http://wiki2.dovecot.org/HowTo/ConvertPasswordSchemes Thanks, I started to look into that and got stopped no the first step > userdb { > driver = prefetch > } If I set that and reload dovecot users cannot login. dovecot: auth: Fatal: userdb prefetch: No args are supported: /etc/dovecot/dovecot-sql.conf.ext dovecot: master: Error: service(auth): command startup failed, throttling for 8 secs dovecot: imap-login: Disconnected: Auth process broken (disconnected before auth was ready, waited 4 secs): user=<>, # 2.2.27 (c0f36b0): /usr/local/etc/dovecot/dovecot.conf # OS: FreeBSD 10.3-RELEASE-p11 i386 auth_failure_delay = 5 secs auth_mechanisms = PLAIN LOGIN default_client_limit = 4096 default_process_limit = 1024 default_vsz_limit = 768 M disable_plaintext_auth = no first_valid_uid = 89 imap_id_log = * lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_log_format_elements = user=<%u> %r %m %c mail_location = maildir:~/Maildir mail_max_userip_connections = 90 namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox NotJunk { auto = subscribe } mailbox Sent { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocols = imap service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } } service imap-login { inet_listener imaps { port = 993 ssl = yes } } service imap-postlogin { executable = script-login /usr/local/etc/dovecot/afterlogin.sh user = $default_internal_user } ssl_cert =
Re: Moving to new password scheme
Il 24/01/2017 23:29, @lbutlr ha scritto: dovecot is setup on a system with MD5-CRYPT password scheme for all users, and I would like to update this to something that is secure, probably SSHA256-CRYPT, but I want to do this seamlessly without the users having to jump through any hoops. The users are in mySQL (managed via postfixadmin) and the mailbox record simply stores the hash in the password field. Users access their accounts though IMAP MUAs or Roundcube. How would I setup my system so that if a user logs in and still has a $1$ password (MD5-CRYPT) their password will be encoded to the new SHCEME and then the SQL row updated with the $5$ password instead? Something where they are redirected after authentication to a page that forces them to renter their password (or choose a new one) is acceptable. And, while I am here, is it worthwhile to set the -r flag to a large number (like something over 100,000 which sets takes about 0.25 seconds to do on my machine)? Hi, you can convert password scheme during the login: http://wiki2.dovecot.org/HowTo/ConvertPasswordSchemes Ciao -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice