Re: [Dovecot] Auth Issues - Urgent - Help!
No one else with opinions on this? Dave Quoting David Cunningham [EMAIL PROTECTED]: Yes, i telnet to port 143 and enter everything manually. Dave Quoting Charles Marcus [EMAIL PROTECTED]: On 11/19/2008 10:17 PM, David Cunningham wrote: Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered. Is this something wrong with my configuraiton, or the auth code itself? Maybe it is the mail client doing the caching... have you tested this on the command line? -- Best regards, Charles
Re: [Dovecot] Auth Issues - Urgent - Help!
On Wed, 2008-11-19 at 22:17 -0500, David Cunningham wrote: Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, Do you mean auth_cache_negative_ttl=0 by this? It only affects user not found caching. new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered. Is this something wrong with my configuraiton, or the auth code itself? The way it should work is that: 1) User logs in with password X which succeeds. 2) Password is changed to Y. 3) User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth succeeded, so it'll do an auth lookup, sees that the password was changed and caches it. But this can also happen: 1) User logs in with password X which succeeds. 2) Password is changed to Y. 3) User logs in with password X, which succeeds. Or: 1) User logs in with password X which succeeds. 2) User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth succeeded, so it'll do an auth lookup and sees that the password wasn't changed. 3) Password is changed to Y. 4) User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth failed, so it doesn't bother doing another lookup. Can you consistently make Dovecot behave differently as described above? signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Auth Issues - Urgent - Help!
On Fri, 2008-11-21 at 21:38 +0200, Timo Sirainen wrote: On Wed, 2008-11-19 at 22:17 -0500, David Cunningham wrote: Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, Do you mean auth_cache_negative_ttl=0 by this? It only affects user not found caching. new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered. Is this something wrong with my configuraiton, or the auth code itself? The way it should work is that: I also added these to Wiki with slightly better wording: http://wiki.dovecot.org/Authentication/Caching signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Auth Issues - Urgent - Help!
I think the last thing you say is exactly what is happening to me. I think the user is updating the password, but a slight delay in my LDAP replication is causing them to try the new password before it is actually the new password. Yes, I was refering to auth_cache_negative_ttl=0. I didn't realize that was user not found only. Is there any way to force the cache to check the password for anything that was not previously cached as being the correct password? Dave Quoting Timo Sirainen [EMAIL PROTECTED]: On Wed, 2008-11-19 at 22:17 -0500, David Cunningham wrote: Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, Do you mean auth_cache_negative_ttl=0 by this? It only affects user not found caching. new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered. Is this something wrong with my configuraiton, or the auth code itself? The way it should work is that: 1) User logs in with password X which succeeds. 2) Password is changed to Y. 3) User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth succeeded, so it'll do an auth lookup, sees that the password was changed and caches it. But this can also happen: 1) User logs in with password X which succeeds. 2) Password is changed to Y. 3) User logs in with password X, which succeeds. Or: 1) User logs in with password X which succeeds. 2) User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth succeeded, so it'll do an auth lookup and sees that the password wasn't changed. 3) Password is changed to Y. 4) User logs in with password Y. Dovecot sees that X != Y, but it sees that the previous auth failed, so it doesn't bother doing another lookup. Can you consistently make Dovecot behave differently as described above?
Re: [Dovecot] Auth Issues - Urgent - Help!
On Fri, 2008-11-21 at 14:50 -0500, David Cunningham wrote:
Is there any way to force the cache to
check the password for anything that was not previously cached as
being the correct password?
Nope. Hmm. Perhaps there should be a different TTL for that. I don't
really like adding new settings though.
For now you can at least do it by modifying sources:
src/auth/passdb-cache.c:
if (ret == 0 node-last_success) {
Change it to:
if (ret == 0) {
signature.asc
Description: This is a digitally signed message part
Re: [Dovecot] Auth Issues - Urgent - Help!
Wowa, that's easy enough. I will do that the next time that I upgrade
in a few weeks.
Dave
Quoting Timo Sirainen [EMAIL PROTECTED]:
On Fri, 2008-11-21 at 14:50 -0500, David Cunningham wrote:
Is there any way to force the cache to
check the password for anything that was not previously cached as
being the correct password?
Nope. Hmm. Perhaps there should be a different TTL for that. I don't
really like adding new settings though.
For now you can at least do it by modifying sources:
src/auth/passdb-cache.c:
if (ret == 0 node-last_success) {
Change it to:
if (ret == 0) {
Re: [Dovecot] Auth Issues - Urgent - Help!
On 11/19/2008 10:17 PM, David Cunningham wrote: Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered. Is this something wrong with my configuraiton, or the auth code itself? Maybe it is the mail client doing the caching... have you tested this on the command line? -- Best regards, Charles
Re: [Dovecot] Auth Issues - Urgent - Help!
Yes, i telnet to port 143 and enter everything manually. Dave Quoting Charles Marcus [EMAIL PROTECTED]: On 11/19/2008 10:17 PM, David Cunningham wrote: Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered. Is this something wrong with my configuraiton, or the auth code itself? Maybe it is the mail client doing the caching... have you tested this on the command line? -- Best regards, Charles
Re: [Dovecot] Auth Issues - Urgent - Help!
Well, most of my issues are gone with adding auth cache. However, I am having an issue. Sometimes, even though cache incorrect passwords is disabled, new passwords do not work. It would seem that once a user logs in with one password successfully the cache does not automatically retry if the user tries a different passwords. I would think that the auth cache should check to see if the password changed on the ldap server if something other than the cached password is entered. Is this something wrong with my configuraiton, or the auth code itself? Thanks, Dave Quoting Timo Sirainen [EMAIL PROTECTED]: On Wed, 2008-10-08 at 08:01 -0400, David Cunningham wrote: After a few hours of running, I get tons of the following errors in my logs: dovecot: Oct 08 07:41:50 Error: auth(default): ldap([EMAIL PROTECTED],x.x.x.x): Request queue is full BTW. I improved this error message slightly to also tell how many seconds old data is in the queue. http://hg.dovecot.org/dovecot-1.1/rev/0329dc4df5ed I guess you're using auth binds? If you weren't, I think it wouldn't be possible to fill the queue.
[Dovecot] Auth Issues - Urgent - Help!
After a few hours of running, I get tons of the following errors in my logs: dovecot: Oct 08 07:41:50 Error: auth(default): ldap([EMAIL PROTECTED],x.x.x.x): Request queue is full I removed the username and IP, obviously. Any idea how to stop this? I have about 5 Thousand users using horde that login ever 1-5 minutes to refresh their page. I assume it is a setting, but I am confused as to why it doesn't happen almost right away. It seems to take some time to build up. Please help! This is taking my webmail system down hourly.
Re: [Dovecot] Auth Issues - Urgent - Help!
On Oct 8, 2008, at 8:01 AM, David Cunningham wrote: After a few hours of running, I get tons of the following errors in my logs: dovecot: Oct 08 07:41:50 Error: auth(default): ldap([EMAIL PROTECTED],x.x.x.x): Request queue is full I removed the username and IP, obviously. Any idea how to stop this? I have about 5 Thousand users using horde that login ever 1-5 minutes to refresh their page. I assume it is a setting, but I am confused as to why it doesn't happen almost right away. It seems to take some time to build up. Please help! This is taking my webmail system down hourly. dovecot -n? Hunch is login_max_processes_count is too low. http://wiki.dovecot.org/LoginProcess hth, JL smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] Auth Issues - Urgent - Help!
Here is my dovecot -n: # 1.1.3: /etc/dovecot.conf log_path: /var/log/dovecot info_log_path: /var/log/dovecot-info login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_greeting: MECnet Mail System, Authorized Use Only, Please Log In. login_process_per_connection: no login_process_size: 1024 login_max_processes_count: 1024 login_max_connections: 1024 max_mail_processes: 5 verbose_proctitle: yes first_valid_uid: 50 mail_uid: 93 mail_gid: 12 mail_location: maildir:/var/spool/maildirs/%d/%n/Maildir mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_process_size: 1024 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 namespace: type: private prefix: INBOX. inbox: yes list: yes subscriptions: yes auth default: verbose: yes passdb: driver: ldap args: /etc/dovecot-ldap.conf plugin: quota: maildir My buddy and I found #define DB_LDAP_MAX_QUEUE_SIZE 1024 in the db-ldap.h file in the source. We believe we hitting this threshold for some reason. So, we are looking to increase this to 8192. However, when trying to build the RPM source from atrpms, I get this: # rpmbuild -ba dovecot.spec error: line 1: Unknown tag: %bcond_without inotify Any help? Dave Quoting Jurvis LaSalle [EMAIL PROTECTED]: On Oct 8, 2008, at 8:01 AM, David Cunningham wrote: After a few hours of running, I get tons of the following errors in my logs: dovecot: Oct 08 07:41:50 Error: auth(default): ldap([EMAIL PROTECTED],x.x.x.x): Request queue is full I removed the username and IP, obviously. Any idea how to stop this? I have about 5 Thousand users using horde that login ever 1-5 minutes to refresh their page. I assume it is a setting, but I am confused as to why it doesn't happen almost right away. It seems to take some time to build up. Please help! This is taking my webmail system down hourly. dovecot -n? Hunch is login_max_processes_count is too low. http://wiki.dovecot.org/LoginProcess hth, JL
Re: [Dovecot] Auth Issues - Urgent - Help!
David Cunningham wrote: I have about 5 Thousand users using horde that login ever 1-5 minutes to refresh their page. I assume it is a setting, but I am confused as to This may not related to the real reason of your problem, but I recommend up-imapproxy (http://www.imapproxy.org/) for such setups, which caches connections (and therefore logins). You could try to enable dovecot's auth_cache instead.
Re: [Dovecot] Auth Issues - Urgent - Help!
David Cunningham schrieb: I have about 5 Thousand users using horde that login ever 1-5 minutes to refresh their page. Imapproxy can be configured to cache connections only for a defined amount of time, eg. close connections not used for more than a few minutes. Setting is cache_expiration_time in imapproxy.conf. Remember: Horde creates a new imap connection for every page click, imapproxy heavily reduces this login/logout noise. If your 5k users are all using webmail *simultaneously* I don't know how the proxy will perform.
Re: [Dovecot] Auth Issues - Urgent - Help!
Unfortantely, it just happened again! sigh I am going to implement my increased queue change and see what happens. Dave Quoting Jurvis LaSalle [EMAIL PROTECTED]: On Oct 8, 2008, at 8:01 AM, David Cunningham wrote: After a few hours of running, I get tons of the following errors in my logs: dovecot: Oct 08 07:41:50 Error: auth(default): ldap([EMAIL PROTECTED],x.x.x.x): Request queue is full I removed the username and IP, obviously. Any idea how to stop this? I have about 5 Thousand users using horde that login ever 1-5 minutes to refresh their page. I assume it is a setting, but I am confused as to why it doesn't happen almost right away. It seems to take some time to build up. Please help! This is taking my webmail system down hourly. dovecot -n? Hunch is login_max_processes_count is too low. http://wiki.dovecot.org/LoginProcess hth, JL
Re: [Dovecot] Auth Issues - Urgent - Help!
On Wed, 2008-10-08 at 10:48 -0400, David Cunningham wrote:
I agree. In fact, I may have found a DNS issue that may have been
causing login sessions to hang and thus reach max too quickly. The
last few hours have been stable. So, I am keeping my fingers crossed.
I have also recompiled dovecot and changed the setting in db-ldap.h
that reads:
#define DB_LDAP_MAX_QUEUE_SIZE 1024
to
#define DB_LDAP_MAX_QUEUE_SIZE 8192
If you're getting more than 1024 requests queued, something's wrong or
you have hundreds or logins per second. Which one is it? (5000 users
logging in once per minute is still only 83/sec)
The queue keeps increasing if the LDAP server isn't replying to old
requests. So have you looked at the LDAP server side if it's running too
slow?
Anyway two things you could do here:
1) Enable auth cache with large enough size so Dovecot doesn't consult
LDAP server nearly as much.
2) Increase the number of auth processes (auth { .. count=5 }), so that
you'll use more connections and hopefully the LDAP server likes that
better than one connection sending lots of requests. Or maybe not.
signature.asc
Description: This is a digitally signed message part
Re: [Dovecot] Auth Issues - Urgent - Help!
Thank you, I will try the caching.
Dave
Quoting Timo Sirainen [EMAIL PROTECTED]:
On Wed, 2008-10-08 at 10:48 -0400, David Cunningham wrote:
I agree. In fact, I may have found a DNS issue that may have been
causing login sessions to hang and thus reach max too quickly. The
last few hours have been stable. So, I am keeping my fingers crossed.
I have also recompiled dovecot and changed the setting in db-ldap.h
that reads:
#define DB_LDAP_MAX_QUEUE_SIZE 1024
to
#define DB_LDAP_MAX_QUEUE_SIZE 8192
If you're getting more than 1024 requests queued, something's wrong or
you have hundreds or logins per second. Which one is it? (5000 users
logging in once per minute is still only 83/sec)
The queue keeps increasing if the LDAP server isn't replying to old
requests. So have you looked at the LDAP server side if it's running too
slow?
Anyway two things you could do here:
1) Enable auth cache with large enough size so Dovecot doesn't consult
LDAP server nearly as much.
2) Increase the number of auth processes (auth { .. count=5 }), so that
you'll use more connections and hopefully the LDAP server likes that
better than one connection sending lots of requests. Or maybe not.
Re: [Dovecot] Auth Issues - Urgent - Help!
Simply changing auth_cache_size to a non-zero number enables caching, correct?
How big is too big?
Where does it cache it?
Here is what I set:
auth_cache_size = 1048576
I was hoping for 1GB worth of cache.
I have 16GB of memory on the system, so memory is not an issue if it
stores it in memory as opposed to disk.
Dave
Quoting David Cunningham [EMAIL PROTECTED]:
Thank you, I will try the caching.
Dave
Quoting Timo Sirainen [EMAIL PROTECTED]:
On Wed, 2008-10-08 at 10:48 -0400, David Cunningham wrote:
I agree. In fact, I may have found a DNS issue that may have been
causing login sessions to hang and thus reach max too quickly. The
last few hours have been stable. So, I am keeping my fingers crossed.
I have also recompiled dovecot and changed the setting in db-ldap.h
that reads:
#define DB_LDAP_MAX_QUEUE_SIZE 1024
to
#define DB_LDAP_MAX_QUEUE_SIZE 8192
If you're getting more than 1024 requests queued, something's wrong or
you have hundreds or logins per second. Which one is it? (5000 users
logging in once per minute is still only 83/sec)
The queue keeps increasing if the LDAP server isn't replying to old
requests. So have you looked at the LDAP server side if it's running too
slow?
Anyway two things you could do here:
1) Enable auth cache with large enough size so Dovecot doesn't consult
LDAP server nearly as much.
2) Increase the number of auth processes (auth { .. count=5 }), so that
you'll use more connections and hopefully the LDAP server likes that
better than one connection sending lots of requests. Or maybe not.
Re: [Dovecot] Auth Issues - Urgent - Help!
On Wed, 2008-10-08 at 16:13 -0400, David Cunningham wrote: Simply changing auth_cache_size to a non-zero number enables caching, correct? How big is too big? If it uses up too much of your memory. Although you'll probably also need to change auth_process_size then, because by default it kills the process if it grows larger than 256 MB. You could also disable it by setting it to 0. Where does it cache it? Here is what I set: auth_cache_size = 1048576 I was hoping for 1GB worth of cache. I think you could do with a lot less :) I think a single user takes maybe 50-100 bytes normally. Even with 200 bytes your 5000 users would fit into 1 MB. Of course negative hits are also cached, so the rest of the memory would go to them. I'd decrease it to 10 MB or so :) Also you might want to increase auth_cache_ttl from the default hour. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Auth Issues - Urgent - Help!
On Wed, 2008-10-08 at 08:01 -0400, David Cunningham wrote: After a few hours of running, I get tons of the following errors in my logs: dovecot: Oct 08 07:41:50 Error: auth(default): ldap([EMAIL PROTECTED],x.x.x.x): Request queue is full BTW. I improved this error message slightly to also tell how many seconds old data is in the queue. http://hg.dovecot.org/dovecot-1.1/rev/0329dc4df5ed I guess you're using auth binds? If you weren't, I think it wouldn't be possible to fill the queue. signature.asc Description: This is a digitally signed message part
