Re: [Dovecot] cannot update mailbox - unable to lock for exclusive access
Thanks, all your comments are noted. As I said, you can migrate users individually. You could easily do 10 users a day during coffee breaks etc and be done in a month plus. Do 40 a day and you're done in 10 days. The only time you'll burn is in the learning curve, not the actual mailbox migration which takes no time at all with POP accounts. That's interesting, as I (wrongly) assumed switching from mbox to maildir was an all or nothing process. You're saying we can run half the mailboxes in mbox format and the other half in maildir format? In which case we can get going with this sooner than I thought. Always test with a dummy mailbox first to iron out any issues. Then start migrating the problem users first, the smart phone users who tie up their mailboxes for many minutes during download. Thank you - I would probably start with the CEO's mailbox first and then go from there >:-D
Re: [Dovecot] cannot update mailbox - unable to lock for exclusive access
So this is a step in the right direction. But still far less than optimal. The read/write lock contention on mbox is unnecessarily eating up system resources (mainly memory), and causing unnecessary delivery delays to the mailbox. You should really start looking at migrating to maildir. It's not that difficult (though maybe more so with 1.0.7) if you don't have a ton of mailboxes, and especially with POP since the mailboxes typically wont be holding much mail to migrate. How many do you have? There's around four hundred mail boxes or so. Some used more intensively than others. Our server is with Rackspace, and RHEL5 is the OS they offered us as an upgrade path from RHEL4. So they're getting the support from Red Hat and we're getting the support from Rackspace. The plot thickens again. You're using a rented server. Sigh... This entire thread could have been greatly shortened, saving all of us much time, if you'd have given all these details up front. Is this a cloud server (shared host), or a dedicated server? It's a dedicated server FWIW, you don't have RHEL5, but CentOS 5. Hosting companies don't pay for RHEL licenses for 10s of thousands of hosts. It's RHEL5: $cat /etc/issue Red Hat Enterprise Linux Server release 5.8 (Tikanga) The cost of the license is included in our contract. I have a few salient recommendations for you: 1. Migrate to maildir. It is far more appropriate for a POP workload. Yes, this will be our next course of action 2. Switch to a hosting provider that offers much more recent software. We can upgrade the software if we wish, but will no longer get full support from Rackspace if we do this. 3. Or, get a colo server so you can use whatever software you wish. We can install whatever software we wish at the moment, but see the point above. Finally, if this email service you're providing isn't all that critical to you or your organization, simply prod along as you have been, fighting these problems frequently along the way. It's kind of working ok now but we will go with your recommendation of switching to maildir when we have time. Thanks for your help
Re: [Dovecot] cannot update mailbox - unable to lock for exclusive access
On 23/11/2012 06:07, Stan Hoeppner wrote: On 11/22/2012 3:26 PM, 1st WebDesigns wrote: Output of dovecot -n is as follows: # 1.0.7: /etc/dovecot.conf login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login mail_privileged_group: mail mail_location: mbox:~/mail:INBOX=/var/mail/%u mbox_lock_timeout: 600 mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 auth default: passdb: driver: pam userdb: driver: passwd Are your mailboxes on NFS storage? You haven't stated on what storage your mailboxes reside. NFS complicates locking. If you use an NFS server, did anything on it change recently, such as an upgrade to RHEL5? No they are not on NFS storage, the mailboxes are stored on the local filesystem. I found a thread stating RHEL5 has a bad FCNTL implementation that could be related to your write lock delay problem. Try using dotlock only for read and write and see if that helps. It has additional filesystem IO overhead, but nothing like the many minutes of delay you have now. mbox_read_locks = dotlock mbox_write_locks = dotlock Thank you I will try this. I did read that when using Postfix and Dovecot, both systems should use a matching locking mechanism, which I had already tried. However, I hadn't tried just dotlock, only FCNTL and a combination of FCNTL and dotlock. We upgraded from RedHat 4 to RedHat 5. The problem didn't exist with RH4 and an even older version of Dovecot. That may be, but you're surely not planning on downgrading back to RHEL4. No, not at all. When emails are stuck in the queue, doing this: Dovecot doesn't use queues. It writes directly to the mailbox files. lsof /var/spool/mail/ These are mailbox files, your user inbox mbox files, not spool files. Spool implies temporary storage. Don't let "spool" fool you. On many/most systems /var/spool/mail is a link to /var/mail. Yes that's correct. shows the spool file in use by a pop3 login and the Dovecot deliver process. Since changing mbox_lock_timeout from 300 to 600 the pop3 process eventually finishes before 600 seconds and the deliver process is able to complete. I admit this is masking the problem rather than solving it. Does the larger timeout value completely eliminate the errors? If so this may be the best outcome you can get with Dovecot 1.0.7, mbox storage, on RHEL5, unless a different locking method fixes it. Yes it completely eliminates the errors. If a pop3 connection has the lock, the mail simply sits there and is eventually delivered in (less than) 600 seconds. Whereas before, it would get deferred. When re-delivery was attempted, it's possible that the box would be locked again, and the mail would get deferred again, leading to a delay of several hours on a busy day. As discussed before our version of Dovecot is dated now, however it's the version provided by RedHat and the version supported by our support company (who aren't doing a great job, hence me posting here). It's the version provided by RHEL5. RHEL6.3 has Dovecot 2.0.9. There are 3rd party 1.2.x RPMs available for RHEL5.x as well as 2.x.x RPMs for RHEL5.x. What "support company"? If you're using RHEL, Red Hat provides the support. That's the whole reason for "paying for" a Linux distro. What is preventing you from upgrading to RHEL 6.3, the current release? Which BTW is behind nearly all other distros WRT package versions. For instance Debian stable has Dovecot 2.1.7 available in the backports repo. Our server is with Rackspace, and RHEL5 is the OS they offered us as an upgrade path from RHEL4. So they're getting the support from Red Hat and we're getting the support from Rackspace.
Re: [Dovecot] cannot update mailbox - unable to lock for exclusive access
Output of dovecot -n is as follows: # 1.0.7: /etc/dovecot.conf login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login mail_privileged_group: mail mail_location: mbox:~/mail:INBOX=/var/mail/%u mbox_lock_timeout: 600 mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 auth default: passdb: driver: pam userdb: driver: passwd We upgraded from RedHat 4 to RedHat 5. The problem didn't exist with RH4 and an even older version of Dovecot. When emails are stuck in the queue, doing this: lsof /var/spool/mail/ shows the spool file in use by a pop3 login and the Dovecot deliver process. Since changing mbox_lock_timeout from 300 to 600 the pop3 process eventually finishes before 600 seconds and the deliver process is able to complete. I admit this is masking the problem rather than solving it. As discussed before our version of Dovecot is dated now, however it's the version provided by RedHat and the version supported by our support company (who aren't doing a great job, hence me posting here). Thanks, On 22/11/2012 12:09, Stan Hoeppner wrote: On 11/12/2012 5:15 AM, 1st WebDesigns wrote: Thanks for your replies. I switched to Dovecot LDA this morning, but the issue still persists, albeit logged slightly differently by Dovecot now instead of Postfix: "save failed to INBOX: Timeout while waiting for lock" The reason is because some pop3 clients Full stop. This is the first time you've mentioned POP that I recall. FYI, Dovecot is primarily an IMAP server. Unless an OP states up front that he's using primarily POP, everyone assumes IMAP and counsels accordingly. You should have stated POP in your first post. Actually, you should have included many more details prior to now. Please post your complete 'dovecot -n' output. are holding their connection for 5 or 6 minutes (don't ask me why - and the iPhone seems to be the major culprit). I'm no smartphone POP expert, but old rural tower, poor tower connection, etc, all cause low data rates, which could cause this. However, you state this problem cropped up out of nowhere after a distro upgrade to CentOS 5. Can you confirm that the problem didn't exist before the upgrade? Your definitive answer to this question dictates the troubleshooting course of action. In dovecot.conf I changed: mbox_lock_timeout = 300 to mbox_lock_timeout = 600 Which seems to have helped. I am unclear if this value only applied to Dovecot LDA or if it would have worked previously before switching to Dovecot LDA? This simply changes how long Dovecot will wait to acquire a lock. Increasing this value simply increases delays, masks the underlying problem without really helping much. The only real architectural solution to such a POP/mbox locking problem due to slow/long client downloads is, as you mentioned, moving to a lockless mailbox format, such as maildir or sdbox. Worth noting, we are both/all at fault in the slow progress of this issue, you for not stating POP up front, and me/us for not asking. Your 'dovecot -n' output may allow us to help get mbox working a little better, but the long term solution is very likely moving to maildir/sdbox.
Re: [Dovecot] cannot update mailbox - unable to lock for exclusive access
On 08/11/2012 23:53, Ben Morrow wrote: At 3AM -0600 on 8/11/12 you (Stan Hoeppner) wrote: 1.0.7 is absolutely ancient and no longer officially supported. You need 1.2.x minimum, 2.x.x even better. And you say you just recently upgraded your Linux distro? What planet do you live on son? You're a few light years behind current stable software. [A light-year is a measure of distance, not of time.] LDA completely eliminates lock contention. As we have discussed before, using the LDA does not prevent lock contention, it just prevents the problems that arise when different software is using different locking strategies on the same mailbox (assuming nothing except LDA and imap is touching the mailbox directly). There are valid reasons for not using the LDA: the OP might be already using procmail, for instance, and have users with procmail recipies which sort into IMAP folders. These folders will need to be locked by procmail even if the default delivery to INBOX is changed (globally) to happen through dovecot-lda. While migrating to sieve (and mdbox, and LMTP) would, IMHO, be the best long-term solution, this isn't necessarily something that can be set up overnight. Ben - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.2221 / Virus Database: 2441/5382 - Release Date: 11/08/12 Thanks for your replies. I switched to Dovecot LDA this morning, but the issue still persists, albeit logged slightly differently by Dovecot now instead of Postfix: "save failed to INBOX: Timeout while waiting for lock" The reason is because some pop3 clients are holding their connection for 5 or 6 minutes (don't ask me why - and the iPhone seems to be the major culprit). In dovecot.conf I changed: mbox_lock_timeout = 300 to mbox_lock_timeout = 600 Which seems to have helped. I am unclear if this value only applied to Dovecot LDA or if it would have worked previously before switching to Dovecot LDA?
[Dovecot] cannot update mailbox - unable to lock for exclusive access
Hi there, We're using Dovecot version 1.0.7 and Postfix, and since upgrading our Linux box we're getting this in the maillog: Nov 8 07:49:11 server1 postfix/local[27901]: 04B8E7081DA: to=, orig_to=, relay=local, delay=19, delays=0.07/0/0/19, dsn=4.2.0, status=deferred (cannot update mailbox /var/spool/mail/abc for user abc. unable to lock for exclusive access: Resource temporarily unavailable) Postfix is currently set to: mailbox_delivery_lock = fcntl, dotlock Dovecot has this: mbox_read_locks = fcntl mbox_write_locks = fcntl dotlock I've scoured the web and tried all kinds of different locking mechanisms and combinations to no avail. The mail is eventually delivered but on a busy day this can take several hours. In the evening it generally doesn't happen, which leads me to believe it occurs during the day when people have their mail clients open. However, this particular issue doesn't affect every user that has their mail client open, only some! We currently have the mailboxes in mbox format - yes I know we should upgrade to maildir which we will eventually, however we've been using mbox for the last 10 years and this has only become an issue since upgrading. What I have discovered this morning is a Dovecot connection that is open for 6 minutes before disconnection. During this 6 minutes the above problem occurs (new mail cannot get an exclusive lock). The same user will have connected and disconnected several times before and after, with a connection taking no more than a few seconds. But it seems sometimes the connection is taking longer than it should and I suspect the problem lies here. Please can anyone help! Thank you, G
