Re: How to detect out-of-sync condition
I tried to look at a dovecot.index.log and it was unreadable in a text editor. I didn't see anything in the wiki link about how to view the log. It's not supposed to be human readable -- the point of this exercise was to check whether your indices are really out of date. You can perhaps compare the mtime of the index files and the mailbox. If they're close or the same, then your indices are not out of sync and don't require sync'ing. So I guess I'll just be guilty of excessive optimizing. Should I run a daily doveadm index or resync? You've skipped over the first 3 steps (is it really out of sync, is there a better way to avoid it, does the un-sync really cause probems) and went right to a solution for a possibly non-existent problem. Be that as it may, the frequency of re-syncing is up to you. I have no idea how large your userbase is or how intensive this operation will be. If it takes only a minute, you could run it more frequently. If you're going through terabytes of data, less frequently. I'm not quite convince you need it at all: I never had to do a global re-indexing except for a one-time dovecot initializing, or dealing with the odd crash that left indices in a unfixable state (which is more a bug than an unsync problem). Joseph Tam jtam.h...@gmail.com
Re: How to detect out-of-sync condition
Yes ... it is maillog per below: # doveadm log find Looking for log files from /var/log Debug: /var/log/maillog Info: /var/log/maillog Warning: /var/log/maillog Error: /var/log/maillog Fatal: /var/log/maillog On 3/13/2015 9:20 AM, Steffen Kaiser wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, 13 Mar 2015, Reindl Harald wrote: Am 13.03.2015 um 14:29 schrieb Cliff Hayes: I looked in the place where dovecot logs everything ... the maillog. Are you sure about maillog? I mean: doveadm log find gives you that file? I didn't see anything but the log is huge and I could have easily missed it. Is there a certain error or phrase I should look for? If so please advise. man grep grep -i 'sync' maillog grep -i 'fail' maillog grep -i 'error' maillog grep -i 'warn' maillog (Y) On 3/13/2015 3:05 AM, Steffen Kaiser wrote: On Thu, 12 Mar 2015, Cliff Hayes wrote: I recently had a user whose mailbox had gone out of sync. There are no log entries about broken sync or something something like that? - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBVQLyJXz1H7kL/d9rAQK7YggAtWJz0FEylF3I8r7qUZsRjraPuXcHUAHM bDGkrr/9BVUTeFF+wVK51k8iBBb7P2VJo8tefJ1gho00gfJpZo2EE8VG/LEt2LeH chq7v8iCNBaTRMXLlQwzUMii5JjSfoKuFRFEhphuv0yLacWT5AICPiqTqrLz+wQh UW7KMfxZuXDxR4zNj58Da8HQmiTSNm7BQv4N1GpExkGNxcE14X7By4iPFxkKq1OI O24iPL1WSwwhOftdBGuXFO2oCOqIQKHu/aZdoElezobxlNXgegDMk6ui9LPu9V1T 44IF1sICPEjpQyf7SgEvJcpBarXeaUy2NtZGMkWHWU0LNl5XQrETcA== =OGb3 -END PGP SIGNATURE-
Re: How to detect out-of-sync condition
I searched with your recommended criteria and there was nothing relevant returned. On 3/13/2015 8:36 AM, Reindl Harald wrote: Am 13.03.2015 um 14:29 schrieb Cliff Hayes: I looked in the place where dovecot logs everything ... the maillog. I didn't see anything but the log is huge and I could have easily missed it. Is there a certain error or phrase I should look for? If so please advise. man grep grep -i 'sync' maillog grep -i 'fail' maillog grep -i 'error' maillog grep -i 'warn' maillog On 3/13/2015 3:05 AM, Steffen Kaiser wrote: On Thu, 12 Mar 2015, Cliff Hayes wrote: I recently had a user whose mailbox had gone out of sync. There are no log entries about broken sync or something something like that?
Re: How to detect out-of-sync condition
I tried to look at a dovecot.index.log and it was unreadable in a text editor. I didn't see anything in the wiki link about how to view the log. So I guess I'll just be guilty of excessive optimizing. Should I run a daily doveadm index or resync? On 3/13/2015 6:00 PM, Joseph Tam wrote: On Fri, 13 Mar 2015, Cliff Hayes writes: By closing off other avenues other than dovecot imap/pop/lda/etc., the indices will stay sync'd. I use dovecot's lda and dovecot's sieve filter. Then I'm not sure how mailboxes ever get out of sync. So it looks like I need to compare the index/mailbox mtimes as you suggest. What am I looking for? I see that the indexes are updated when I run the resync. I checked my mailbox (that was not resynced) and noticed that dovecot.index last update was 16 days ago. Just to be clear, you're talking doveadm index ..., yes? You might have to check dovecot.index.log* as well as they could contain the latest changes that have yet to propagate to the main index file. http://wiki2.dovecot.org/Design/Indexes/TransactionLog So am I resyncing if the gap is over x days? If so, is there a way to resync just those mailboxes with a doveadm command or do I have to write a program look for that condition and run doveadm when matched? It's possible that doveadm index checks modification times as an optimization measure. My recommended game plan: 1) Are your indices *really* out of date (checks logs as Steffen recommends)? 2) If so, how do they get out of date and can you avoid it? 3) If you can't avoid it, does it cause real problems? Most of the time, dovecot will seemlessly rebuild it and it's transparent to the user. 4) If you got this far, run a trial doveadm index -A INBOX (assuming you're just concerned about INBOXs) to see if it's really such a bad operation. As Knuth (hallowed be thy name!) said, premature optimization is the root of all evil. Only after all these steps are found to be unsatisfactory would I consider writing your own scan and fix tool. Joseph Tam jtam.h...@gmail.com
Re: How to detect out-of-sync condition
On Fri, 13 Mar 2015, Cliff Hayes writes: By closing off other avenues other than dovecot imap/pop/lda/etc., the indices will stay sync'd. I use dovecot's lda and dovecot's sieve filter. Then I'm not sure how mailboxes ever get out of sync. So it looks like I need to compare the index/mailbox mtimes as you suggest. What am I looking for? I see that the indexes are updated when I run the resync. I checked my mailbox (that was not resynced) and noticed that dovecot.index last update was 16 days ago. Just to be clear, you're talking doveadm index ..., yes? You might have to check dovecot.index.log* as well as they could contain the latest changes that have yet to propagate to the main index file. http://wiki2.dovecot.org/Design/Indexes/TransactionLog So am I resyncing if the gap is over x days? If so, is there a way to resync just those mailboxes with a doveadm command or do I have to write a program look for that condition and run doveadm when matched? It's possible that doveadm index checks modification times as an optimization measure. My recommended game plan: 1) Are your indices *really* out of date (checks logs as Steffen recommends)? 2) If so, how do they get out of date and can you avoid it? 3) If you can't avoid it, does it cause real problems? Most of the time, dovecot will seemlessly rebuild it and it's transparent to the user. 4) If you got this far, run a trial doveadm index -A INBOX (assuming you're just concerned about INBOXs) to see if it's really such a bad operation. As Knuth (hallowed be thy name!) said, premature optimization is the root of all evil. Only after all these steps are found to be unsatisfactory would I consider writing your own scan and fix tool. Joseph Tam jtam.h...@gmail.com
Re: How to detect out-of-sync condition
Cliff Hayes writes:
I didn't see anything but the log is huge and I could have easily missed
it.
Is there a certain error or phrase I should look for?
If so please advise.
In my setup using mbox, typical log entries would be like
(Common)
Warning: UIDVALIDITY changed ({int} - {int}) in mbox file ...
Error: Next message unexpectedly corrupted in mbox file ...
(Rare)
Error: Corrupted index cache file {cache-file}: Broken physical size
for mail UID ...
Error: read({file}) failed: Next message unexpectedly lost from mbox
file ...
Error: read({file}) failed: Cached message size larger than expected ...
Joseph Tam jtam.h...@gmail.com
Re: How to detect out-of-sync condition
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Thu, 12 Mar 2015, Cliff Hayes wrote: I recently had a user whose mailbox had gone out of sync. There are no log entries about broken sync or something something like that? - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBVQKaP3z1H7kL/d9rAQKkCQf/VwbytrDRGDJT8Ado8J5L/melZ751jqME pmRQ51McVo80SFvpNuQe93tLlJcGwltgr4He2scxBMelzBlfmQ9Vqrl0EH9R+RpE UxPGHbhWa6Dm23gRMgeYiLx9YlZXnnKtnpUL4MriMwIrrTGaN1hq7yX4Ot2k+ImM vHI3aFSTg35lklKfrzfDwKaQKQMMVf8fz4fZJICL2Vas2df5XaSEyQY1wM/PUqoj Bev0POPfvKhba8adY5yAM9fWpBmStwspIwKR1ZWqflOlz41WfOKeQBWkMqXK7hpV LpF5dY1omVE0BORE5+4T9hY03RXtBcepVczfS4MhX4WrNkJ3TMrjvw== =gjO3 -END PGP SIGNATURE-
Re: How to detect out-of-sync condition
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, 13 Mar 2015, Reindl Harald wrote: Am 13.03.2015 um 14:29 schrieb Cliff Hayes: I looked in the place where dovecot logs everything ... the maillog. Are you sure about maillog? I mean: doveadm log find gives you that file? I didn't see anything but the log is huge and I could have easily missed it. Is there a certain error or phrase I should look for? If so please advise. man grep grep -i 'sync' maillog grep -i 'fail' maillog grep -i 'error' maillog grep -i 'warn' maillog (Y) On 3/13/2015 3:05 AM, Steffen Kaiser wrote: On Thu, 12 Mar 2015, Cliff Hayes wrote: I recently had a user whose mailbox had gone out of sync. There are no log entries about broken sync or something something like that? - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBVQLyJXz1H7kL/d9rAQK7YggAtWJz0FEylF3I8r7qUZsRjraPuXcHUAHM bDGkrr/9BVUTeFF+wVK51k8iBBb7P2VJo8tefJ1gho00gfJpZo2EE8VG/LEt2LeH chq7v8iCNBaTRMXLlQwzUMii5JjSfoKuFRFEhphuv0yLacWT5AICPiqTqrLz+wQh UW7KMfxZuXDxR4zNj58Da8HQmiTSNm7BQv4N1GpExkGNxcE14X7By4iPFxkKq1OI O24iPL1WSwwhOftdBGuXFO2oCOqIQKHu/aZdoElezobxlNXgegDMk6ui9LPu9V1T 44IF1sICPEjpQyf7SgEvJcpBarXeaUy2NtZGMkWHWU0LNl5XQrETcA== =OGb3 -END PGP SIGNATURE-
Re: How to detect out-of-sync condition
I looked in the place where dovecot logs everything ... the maillog. I didn't see anything but the log is huge and I could have easily missed it. Is there a certain error or phrase I should look for? If so please advise. On 3/13/2015 3:05 AM, Steffen Kaiser wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Thu, 12 Mar 2015, Cliff Hayes wrote: I recently had a user whose mailbox had gone out of sync. There are no log entries about broken sync or something something like that? - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBVQKaP3z1H7kL/d9rAQKkCQf/VwbytrDRGDJT8Ado8J5L/melZ751jqME pmRQ51McVo80SFvpNuQe93tLlJcGwltgr4He2scxBMelzBlfmQ9Vqrl0EH9R+RpE UxPGHbhWa6Dm23gRMgeYiLx9YlZXnnKtnpUL4MriMwIrrTGaN1hq7yX4Ot2k+ImM vHI3aFSTg35lklKfrzfDwKaQKQMMVf8fz4fZJICL2Vas2df5XaSEyQY1wM/PUqoj Bev0POPfvKhba8adY5yAM9fWpBmStwspIwKR1ZWqflOlz41WfOKeQBWkMqXK7hpV LpF5dY1omVE0BORE5+4T9hY03RXtBcepVczfS4MhX4WrNkJ3TMrjvw== =gjO3 -END PGP SIGNATURE-
Re: How to detect out-of-sync condition
Am 13.03.2015 um 14:29 schrieb Cliff Hayes: I looked in the place where dovecot logs everything ... the maillog. I didn't see anything but the log is huge and I could have easily missed it. Is there a certain error or phrase I should look for? If so please advise. man grep grep -i 'sync' maillog grep -i 'fail' maillog grep -i 'error' maillog grep -i 'warn' maillog On 3/13/2015 3:05 AM, Steffen Kaiser wrote: On Thu, 12 Mar 2015, Cliff Hayes wrote: I recently had a user whose mailbox had gone out of sync. There are no log entries about broken sync or something something like that? signature.asc Description: OpenPGP digital signature
Re: How to detect out-of-sync condition
Thanks. I use dovecot's lda and dovecot's sieve filter. So it looks like I need to compare the index/mailbox mtimes as you suggest. What am I looking for? I see that the indexes are updated when I run the resync. I checked my mailbox (that was not resynced) and noticed that dovecot.index last update was 16 days ago. So am I resyncing if the gap is over x days? If so, is there a way to resync just those mailboxes with a doveadm command or do I have to write a program look for that condition and run doveadm when matched? Cliff Hayes cha...@afo.net writes: I recently had a user whose mailbox had gone out of sync. Webmail and client inbox totals had become vastly different and response times had become unacceptable. A doveadm force-resync fixed the problem nicely. This will no doubt happen again as we grow the service and I would like to have some type of indication at the server level so I can run a resync on a per-user basis as required. Perhaps what you ought to look for is how are they getting out of sync in the first place. Assuming your webmail is a IMAP/POP client, I would look at ways that your user's INBOX can be modified out of Dovecot's sight. For example, does your LDA use dovecot's lda/lmtp, or do your users run mail clients that directly manipulate mailbox files, or do you use procmail/mail filters? By closing off other avenues other than dovecot imap/pop/lda/etc., the indices will stay sync'd. If you really have to do manual resyncing, comparing index/mailbox mtimes might be one way to do it. Joseph Tam jtam.h...@gmail.com
How to detect out-of-sync condition
Hello list, I recently had a user whose mailbox had gone out of sync. Webmail and client inbox totals had become vastly different and response times had become unacceptable. A doveadm force-resync fixed the problem nicely. This will no doubt happen again as we grow the service and I would like to have some type of indication at the server level so I can run a resync on a per-user basis as required. However, I don't know what to look for. I did not see any signs of the problem on the server - but perhaps I did not know what to look for. The only other alternative is for me to do a global resync in the cron.daily but my fear is that this won't scale well for thousands of users. Please advise. Thanks in advance.
Re: How to detect out-of-sync condition
Cliff Hayes cha...@afo.net writes: I recently had a user whose mailbox had gone out of sync. Webmail and client inbox totals had become vastly different and response times had become unacceptable. A doveadm force-resync fixed the problem nicely. This will no doubt happen again as we grow the service and I would like to have some type of indication at the server level so I can run a resync on a per-user basis as required. Perhaps what you ought to look for is how are they getting out of sync in the first place. Assuming your webmail is a IMAP/POP client, I would look at ways that your user's INBOX can be modified out of Dovecot's sight. For example, does your LDA use dovecot's lda/lmtp, or do your users run mail clients that directly manipulate mailbox files, or do you use procmail/mail filters? By closing off other avenues other than dovecot imap/pop/lda/etc., the indices will stay sync'd. If you really have to do manual resyncing, comparing index/mailbox mtimes might be one way to do it. Joseph Tam jtam.h...@gmail.com
