Re: Inconsistency in map index with dovecot v2.3.21
Apologies for the formatting. Retrying with HTML formatting disabled: We are getting the same warnings on 2 separate test servers; one with local ZFS storage, the other one with NFS storage (single server, data not shared with any other servers). We are imapsync'ing a few hundred user accounts from the production server so there is a lot of emails transfer a day. The warnings occur at random times and with random users. I reinstalled 2.3.21 from source (originally was using Dovecot repo packages) and tested by reverting that commit. The warnings continued. Next I tried with Dovecot 2.3.20 and still got the same warnings after a while. # 2.3.20 (80a5ac675d): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.20 (149edcf2) # OS: Linux 5.4.0-171-generic x86_64 Ubuntu 20.04.6 LTS nfs4 Feb 13 15:38:25 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=272469, TLS, session= Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: mdbox /mail/spool/username/storage: Inconsistency in map index (3,40 != 3,56) Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: fscking index file /mail/spool/username/storage/dovecot.map.index Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: mdbox /mail/spool/username/storage: rebuilding indexes Feb 13 15:38:29 dovecot: imap(username)<272469>: Error: Mailbox INBOX: /mail/spool/username/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is now inconsistent Feb 13 15:38:29 dovecot: imap(username)<272469>: Warning: fscking index file /mail/spool/username/storage/dovecot.map.index I will try with an even older version to continue testing... ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
On 2024-02-10 2:24 a.m., Timo Sirainen wrote: On 12. Jan 2024, at 9.37, John Alex. via dovecot wrote: since I upgraded dovecot to version 2.3.21 on a FreeBSD 13.2 system, I've been seeing a few occurences of the following warnings daily: Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp (user1)<8209>: Warning: mdbox / var/vmail/users/user1/mdbox/storage: Inconsistency in map index (5,40 != 5,56) Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp (user1)<8209>: Warning: fscking index file /var/vmail/users/user1/mdbox/storage/ dovecot.map.index Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp (user1)<8209>: Warning: mdbox / var/vmail/users/user1/mdbox/storage: rebuilding indexes Jan 11 13:13:26 dimokritos dovecot[33170]: lmtp (user1)<8209>: Warning: fscking index file /var/vmail/users/user1/mdbox/storage/ dovecot.map.index Hmm. I can't seem to be able to reproduce it easily at least. Looking at the code changes, I do see now that there is one related change that may be wrong, or at least the commit message is wrong: https:// github.com/dovecot/core/commit/ aae93d3ea73004ba1883bbf91c9e40f613ffc857 - although it shouldn't affect anything unless there are already some other errors. But maybe there's something I don't see right now. Can you try if the errors go away by reverting this change? We are getting the same warnings on 2 separate test servers; one with local ZFS storage, the other one with NFS storage (single server, data not shared with any other servers). We are imapsync'ing a few hundred user accounts from the production server so there is a lot of emails transfer a day. The warnings occur at random times and with random users. I reinstalled 2.3.21 from source (originally was using Dovecot repo packages) and tested by reverting that commit. The warnings continued. Next I tried with Dovecot 2.3.20 and still got the same warnings after a while. # 2.3.20 (80a5ac675d): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.20 (149edcf2) # OS: Linux 5.4.0-171-generic x86_64 Ubuntu 20.04.6 LTS nfs4 Feb 13 15:38:25 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=272469, TLS, session= Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: mdbox /mail/spool/username/storage: Inconsistency in map index (3,40 != 3,56) Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: fscking index file /mail/spool/username/storage/dovecot.map.index Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: mdbox /mail/spool/username/storage: rebuilding indexes Feb 13 15:38:29 dovecot: imap(username)<272469>: Error: Mailbox INBOX: /mail/spool/username/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is now inconsistent Feb 13 15:38:29 dovecot: imap(username)<272469>: Warning: fscking index file /mail/spool/username/storage/dovecot.map.index I will try with an even older version to continue testing... ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
On 12. Jan 2024, at 9.37, John Alex. via dovecot wrote: > > Hi, > > since I upgraded dovecot to version 2.3.21 on a FreeBSD 13.2 system, I've > been seeing a few occurences of the following warnings daily: > > Jan 11 13:13:25 dimokritos dovecot[33170]: > lmtp(user1)<8209>: Warning: mdbox > /var/vmail/users/user1/mdbox/storage: Inconsistency in map index (5,40 != > 5,56) > Jan 11 13:13:25 dimokritos dovecot[33170]: > lmtp(user1)<8209>: Warning: fscking index file > /var/vmail/users/user1/mdbox/storage/dovecot.map.index > Jan 11 13:13:25 dimokritos dovecot[33170]: > lmtp(user1)<8209>: Warning: mdbox > /var/vmail/users/user1/mdbox/storage: rebuilding indexes > Jan 11 13:13:26 dimokritos dovecot[33170]: > lmtp(user1)<8209>: Warning: fscking index file > /var/vmail/users/user1/mdbox/storage/dovecot.map.index Hmm. I can't seem to be able to reproduce it easily at least. Looking at the code changes, I do see now that there is one related change that may be wrong, or at least the commit message is wrong: https://github.com/dovecot/core/commit/aae93d3ea73004ba1883bbf91c9e40f613ffc857 - although it shouldn't affect anything unless there are already some other errors. But maybe there's something I don't see right now. Can you try if the errors go away by reverting this change? > Should I be worried about data loss due to these index inconsistencies? No data loss, but it can lead to inefficient rebuilds. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
I have the same problem after updating dovecot 2.3.21, I am receiving the errors below for the accounts when I perform backups. doveadm(u...@user.com.br): Error: Purging namespace '' failed: BUG: Unknown internal error Error: failed sync dir: doveadm(us...@user2.com.br): Warning: mdbox /data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2.com.br/mdbox/storage: Inconsistency in map index (42,40 != 42,56) doveadm(us...@user2.com.br): Warning: fscking index file /data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2/mdbox/storage/dovecot.map.index doveadm(us...@user2.com.br): Warning: mdbox /data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2.com.br/mdbox/storage: rebuilding indexes doveadm(us...@user2.com.br): Warning: Transaction log file /data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2/mdbox/storage/dovecot.map.index.log was locked for 284 seconds (mdbox storage rebuild) doveadm(us...@user2.com.br): Error: Purging namespace '' failed: BUG: Unknown internal error As in the case of our friend the other day, the error no longer happens on the same account, it happens on another random one. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Inconsistency in map index with dovecot v2.3.21
Hi, since I upgraded dovecot to version 2.3.21 on a FreeBSD 13.2 system, I've been seeing a few occurences of the following warnings daily: Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp(user1)<8209>: Warning: mdbox /var/vmail/users/user1/mdbox/storage: Inconsistency in map index (5,40 != 5,56) Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp(user1)<8209>: Warning: fscking index file /var/vmail/users/user1/mdbox/storage/dovecot.map.index Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp(user1)<8209>: Warning: mdbox /var/vmail/users/user1/mdbox/storage: rebuilding indexes Jan 11 13:13:26 dimokritos dovecot[33170]: lmtp(user1)<8209>: Warning: fscking index file /var/vmail/users/user1/mdbox/storage/dovecot.map.index The numbers in parentheses seem to always be of the format (x,40 != x,56) or (x,40 != x,72). I also get such errors on some users when purging (doveadm-purge fails with error 75): doveadm(user2): Warning: mdbox /var/vmail/users/user2/mdbox/storage: Inconsistency in map index (5,40 != 5,72) doveadm(user2): Warning: fscking index file /var/vmail/users/user2/mdbox/storage/dovecot.map.index doveadm(user2): Warning: mdbox /var/vmail/users/user2/mdbox/storage: rebuilding indexes doveadm(user2): Error: Purging namespace '' failed: BUG: Unknown internal error If I rerun doveadm purge for this user it completes successfully. Can anyone shed some light on what might be causing these messages? I didn't have these with 2.3.20. Should I be worried about data loss due to these index inconsistencies? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index
On Fri, Aug 18, 2017 at 1:30 PM, Timo Sirainenwrote: > > That would work. Also as a different workaround you could just rm > storage/dovecot.map.index* and doveadm force-resync -u user@domain '*'. Thank you Timo, that did the trick without the need of a sudden upgrade. Mailbox was fixed :) Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil* > >
Re: Inconsistency in map index
On 18 Aug 2017, at 16.43, Webert de Souza Limawrote: > > On Fri, Aug 18, 2017 at 9:03 AM, Aki Tuomi wrote: > >> This is fixed in next release (2.2.32) with >> https://github.com/dovecot/core/commit/c8be394 >> >> Aki Tuomi >> > > As this is still a release candidate, I'm thinking of running an isolated > instance of this version, and do doveadm force-resync just to fix just this > user's mailbox. > What do you think? That would work. Also as a different workaround you could just rm storage/dovecot.map.index* and doveadm force-resync -u user@domain '*'. Should be also safe to do (won't lose any information), although I guess it has some potential of race conditions causing temporary problems if the user accesses mails at the same time.
Re: Inconsistency in map index
On Fri, Aug 18, 2017 at 9:03 AM, Aki Tuomiwrote: > This is fixed in next release (2.2.32) with > https://github.com/dovecot/core/commit/c8be394 > > Aki Tuomi > As this is still a release candidate, I'm thinking of running an isolated instance of this version, and do doveadm force-resync just to fix just this user's mailbox. What do you think? Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil*
Re: Inconsistency in map index
Oh, so that's likely a bug. I was thinking it would require manual intervention to fix. Great, I'll do an upgrade ASAP. Praised be Docker. Thank you very much. Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil* On Fri, Aug 18, 2017 at 9:03 AM, Aki Tuomi <aki.tu...@dovecot.fi> wrote: > > > On 18.08.2017 14:55, Webert de Souza Lima wrote: > > Hello, > > > > The following errors are constantly popping up for 2 accounts. I can't > get > > it fixed, > > I did doveadm backup to another account, the same happens in the new > > account. > > I did doveadm force-resync, the problem persists. > > > > I'm using dovecot 2.2. > > > > 2017-08-18T11:46:12.472821881Z Aug 18 11:46:12 lmtp( > ramon.lace...@alliar.com): > > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage: > > Inconsistency in map index (647,6288 != 647,28333584) > > > > 2017-08-18T11:46:12.651002372Z Aug 18 11:46:12 lmtp( > ramon.lace...@alliar.com): > > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage: > > Inconsistency in map index (647,6288 != 647,28333708) > > > > 2017-08-18T11:46:12.651059432Z Aug 18 11:46:12 lmtp( > ramon.lace...@alliar.com): > > Warning: fscking index file /srv/dovecot2/index/ > > alliar.com/ramon.lacerda/storage/dovecot.map.index > > > > 2017-08-18T11:46:12.764926940Z Aug 18 11:46:12 lmtp( > ramon.lace...@alliar.com): > > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage: > > rebuilding indexes > > > > > > Regards, > > > > Webert Lima > > DevOps Engineer at MAV Tecnologia > > *Belo Horizonte - Brasil* > > Hi! > > This is fixed in next release (2.2.32) with > https://github.com/dovecot/core/commit/c8be394 > > Aki Tuomi >
Re: Inconsistency in map index
On 18.08.2017 14:55, Webert de Souza Lima wrote: > Hello, > > The following errors are constantly popping up for 2 accounts. I can't get > it fixed, > I did doveadm backup to another account, the same happens in the new > account. > I did doveadm force-resync, the problem persists. > > I'm using dovecot 2.2. > > 2017-08-18T11:46:12.472821881Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com): > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage: > Inconsistency in map index (647,6288 != 647,28333584) > > 2017-08-18T11:46:12.651002372Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com): > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage: > Inconsistency in map index (647,6288 != 647,28333708) > > 2017-08-18T11:46:12.651059432Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com): > Warning: fscking index file /srv/dovecot2/index/ > alliar.com/ramon.lacerda/storage/dovecot.map.index > > 2017-08-18T11:46:12.764926940Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com): > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage: > rebuilding indexes > > > Regards, > > Webert Lima > DevOps Engineer at MAV Tecnologia > *Belo Horizonte - Brasil* Hi! This is fixed in next release (2.2.32) with https://github.com/dovecot/core/commit/c8be394 Aki Tuomi
Inconsistency in map index
Hello, The following errors are constantly popping up for 2 accounts. I can't get it fixed, I did doveadm backup to another account, the same happens in the new account. I did doveadm force-resync, the problem persists. I'm using dovecot 2.2. 2017-08-18T11:46:12.472821881Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com): Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage: Inconsistency in map index (647,6288 != 647,28333584) 2017-08-18T11:46:12.651002372Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com): Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage: Inconsistency in map index (647,6288 != 647,28333708) 2017-08-18T11:46:12.651059432Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com): Warning: fscking index file /srv/dovecot2/index/ alliar.com/ramon.lacerda/storage/dovecot.map.index 2017-08-18T11:46:12.764926940Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com): Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage: rebuilding indexes Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil*
mdbox Inconsistency in map index
After a (power)crash two accounts have been corrupted. I tried to rescue things by running force-resync multiple times, but that didn't work out. Searching the archives, I found a recent suggestion (by Timo) to delete the index files, but I'm not sure which files to delete and what the consequences will be. When I deleted ``storage/dovecot.map.index'' in an unimportant account of my own, things only grew worse and I decided to delete the account and start fresh... Something I obviously don't want to do with the other account as it belongs to a client with about 2.5G mails (856 m.* files), which we don't want to lose. I have observed the following behavior in the logs: When a new mail arrives: (The second line below is always the same, however there's no mail with uid 12819 according to doveadm search uid 12819) Mar 04 11:43:28 lda(user): Warning: mdbox /var/mail/user/storage: Inconsistency in map index (117,1060 != 117,1883660) Mar 04 11:43:28 lda(user): Error: Log synchronization error at seq=117,offset=1546660 for /var/mail/user/storage/dovecot.map.index: Extension record inc drops number below zero (uid=12819, diff=-1, orig=0) Mar 04 11:43:28 lda(user): Warning: fscking index file /var/mail/user/storage/dovecot.map.index Mar 04 11:43:28 lda(user): Info: msgid=<d760229d563e99dc787acd0746b18...@feedback.1dayfly.net>: saved mail to INBOX Mar 04 11:43:28 lda(user): Warning: mdbox /var/mail/user/storage: Inconsistency in map index (117,1060 != 117,1883784) Mar 04 11:43:28 lda(user): Warning: fscking index file /var/mail/user/storage/dovecot.map.index Mar 04 11:43:28 lda(user): Warning: mdbox /var/mail/user/storage: rebuilding indexes Mar 04 11:43:28 lda(user): Warning: fscking index file /var/mail/user/storage/dovecot.map.index Mar 04 11:43:28 imap(user): Error: /var/mail/user/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is now inconsistent Mar 04 11:43:28 imap(user): Info: IMAP session state is inconsistent, please relogin. in=2011 out=108358 deleted=0 expunged=0 trashed=0 When user logs in next (might be from another box/MUA) Mar 04 11:44:36 imap-login: Info: Login: user=, method=PLAIN, rip=82.95.XX.XX, lip=37.97.XX.XX, mpid=14884, TLS, session=<VfvML+ZJuO9SX18R> Mar 04 11:44:37 imap(user): Warning: mdbox /var/mail/user/storage: Inconsistency in map index (117,1060 != 117,1884336) Mar 04 11:44:37 imap(user): Warning: fscking index file /var/mail/user/storage/dovecot.map.index Mar 04 11:44:37 imap(user): Warning: mdbox /var/mail/user/storage: rebuilding indexes Mar 04 11:44:37 imap(user): Error: /var/mail/user/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is now inconsistent Mar 04 11:44:37 imap(user): Info: IMAP session state is inconsistent, please relogin. in=331 out=178391 deleted=0 expunged=0 trashed=0 Mar 04 11:44:37 imap(user): Warning: mdbox /var/mail/user/storage: Inconsistency in map index (117,1060 != 117,1884396) Mar 04 11:44:37 imap(user): Warning: fscking index file /var/mail/user/storage/dovecot.map.index Mar 04 11:44:37 imap(user): Warning: fscking index file /var/mail/user/storage/dovecot.map.index Mar 04 11:44:37 imap-login: Info: Login: user=, method=PLAIN, rip=82.95.XX.XX, lip=37.97.XX.XX, mpid=14886, TLS, session=<AUTbL+ZJuu9SX18R> Besides the nuisance of getting kicked of every time a new mail arrives, user cannot delete mails (as in: move to Trash). TL;DR: which files to delete and what will be the consequences? -- b.