Re: Inconsistency in map index with dovecot v2.3.21
Hello Timo, We had been running version 2.3.20 previously, not 2.3.19. We will upgrade to version 2.3.22 when it is out and monitor if these errors stop. Thank you, Nikos ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
On 30. Aug 2024, at 16.39, Nikolaos Pyrgiotis via dovecot wrote: > > But is there a possible bug in dovecot 2.3.21 version linked with the mdbox > format that is causing the `inconsistency in map index` in the first place or > it is just a configuration error? Other users have also reported these error > messages on this older thread > https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/73CEPDRB7TWP6BJABZL6VBZZH66HQ6S6/#73CEPDRB7TWP6BJABZL6VBZZH66HQ6S6 > . What was the previous version you were running? 2.3.20? I don't think there are really much of any changes related to dbox or index file handling. We did do pretty large fixes to mdbox corruption handling, but looks like they're still waiting for v2.3.22 release. Maybe those would help with these inconsistency issues also, or at least fixing them. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
Thank you for your recommendation. We will try to increase the vsz_limit for these services and see if it helps to mitigate the issue. But is there a possible bug in dovecot 2.3.21 version linked with the mdbox format that is causing the `inconsistency in map index` in the first place or it is just a configuration error? Other users have also reported these error messages on this older thread https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/73CEPDRB7TWP6BJABZL6VBZZH66HQ6S6/#73CEPDRB7TWP6BJABZL6VBZZH66HQ6S6 . Is there anything i can do to assist in identifying the cause of these errors? Thank you, Nikos Pyrgiotis ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
> On 30/08/2024 16:11 EEST Nikolaos Pyrgiotis via dovecot
> wrote:
>
>
> We are using the Shared Mailboxes in Dovecot Cluster architecture so the
> user`s mailbox should only be accessed by a single backend. We have been
> running with this architecture for about 2 years with dovecot version 2.3.19
> and we had not seen any `Inconsistency in map index` messages in the logs.
>
> We had already increased the default vsize_limit to 512M for the lmtp process
> but maybe we have to increase it more if this error will be occurring
> repeatedly.
> This is the relevant configuration stanza for the lmtp service for all
> backends.
> service lmtp {
> unix_listener lmtp {
> mode = 0666
> }
>
> inet_listener lmtp-mod {
> port = 24
> ssl = no
> }
> service_count = 0
>
> # Number of processes to always keep waiting for more connections.
> process_min_avail = 0
> process_limit = 0
>
> # If you set service_count=0, you probably need to grow this.
> vsz_limit = 512M
> client_limit = 1
> }
> But the question is what causes this inconsistency and if there is any change
> on 2.3.21 version that might have introduced it, because we did not get any
> similar logs on previous versions.
> When this does occur is there a chance that a mailbox might get corrupted? Is
> the only way to safely mitigate this to wait for the index rebuild to finish?
>
> Thank you,
> Nikos Pyrgiotis
You can try running force-resync on the affected mailbox.
Also I would recommend upping the limit to 2G for lmtp, indexer,
indexer-worker,lmtp etc. because otherwise user's index cache might not fit
into memory.
Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
We are using the Shared Mailboxes in Dovecot Cluster architecture so the user`s
mailbox should only be accessed by a single backend. We have been running with
this architecture for about 2 years with dovecot version 2.3.19 and we had not
seen any `Inconsistency in map index` messages in the logs.
We had already increased the default vsize_limit to 512M for the lmtp process
but maybe we have to increase it more if this error will be occurring
repeatedly.
This is the relevant configuration stanza for the lmtp service for all backends.
service lmtp {
unix_listener lmtp {
mode = 0666
}
inet_listener lmtp-mod {
port = 24
ssl = no
}
service_count = 0
# Number of processes to always keep waiting for more connections.
process_min_avail = 0
process_limit = 0
# If you set service_count=0, you probably need to grow this.
vsz_limit = 512M
client_limit = 1
}
But the question is what causes this inconsistency and if there is any change
on 2.3.21 version that might have introduced it, because we did not get any
similar logs on previous versions.
When this does occur is there a chance that a mailbox might get corrupted? Is
the only way to safely mitigate this to wait for the index rebuild to finish?
Thank you,
Nikos Pyrgiotis
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
> On 30/08/2024 15:32 EEST Nikolaos Pyrgiotis via dovecot > wrote: > > > Hello, > > We have been running dovecot version 2.3.21 since July on debian > bookworm system. After upgrading we have seen in the logs similar > messages to the following: > > 2024-08-29T19:33:27.755399+03:00 server1 dovecot: > lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Warning: > mdbox /path/user1/mdbox/storage: Inconsistency in map index (2213,40 != > 2213,56). > 2024-08-29T19:33:27.755399+03:00 server1 dovecot: > lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Warning: > mdbox /path/user1/mdbox/storage: Inconsistency in map index (2213,40 != > 2213,56) > 2024-08-29T19:33:27.838990+03:00 server1 dovecot: > lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: sieve: > msgid=<39f9c444-0c08-4bda-bfc9-512c34584...@modulus.gr>: fileinto > action: stored mail into mailbox 'Replies' > 2024-08-29T19:33:27.840222+03:00 server1 dovecot: > lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Warning: > fscking index file /path/user1/mdbox/storage/dovecot.map.index > 2024-08-29T19:33:27.953281+03:00 server1 dovecot: > lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Warning: > mdbox /path/user1/mdbox/storage: rebuilding indexes > 2024-08-29T19:37:27.989792+03:00 server1 dovecot: > lmtp(us...@modulus.gr)<481249>: Error: > Timeout (180s)while waiting for lock for transaction log file > /path/user1/mdbox/storage/dovecot.map.index.log (WRITE lock held by pid > 481240) > 2024-08-29T19:37:27.995913+03:00 server1 dovecot: > lmtp(us...@modulus.gr)<481249>: Error: sieve: > msgid=<39f9c444-0c08-4bda-bfc9-512c34584...@modulus.gr>: fileinto > action: failed to store into mailbox 'Mailbox': Timeout (180s) while > waiting for lock for transaction log file > /path/user1/mdbox/storage/dovecot.map.index.log (WRITE lock held by pid > 481240) > > > This has happened a few times since we upgraded always by a lmtp > process. It seems to be happening randomly and we have not been able to > reproduce it. > > The mailboxes are stored in a glusterfs server, and we use the dovecot > director with shared mailboxes architecture. > > Usually this was mitigated automatically by dovecot index rebuilding, > but last time it happened on a mailbox with a lot of giga byte of mail > data the following logs were produced for more than an hour: > > 2024-08-29T19:45:30.506694+03:00 server1 dovecot: > lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Error: > mmap_anon(/path/user/mdbox/mailboxes/Mailbox/dbox-Mails/dovecot.index.cache, > 74076160) failed: Cannot allocate memory > > During that period segfaults starting being generated for the imap > processes accessing that mailbox probably caused by timeouts from imap > processes attempting to fetch emails from the affected mailbox: > > [3912187.557029] imap[481783]: segfault at 38 ip 7fb03248abf8 sp > 7ffc88b264f0 error 4 in > libdovecot-storage.so.0.0.0[7fb0323c7000+d8000] likely on CPU 6 (core 3, > socket 0) Are you by chance accessing user's mailbox from multiple different servers, like delivering via LMTP on one, and user accessing on another? Also you probably want to hike up the vsize_limit, the default 256M is usually too little for mail processses. mmap'd files are counted against this limit. Aki ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Inconsistency in map index with dovecot v2.3.21
Hello, We have been running dovecot version 2.3.21 since July on debian bookworm system. After upgrading we have seen in the logs similar messages to the following: 2024-08-29T19:33:27.755399+03:00 server1 dovecot: lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Warning: mdbox /path/user1/mdbox/storage: Inconsistency in map index (2213,40 != 2213,56). 2024-08-29T19:33:27.755399+03:00 server1 dovecot: lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Warning: mdbox /path/user1/mdbox/storage: Inconsistency in map index (2213,40 != 2213,56) 2024-08-29T19:33:27.838990+03:00 server1 dovecot: lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: sieve: msgid=<39f9c444-0c08-4bda-bfc9-512c34584...@modulus.gr>: fileinto action: stored mail into mailbox 'Replies' 2024-08-29T19:33:27.840222+03:00 server1 dovecot: lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Warning: fscking index file /path/user1/mdbox/storage/dovecot.map.index 2024-08-29T19:33:27.953281+03:00 server1 dovecot: lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Warning: mdbox /path/user1/mdbox/storage: rebuilding indexes 2024-08-29T19:37:27.989792+03:00 server1 dovecot: lmtp(us...@modulus.gr)<481249>: Error: Timeout (180s)while waiting for lock for transaction log file /path/user1/mdbox/storage/dovecot.map.index.log (WRITE lock held by pid 481240) 2024-08-29T19:37:27.995913+03:00 server1 dovecot: lmtp(us...@modulus.gr)<481249>: Error: sieve: msgid=<39f9c444-0c08-4bda-bfc9-512c34584...@modulus.gr>: fileinto action: failed to store into mailbox 'Mailbox': Timeout (180s) while waiting for lock for transaction log file /path/user1/mdbox/storage/dovecot.map.index.log (WRITE lock held by pid 481240) This has happened a few times since we upgraded always by a lmtp process. It seems to be happening randomly and we have not been able to reproduce it. The mailboxes are stored in a glusterfs server, and we use the dovecot director with shared mailboxes architecture. Usually this was mitigated automatically by dovecot index rebuilding, but last time it happened on a mailbox with a lot of giga byte of mail data the following logs were produced for more than an hour: 2024-08-29T19:45:30.506694+03:00 server1 dovecot: lmtp(us...@modulus.gr)<481240><7TMeHNei0GYpeTkA1g8ogQ:P1>: Error: mmap_anon(/path/user/mdbox/mailboxes/Mailbox/dbox-Mails/dovecot.index.cache, 74076160) failed: Cannot allocate memory During that period segfaults starting being generated for the imap processes accessing that mailbox probably caused by timeouts from imap processes attempting to fetch emails from the affected mailbox: [3912187.557029] imap[481783]: segfault at 38 ip 7fb03248abf8 sp 7ffc88b264f0 error 4 in libdovecot-storage.so.0.0.0[7fb0323c7000+d8000] likely on CPU 6 (core 3, socket 0) What might be the reason that inconsistency of the map index is introduced on random users? We have seen some other users had similar issues and have mentioned them on this mailing list(dovecot Digest, Vol 249, Issue 4, dovecot Digest, Vol 250, Issue 8)? Is there any update for a mitigation or bug confirmation for dovecot version 2.3.21 that might be responsible? Thank you, Nikos Pyrgiotis ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
Apologies for the formatting. Retrying with HTML formatting disabled: We are getting the same warnings on 2 separate test servers; one with local ZFS storage, the other one with NFS storage (single server, data not shared with any other servers). We are imapsync'ing a few hundred user accounts from the production server so there is a lot of emails transfer a day. The warnings occur at random times and with random users. I reinstalled 2.3.21 from source (originally was using Dovecot repo packages) and tested by reverting that commit. The warnings continued. Next I tried with Dovecot 2.3.20 and still got the same warnings after a while. # 2.3.20 (80a5ac675d): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.20 (149edcf2) # OS: Linux 5.4.0-171-generic x86_64 Ubuntu 20.04.6 LTS nfs4 Feb 13 15:38:25 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=272469, TLS, session= Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: mdbox /mail/spool/username/storage: Inconsistency in map index (3,40 != 3,56) Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: fscking index file /mail/spool/username/storage/dovecot.map.index Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: mdbox /mail/spool/username/storage: rebuilding indexes Feb 13 15:38:29 dovecot: imap(username)<272469>: Error: Mailbox INBOX: /mail/spool/username/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is now inconsistent Feb 13 15:38:29 dovecot: imap(username)<272469>: Warning: fscking index file /mail/spool/username/storage/dovecot.map.index I will try with an even older version to continue testing... ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
On 2024-02-10 2:24 a.m., Timo Sirainen wrote: On 12. Jan 2024, at 9.37, John Alex. via dovecot wrote: since I upgraded dovecot to version 2.3.21 on a FreeBSD 13.2 system, I've been seeing a few occurences of the following warnings daily: Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp (user1)<8209>: Warning: mdbox / var/vmail/users/user1/mdbox/storage: Inconsistency in map index (5,40 != 5,56) Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp (user1)<8209>: Warning: fscking index file /var/vmail/users/user1/mdbox/storage/ dovecot.map.index Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp (user1)<8209>: Warning: mdbox / var/vmail/users/user1/mdbox/storage: rebuilding indexes Jan 11 13:13:26 dimokritos dovecot[33170]: lmtp (user1)<8209>: Warning: fscking index file /var/vmail/users/user1/mdbox/storage/ dovecot.map.index Hmm. I can't seem to be able to reproduce it easily at least. Looking at the code changes, I do see now that there is one related change that may be wrong, or at least the commit message is wrong: https:// github.com/dovecot/core/commit/ aae93d3ea73004ba1883bbf91c9e40f613ffc857 - although it shouldn't affect anything unless there are already some other errors. But maybe there's something I don't see right now. Can you try if the errors go away by reverting this change? We are getting the same warnings on 2 separate test servers; one with local ZFS storage, the other one with NFS storage (single server, data not shared with any other servers). We are imapsync'ing a few hundred user accounts from the production server so there is a lot of emails transfer a day. The warnings occur at random times and with random users. I reinstalled 2.3.21 from source (originally was using Dovecot repo packages) and tested by reverting that commit. The warnings continued. Next I tried with Dovecot 2.3.20 and still got the same warnings after a while. # 2.3.20 (80a5ac675d): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.20 (149edcf2) # OS: Linux 5.4.0-171-generic x86_64 Ubuntu 20.04.6 LTS nfs4 Feb 13 15:38:25 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=272469, TLS, session= Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: mdbox /mail/spool/username/storage: Inconsistency in map index (3,40 != 3,56) Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: fscking index file /mail/spool/username/storage/dovecot.map.index Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning: mdbox /mail/spool/username/storage: rebuilding indexes Feb 13 15:38:29 dovecot: imap(username)<272469>: Error: Mailbox INBOX: /mail/spool/username/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is now inconsistent Feb 13 15:38:29 dovecot: imap(username)<272469>: Warning: fscking index file /mail/spool/username/storage/dovecot.map.index I will try with an even older version to continue testing... ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
On 12. Jan 2024, at 9.37, John Alex. via dovecot wrote: > > Hi, > > since I upgraded dovecot to version 2.3.21 on a FreeBSD 13.2 system, I've > been seeing a few occurences of the following warnings daily: > > Jan 11 13:13:25 dimokritos dovecot[33170]: > lmtp(user1)<8209>: Warning: mdbox > /var/vmail/users/user1/mdbox/storage: Inconsistency in map index (5,40 != > 5,56) > Jan 11 13:13:25 dimokritos dovecot[33170]: > lmtp(user1)<8209>: Warning: fscking index file > /var/vmail/users/user1/mdbox/storage/dovecot.map.index > Jan 11 13:13:25 dimokritos dovecot[33170]: > lmtp(user1)<8209>: Warning: mdbox > /var/vmail/users/user1/mdbox/storage: rebuilding indexes > Jan 11 13:13:26 dimokritos dovecot[33170]: > lmtp(user1)<8209>: Warning: fscking index file > /var/vmail/users/user1/mdbox/storage/dovecot.map.index Hmm. I can't seem to be able to reproduce it easily at least. Looking at the code changes, I do see now that there is one related change that may be wrong, or at least the commit message is wrong: https://github.com/dovecot/core/commit/aae93d3ea73004ba1883bbf91c9e40f613ffc857 - although it shouldn't affect anything unless there are already some other errors. But maybe there's something I don't see right now. Can you try if the errors go away by reverting this change? > Should I be worried about data loss due to these index inconsistencies? No data loss, but it can lead to inefficient rebuilds. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Inconsistency in map index with dovecot v2.3.21
I have the same problem after updating dovecot 2.3.21, I am receiving the errors below for the accounts when I perform backups. doveadm(u...@user.com.br): Error: Purging namespace '' failed: BUG: Unknown internal error Error: failed sync dir: doveadm(us...@user2.com.br): Warning: mdbox /data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2.com.br/mdbox/storage: Inconsistency in map index (42,40 != 42,56) doveadm(us...@user2.com.br): Warning: fscking index file /data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2/mdbox/storage/dovecot.map.index doveadm(us...@user2.com.br): Warning: mdbox /data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2.com.br/mdbox/storage: rebuilding indexes doveadm(us...@user2.com.br): Warning: Transaction log file /data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2/mdbox/storage/dovecot.map.index.log was locked for 284 seconds (mdbox storage rebuild) doveadm(us...@user2.com.br): Error: Purging namespace '' failed: BUG: Unknown internal error As in the case of our friend the other day, the error no longer happens on the same account, it happens on another random one. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Inconsistency in map index with dovecot v2.3.21
Hi, since I upgraded dovecot to version 2.3.21 on a FreeBSD 13.2 system, I've been seeing a few occurences of the following warnings daily: Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp(user1)<8209>: Warning: mdbox /var/vmail/users/user1/mdbox/storage: Inconsistency in map index (5,40 != 5,56) Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp(user1)<8209>: Warning: fscking index file /var/vmail/users/user1/mdbox/storage/dovecot.map.index Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp(user1)<8209>: Warning: mdbox /var/vmail/users/user1/mdbox/storage: rebuilding indexes Jan 11 13:13:26 dimokritos dovecot[33170]: lmtp(user1)<8209>: Warning: fscking index file /var/vmail/users/user1/mdbox/storage/dovecot.map.index The numbers in parentheses seem to always be of the format (x,40 != x,56) or (x,40 != x,72). I also get such errors on some users when purging (doveadm-purge fails with error 75): doveadm(user2): Warning: mdbox /var/vmail/users/user2/mdbox/storage: Inconsistency in map index (5,40 != 5,72) doveadm(user2): Warning: fscking index file /var/vmail/users/user2/mdbox/storage/dovecot.map.index doveadm(user2): Warning: mdbox /var/vmail/users/user2/mdbox/storage: rebuilding indexes doveadm(user2): Error: Purging namespace '' failed: BUG: Unknown internal error If I rerun doveadm purge for this user it completes successfully. Can anyone shed some light on what might be causing these messages? I didn't have these with 2.3.20. Should I be worried about data loss due to these index inconsistencies? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
