Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Jun 9, 2009, at 6:40 AM, Jonathan Siegle wrote: But of course there are no problems (well, some very rare random ones maybe) having multiple processes accessing the same mailbox on the same server. Ever since I wrote my imaptest tool (a few years ago?) I've been heavily stress testing multiple connections modifying the mailbox at the same time. Timo, Can you tell me what platform/filesystem you use for testing? Linux/ext3 mainly. I also use Solaris/ufs for my own mails. Once in a while I also try OS X/HFS+. Oh and can you hint at what bad things may happen when I get the error record points outside file? When you get that error it's not that bad. Dovecot just deletes the cache file which can cause small slowdowns. More problematic is when there's a corruption that Dovecot doesn't detect and sends garbage to client.
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Jun 3, 2009, at 12:17 PM, Timo Sirainen wrote: On Wed, 2009-06-03 at 12:11 -0400, Timo Sirainen wrote: On Wed, 2009-06-03 at 10:14 -0400, Jonathan Siegle wrote: On Jun 3, 2009, at 9:35 AM, Timo Sirainen wrote: On Wed, 2009-06-03 at 09:31 -0400, Jonathan Siegle wrote: Are you saying that multiple processes on the same folder(INBOX) on the same IMAP server can cause this collision as well? Is there a difference between running multiple processes on the same folder(INBOX) on multiple IMAP servers vs running multiple processes on the same folder on a single IMAP server? I don't know. That depends on how GPFS is implemented. Pick a local filesystem, say ext3? But with ext3 you can't have multiple servers accessing the same filesystem. But of course there are no problems (well, some very rare random ones maybe) having multiple processes accessing the same mailbox on the same server. Ever since I wrote my imaptest tool (a few years ago?) I've been heavily stress testing multiple connections modifying the mailbox at the same time. Timo, Can you tell me what platform/filesystem you use for testing? Oh and can you hint at what bad things may happen when I get the error record points outside file? Thanks, Jonathan smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Jun 2, 2009, at 1:31 PM, Timo Sirainen wrote: On Tue, 2009-06-02 at 13:24 -0400, jsie...@psu.edu wrote: dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file So you're using AIX? Do you also happen to use NFS? Can you reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTest No NFS here. The filesystem is called GPFS. It is a clustered FS. Ah, that probably explains it. Can multiple different servers modify the same mailbox? Cache file is the part of Dovecot that demands the most from the OS/filesystem. The most difficult part is probably that it writes to the file without locking. It first reserves a space and then starts writing there. Multiple processes can write to the same file at the same time. Timo, Are you saying that multiple processes on the same folder(INBOX) on the same IMAP server can cause this collision as well? Is there a difference between running multiple processes on the same folder(INBOX) on multiple IMAP servers vs running multiple processes on the same folder on a single IMAP server? This should probably be changed at some point, since it could just buffer more data to memory and then lock, write, unlock. That would also make the code simpler, since it can currently leave holes to the file because it has to guess initially how much space to reserve.. This is definitely on my wish list. Thanks, Jonathan smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Wed, 2009-06-03 at 09:31 -0400, Jonathan Siegle wrote: Are you saying that multiple processes on the same folder(INBOX) on the same IMAP server can cause this collision as well? Is there a difference between running multiple processes on the same folder(INBOX) on multiple IMAP servers vs running multiple processes on the same folder on a single IMAP server? I don't know. That depends on how GPFS is implemented. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Jun 3, 2009, at 9:35 AM, Timo Sirainen wrote: On Wed, 2009-06-03 at 09:31 -0400, Jonathan Siegle wrote: Are you saying that multiple processes on the same folder(INBOX) on the same IMAP server can cause this collision as well? Is there a difference between running multiple processes on the same folder(INBOX) on multiple IMAP servers vs running multiple processes on the same folder on a single IMAP server? I don't know. That depends on how GPFS is implemented. Pick a local filesystem, say ext3? smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Wed, 2009-06-03 at 12:11 -0400, Timo Sirainen wrote: On Wed, 2009-06-03 at 10:14 -0400, Jonathan Siegle wrote: On Jun 3, 2009, at 9:35 AM, Timo Sirainen wrote: On Wed, 2009-06-03 at 09:31 -0400, Jonathan Siegle wrote: Are you saying that multiple processes on the same folder(INBOX) on the same IMAP server can cause this collision as well? Is there a difference between running multiple processes on the same folder(INBOX) on multiple IMAP servers vs running multiple processes on the same folder on a single IMAP server? I don't know. That depends on how GPFS is implemented. Pick a local filesystem, say ext3? But with ext3 you can't have multiple servers accessing the same filesystem. But of course there are no problems (well, some very rare random ones maybe) having multiple processes accessing the same mailbox on the same server. Ever since I wrote my imaptest tool (a few years ago?) I've been heavily stress testing multiple connections modifying the mailbox at the same time. signature.asc Description: This is a digitally signed message part
[Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
I use mbox with mbox-snarf plugin.
I get errors like:
dovecot: IMAP(testuser): Corrupted index cache file /full/path/
dovecot.index.cache: record points outside file
The errors can be in either folders or INBOX.
I put some syslog statements in src/lib-index/mail-cache-lookup.c to
help understand this.
if (offset + sizeof(*rec) cache-mmap_length) {
mail_cache_set_corrupted(cache, record points
outside file);
syslog(LOG_DEBUG,mail_cache_get_record rec-size is
%d , rec-size);
syslog(LOG_DEBUG,cache-mmap_length is %d , cache-
mmap_length);
syslog(LOG_DEBUG,offset is %d , offset);
return -1;
}
syslog: mail_cache_get_record rec-size is 268595472
syslog: cache-mmap_length is 4096
syslog: offset is 1630760037
Oh I forgot to print off sizeof(*rec). Would that help?
Thanks,
Jonathan
smime.p7s
Description: S/MIME cryptographic signature
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Tue, 2009-06-02 at 09:51 -0400, Jonathan Siegle wrote: I use mbox with mbox-snarf plugin. I get errors like: dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file So you're using AIX? Do you also happen to use NFS? Can you reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTest signature.asc Description: This is a digitally signed message part
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Tue, 2 Jun 2009, Timo Sirainen wrote: On Tue, 2009-06-02 at 09:51 -0400, Jonathan Siegle wrote: I use mbox with mbox-snarf plugin. I get errors like: dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file So you're using AIX? Do you also happen to use NFS? Can you reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTest No NFS here. The filesystem is called GPFS. It is a clustered FS. I'll work on reproducing with imaptest. Once I reproduce, I'll give more details about how I run dovecot. thanks, Jonathan
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Tue, 2009-06-02 at 13:24 -0400, jsie...@psu.edu wrote: dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file So you're using AIX? Do you also happen to use NFS? Can you reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTest No NFS here. The filesystem is called GPFS. It is a clustered FS. Ah, that probably explains it. Can multiple different servers modify the same mailbox? Cache file is the part of Dovecot that demands the most from the OS/filesystem. The most difficult part is probably that it writes to the file without locking. It first reserves a space and then starts writing there. Multiple processes can write to the same file at the same time. This should probably be changed at some point, since it could just buffer more data to memory and then lock, write, unlock. That would also make the code simpler, since it can currently leave holes to the file because it has to guess initially how much space to reserve.. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Tue, 2 Jun 2009, Timo Sirainen wrote: On Tue, 2009-06-02 at 13:24 -0400, jsie...@psu.edu wrote: dovecot: IMAP(testuser): Corrupted index cache file /full/path/ dovecot.index.cache: record points outside file So you're using AIX? Do you also happen to use NFS? Can you reproduce this error by running imaptest for a while? http://imapwiki.org/ImapTest No NFS here. The filesystem is called GPFS. It is a clustered FS. Ah, that probably explains it. Can multiple different servers modify the same mailbox? Yes. I have N+1 machines that access the same filesystem and it looks local to both machines. Cache file is the part of Dovecot that demands the most from the OS/filesystem. The most difficult part is probably that it writes to the file without locking. It first reserves a space and then starts writing there. Multiple processes can write to the same file at the same time. This should probably be changed at some point, since it could just buffer more data to memory and then lock, write, unlock. That would also make the code simpler, since it can currently leave holes to the file because it has to guess initially how much space to reserve.. Ok. Should I disable Cache file or what would be the best fix? Thanks, Jonathan
Re: [Dovecot] record points outside file error with dovecot revision 9116:9ae55b68cf61
On Tue, 2009-06-02 at 13:40 -0400, jsie...@psu.edu wrote: Cache file is the part of Dovecot that demands the most from the OS/filesystem. The most difficult part is probably that it writes to the file without locking. It first reserves a space and then starts writing there. Multiple processes can write to the same file at the same time. This should probably be changed at some point, since it could just buffer more data to memory and then lock, write, unlock. That would also make the code simpler, since it can currently leave holes to the file because it has to guess initially how much space to reserve.. Ok. Should I disable Cache file or what would be the best fix? If you do that, you'll lose all the performance advantages of Dovecot. Do you already have mmap_disable=yes? You could also try if setting mail_nfs_index=yes happens to help. signature.asc Description: This is a digitally signed message part
