Re: cyrus.expunge/cyrus.cache mistmatch on replica
On Mon, 24 Sep 2007, Bron Gondwana wrote: Picture if you will... sync_client finds a message missing on the replica during a mailboxes event and causes a sync_combine_commit to be called on the replica to absorb the new messages. * cyrus.index is re-written with new cache_offset values for each message * cyrus.cache is re-written with just the records in cyrus.index * cyrus.expunge is ***ERROR, BOGUS CACHE FILE OFFSETS*** The common cause of sync_combine_commit() will be unexpunge on the master. I imagine that a single GUID appearing in cyrus.index and cyrus.expunge is bad news. sync_server needs to know about the expunge index. It did in my original code base, but replication got merged before delayed expunge. More later. In theory I'm on holiday this week. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH.
Re: cyrus.expunge/cyrus.cache mistmatch on replica
On Tue, 25 Sep 2007, David Carter wrote: I imagine that a single GUID appearing in cyrus.index and cyrus.expunge is bad news. sync_server needs to know about the expunge index. It did in my original code base, but replication got merged before delayed expunge. Single _UID_ appearing in cyrus.index and cyrus.expunge. GUIDs on the brain. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH.
Re: cyrus.expunge/cyrus.cache mistmatch on replica
On Tue, 25 Sep 2007 12:15:04 +0100 (BST), David Carter [EMAIL PROTECTED] said: On Tue, 25 Sep 2007, David Carter wrote: I imagine that a single GUID appearing in cyrus.index and cyrus.expunge is bad news. sync_server needs to know about the expunge index. It did in my original code base, but replication got merged before delayed expunge. Single _UID_ appearing in cyrus.index and cyrus.expunge. GUIDs on the brain. Yeah - that appears to be happening a lot more than I expected. At least by the output that check_cyrus_index.pl has been givin me. I need to spend some time auditing the causes of it, but it seems to think there are a lot of duplicates across those two files sometimes (I remove the one in cyrus.expunge if there are) Bron. -- Bron Gondwana [EMAIL PROTECTED]
Re: cyrus.expunge/cyrus.cache mistmatch on replica
On Mon, 24 Sep 2007, Bron Gondwana wrote: Do you guys suggest anything? Ideas I've come up with include: Ho hum. My original two phase expunge code had separate expunge.index and expunge.cache files, which keeps cache entries for expunged messages well out of the way. I think this is probably the simplest solution. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH.
Re: cyrus.expunge/cyrus.cache mistmatch on replica
On Mon, 24 Sep 2007 08:20:50 +0100 (BST), David Carter [EMAIL PROTECTED]
said:
On Mon, 24 Sep 2007, Bron Gondwana wrote:
Do you guys suggest anything? Ideas I've come up with include:
Ho hum. My original two phase expunge code had separate expunge.index
and expunge.cache files, which keeps cache entries for expunged messages
well out of the way. I think this is probably the simplest solution.
Any reason to actually keep them though? It's not as if unexpunge is
common enough to be worth optimising, and they're all auto-generated
content that we can recreate from the message if needed.
Bron ( besides, if I see anything like this copy and pasted to another
place in the code I will _cry_ )
path = (mailbox.mpath
(config_metapartition_files
IMAP_ENUM_METAPARTITION_FILES_CACHE)) ?
mailbox.mpath : mailbox.path;
strlcpy(fnamebuf, path, sizeof(fnamebuf));
strlcat(fnamebuf, FNAME_CACHE, sizeof(fnamebuf));
strlcat(fnamebuf, .NEW, sizeof(fnamebuf));
newcache_fd = open(fnamebuf, O_RDWR|O_TRUNC|O_CREAT, 0666);
if (newcache_fd == -1) {
fclose(newindex);
mailbox_close(mailbox);
return IMAP_IOERROR;
}
--
Bron Gondwana
[EMAIL PROTECTED]
Re: cyrus.expunge/cyrus.cache mistmatch on replica
On Mon, 24 Sep 2007, Bron Gondwana wrote: Any reason to actually keep them though? It's not as if unexpunge is common enough to be worth optimising, and they're all auto-generated content that we can recreate from the message if needed. My original motivation was user access to expunged messages and deleted mailboxes. But Cyrus 2.3, doesn't do this, so you are quite correct. Not having to rewrite the entire cyrus.cache file on every expunge is a fairly substantial performance boost. We can keep this benefit by leaving unreferenced junk in the cache file and then garbage collecting (rewrite on expunge and/or cyr_expire) when the file reaches some threshold. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH.
