expunge_days option in Cyrus 2.4

2012-07-17 Thread David Carter
We moved from Cyrus 2.3 to 2.4 earlier in the year. We run with delayed 
expunge, with a nightly job to expire old expunged messages after 28 days:

   cyr_expire -E 3 -D 28 -X 28

This worked nicely under Cyrus 2.3.

A few days back we had our first request to restore email which has been 
expunged more than seven days previously, and I made the unpleasant 
discovery that we were only actually keeping expunged messages for 7 days.

A bit of digging through source code uncovered:

   expunge_days: 7
 Number of days to retain expunged messages before cleaning up
 their index  records. The  default is 7. This is a necessary for
 QRESYNC to work  correctly. If combined with delayed expunge (above)
 you will also be able to unexpunge messages during this time.

which seems to expire messages as soon as a mailbox is closed after 7 
days, regardless of what cyr_expire might be up to. I believe increasing 
expunge_days to 30 days will make things work the way that I expected.

I put this down to a bad experience and failing to trawl through 
imap.conf.5 carefully enough looking for new options (it isn't mentioned 
in changes.html, but a lot changed between 2.3 and 2.4). I have since 
found a short thread on this list from January discussing the new option.

However I had a second bad experience with expunge_days over the weekend.

We use the replication engine to seed new mailstores and move users around 
the cluster. This worked nicely using my own code in Cyrus 2.1 and 2.3.
In 2.4 the following innocent looking test:

sync_client.c :

 /* check that replication stands a chance of succeeding */
 if (remote  !is_repeat) {
if (mailbox-i.deletedmodseq  remote-highestmodseq) {
syslog(LOG_NOTICE, inefficient replication (
   MODSEQ_FMTMODSEQ_FMT ) %s,
   mailbox-i.deletedmodseq, remote-highestmodseq,
   local-name);
r = IMAP_AGAIN;
goto done;
}

appears to kick in if you run sync_client -u between a master and 
replica or backup mailstore with a gap of more than 7 days between the two 
runs. mailbox_full_update() gets rather excited about messages on the 
replica which have been deleted, expunged and unlinked from the master.

Again expunge_days: 30 should help, but I thought that others might want 
to be warned before they try to use sync_client to move users around.

-- 
David Carter Email: david.car...@ucs.cam.ac.uk
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Re: expunge_days option in Cyrus 2.4

2012-07-17 Thread Bron Gondwana
On Tue, Jul 17, 2012, at 04:45 PM, David Carter wrote:
 A bit of digging through source code uncovered:
 
expunge_days: 7
  Number of days to retain expunged messages before cleaning up
  their index  records. The  default is 7. This is a necessary for
  QRESYNC to work  correctly. If combined with delayed expunge (above)
  you will also be able to unexpunge messages during this time.

Sorry!  This was to stop sites having their disk fill up unexpectedly, but I
should have probably set it to something higher.  We use 7 days at FastMail
currently.  It will actually check for double that before it triggers usually,
but your value is significantly higher than that.

The sync_client thing is a separate matter... it should consider those to be
bogus and clean them up I think... hmm.

Magic is generally bad, but I don't think we can guarantee cyr_expire will run
with sensible options, so the code has to have some sort of sanity check.
I guess it COULD just trigger a syslog, but then the sites not doing cyr_expire
probably aren't reading syslog either :(

Bron.
-- 
  Bron Gondwana
  br...@fastmail.fm


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Re: expunge_days option in Cyrus 2.4

2012-07-17 Thread David Carter
On Tue, 17 Jul 2012, Bron Gondwana wrote:

 The sync_client thing is a separate matter... it should consider those 
 to be bogus and clean them up I think... hmm.

It definitely cleaned up many cases. I found a couple of examples where 
messages seemed to get pulled back to the master, fortunately not on one 
of the live systems. The problem of course is that you only get to see the 
after state. The logs do indicate a lot of UID and MODSEQ ping pong took 
place as the two ends worked out their differences.

Noone has come to ask me what happened, which I take as a good sign :).

-- 
David Carter Email: david.car...@ucs.cam.ac.uk
University Computing Service,Phone: (01223) 334502
New Museums Site, Pembroke Street,   Fax:   (01223) 334679
Cambridge UK. CB2 3QH.

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus