unexpunge time interval
http://www-uxsup.csx.cam.ac.uk/~dpc22/cyrus/patches/2.3cvs/ unexpunge-since.patch = Finally got fed up interpreting unexpunge -l with magic sed scripts. Here's a patch which adds a -t time-interval argument, so that: unexpunge -t 24h user/dpc22 restores everything expunged in the last 24 hours for that mailbox. That's the way that my old two phase expunge stuff used to work. -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH.
statuscache
Looking (again) at integrating this. Two observations, the first somewhat minor, and the second somewhat major: - statuscache v.2 doesn't have support for HIGHESTMODSEQ. This is trivial to add to a v.3. - statuscache v.2 stores statuscache_data as a binary blob, which is platform dependent (byte-order word size). This make statuscache.db non-portable. I believe that this might be the first cyrusdb that would be non-portable. Does anybody care? -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University
Re: statuscache
Bron Gondwana wrote: On Thu, Jan 17, 2008 at 02:47:27PM -0500, Ken Murchison wrote: Looking (again) at integrating this. Two observations, the first somewhat minor, and the second somewhat major: - statuscache v.2 doesn't have support for HIGHESTMODSEQ. This is trivial to add to a v.3. Yeah, shouldn't be a big problem. - statuscache v.2 stores statuscache_data as a binary blob, which is platform dependent (byte-order word size). This make statuscache.db non-portable. I believe that this might be the first cyrusdb that would be non-portable. Does anybody care? Hmm - it would be a very minor cost to make it platform independent. On the flip side, it's not supposed to persist over shutdowns of Cyrus anyway. Still, I would vote for fixing that too - might as well keep the platform safeness. Processor time is cheap compared to disk IO at least in our setup. The only tricky part is going to be upgrading from v.2 to v.3. If its not designed to be persistent and/or migrated between platforms, I'm not sure I care, since your design is definitely quicker. I'm in the process of refactoring your patch a little, and I'm going to leave the storage method alone for now. I'll send you my patch when complete. I'll have it done later tonight, after I make dinner for my kids. BTW, is statuscache:log necessary, or is it just for debugging? -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University
Re: statuscache
- statuscache v.2 stores statuscache_data as a binary blob, which is platform dependent (byte-order word size). This make statuscache.db non-portable. I believe that this might be the first cyrusdb that would be non-portable. Does anybody care? The only tricky part is going to be upgrading from v.2 to v.3. If its not designed to be persistent and/or migrated between platforms, I'm not sure I care, since your design is definitely quicker. Basically our startup scripts completely delete the statuscache.db file on startup, so migrating isn't an issue for us at all. So if you want to make it platform independent for others in the future, I'm fine with that. BTW, is statuscache:log necessary, or is it just for debugging? It's not necessary, it was just helpful for debugging rather than having to recompile everything. From memory, clients make SO many status calls, that it can completely flood the log if you leave the syslog calls in. Rob
Re: statuscache
Rob Mueller wrote: Basically our startup scripts completely delete the statuscache.db file on startup, so migrating isn't an issue for us at all. Why do you delete the cache? I doesn't appear from the code that the cache entries would be invalid if the mailbox and seen state aren't touched. BTW, is statuscache:log necessary, or is it just for debugging? It's not necessary, it was just helpful for debugging rather than having to recompile everything. From memory, clients make SO many status calls, that it can completely flood the log if you leave the syslog calls in. I'm changing the statuscache option to a switch and leaving the logging in as LOG_DEBUG -- Kenneth Murchison Systems Programmer Project Cyrus Developer/Maintainer Carnegie Mellon University
Re: statuscache
Why do you delete the cache? I doesn't appear from the code that the cache entries would be invalid if the mailbox and seen state aren't touched. Historical. We only use berkeley db for statuscache and duplicate delivery, and because of problems in the past with berkeley db's corrupting themselves, we found it easier to just delete them on startup. It avoids the case of cyrus failing to start if one of the bdb's is corrupted I'm changing the statuscache option to a switch and leaving the logging in as LOG_DEBUG My only concern with this is that it can cause a LOT of log messages. I've never been entirely sure how well syslog handles the case of lots of log messages. I think it still has to send them over the socket to the syslogd. Most people then have a syslog.conf that discards debug messages, so it always seemed like a possible way to overflow the syslog socket buffer and loose important messages (i'm pretty sure syslog never guarantees that a message will be logged, which is why djb implemented his own way that uses streams rather than dgrams). This is probably all entirely academic and not a real issue... Rob
Re: statuscache
Here's my modified patch against CVS. I suppose some code could be
added to statuscache_lookup() to read both v.2 and v.3. There isn't any
reason to invalidate all v.2 caches just because they don't have
highestmodseq
--
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University
Index: doc/changes.html
===
RCS file: /afs/andrew/system/cvs/src/cyrus/doc/changes.html,v
retrieving revision 1.152
diff -u -r1.152 changes.html
--- doc/changes.html 17 Jan 2008 18:51:39 - 1.152
+++ doc/changes.html 18 Jan 2008 04:16:11 -
@@ -10,6 +10,10 @@
h1Changes to the Cyrus IMAP Server since 2.3.11/h1
ul
+liAdded ttstatuscache.db/tt to cache IMAP STATUS data which
+significantly reduces the amount of I/O necessary when neither the
+mailbox nor \Seen state has changed
+i-- courtesy of Fastmail.fm/i./li
liAdded option to ttunexpunge/tt to restore messages by time
interval
i-- courtesy of David Carter/i./li
Index: imap/Makefile.in
===
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/Makefile.in,v
retrieving revision 1.191
diff -u -r1.191 Makefile.in
--- imap/Makefile.in 18 Oct 2007 18:48:02 - 1.191
+++ imap/Makefile.in 18 Jan 2008 04:16:11 -
@@ -101,7 +101,8 @@
convert_code.o duplicate.o saslclient.o saslserver.o signals.o \
annotate.o search_engines.o squat.o squat_internal.o mbdump.o \
imapparse.o telemetry.o user.o notify.o protocol.o idle.o quota_db.o \
- sync_log.o $(SEEN) mboxkey.o backend.o tls.o message_guid.o
+ sync_log.o $(SEEN) mboxkey.o backend.o tls.o message_guid.o \
+ statuscache_db.o
IMAPDOBJS=pushstats.o imapd.o proxy.o imap_proxy.o index.o version.o
Index: imap/ctl_cyrusdb.c
===
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/ctl_cyrusdb.c,v
retrieving revision 1.30
diff -u -r1.30 ctl_cyrusdb.c
--- imap/ctl_cyrusdb.c 4 Jan 2008 12:24:05 - 1.30
+++ imap/ctl_cyrusdb.c 18 Jan 2008 04:16:11 -
@@ -80,6 +80,7 @@
#include libcyr_cfg.h
#include mboxlist.h
#include seen.h
+#include statuscache.h
#include tls.h
#include util.h
#include xmalloc.h
@@ -99,7 +100,8 @@
{ FNAME_ANNOTATIONS, config_annotation_db, 1 },
{ FNAME_DELIVERDB, config_duplicate_db, 0 },
{ FNAME_TLSSESSIONS, config_tlscache_db, 0 },
-{ FNAME_PTSDB, config_ptscache_db,0 },
+{ FNAME_PTSDB, config_ptscache_db, 0 },
+{ FNAME_STATUSCACHEDB, config_statuscache_db, 0 },
{ NULL, NULL, 0 }
};
Index: imap/global.c
===
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/global.c,v
retrieving revision 1.24
diff -u -r1.24 global.c
--- imap/global.c 28 Sep 2007 02:27:46 - 1.24
+++ imap/global.c 18 Jan 2008 04:16:11 -
@@ -95,6 +95,7 @@
struct cyrusdb_backend *config_duplicate_db;
struct cyrusdb_backend *config_tlscache_db;
struct cyrusdb_backend *config_ptscache_db;
+struct cyrusdb_backend *config_statuscache_db;
/* Called before a cyrus application starts (but after command line parameters
* are read) */
@@ -190,6 +191,8 @@
cyrusdb_fromname(config_getstring(IMAPOPT_TLSCACHE_DB));
config_ptscache_db =
cyrusdb_fromname(config_getstring(IMAPOPT_PTSCACHE_DB));
+ config_statuscache_db =
+ cyrusdb_fromname(config_getstring(IMAPOPT_STATUSCACHE_DB));
/* configure libcyrus as needed */
libcyrus_config_setstring(CYRUSOPT_CONFIG_DIR, config_dir);
Index: imap/global.h
===
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/global.h,v
retrieving revision 1.9
diff -u -r1.9 global.h
--- imap/global.h 27 Mar 2007 19:53:08 - 1.9
+++ imap/global.h 18 Jan 2008 04:16:11 -
@@ -154,5 +154,6 @@
extern struct cyrusdb_backend *config_duplicate_db;
extern struct cyrusdb_backend *config_tlscache_db;
extern struct cyrusdb_backend *config_ptscache_db;
+extern struct cyrusdb_backend *config_statuscache_db;
#endif /* INCLUDED_GLOBAL_H */
Index: imap/imapd.c
===
RCS file: /afs/andrew/system/cvs/src/cyrus/imap/imapd.c,v
retrieving revision 1.539
diff -u -r1.539 imapd.c
--- imap/imapd.c 17 Jan 2008 13:07:40 - 1.539
+++ imap/imapd.c 18 Jan 2008 04:16:12 -
@@ -95,6 +95,7 @@
#include mkgmtime.h
#include mupdate-client.h
#include quota.h
+#include statuscache.h
#include sync_log.h
#include telemetry.h
#include tls.h
@@ -685,6 +686,10 @@
annotatemore_init(0, NULL, NULL);
annotatemore_open(NULL);
+if (config_getswitch(IMAPOPT_STATUSCACHE)) {
+ statuscache_open(NULL);
+}
+
/* Create a protgroup for input from the client and selected backend */
protin = protgroup_new(2);
@@ -866,6 +871,11 @@
annotatemore_close();
annotatemore_done();
+if (config_getswitch(IMAPOPT_STATUSCACHE))
