Deliver.db and Checkpoint/Recovery
Hi, We recently encountered a problem with our Cyrus installation involving the checkpoint/recovery of the deliver.db file. We are running Cyrus 2.3.8 on Solaris 10. The server houses about 28,000 users, with a large volume of email. After our latest round of patching, we ended up removing the existing deliver.db database (skiplist). This didn't seem to be a problem at the time. Shortly thereafter, we noted that several lmtpd processes would run a checkpoint/recovery of the deliver.db database. This happens every time the database doubles in size (which is quite often when starting from 0). This wouldn't be an issue, except that mail delivery stops while this process is going on. Also, instead of one instance of lmtpd performing this action, anywhere from 1 to all of the currently running lmtpd processes will do this in succession. This can prevent mail delivery for up to an hour (the database is still small ... around 40 MB now). I am wondering if anyone has experienced this issue and how they dealt with it? Since we don't use duplicate suppression, we are looking at the possibility of patching the code to prevent the deliver.db from being check-pointed automatically. Another possibility is to change the code to only allow checkpoints to be done during the evening hours. Thoughts? Thanks, Joshua Van Horn Data Center & Client Services [EMAIL PROTECTED] (530) 754-6699 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: better techniques to identify and remove zero-day viruses from cyrus store sought
John Crawford wrote: > What's the best way, and second best way to react to zero-day virus > threats - messages that are delivered to the mail store before the > detection is in place? Any detection that can take place in the mail store can (and should) be moved up the chain, preferably to the MTA. > Is there a best practice that functions nicely > within the cyrus community? Yes, once a message is delivered, leave it alone. The most you should do at that point is maybe provide an opt-in sieve rule that stores suspicious messages in a special folder. But even this has caveats, and I prefer to let the users sort their own mail. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: better techniques to identify and remove zero-day viruses from cyrus store sought
John Crawford <[EMAIL PROTECTED]> wrote: > What's the best way, and second best way to react to zero-day virus > threats - messages that are delivered to the mail store before the > detection is in place? Refuse mail with executable attachments. List is at: http://support.microsoft.com/kb/262631 http://support.microsoft.com/kb/829982 This 100% effective and there are no zero-day or zero-hour problems. Done here since February 2003. Joseph Brennan Lead Email Systems Engineer Columbia University Information Technology Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
better techniques to identify and remove zero-day viruses from cyrus store sought
Hello. What's the best way, and second best way to react to zero-day virus threats - messages that are delivered to the mail store before the detection is in place? Is there a best practice that functions nicely within the cyrus community? Like a perl script that traverses the mail store (via imap or cyrus utilities) and checks message content against a antivirus command line? And then safely within the cyrus system deletes the message? I can't do that. I don't have that functionality. Can anyone share code or ideas to help make this happen? The second best I can think of is traverse and locate the storage system from the command line, remove found exploited messages and recontruct ASAP the mailboxes with changes. Here we are working outside of cyrus, so it's discouraged. long line: find /var/spool/imap/user/ -name "*\." -ctime -6h -print0 | xargs -0 clamscan | grep FOUND > badones remove and reconstruct based on file badones. As a related question, if I locate a message in the mail store from command mode, and neuter some aspect of the vulnerability presented by modifying the text of the stored message yet retaining the same message size, are then internal cyrus structures that require updating? I seem to get good results from this, but I'm guessing reconstruct afterwards is best. (and that the idea of modification from the file system is, yes, discouraged). thanks for any thoughts or ideas, John Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: 'batch move' of a few thousand mailboxes to different partitions
> On Wed, 18 Apr 2007, Eric Luyten wrote: > > > Cyrus 2.2.13 on a Solaris 9 server > > 4 Cyrus spool partitions, 350 GB each, nearing 80% occupation > > (Hashing performed on first letter basis of account name) > > > > I have two extra 350 GB partitions ready and 'partition' lines > > already added to /etc/imapd.conf > > > > > > To spread out the mail data over six partitions I would need > > to move about 350 GB of mail. > > > > Two scenarios : > > > > a) use rename_mailbox in cyradm <...> > > b) use 'cp -p -r', update a text dump of mailboxes.db and regenerate [Andrew Morgan] > I would use method "a". You can easily do the moves in batch using a perl > script. Here is a simple perl function we use to move users between > murder backends: Hello, I just wanted to report to the list that I had a thorough look at your code and the code offered by another info-cyrus participant ; I then used the combined knowledge to modify imapcreate.pl (by Garry Mills and Clement Nodens) to suit our needs. We already used the latter on our system and feel entirely comfortable with it, so... On 25, 26 and 27 July about 675 GB (this is NO typo) of Cyrus data and meta- data were moved to fresh partitions. We now operate a server with 4 x 350 GB and 4 x 420 GB of Cyrus spool. There are just under 45.000 users, 22 million messages on the server. This is Solaris 9 on an eight-processor SunFire V1280 ; CPU load average varies between 1 and 2. (September will bring more students and more load.) Yeah, we really ought to move to a cluster setup. That's a 2008 project. Eric Luyten, Computing Centre VUB/ULB. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
[PATCH] Changing folder paths on disk to support fast-rename
This is a very small patch for a very big change (and incomplete,
it doesn't alter rehash, dohash, etc)
Anyway...
Considering the fast-rename and extending it to delete, we have
the following problem:
folders:
user.brong
user.brong.foo
user.brong.foo.bar
Delete/rename user.brong.foo WITHOUT affecting user.brong.foo.bar.
On disk these are:
BASEDIR/b/user/brong/
BASEDIR/b/user/brong/foo/
BASEDIR/b/user/brong/foo/bar/
So you either have to:
a) move every single non "bar" file to the new location, or
b) move user/brong/foo to the new location, then create a new
user/brong/foo and move bar back into it.
I realised there was:
c) change folder layout on disk such that sub folders in the
IMAP world aren't sub-folders in the on-disk world.
(c) started looking more attractive all the time, especially
since I've seen it at work with my Maildir copy which
offlineimap keeps up to date for me.
The following patch creates:
BASEDIR/user/b/brong/user.brong/
BASEDIR/user/b/brong/user.brong.foo/
BASEDIR/user/b/brong/user.brong.foo.bar/
and handles shared namespace, etc as:
BASEDIR/shared/shared
BASEDIR/shared/shared.foo
BASEDIR/shared/shared.foo.bar
Basically, if mboxlist_isusermailbox(name, 0) is true then
it gets hashed the first way, otherwise the second.
This has the nice property with our rename/delete patch
(also attached) and required for this patch you get:
BASEDIR/user/b/brong/DELETED.user.brong.foo.TIMESTAMP/
keeping the user's stuff all together on disk.
I really do prefer the "each IMAP folder is a different
folder" and "hashing based on realistic use patterns" (at
least for us... and I'm a believer in Hans Reiser's attitude
that if your filesystem can't handle a folder with thousands
of items then your filesystem needs fixing)
I'll be working on fast-rename in this new universe next,
but I thought I should throw this out there for comments.
So, what do you think (and yes, I'll be making rehash at
least work happily with this, because we'll need to do it
ourselves. Current plan - down the replica, rehash it,
change the config, bring it back up. Failover, rinse,
repeat)
Bron.
P.S. I'm using quilt now to work on these patches. It's an
order of magnitude nicer than working on each patch in
isolation and using diff/patch by hand, but it means that
patches are now in a series that apply one after the other
rather than all against the raw upstream source. I hope to
post my entire quilt series once I fix the one thing that
has a FastMail specific key encoded in it to use a config
option instead.
--
Bron Gondwana
[EMAIL PROTECTED]
Index: cyrus-imapd-2.3.9/imap/mailbox.c
===
--- cyrus-imapd-2.3.9.orig/imap/mailbox.c 2007-08-21 01:28:56.0
-0400
+++ cyrus-imapd-2.3.9/imap/mailbox.c2007-08-21 01:36:23.0 -0400
@@ -3362,6 +3362,7 @@
{
const char *idx;
char c, *p;
+char itemname[MAX_MAILBOX_PATH+1];
snprintf(buf, buf_len, "%s", root);
buf_len -= strlen(buf);
@@ -3369,7 +3370,7 @@
if (config_virtdomains && (p = strchr(name, '!'))) {
*p = '\0'; /* split domain!user */
- if (config_hashimapspool) {
+ if (config_hashimapspool != IMAP_ENUM_HASHIMAPSPOOL_OFF) {
c = (char) dir_hash_c(name, config_fulldirhash);
snprintf(buf, buf_len, "%s%c/%s", FNAME_DOMAINDIR, c, name);
}
@@ -3382,7 +3383,7 @@
buf += strlen(buf);
}
-if (config_hashimapspool) {
+if (config_hashimapspool == IMAP_ENUM_HASHIMAPSPOOL_ON) {
idx = strchr(name, '.');
if (idx == NULL) {
idx = name;
@@ -3392,13 +3393,30 @@
c = (char) dir_hash_c(idx, config_fulldirhash);
snprintf(buf, buf_len, "/%c/%s", c, name);
+
+ /* change all '.'s to '/' */
+ for (p = buf; *p; p++) {
+ if (*p == '.') *p = '/';
+ }
+} else if (config_hashimapspool == IMAP_ENUM_HASHIMAPSPOOL_USERID) {
+ if (idx = mboxname_isusermailbox(name, 0)) {
+ c = (char) dir_hash_c(idx, config_fulldirhash);
+ strncpy(itemname, idx, MAX_MAILBOX_PATH);
+ if (p = strchr(itemname, '.')) *p = '\0'; /* trim to username */
+ snprintf(buf, buf_len, "/user/%c/%s/%s", c, itemname, name);
+ } else {
+ strncpy(itemname, name, MAX_MAILBOX_PATH);
+ if (p = strchr(itemname, '.')) *p = '\0'; /* trim to first part */
+ snprintf(buf, buf_len, "/%s/%s", itemname, name);
+ }
} else {
/* standard mailbox placement */
snprintf(buf, buf_len, "/%s", name);
-}
-/* change all '.'s to '/' */
-for (p = buf; *p; p++) {
- if (*p == '.') *p = '/';
+ /* change all '.'s to '/' */
+ for (p = buf; *p; p++) {
+ if (*p == '.') *p = '/';
+ }
}
+
}
Index: cyrus-imapd-2.3.9/lib/imapoptions
===
--- cyrus-imapd-2.
Re: Automatic Sieve Script Subscription
Thanks a lot! I will try it. > Message du 21/08/07 12:12 > De : "Christian Kratzer" <[EMAIL PROTECTED]> > A : "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > Copie à : "info-cyrus" > Objet : Re: Automatic Sieve Script Subscription > > Hi, > > On Tue, 21 Aug 2007, [EMAIL PROTECTED] wrote: > > > Hy all, > > > > > > Do you know if there's a way for any mailboxes to subscribe > > automaticaly to a sieve script at creation of the mailbox? > > > > I would like for all users to transfert SPAM tagged subject to > > a SPAM folders, or mailling list mail to mailling list folder > > without having to explain all users to write it with avelsieve. > > > > I tought Autosieve patch was able to do it but it "only" > > creates folders if it doesn't exist. > > we use a perl script to automatically create new accounts, > create and subscribe the Spam mailbox and also install and > activate a default sieve script. > > We use following perl modules: > > use Cyrus::IMAP::Admin; > use Cyrus::SIEVE::managesieve; > use IMAP::Admin; > > IMAP::Admin required a patch to allow logging in with separate > authentication and authorisation rules for subscribing to the > newly created Spam mailbox. > > Apart from that it was straight forward. > > Greetings > Christian > > -- > Christian Kratzer CK Software GmbH > Email: [EMAIL PROTECTED] Schwarzwaldstr. 31 > Phone: +49 7452 889 135 D-71131 Jettingen > Fax: +49 7452 889 136 HRB 245288, Amtsgericht Stuttgart > Web: http://www.cksoft.de/ Geschaeftsfuehrer: Christian Kratzer > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Converting a unified murder to a traditional one
Hi! I'd set up a unified murder config, deciding not to care about the XFER bug, but when deploying it, I stumbled over an annoying bug: copying a message from a mailbox on the local server to a mailbox on a remote one doesn't work, nor does copying a message from a mailbox on a remote server to a mailbox on the local one (see bug #2976). Moreover, I was unable to get proxying to work if the remote node only supported plaintext authentication (even if it was only supported over an encrypted layer) - I had to make the nodes support digest-md5 authentication, and put the administrative accounts into /etc/sasldb. But this resulted in my clients trying to authenticate using digest-md5, too, and resulted in stupid delays while they waited for the authentication to fail so that they could authenticate using plaintext, which was the method that could be integrated into our existing authentication system (radius). So, I ended up having to add a couple of frontends for the clients to connect to, with support only for plaintext auth, and have no clients connect directly to the backends - so now I'm running a system that is, for all intents and purposes, a traditional Murder config, but with all the bugs and limitations of a unified Murder config. So the question is: can I turn a unified Murder config into a traditional one safely, editing the imapd.conf on one node at a time, restarting (or maybe even kill -HUPing) one node at a time? Any experiences, or do I just try this out? Thanks. --Janne Peltonen email admin Univ. of Helsinki -- Janne Peltonen <[EMAIL PROTECTED]> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Automatic Sieve Script Subscription
Hy all, Do you know if there's a way for any mailboxes to subscribe automaticaly to a sieve script at creation of the mailbox? I would like for all users to transfert SPAM tagged subject to a SPAM folders, or mailling list mail to mailling list folder without having to explain all users to write it with avelsieve. I tought Autosieve patch was able to do it but it "only" creates folders if it doesn't exist. Thanks a lot. Thanks again for your good work on Cyrus! Créez votre adresse électronique [EMAIL PROTECTED] 1 Go d'espace de stockage, anti-spam et anti-virus intégrés. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
