Re: CVS to GIT
On Mon, 13 Sep 2010 16:34:02 +0200, Jeroen van Meeuwen (Kolab Systems) vanmeeu...@kolabsys.com wrote: Mark Cave-Ayland wrote: My other point, of course, was that since the PostgreSQL guys worked to fix a couple of bugs in cvs2git over the past couple of weeks, you may get a better result if you grab the tip version of cvs2git and re-run the conversion. If we were using cvs2git, sure! But we're not using cvs2git, we're using git-cvsimport ;-) I would highly recommend cvs2git over git-cvsimport for reproducing an accurate history of the cvs repository. cvs2git doesn't do incremental imports like cvsimport, but I don't think that is needed in this situation. You may want to give it a try, just for a comparison anyway. -Brian Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Backend attempting to proxy to itself?
imapd is trying to proxy because the entry 1 store-101.internal.example.com tells it that it's remote, even though it is not. Theoretically this would work correctly with a unified murder configuration, where any machine can proxy for another, but it isn't implemented. The mailbox entry on the backend should look like; user.simon 0default simon lrswipkxtecda I'm not sure how the mailbox list ended up with entries like that on your backend. Are you running mupdate there? There should probably be a warning in the docs about not starting mupdate on a backend, if there isn't already. To fix it, you may need to dump the db to text, use sed/awk/perl (pick your favorite) and change all the 1 servername!default to 0 default, remove the old db and reload it. Hope that helps. -Brian On Mon, 26 Apr 2010 12:44:35 +0100 (BST), Simon Beale si...@minos.org.uk wrote: I'm having problems with getting the backend responding correctly in a murder cluster (using Simon Matter's 2.3.16 rpm built on CentOS 5.4). I've got it so that I can run cyradm and issue 'cm user.simon' on the frontend, see it make the mailbox on the backend, and doing 'ctl_mboxlist -d' on murder, frontend and backend all list the relevant backend location: user.simon 1 store-101.internal.example.com!default simon lrswipkxtecda However, when I run imtest and login on the frontend: . LIST * * LIST (\HasNoChildren) . INBOX . OK Completed (0.000 secs 2 calls) . SELECT INBOX . NO Server(s) unavailable to complete operation Looking at the output of strace and syslogs on the backend, it appears that the backend is trying to make a new TLS connection back to itself rather than directly answering the incoming SELECT. Apr 26 13:24:09 store-101 imap[26128]: accepted connection Apr 26 13:24:09 store-101 master[26615]: about to exec /usr/lib/cyrus-imapd/imapd Apr 26 13:24:09 store-101 imap[26128]: login: switch-101.internal.example.com [10.10.10.37] simon DIGEST-MD5 User logged in Apr 26 13:24:09 store-101 imap[26615]: executed Apr 26 13:24:09 store-101 imap[26615]: accepted connection Apr 26 13:24:09 store-101 master[26616]: about to exec /usr/lib/cyrus-imapd/imapd Apr 26 13:24:09 store-101 imap[26616]: executed Apr 26 13:24:09 store-101 imap[26615]: skiplist: checkpointed /var/lib/imap/tls_sessions.db (1124 records, 206900 bytes) in 0 seconds Apr 26 13:24:09 store-101 imap[26615]: imapd:Loading hard-coded DH parameters Apr 26 13:24:09 store-101 imap[26615]: SSL_accept() incomplete - wait Apr 26 13:24:09 store-101 imap[26128]: Doing a peer verify Apr 26 13:24:09 store-101 imap[26128]: Doing a peer verify Apr 26 13:24:09 store-101 imap[26128]: received server certificate Apr 26 13:24:09 store-101 imap[26128]: starttls: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits new client) no authentication Apr 26 13:24:09 store-101 imap[26615]: SSL_accept() succeeded - done Apr 26 13:24:09 store-101 imap[26615]: starttls: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits new) no authentication Apr 26 13:24:09 store-101 imap[26128]: couldn't authenticate to backend server: no mechanism available Can anyone help me work out why the backend appears to be attempting to proxy onwards rather than answering the SELECT itself? === Backend imapd.conf: admins: cyrus cyrus-frontend allowallsubscribe: true allowplaintext: true allowusermoves: true configdirectory:/var/lib/imap delete_mode:delayed duplicate_db: skiplist expunge_mode: delayed hashimapspool: true improved_mboxlist_sort: true lmtp_downcase_rcpt: true mupdate_authname: cyrus-frontend mupdate_password: mupdate_server: switch-102.internal.example.com mupdate_username: cyrus-frontend normalizeuid: true partition-default: /var/spool/imap proxyservers: cyrus-frontend ptscache_db:skiplist sasl_mech_list: PLAIN LOGIN DIGEST-MD5 sasl_pwcheck_method:auxprop servername: store-101.internal.example.com sievedir: /var/lib/imap/sieve statuscache_db: skiplist tlscache_db:skiplist tls_ca_file:/etc/pki/tls/certs/ca-bundle.crt tls_cert_file: /etc/ssl/certs/wildcard.pem tls_key_file: /etc/ssl/certs/wildcard.pem unix_group_enable: false Frontend imapd.conf: admins: cyrus allowplaintext: false allowusermoves: true configdirectory:/var/lib/imap delete_mode:delayed duplicate_db: skiplist expunge_mode: delayed improved_mboxlist_sort: true lmtp_downcase_rcpt: true mupdate_authname: cyrus-frontend mupdate_password: mupdate_server: switch-102.internal.example.com mupdate_username: cyrus-frontend normalizeuid:
Re: Backend attempting to proxy to itself?
You need to remove mupdate entry from the Services section of the cyrus
config on the backend servers. mupdate always assumes mailboxes are remote
so it is going through and changing all the mailbox entries to remote ones.
I'm not sure about the autocreate feature though, I assume that is what you
are trying to use by creating the mailbox while connected to the frontend.
Traditionally you would connect to the backend where you wanted the mailbox
to live and create it there.
-Brian
On Mon, 26 Apr 2010 17:27:46 +0100 (BST), Simon Beale
si...@minos.org.uk
wrote:
However, if I restart the backend at this point, I get the old entries
back again in addition to the fixed entries.
Given this cluster isn't yet in production, I've just stopped the entire
cluster, deleted mailboxes.db from everything and rm -rf
${partition-default}/* in case there was something bad lurking around
from previous experiments. But it's still the case that if I cm
user.simon on the frontend with cyradm, the mailboxes.db on the backend
appears as ... 1 store-101...!default...
So for some reason I'm not getting correct mailbox location information
created on the backends.
I've included my cyrus.conf files and the murder master's imapd.conf
below
in case there's something wrong I've put in any of those.
Cheers
Simon
=
Backend/frontend cyrus.conf
START {
recover cmd=ctl_cyrusdb -r
idled cmd=idled
#the next line is only present on the backend
mupdatepush cmd=ctl_mboxlist -m
}
SERVICES {
imap cmd=imapd listen=imap proto=tcp4 prefork=2
imaps cmd=imapd -s listen=imaps proto=tcp4 prefork=5
pop3 cmd=pop3d listen=pop3 proto=tcp4 prefork=2
pop3s cmd=pop3d -s listen=pop3s proto=tcp4 prefork=2
sieve cmd=timsieved listen=sieve proto=tcp4 prefork=2
mupdate cmd=mupdate listen=3905 proto=tcp4 prefork=2
fud cmd=fud listen=4201 proto=udp4 prefork=1
maxchild=10
lmtp cmd=lmtpd -a listen=127.0.0.1:2003 prefork=1
}
EVENTS {
checkpointcmd=ctl_cyrusdb -c period=30
delprune cmd=cyr_expire -D 3 -E 3 -X 3 at=0400
tlsprune cmd=tls_prune at=0400
}
==
Murder master /etc/cyrus.conf
START {
recover cmd=ctl_cyrusdb -r
idled cmd=idled
}
SERVICES {
mupdate cmd=mupdate -m listen=mupdate prefork=1
}
EVENTS {
checkpointcmd=ctl_cyrusdb -c period=30
delprune cmd=cyr_expire -D 3 -E 3 -X 3 at=0400
tlsprune cmd=tls_prune at=0400
}
==
Murder master /etc/imapd.conf
admins: cyrus cyrus-frontend
allowplaintext: true
configdirectory:/var/lib/imap
duplicate_db: skiplist
improved_mboxlist_sort: true
lmtp_downcase_rcpt: true
normalizeuid: true
partition-default: /var/spool/imap
ptscache_db:skiplist
sasl_mech_list: DIGEST-MD5 PLAIN LOGIN
sasl_pwcheck_method:auxprop
sievedir: /var/lib/imap/sieve
statuscache_db: skiplist
tlscache_db:skiplist
tls_ca_file:/etc/pki/tls/certs/ca-bundle.crt
tls_cert_file: /etc/ssl/certs/wildcard.pem
tls_key_file: /etc/ssl/certs/wildcard.pem
unix_group_enable: false
imapd is trying to proxy because the entry 1
store-101.internal.example.com tells it that it's remote, even though
it
is not. Theoretically this would work correctly with a unified murder
configuration, where any machine can proxy for another, but it isn't
implemented. The mailbox entry on the backend should look like;
user.simon 0default simon lrswipkxtecda
I'm not sure how the mailbox list ended up with entries like that on
your
backend. Are you running mupdate there? There should probably be a
warning in the docs about not starting mupdate on a backend, if there
isn't
already. To fix it, you may need to dump the db to text, use
sed/awk/perl
(pick your favorite) and change all the 1 servername!default to 0
default, remove the old db and reload it. Hope that helps.
-Brian
On Mon, 26 Apr 2010 12:44:35 +0100 (BST), Simon Beale
si...@minos.org.uk
wrote:
I'm having problems with getting the backend responding correctly in a
murder cluster (using Simon Matter's 2.3.16 rpm built on CentOS 5.4).
I've
got it so that I can run cyradm and issue 'cm user.simon' on the
frontend,
see it make the mailbox on the backend, and doing 'ctl_mboxlist -d' on
murder, frontend and backend all list the relevant backend location:
user.simon 1 store-101.internal.example.com!default simon
lrswipkxtecda
However, when I run imtest and login on the frontend:
. LIST *
* LIST (\HasNoChildren) . INBOX
. OK Completed (0.000 secs 2 calls)
. SELECT INBOX
. NO Server(s) unavailable to complete operation
Looking at the output of strace and syslogs on the backend, it appears
that the
Re: Building an interface to unexpunge for users
On Mon, 29 Mar 2010 21:17:08 -0400, Wesley Craig w...@umich.edu wrote: On 29 Mar 2010, at 18:18, Bron Gondwana wrote: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3008 Yeah, that should be the default. Search form is not for presentation to users! That patch has been waiting since Nov 2008 to be committed. But I'd be happy to adjust it so human readable is the default. Perhaps Brian could put the rest of those patches in BZ as well. Submitted to BZ, as I commented there the human readable patch may be dependent on the first (-r,-c) patch because of the code refactoring. If people are only interested in the human readable patch I can probably reorder them with git. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Nested INBOX folders - hard to delete
What version of cyrus are you running? If it's a version 2.3 with delayed delete enabled, you are likely running into a bug in cyrus. We have run into this issue, where you have a mailbox that is near the max length and when you try to delete it, cyrus adds the extra DELETED/timestamp to the mailbox name and finds it's longer than the max allowed, then returns an error. I've had a patch for this in bugzilla for a while; https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3154 If you don't want to shut cyrus down, you can probably use cyr_dbtool to delete the mailbox entry, then manually cleanup the filesystem. -Brian On Mon, 29 Mar 2010 16:53:51 +0300 (EEST), Jukka Huhta jukka.hu...@helsinki.fi wrote: We have several users with folders like this: user.username.INBOX.INBOX.INBOXINBOX.INBOX.INBOX.INBOX.Deleted Messages (total of 76 INBOXes or something). No doubt the folders are generated by a bug in Apple Mail, but how to get rid of them? Users can't do that by themselves, and no MUA I've tried is able to handle the folders either. I wonder what Cyrus thinks of them, probably doesn't like too long names or something. Cyradm is just giving an error from Cyrus: cyradm dm user.username.INBOX.* Deleting mailbox user.username.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.Deleted Messages...Invalid mailbox name I'd really like to fix the problem without shutting any Cyrus instances down and manually dumping, editing and undumping the mailbox list. Any suggestions? -Jukka Huhta Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Building an interface to undelete for users
We implemented a self service restore page for our users, I'm not sure if
the code is anywhere publicly accessibe and I think it is fairly specific
to our installation since we use kerberos/murder. Our implementation uses
remctl to make calls to unexpunge on the appropriate backend and we made
some changes to unexpunge to make it more user friendly (prints the output
in human readable form) and efficient (-r recursive + -c message count).
I've attached the patches for unexpunge if your interested, since I haven't
gotten around to posting them in bugzilla.
-Brian
On Mon, 29 Mar 2010 11:28:50 +0100, David Mayo d.j.m...@bath.ac.uk wrote:
We have recently upgraded to Cyrus 2.3 and are making full use of the
delayed delete feature, and we are considering writing an interface to
allow users to undelete their own messages and mailboxes.
Before I start work on this myself, I thought I'd check with people here
to see if anyone has any tips or code they are willing to share. I hope
we will be able to publish the product we create.
Regards,
Dave.
David Mayo
Networks/Systems Administrator
University of Bath Computing Services, UK
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.htmlFrom bf080b28e2b4389558d3bc6f25f605f6924773f9 Mon Sep 17 00:00:00 2001
From: Brian Awood baw...@umich.edu
Date: Thu, 24 Sep 2009 15:43:03 -0400
Subject: [PATCH 1/3] Add recursive (-r) and count expunged (-c) to unexpunge
command
The -c flag prints a count of the messages in the expunged state
for a mailbox. The output is in the form mailbox: count.
The -r flag causes unexpunge to recurse from the specified
mailbox, it can be combined with the new -c flag, or the existing
-l and -a but not -u.
diff --git a/imap/unexpunge.c b/imap/unexpunge.c
index afbc6df..e9dd19e 100644
--- a/imap/unexpunge.c
+++ b/imap/unexpunge.c
@@ -79,17 +79,24 @@
/* global state */
const int config_need_data = 0;
+int unsetdeleted = 0;
+time_t time_since;
+
/* current namespace */
static struct namespace unex_namespace;
+int do_unexpunge(char *name, int matchlen, int maycreate, void *rock);
+int unexpunge(char *mboxname, unsigned long *uids, unsigned nuids);
+
int verbose = 0;
void usage(void)
{
fprintf(stderr,
- unexpunge [-C altconfig] -l mailbox\n
+ unexpunge [-C altconfig] -l [-r] mailbox\n
+ unexpunge [-C altconfig] -c [-r] mailbox\n
unexpunge [-C altconfig] -t time-interval [ -d ] [ -v ]
mailbox\n
- unexpunge [-C altconfig] -a [-d] [-v] mailbox\n
+ unexpunge [-C altconfig] -a [-d] [-v] [-r] mailbox\n
unexpunge [-C altconfig] -u [-d] [-v] mailbox uid...\n);
exit(-1);
}
@@ -99,8 +106,9 @@ enum {
MODE_LIST,
MODE_ALL,
MODE_TIME,
-MODE_UID
-};
+MODE_UID,
+MODE_COUNT
+} mode = MODE_UNKNOWN;
struct msg {
unsigned recno;
@@ -162,7 +170,7 @@ void list_expunged(struct mailbox *mailbox,
int restore_expunged(struct mailbox *mailbox,
struct msg *msgs, unsigned long eexists,
const char *expunge_index_base,
-unsigned *numrestored, int unsetdeleted)
+unsigned *numrestored)
{
int r = 0;
const char *irec;
@@ -415,114 +423,33 @@ int restore_expunged(struct mailbox *mailbox,
return r;
}
-int main(int argc, char *argv[])
+/*
+ * mboxlist_findall() callback function for unexpunge recursive features
+ */
+int
+do_unexpunge(char *name,
+ int matchlen __attribute__((unused)),
+ int maycreate __attribute__((unused)),
+ void *rock __attribute__((unused)))
{
-extern char *optarg;
-int opt, r = 0;
-char *alt_config = NULL;
-char buf[MAX_MAILBOX_PATH+1];
-struct mailbox mailbox;
-int doclose = 0, mode = MODE_UNKNOWN, unsetdeleted = 0;
-char expunge_fname[MAX_MAILBOX_PATH+1];
+return unexpunge(name, NULL, 0);
+}
+
+int
+unexpunge(char *mboxname, unsigned long *uids, unsigned nuids)
+{
+int doclose = 0, r;
int expunge_fd = -1;
-struct stat sbuf;
+char expunge_fname[MAX_MAILBOX_PATH+1];
const char *lockfailaction;
-struct msg *msgs;
unsigned numrestored = 0;
-time_t last_update, time_since = time(NULL);
-int len, secs = 0;
-
-if ((geteuid()) == 0 (become_cyrus() != 0)) {
- fatal(must run as the Cyrus user, EC_USAGE);
-}
-
-while ((opt = getopt(argc, argv, C:laudt:v)) != EOF) {
- switch (opt) {
- case 'C': /* alt config file */
- alt_config = optarg;
- break;
-
- case 'l':
- if (mode != MODE_UNKNOWN) usage();
- mode = MODE_LIST;
- break;
-
- case 'a':
- if (mode != MODE_UNKNOWN) usage();
- mode = MODE_ALL;
- break;
-
- case 't':
- if (mode != MODE_UNKNOWN) usage
Re: Setting TCP keepalive for Cyrus daemons
I've had a patch against lmptd in bugzilla for a while; https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3158 I think the code probably just needs to be audited, to find the places where setting a read timeout on the stream would be sufficient. -Brian On Friday 12 February 2010 16:59:31 Paul M Fleming wrote: Shouldn't these client connections already be handled by the poptimeout timeout options? unless you have it set to zero... We have had problems within the murder (old code had several spots where murder front - back communications could deadlock).. On Sat, 13 Feb 2010, Bron Gondwana wrote: On Fri, Feb 12, 2010 at 09:45:02AM -0600, Gary Mills wrote: I've been noticing idle pop3d processes on our Cyrus front end server for some time. These should be transient. One that was several days old had an established TCP connection to a wireless client that had disappeared. Presumably the client never closed the connection. Setting TCP keepalive on the file descriptor should permit the kernel to close the connection in this situation. Does this sound reasonable? Perhaps it's already been addressed in a later Cyrus version. We're running cyrus-imapd-2.3.8. I'm willing to add a `keepalive' option to Cyrus master along with the setsockopt() system call to enable that setting. This option could be added to the cyrus.conf file for any services that could benefit from it. Would this be a reasonable addition to Cyrus? This is something I've been wanting to look in to as well. If a machine crashes, it can leave sync_clients hanging for ever, thinking they are still talking to the replica - meaning that they hold the lock and replication doesn't start back up. Quite annoying. Is there any reason to make it an option rather than just always having it on? Or at least not to make it the default. If it's a good idea, it SHOULD be the default. I'm strongly against having hundreds of lines of config file required to get the sanest defaults! Hmm... oh: http://tools.ietf.org/html/rfc1122#page-101 Implementors MAY include keep-alives in their TCP implementations, although this practice is not universally accepted. If keep-alives are included, the application MUST be able to turn them on or off for each TCP connection, and they MUST default to off. Keep-alive packets MUST only be sent when no data or acknowledgement packets have been received for the connection within an interval. This interval MUST be configurable and MUST default to no less than two hours. Sounds like it does need to be configurable! Also, the defaults are crazy high amounts for a local reliable network. I think I'd prefer about 5 minutes than 2 hours! Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Delays in collecting messages with Mac Mail
We've seen somewhat similar behavior with the Apple Mail client in the past. My guess is it's IDLE related. If your using Apple Mail that came with 10.5, it's IDLE implementation appears to work at first. However it has a bug in it so that it stops working after a while. It seemed related to how it managed different connections to the server, using a different connection for each folder you opened. It is more reliable in the version that comes with OS X 10.6. However we've also seen very strange behavior in Apple Mail when it's local cache gets corrupted somehow. You can try using the Mailbox-Rebuild option, but typically the most effective way I've found to correct it is to go into Preferences:Accounts and remove the account that has the problem, then restart Apple Mail and reconfigure the account. -Brian On Thursday 04 February 2010 11:06:41 Simon Fraser wrote: Hi folks, I'm running the debian package of cyrus in lenny, 2.2.13-14, and I've encountered some odd behaviour with Mac Mail that I'm struggling to explain. Emails that have been successfully delivered sometimes fail to appear in Mac Mail clients for many hours. I'm currently uncertain if it's being caused by cyrus or Mail.app, although it has only started happening since we moved from a compiled 2.2.12 version to the debian package on new hardware - we migrated the mailboxes using the cyradm 'xfer' feature. Exim's mail logs report successful delivery, and the timestamp of the file in the cyrus spool matches the delivery time. If checked with another email client, such as squirrelmail, the emails are in the mailbox. I'll mention that our squirrelmail installation connects using imap through a perdition proxy, just for completeness, but there are some users who have experienced this delay without using any other email clients. Using the example of one of our sysadmins yesterday, an email was delivered into his mailbox at 10:09. It appeared in Mac Mail at approximately 4pm. In the intervening hours, other messages were delivered and appeared in Mac Mail; I also see several successful logins from his mac to the imap server, regular opening of INBOX.Apple Mail To Do and so forth. During that time he had no other email client open. I also checked and found that Mac Mail was configured to use the 'IDLE' command, which, when manually testing it, appears to be working satisfactorily. Has anyone encountered something like this before? Thanks, Simon. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Question about reconstruct
On Tuesday 12 January 2010 18:15:29 Bron Gondwana wrote:
On Tue, Jan 12, 2010 at 08:57:28AM -0800, wcoo...@nakedape.cc wrote:
Perhaps it would be a good idea, then, to make the '-k' behavior the
default and use another option to invert the logic?
Absolutely... it will probably happen in 2.4 or 2.5... it's on my list
of incompatible changes!
If anyone is interested, here's a patch that we use that turns on -k by
default when delayed expunge is enabled. We've never seen a need for the non
-k behavior since there are other ways to achieve the same results, if you
really do need to delete messages meta data. Of course this patch is a lot
larger than it needs to be for just the -k change, since it adds a -w
(warning) and -e (add un-indexed files to expunge). Also, it looks like a lot
of changes, but it's mostly additional if statements and some corresponding
formatting fixes.
-Brian
commit 989d72de5762dcc0962d0919f8c4652798816cf4
Author: Brian Awood baw...@umich.edu
Date: Mon Aug 10 17:07:02 2009 -0400
Add -w and -e options to reconstruct
Added -w to warn admins about potential user visible changes, like
an increase or decrease in the number of messages indexed in a
mailbox.
Added -e to enable reconstructing an expunge file. Using -e will put
all unindexed files into the expunge file.
This patch also enables the -k behavior with delayed expunge is
enabled.
diff --git a/imap/reconstruct.c b/imap/reconstruct.c
index 6a115e0..23be7e9 100644
--- a/imap/reconstruct.c
+++ b/imap/reconstruct.c
@@ -134,10 +134,13 @@ struct uniqmailid * find_uniqid (char * mailboxname, char * mailboxid);
extern cyrus_acl_canonproc_t mboxlist_ensureOwnerRights;
int code = 0;
+int status = 0;
int keepflag = 0;
int syncflag = 0;
int guid_clear = 0;
int guid_set = 0;
+int warn_only = 0;
+int add_expunge = 0;
int main(int argc, char **argv)
{
@@ -163,7 +166,7 @@ int main(int argc, char **argv)
assert(INDEX_HEADER_SIZE == (OFFSET_SPARE4+4));
assert(INDEX_RECORD_SIZE == (OFFSET_MODSEQ+4));
-while ((opt = getopt(argc, argv, C:kp:rmfsxgG)) != EOF) {
+while ((opt = getopt(argc, argv, C:kp:rmfsxgGwe)) != EOF) {
switch (opt) {
case 'C': /* alt config file */
alt_config = optarg;
@@ -205,6 +208,14 @@ int main(int argc, char **argv)
guid_set = 1;
break;
+ case 'w':
+ warn_only = 1;
+ break;
+
+ case 'e':
+ add_expunge = 1;
+ break;
+
default:
usage();
}
@@ -213,6 +224,11 @@ int main(int argc, char **argv)
cyrus_init(alt_config, reconstruct, 0);
global_sasl_init(1,0,NULL);
+if( config_getenum(IMAPOPT_EXPUNGE_MODE) ==
+ IMAP_ENUM_EXPUNGE_MODE_DELAYED ) {
+ keepflag = 1;
+}
+
/* Set namespace -- force standard (internal) */
if ((r = mboxname_init_namespace(recon_namespace, 1)) != 0) {
syslog(LOG_ERR, error_message(r));
@@ -302,10 +318,14 @@ int main(int argc, char **argv)
(*recon_namespace.mboxname_tointernal)(recon_namespace, argv[i],
NULL, buf);
- r = mboxlist_createmailbox(buf, 0, start_part, 1,
+ if( warn_only ) {
+ printf( Add to mailbox list: %s\n, buf );
+ } else {
+ r = mboxlist_createmailbox(buf, 0, start_part, 1,
cyrus, NULL, 0, 0, !xflag);
- if(r) {
- fprintf(stderr, could not create %s\n, argv[i]);
+ if(r) {
+ fprintf(stderr, could not create %s\n, argv[i]);
+ }
}
}
}
@@ -363,10 +383,14 @@ int main(int argc, char **argv)
p = head.next;
head.next = p-next;
- /* create p (database only) and reconstruct it */
- /* partition is defined by the parent mailbox */
- r = mboxlist_createmailbox(p-name, 0, NULL, 1,
+ if( warn_only ) {
+ printf( Discovered mailbox %s\n, p-name );
+ } else {
+ /* create p (database only) and reconstruct it */
+ /* partition is defined by the parent mailbox */
+ r = mboxlist_createmailbox(p-name, 0, NULL, 1,
cyrus, NULL, 0, 0, !xflag);
+ }
if (!r) {
do_reconstruct(p-name, strlen(p-name), 0, head);
} else {
@@ -387,7 +411,8 @@ int main(int argc, char **argv)
cyrus_done();
-return code;
+if(code) return code;
+return ( status ? 1: 0);
}
void usage(void)
@@ -833,11 +858,16 @@ int reconstruct(char *name, struct discovered *found)
snprintf(mbpath, sizeof(mbpath), %s%s, path, FNAME_HEADER);
if(stat(mbpath, sbuf) == -1) {
/* Header doesn't exist, create it! */
- r = mailbox_create(name, mypart, myacl, NULL,
+ if( warn_only ) {
+ printf( stat() error of header file, does it exist?\n );
+ return IMAP_IOERROR;
+ } else {
+ r = mailbox_create(name, mypart, myacl, NULL,
((mytype MBTYPE_NETNEWS) ?
MAILBOX_FORMAT_NETNEWS :
MAILBOX_FORMAT_NORMAL), NULL);
- if(r) return r;
+ if(r) return r;
+ }
}
/* Now open just the header (it will hopefully be valid) */
@@ -859,8 +889,7 @@ int reconstruct(char *name, struct discovered *found
Re: prevent stuck processes with large folder manipulations
Hi Paul On Sunday 03 January 2010 @ 13:35, Paul Dekkers wrote: Are you running 32 or 64-bits? We run 64-bits, and I realized that this allows a single imapd process to consume a considerable amount of memory (eg. all) instead of just 2G or so per process. (The server I'm talking about has 6G of memory and 2G swap, should be ok for ~50 active users.) But now there's nothing limiting the memory consumption by just one single user/process, I guess. We're running 32-bit, with 8G of ram and 8G of swap but around ~4k active users per backend machine. It suspect it would probably harmless to limit the cyrus processes to 2G with ulimit, I doubt any one imapd would ever legitimately need to be larger than that. We are not running any proxies, well, other than stunnel doing SSL offloading on another box. So apart from SSL the clients are directly talking to imapd. I do believe it's just Thunderbird timing-out. This box is running 2.3.13 (Simon's RPM on 64-bit RHEL 4), but I recall seeing it with olders versions before (of both Cyrus and TB). Actually; the mozilla bugtracker reference sounds very similar. It happens with large deletions too, just like with large copies, as described in this bug. But we see this with much more recent versions, Thunderbird 3 and the recent 2.3 cyrus. The behavior I observed was that TB was hitting a timeout, even though the server was responding almost immediately. TB just wasn't recognizing the response and it seemed related to the length of that response. If you still see the behavior with Thunderbird 3, and your able to reproduce the problem, you may be able to provide more info to the TB developers. They seem pretty responsive if they get good feedback, so there's a good chance it could be solved once and for all if you can give provide them some debugging info. Regarding the last suggestion in this bug; for the deletes I did consider the delayed expunge and/or delete, but that wouldn't help with the large copies. Even large copies should occur fairly quickly, unless you have disabled singleinstancestore. singleinstancestore is enabled by default, and causes cyrus to create hardlinks rather than actually creating copies. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: prevent stuck processes with large folder manipulations
We used to run into this fairly frequently when we were running 2.2 proxy hosts. Although it never reached the point where it caused memory exhaustion, usually we would catch it at an early stage because of a slowdown in replication. It always appeared to be a Thunderbird client, and at one time I was able to reproduce the behavior. Do you have any proxies in your configuration, if so what version are they? I believe there is some sort of weird interaction between cyrus 2.2 and thunderbird (at least 2.x). I had posted some info to the mozilla bugtracker; https://bugzilla.mozilla.org/show_bug.cgi?id=340265 But before I really had the time to investigate we upgraded our proxies to 2.3 and as far as I know the problem seems to have stopped. -Brian On Friday 01 January 2010 @ 15:45, Paul Dekkers wrote: Hi, From time to time (but mostly at the start of the year ;-)), I notice a lot of load caused by people archiving their mail-folders. Maybe this is mostly caused by Thunderbird going mad, but I was wondering if I could do anything on the server-side to prevent things from going bad. Because now I see memory (and swap) exhaustion and the side-effects of that (Linux kernel killing processes)... One example: someone was moving tens of thousands of messages from 2009 to a new 2009 folder. Apparently Thunderbird was stuck, maybe because these things don't happen instantly moving this number of messages so the server doesn't finish quickly: but Thunderbird created a lot (~100) of sessions / imapd-processes for this user, maybe after timeouts. (I think) Only one process was active doing the link's, it looked like the others were mostly waiting for a write lock (fortunately), waiting to do the same thing. (Inspected with strace.) But when the process that hogged the CPU was killed, the next process took over, until all similar processes were killed. And the new archive-folder now ended up with several duplicates, taking about millions instead of tens of thousands. (We'll have to see how to dedup that, any ideas are appreciated otherwise I'll write something for that.) It just happened, but it happened before. This mail-server is not that busy, 100 users, but it happens at least a few times per year. Any idea how to prevent things like this? Judging from the man-pages I don't think I could do this from within cyrus, but that I would have to prevent from linux's ulimit or so and tune that (sounds like a tough job)... or could I actually do this with cyrus parameters? Curious if people have similar experiences :-) Regards, Paul P.S. This specific machine is running Red Hat 4 and a version of Simon's (s)rpm. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: prevent stuck processes with large folder manipulations
On Friday 01 January 2010 @ 15:45, Paul Dekkers wrote: similar processes were killed. And the new archive-folder now ended up with several duplicates, taking about millions instead of tens of thousands. (We'll have to see how to dedup that, any ideas are appreciated otherwise I'll write something for that.) I forgot to add, we used to use a perl script called dupseek to clean these up. It has some nice optimization that make it quite fast. http://freshmeat.net/projects/dupseek/ -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Strange bug in cyrus (missing folders with reconstruct !)
On Thursday 17 December 2009 @ 08:28, Denis BUCHER wrote: I think I copied with scp. Does the CLIENT folder have at least a cyrus.index file in it? No, the CLIENT folder has only subfolders... Normally reconstruct doesn't recurse through directories that aren't mailboxes unless you use the -p partition option. If you don't have a partition defined, try -p default. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Strange bug in cyrus (missing folders with reconstruct !)
On Thursday 17 December 2009 @ 09:14, Denis BUCHER wrote: Hello Brian, Brian Awood a écrit : I think I copied with scp. Does the CLIENT folder have at least a cyrus.index file in it? No, the CLIENT folder has only subfolders... Normally reconstruct doesn't recurse through directories that aren't mailboxes unless you use the -p partition option. If you don't have a partition defined, try -p default. This looks intersting, but this is what I get : $ /usr/sbin/cyrreconstruct -p default -rf user.psmith.CLIENTS user.psmith.CLIENTS does not appear to be a mailbox (no /var/spool/cyrus/mail/c/user/psmith/CLIENTS/cyrus.header). Sorry about that, my recollection of -p functionality was incorrect. It will add mailboxes whose parent directory isn't a mailbox, but you still have to specify the path to the mailbox. So for example if you know /var/spool/cyrus/mail/c/user/psmith/CLIENTS/submailboxname is valid, then; /usr/sbin/cyrreconstruct -p default user/psmith/CLIENTS/submailboxname should add submailboxname even if CLIENTS is just a directory without cyrus metadata. The -rf functionality is slightly different because it needs a valid mailbox to start from. If you have a lot of these cases, you might be able to automate this by running ctl_mboxlist -v and cutting out the mailboxes that are listed as being on the filesystem but not in the database. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Strange bug in cyrus (missing folders with reconstruct !)
On Thursday 17 December 2009 @ 18:38, Denis BUCHER wrote: Hello Brian, Correct ! It worked (I tried with a folder) The -rf functionality is slightly different because it needs a valid mailbox to start from. If you have a lot of these cases, you might be able to automate this by running ctl_mboxlist -v and cutting out the mailboxes that are listed as being on the filesystem but not in the database. Do you mean something like /usr/sbin/ctl_mboxlist -d -x -p default ? I didn't find how to list folder present in system but not it database... Denis No, at least in cyrus 2.3 just /usr/sbin/ctl_mboxlist -v will scan the entire filesystem and look for mailboxes that are not in the mailbox database or vis-versa. eg. # /usr/sbin/ctl_mboxlist -v 'user.ddjdjdj.498' has a directory '/var/spool/imap/L/user/ddjdjdj/498' but no DB entry depending on how many mailboxes you have, it may take quite a while to run. But you should be able to take that output and use your favorite text processing utility to take the first field and pass it to reconstruct. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Question about 'flagged' attribute
This was just discussed on the list, check the thread ANNOTATEMORE = METADATA and rfc 5464 in the archive. -Brian On Monday 23 November 2009 @ 04:17, Julien Vehent wrote: Hello Cyrus guys, I was wondering if there were any undergoing work to extend the flagged attribute of IMAP into something more configurable ? I am thinking of some sort of labelling similar to what is implemented on gmail, for example, but also with the wirtual folders in Outlook and extended tagging in Thunderbird. AFAIK, so far these features have been implemented in the email clients. But because I consult my emails with, at least, 4 different clients (cellphone on windows mobile, laptop with kmail, webmail with roundcube, other laptop with thunderbird), I would like to have the categories (flags, labels, whatever) an email belongs to stored inside the IMAP server. I'm not really aware of the work going on around IMAP, I figured you guys might know a little more about this, and maybe be working on something specific to cyrus-imap? Thanks, Julien Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: setting up replication
On Tuesday 17 November 2009 @ 13:11, Patrick Boutilier wrote: On 11/17/2009 01:34 PM, Johannes Rußek wrote: Nope, i'm not. is this necessary? it's 2.3.9 on the master and 2.3.7 on the replica. if that's the problem i will take care of that first. They are both 2.3.x so that shouldn't be a problem. However those versions are relatively old. 2.3.15 is the latest in the 2.3 series. I just setup replication a couple of weeks ago and haven't seen any of those errors. I believe there were major changes in the sync protocol between 2.3.7 2.3.9, enough to probably make them incompatible. Although, I would also recommend running 2.3.15 over anything older. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Murder confusion -- two mupdate slaves, lmtpproxyd's always connecting to master
You can avoid the behavior in your second question by setting
mupdate_config: unified
in your proxies imapd.conf. That was a long standing issue with
murder in a large scale environment with cyrus 2.2.
On Tuesday 10 November 2009 @ 11:51, Michael Bacon wrote:
The second one is that the code for lmtpproxyd very explicitly
connects to the mupdate master rather than the local slave. I
can't really figure out why it would do this, but here are the two
relevant snippets from lmtpd.c:
This is from service_init:
if (config_mupdate_server
(config_mupdate_config ==
IMAP_ENUM_MUPDATE_CONFIG_STANDARD)
!config_getstring(IMAPOPT_PROXYSERVERS)) {
/* proxy only -- talk directly to mupdate master */
r = mupdate_connect(config_mupdate_server, NULL, mhandle,
NULL); if (r) {
syslog(LOG_ERR, couldn't connect to MUPDATE server %s:
%s, config_mupdate_server, error_message(r)); fatal(error
connecting with MUPDATE server, EC_TEMPFAIL); }
}
and it appears that this runs every time a message delivery is
attempted:
static int mlookup(const char *name, char **server, char **aclp,
void *tid) {
int r, type;
if (server) *server = NULL;
if (mhandle) {
/* proxy only, so check the mupdate master */
struct mupdate_mailboxdata *mailboxdata;
/* find what server we're sending this to */
r = mupdate_find(mhandle, name, mailboxdata);
In short, it looks like, unlike proxyd and pop3proxyd, lmtpproxyd
never even bothers to check the local mailboxes database, and when
it wants an answer, feels the need to go directly to the mupdate
master, rather than querying the handy dandy local slave. Is this
intentional? Why can't it use the local cache?
Thanks much,
Michael Bacon
UNC Chapel Hill
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Murder confusion -- two mupdate slaves, lmtpproxyd's always connecting to master
On Tuesday 10 November 2009 @ 12:37, Michael Bacon wrote: I saw this recommendation elsewhere, but I don't understand why one would set mupdate_config: unified if I'm not, in fact, running a unified murder. What's the gain here? We have dedicated front-end boxes and dedicated back-end boxes. Is this just sort of a, yeah, it's not for that, but it does what you want anyway kind of things? Unfortunately it's not well documented, but the unified murder config currently only works on a proxy host. Don't try to configure it on a backend machine that has local mailboxes!!! Unless you want to manually fix up the mailboxes.db and mupdate master. :) There are a few reports in the bugzilla from people who have tried to do that and mucked things up. So to answer your question, basically yes. The gain is that using it on a proxy gives you the behavior you want, which is lmtpd checks it's local db copy first instead of asking mupdate on every delivery. It seems to me like the behavior anyone running murder would want, but for some reason that isn't apparently true. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: painful mupdate syncs between front-ends and database server
On Monday 19 October 2009 @ 16:38, Michael Bacon wrote: I say mostly because while most of the times the thing handles our 80,000 users and 14,000+ simultaneous connections like a champ, some of the time, we get some extreme pain, mostly due to syncs between the MUPDATE master and the front-end servers. We have similar usage levels, but don't have any issues with proxies and MUPDATE. What is the load on your mupdate master like? When a proxy mupdate first connects it should get a dump of the database then sit and wait for any changes to be sent to it. If the load on your mupdate master is high, have you tried setting; mupdate_config: unified on your proxies? I suppose this is Fastmail and others ripped out the proxyd's and replaced them with nginx or perdition. Currently we still support GSSAPI as an auth mechanism, which kept me from going that direction, but given the problems we're seeing, I'd be open to architectural suggestions on either how to tie perdition or nginx to the MUPDATE master (because we don't have the back-ends split along any discernable lines at this point), or suggestions on how to make the master-to-frontend propagation faster or less painful. For comparison, we also support GSSAPI. We have 3 IPs behind our mail hostname, each IP is a hardware load balancer with 2 machines behind it. The system load on each of the 6 machines is generally not much more than 1. Before the load balancers, when we just used a muti-A record we would tend to have some proxyd related load issues because mail clients tend to prefer the lowest IP in a set. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Implement Cyrus IMAPD in High Load Enviromment
On Tuesday 29 September 2009 @ 06:59, Bernd Petrovitsch wrote: On Mon, 2009-09-28 at 15:33 -0700, Vincent Fox wrote: [...] Really I've looked at fsck too many times in my life and don't ever want to again. Anyone who tells me oh yes but Especially not in the 100GB area. We haven't looked at ZFS, though as Bron suggested, I doubt it will solve all filesystem issues. We use ext3 on large partitions, ranging from 2-5TB. While it takes 14-18hrs to fsck, that doesn't really matter if you have replication, we can promote a replica to a primary in about 15minutes. How much performance do you gain and what are the risks? So - in a larger environment - buying a few disks more and/or faster disks and/or battery-backed controllers and more RAM usually outweighs the risk of losing reputation and (commercial) customers. The next question is: Why do I - as the techie/admin/.. - win by saving a few 100€ (or 2.000€) on the hardware (and how much is the total hardware cost?) for *my* decision to use $BRAND_NEW_FAST_FS instead of ext3 and what can I loose (like personal reputation or some sleepless nights and killed weekends in the future)? Does anyone has scripts/tools to - at least - simulate 1000s of (semi-realistic) parallel IMAP clients on a big setup? With cyrus, murder+replication+comodity hardware is the least expensive and probably most scalable way to go. We have some perl scripts for imap load testing, you can find links to them at the bottom of this page, http://blackops.mail.umich.edu/cyrus along with other info on our cyrus implementation. Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Implement Cyrus IMAPD in High Load Enviromment
On Tuesday 29 September 2009 @ 18:41, Bron Gondwana wrote: Possibly the secret is that we use IPAddr2 from linux-ha to force ARP flushes, and we transfer the primary IP address between machines, so nothing else needs to know - we just shut down one end and bring up the other with the IP and it's all good. Our primaries and replicas are located in different data centers, and since we have not control over how the network is subdivided it's impossible for them to take the same IPs. Our process is: a) check there are less than 10kb of files in $conf/sync/ - else abort b) shut down the master (host A) c) run sync_client -f $file on each file in $conf/sync (if any) c2) (if any sync fails, restart the master (host A)) d) shut down the replica (host B) e) update the database with the new master location f) start up the replica (host A) g) start up the master (host B) This means replication starts immediately, because the replica is already there when the master starts. So you just immediately start replicating back to a host (or site) that just failed? How does that work? We have a third level of machines that we sync to, in an out of band process, but the data is stored exactly the same way so we can start replicating to them immediately. So even if a entire data center failed, we can still be running a fully replicated service with almost no downtime visible to users. Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: upgrading a 2.2.12 murder to 2.3.14
On Thursday 16 July 2009 @ 11:05, Nic Bernstein wrote: Brian, I certainly would be interested in seeing those. Please post them here, or to the Wiki. Best regards, -nic Our email operations group manager put together a summary of our cyrus implementation recently for a meeting we have yearly with several other Universities. It has links to the scripts we use, the caveat being they are all written for our environment and likely wouldn't run at another site the way they are currently. However, they probably wouldn't be too difficult to port. http://blackops.mail.umich.edu/cyrus -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: upgrading a 2.2.12 murder to 2.3.14
On Thursday 16 July 2009 @ 05:21, Gavin Gray wrote: 2. We should end up then with our existing murder but with three backends running 2.3.14. We then plan to upgrade the other machines in the murder to 2.3.14 in the following order: frontends then lmtp and finally the mupdate server. Does this make sense? This is the order we did our 2.2-2.3 migration in, although technically it shouldn't make much difference. We ran a mixed environment for quite a while (2.2 frontends and 2.3 backends) because we wanted to enable unified murder on the 2.3 frontends so they wouldn't ask mupdate for a mailbox location on every connection. We had a separate patch that gave us this behavior in 2.2. 3. As part of our preparation for this work we have been experimenting with cyrus replication. The replication protocol seems pretty solid, however we have some concerns about how to make use of it in our upgrade. We are considering having a replicant machine for each of the new backends. But this makes the migration even slower. In our tests, if we migrate users via xfer to a machine that is doing rolling replication, the replication takes around three times as long to complete as the xfer. Does anyone have any experience of migrating to a replicating environment? replication is significantly slower than xfer, but it shouldn't affect the speed of xfer. We wrote some scripts to prevent xfer from getting too far ahead of replication. Since our replicas are a key part of our DR/backup strategy, we never would want to get in a situation where a large amount of data wasn't replicated. In your situation it shouldn't matter since the data is not replicated currently, you should just be able to enable rolling replication and let it catch up. If you still would like to keep xfer from getting too far ahead of replication, I can probably post the scripts we use for this. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: upgrading a 2.2.12 murder to 2.3.14
On Thursday 16 July 2009 @ 09:53, Rudy Gevaert wrote: I wonder if it is possible to run sync to two different servers. Anyone? AFAIK, Not easily. We considered trying to do this when we moved from tape backup to disk based backup. We decided on an out of band process to replicate to our third level machines, basically a script that runs rsync on a regular basis. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: unexpunge Bus Error (signal 7) [Was: Re: ipurge and delayed expunge]
On Friday 26 June 2009 @ 20:47, Adam Tauno Williams wrote: A reconstruct works, but then none of the expunged messages appear (list from unexpunge -l is empty). Should a reconstruct loose expunged messages in delayed expunge mode? reconstruct will remove the expunged data if you don't specify the -k option, also if it's not able to verify the cyrus.expunge file as valid, it will delete it and put all messages back into the index. I posted a patch to the dev list which addresses this and a couple of other issues in reconstruct. In this case it looks like the cyrus.expunge file was corrupted since the expunge_index_base isn't valid. Possibly due to a previous issue with the expunge file. -Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: ipurge and delayed expunge
This issue is fixed in CVS; http://lists.andrew.cmu.edu/pipermail/info-cyrus/2009-March/030753.html -Brian On Friday 26 June 2009 @ 11:23, Adam Tauno Williams wrote: We use delayed expunge with a delay of 120 days. I was just playing with ipurge and it appears to 'physically' delete the messages rather than just marking messages as expunged. Is it possible to have ipurge just mark messages as expunged rather than removing them? Looking at the man page I assume not. $ sudo -u cyrus /usr/lib/cyrus-imapd/ipurge -s -X -f -d 365 user.adam.sent-mail Working on user.adam.sent-mail... total messages 713 total bytes7683241 Deleted messages 0 Deleted bytes 0 Remaining messages 713 Remaining bytes7683241 $ ls /var/spool/imap/a/user/adam/SPAM/ | wc -l 1703 $ sudo -u cyrus /usr/lib/cyrus-imapd/ipurge -s -X -f -d 5 user.adam.SPAM Working on user.adam.SPAM... total messages 205 total bytes6493756 Deleted messages 130 Deleted bytes 4143062 Remaining messages 75 Remaining bytes2350694 $ ls /var/spool/imap/a/user/adam/SPAM/ | wc -l 1573 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.htmlThis issue is fixed in CVS; On Friday 26 June 2009 @ 11:23, Adam Tauno Williams wrote: We use delayed expunge with a delay of 120 days. I was just playing with ipurge and it appears to 'physically' delete the messages rather than just marking messages as expunged. Is it possible to have ipurge just mark messages as expunged rather than removing them? Looking at the man page I assume not. $ sudo -u cyrus /usr/lib/cyrus-imapd/ipurge -s -X -f -d 365 user.adam.sent-mail Working on user.adam.sent-mail... total messages 713 total bytes7683241 Deleted messages 0 Deleted bytes 0 Remaining messages 713 Remaining bytes7683241 $ ls /var/spool/imap/a/user/adam/SPAM/ | wc -l 1703 $ sudo -u cyrus /usr/lib/cyrus-imapd/ipurge -s -X -f -d 5 user.adam.SPAM Working on user.adam.SPAM... total messages 205 total bytes6493756 Deleted messages 130 Deleted bytes 4143062 Remaining messages 75 Remaining bytes2350694 $ ls /var/spool/imap/a/user/adam/SPAM/ | wc -l 1573 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus APIs ?
On Thursday 28 May 2009 @ 18:27, Thomas Cataldo wrote: Hi, We are building webmail groupware software using cyrus for the mail storage part. I'm wondering if any programming interface existed to extend cyrus parts ? Interesting things for us would be : - extending sieve (for exemple to implement in my organisation / out of my organisation vacation messages) - IMAP protocol extensions (most needed thing would be to idle on every folders, not just inbox) - custom authentification mechanism (for single sign-on purpose, because kerberos doesn't fit everywhere) We use Kerberos and single sign-on for webmail. Have you looked at CoSign? http://weblogin.org/ Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: finding by message id
On Tuesday 05 May 2009 @ 09:45, Gabriele Bulfon wrote: Hello, I've got a message logged by posfix with a specific message id, and stated as correctly sent to cyrus. But the user of the imap mailbox is sure that he never received that message. No filter is setup to move the message elsewhere, and no error is visible anywhere. So, I've got the message id from the postfix log. I know that cyrus checks for duplicate ids, so he's got quick knoweledge of every id in the imap folders. How can I ask cyrus for an ID and see if it actually exists somewhere? And how to find it without having to scan all imap folders? Thanx for any help. You can try cyr_dbtool, whether the message ID is still listed or not will depend on how long ago it was delivered and the time period used for clearing the duplicate database. Usually cyr_expire -E somevalue is run on a regular basis to prune old entries from the database. The message IDs are written to the file deliver.db, the location and format depends on your configuration. For example if it's in /var/imap and a berkeley-nosync database you could run; cyr_dbtool /var/imap/deliver.db berkeley-nosync show |grep MSGID Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Bug? xfermailbox seems to be broken but using rename to xfer a mailbox works just fine.
On Thursday 30 April 2009 @ 03:55, Janne Peltonen wrote: Hi! Am I doing something wrong? If I try to do an xfer from one backend to another on a user, like so, xfer user.atest001r m2v3t.mappi.helsinki.fi I get Apr 30 10:24:45 m2cn1t m2v1t/imap[3793]: could not dump mailbox in m2v2t.mappi.helsinki.fi (unknown error) Apr 30 10:24:45 m2cn1t m2v1t/imap[3793]: Could not move mailbox: user.atest001r, dump_mailbox() failed in my murder frontend log, the command (in cyradm) never returns, and the mailbox list on the frontend ends gets corrupted, like this: user.atest001r 1 m2v1t.mappi.helsinki.fi!m2v2t.mappi.helsinki.fi atest001r lrswipkxtecda anyone p (m2v1t.mappi.helsinki.fi is the murder frontend, the murder backends are m2v2t.mappi.helsinki.fi and m2v3t.mappi.helsinki.fi, and m2v2t was the original backend on atest001r). It looks like error was logged from your frontend host m2v1t, so I would guess that the xfer command was issued to it, which isn't correct. You have to connect to the backend where the mailbox is stored and give the xfer command there. But the fact that your frontend database gets corrupted like and that you didn't get a better error (like mailbox not local), definitely seems like a bug. What is you mupdate_config set to on your frontend? Brian Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
