Removing orphaned mailboxes from MUPDATE server
I ran ctl_mboxlist -maw on one of my back end IMAP servers this morning and saw this output: Remove from MUPDATE: user.xx.root.incron Remove from MUPDATE: user.xx.Trash.HCWG.Minutes Remove from MUPDATE: user.xx.freshers-week.fwem.FW09.Rep emails They do not show up if I search for them in cyradm but I found them lurking on the MUPDATE server: $ ctl_mboxlist -d | grep 3 user.xx.Trash.HCWG.Minutes 3 imap-backend.bath.ac.uk!00 user.xx.root.incron 3 imap-backend.bath.ac.uk!04 user.xx.freshers-week.fwem.FW09.Rep emails 3 imap-backend.bath.ac.uk!04 Running ctl_mboxlist -ma on the back end doesn't clear these up. How did these mailbox entries get into state and what is the safest downtime-free way to remove them from the MUPDATE server? ctl_mboxlist doesn't have an advertised way of doing this. The front end is 2.4.16 and the back end is 2.3.13. Many thanks, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
admins access from Cyrus 2.4 proxy
We have started the process of upgrading our Cyrus IMAP service from 2.3
to 2.4. We upgraded the proxy IMAP server earlier this week and
subsequently noticed that quotas can no longer be viewed or set from the
proxy IMAP server.
Cyrus 2.3 front end behaviour:
== cyradm ==
proxy lq user.test
STORAGE 121/25 (0.0484%)
proxy
== Proxy ==
C: 5 GETQUOTA user.test
S: 5 NO [REFERRAL imap://;AUTH=*@backend.bath.ac.uk/user.test] Remote
mailbox.
== Backend ==
C: 5 GETQUOTA user.test
S: * QUOTA user.test (STORAGE 121 25)
Cyrus 2.4 front end behaviour:
== cyradm ==
proxy lq user.test
proxy
== Proxy ==
C: 4 GETQUOTA user.test
S: 4 NO Permission denied
== Backend ==
C: 4 Getquota {11+}
user.test
S: 4 NO Permission denied
Direct to Cyrus 2.3 backend using the same admin user:
== cyradm ==
backend getacl user.test
STORAGE 121/25 (0.0484%)
backend
== Backend ==
C: 1 Getquota {11+}
user.test
S: * QUOTA user.test (STORAGE 121 25)
1 OK Completed
Server details: 2x Solaris 10 x86 machines, proxy server 2.4.16, backend
server 2.3.13.
proxyd_disable_mailbox_referrals is set to true on imapd.conf on the
proxy server for both versions.
It's good to see that an IMAP referral isn't being generated by the
front end server - we use Kerberos authentication so the login was
happening automatically anyway - but is it expected behaviour that the
admin user be denied admin rights when logging in from a proxy server?
Or should I raise a bug about this?
Thanks,
Dave.
David Mayo
Networks/Systems Administrator
University of Bath Computing Services, UK
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Upgrading Murder from 2.3 to 2.4
We are looking at upgrading from Cyrus IMAP 2.3 to 2.4. We run a Murder with one front end server and one back end server. We plan to XFER mailboxes from the old back end to a new back end server with more storage. From the mailing list I understand we should upgrade the front end server first before transferring mailboxes. My planned changes for the front end server are to add the suppress_capabilities directive to imapd.conf, ensure the lock directory is created on tmpfs and touch a blank user_deny.db. My expectation is that I can stop the front end server, upgrade the software and restart it with the updated imapd.conf. As far as the front end upgrade is concerned, is there really any difference between the behaviour of 2.3 vs 2.4? Most of the changes appear to be focused on the mailbox handling code. Does anyone running a Murder have experience of the upgrade from 2.3 to 2.4 they could share? Will users notice any difference at this stage? Thanks, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Can't see messages, reconstruct doesn't work
Eric, On 19/04/11 11:01, Eric Knudstrup wrote: On 04/19/2011 02:43 AM, Patrick Boutilier wrote: On 04/19/2011 06:34 AM, Eric Knudstrup wrote: I have a problem in which there are messages in my main INBOX directory, but they aren't seen in my IMAP client. Unfortunately, when I tell it to reconstruct through cyradm, I see a pair of .NEW files created, but they're empty and the reconstruct returns? This is with 2.4.7. Are the .NEW files still there after reconstruct ends? If so it sounds like reconstruct is crashing. Anything in your logs files? Nothing. I just tried deleting the cyrus.* files and that works. I was replicating this installation to another (new) 2.4.7 installation, starting about a week ago. Are the messages marked as deleted? Try checking with unexpunge -l mailbox - if you deleted cyrus.expunge and reconstructed, the messages would reappear in the mailbox. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Renaming top level/sub mailboxes
When I try to move a top level mailbox to be a sub mailbox, or vice versa, the IMAP server returns: NO Operation is not supported on mailbox eg. renm user.toplevel user.aa.toplevel renm user.aa.toplevel user.toplevel Why is this? Can I get around it other than creating a new mailbox and copying the messages over (which loses the \Seen state). We are running Cyrus 2.3.13. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Renaming top level/sub mailboxes
Simon, On 25/01/11 15:40, Simon Matter wrote: When I try to move a top level mailbox to be a sub mailbox, or vice versa, the IMAP server returns: NO Operation is not supported on mailbox eg. renm user.toplevel user.aa.toplevel renm user.aa.toplevel user.toplevel Why is this? Can I get around it other than creating a new mailbox and copying the messages over (which loses the \Seen state). Is 'allowusermoves' enabled in imapd.conf? Yes. I can rename top level mailboxes to other top level mailboxes, or sub mailboxes to other sub mailboxes - it's just renaming between the two types that fails. Regards, Dave Networks/Systems Administrator, BUCS Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Removing a mailbox with a very long name
Bron, On 24/11/10 23:01, Bron Gondwana wrote: On Wed, Nov 24, 2010 at 04:13:11PM +0100, Michael Menge wrote: We're running Cyrus 2.3.13. Any reason why so old? There were some fixes for this shipped more in the later 2.3 series releases, because we had the same issue at FastMail! 2.3.14 had bugs (didn't work with group: ACLs) and 2.3.15 wouldn't compile happily on Solaris due to issues with compression libraries, as I recall. By then, the servers were live so we stayed at that version. If we did upgrade from 2.3.13 - 2.3.16 is there anything that would prevent us from downgrading again if we hit problems? The changelog does show quite a few improvements! Do you use delete_mode: delayed ? this will rename the folder to $DELETEPREFIX.$OLDNAME.$TIMESTAMP which would result in a longer Name. You can try to rename it to a shorter name and deleting the renamed folder? That's what I was going to say! That's how we wound up dealing with the worst offenders. cyradm wouldn't let me even type a name that long! Perhaps I could have done it via a telnet session, but after DELETE returned Invalid mailbox name I kinda assumed nothing would work. If you live anywhere near the Apple campus you could go over and have a few choice words with thier mail client developers too... FYI: more recent Cyrus rejects the creation of any mailbox with INBOX.INBOX (any case) in the internal name (so user.foo.Inbox is still fine, but user.foo.Inbox.Inbox is not) Yes - this would be handy. An other ideas is to remove the mailboxname from the mailbox.db and deleting it on the filesystem. Use on your own risk cyr_dbtool $configdirectory/mailboxes.db skiplist delete user.abc20.INBOX.. Yep! That will definitely work. That's what I did in the end, and it worked a treat! Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Removing a mailbox with a very long name
Can someone suggest the best way of deleting a mailbox that has gotten too long? This one is stopping our daily quota fix command from running: Nov 24 06:40:07 imap.bath.ac.uk quota[12707]: [ID 240394 mail.error] IOERROR: opening quota file /opt/etc/imapd/quota/a/user.abc20.INBOX.INBOX.INBOX.INBOX.INBOX.toINBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.Deleted Messages: File name too long Nov 24 06:40:07 imap.bath.ac.uk quota[12707]: [ID 335833 mail.error] DBERROR: error fetching user.abc20.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.Deleted Messages: cyrusdb error Nov 24 06:40:07 imap.bath.ac.uk quota[12707]: [ID 857756 mail.notice] skiplist: unlock while not locked Nov 24 06:40:07 imap.bath.ac.uk quota[12707]: [ID 809228 mail.error] failed building quota list for '*': System I/O error: Bad file number I have managed to delete all the parent folders, but not this one. cyradm won't even let me enter a mailbox name this long, but I can use a * which LISTs and DELETEs all the mailboxes apart from this one. cyradm output: imap.bath.ac.uk dm user.abc20.INBOX.* Deleting mailbox user.abc20.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.Deleted Messages...Invalid mailbox name And IMAP telemetry log: 12906057089 RLIST user.abc20.INBOX.* 1290605708* LIST (\HasNoChildren) . user.abc20.INBOX.INBOX.INBOX.INBOX.INBO X.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX. INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.IN BOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBO X.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX. INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.IN BOX.INBOX.INBOX.INBOX.Deleted Messages 9 OK Completed (0.000 secs 2 calls) 129060570810 DELETE user.abc20.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBO X.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX. INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.IN BOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBO X.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX. INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.IN BOX.Deleted Messages 129060570810 NO Invalid mailbox name My cyrus user has create permissions on the mailbox, so it's not an ACL issue. I have removed the mailbox from the filesystem and run a reconstruct, but this (predictably) just recreates the folder on the filesystem. We're running Cyrus 2.3.13. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: sync_client IOERROR
Mike, On 16/11/10 18:36, Michael D. Sofka wrote: I am seeing occasional messages of the form: Nov 16 13:01:36 imap-be4 sync_client[21977]: IOERROR: index record 0 for user.userid past end of file I have fixed these in the past by reconstructing the mailbox. It most often happened when we were transferring mailboxes from the old Cyrus 2.2 backend, but we've seen it once or twice since. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: lmtpd: deliver.db checkpointing
On 15/11/2010 14:46, Patrick Boutilier wrote: On 11/15/2010 10:36 AM, David Mayo wrote: Having looked at the source code, we can see that ctl_cyrusdb does not touch deliver.db and it is not possible to force checkpoints on or off within lmtpd. Ideally we want to schedule the checkpointing to a less sociable time. On our setup, the checkpoint runs every couple of days and usually takes 4-6 minutes[1]. We run cyr_expire every night to remove entries older than 3 days from the duplicate deliveries database but this is the only maintenance we can schedule on this database. No solution, but I wonder why your checkpoint takes so long? With similar DB size it take only seconds here. Nov 8 21:18:04 student2 lmtpunix[6541]: skiplist: checkpointed /var/imap/deliver.db (650493 records, 67180160 bytes) in 5 seconds Nov 12 23:14:06 student2 lmtpunix[26425]: skiplist: checkpointed /var/imap/deliver.db (768262 records, 80010936 bytes) in 11 seconds Looking back further through our logs, I found a checkpoint that was run overnight which completes somewhat faster, but nowhere near your times! Our IMAP back-end is running on a 4x Quad-Core AMD Opteron 2.3 GHz processor with 16 GB RAM: Oct 14 00:17:47 imap.bath.ac.uk lmtp[29924]: [ID 301543 mail.info] skiplist: checkpointed /opt/etc/imapd/deliver.db (715978 records, 68761076 bytes) in 76 seconds This presumably doesn't cause anyone much trouble overnight, but it's a bit more noticeable during the day. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Do other people suffer from this? Is there a patch that anyone has written and/or will be submitting upstream any time soon? We are running Cyrus 2.3.13 across a front end, a back end and a replication host for around 25,000 users. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK [1] Nov 4 15:32:33 imap.bath.ac.uk lmtp[29595]: [ID 301543 mail.info] skiplist: checkpointed /opt/etc/imapd/deliver.db (787618 records, 74775536 bytes) in 386 seconds Nov 9 14:26:06 imap.bath.ac.uk lmtp[1676]: [ID 301543 mail.info] skiplist: checkpointed /opt/etc/imapd/deliver.db (386167 records, 36638408 bytes) in 232 seconds Nov 11 12:45:35 imap.bath.ac.uk lmtp[2497]: [ID 301543 mail.info] skiplist: checkpointed /opt/etc/imapd/deliver.db (657254 records, 62540568 bytes) in 396 seconds Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
lmtpd: deliver.db checkpointing
One day last week I noticed a performance dip on our IMAP server. We tracked the cause down to periodic checkpointing of the deliveries database by the LMTP daemon on our backend server. The only real effect on users is that deliveries don't happen for the duration of the checkpointing cycle, although on this occasion I noticed the IMAP server was running unusually slow during this period and I also discovered the MUPDATE server was unavailable shortly afterwards for creating new mailboxes. This is presumably because the MUPDATE server was busy in the backed-up delivery process. Having looked at the source code, we can see that ctl_cyrusdb does not touch deliver.db and it is not possible to force checkpoints on or off within lmtpd. Ideally we want to schedule the checkpointing to a less sociable time. On our setup, the checkpoint runs every couple of days and usually takes 4-6 minutes[1]. We run cyr_expire every night to remove entries older than 3 days from the duplicate deliveries database but this is the only maintenance we can schedule on this database. Do other people suffer from this? Is there a patch that anyone has written and/or will be submitting upstream any time soon? We are running Cyrus 2.3.13 across a front end, a back end and a replication host for around 25,000 users. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK [1] Nov 4 15:32:33 imap.bath.ac.uk lmtp[29595]: [ID 301543 mail.info] skiplist: checkpointed /opt/etc/imapd/deliver.db (787618 records, 74775536 bytes) in 386 seconds Nov 9 14:26:06 imap.bath.ac.uk lmtp[1676]: [ID 301543 mail.info] skiplist: checkpointed /opt/etc/imapd/deliver.db (386167 records, 36638408 bytes) in 232 seconds Nov 11 12:45:35 imap.bath.ac.uk lmtp[2497]: [ID 301543 mail.info] skiplist: checkpointed /opt/etc/imapd/deliver.db (657254 records, 62540568 bytes) in 396 seconds Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Stuck mailboxes.db entries with mbtype=2
Hi Michael, On 24/08/10 16:21, Michael Bacon wrote: Do to an error I made in migrating a file system during some system work, we ended up with our configdirectory with permissions that the cyrus user couldn't write to on a few of our back-end servers. Amazingly, we were about 90% functional during this time, but several mailboxes that got created during that time ended up with some decidedly messed-up characteristics. Most we've been able to fix with reconstruct, but we're stuck with a few hundred mailboxes where the backend created the mailbox on disk, registered it with the MUPDATE server, but left it in reserved state in the local (backend) mailboxes.db with mbtype=2. This means that it shows up in a LIST or LSUB with the \NoSelect flag, and the users can't do anything with it, including delete it. You should be able to correct this by running ctl_mboxlist -ma on the back-end server. I would advise using the -w switch at first to see what it would do. If the mailboxes are missing on the filesystem, you can fix that by creating the path to the mailbox on the filesystem and running the reconstruct command: mkdir `mbpath user.brokenmailbox`; reconstruct user.brokenmailbox Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus backend crashing (Solaris)
On 20/07/10 17:13, Gary Mills wrote: When trying to diagnose the issue, on any attempt to run ps, prstat or to HUP the syslogd process (to set the log level for imapd to debug) the command hangs and cannot be exited with Ctrl+C. Similarly, attempts to kill the master process or shut down the system (even bypassing the shutdown scripts by using reboot) do not have any effect This happened to me about a year ago with Cyrus on a Solaris 10 server. The cause was a deadlock in one of the kernel ZFS modules. The problem is fixed in the current Solaris patches. Thanks, this feels the most likely cause to me. I'll schedule some upgrade time! To answer the others who replied, we're not running out of swap space or using Berkeley DBs at all. I hadn't considered hardware problems, but if the issues persist after the upgrade we'll have to involve Sun^H^H^HOracle who own the hardware anyway so they will know what tests to run to investigate this option further. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus backend crashing (Solaris)
We are running a Cyrus Murder with one proxy/MUPDATE master and one backend server which uses Cyrus Replication to replicate onto another host. All hosts are running Cyrus imapd 2.3.13 on Solaris 10 Update 7. Since moving all our mailboxes to the backend server in March we have seen 4 crashes on the backend server where it has refused to accept new LMTP connections[1] and, although the logs show it is accepting IMAP connections, no clients can get any response to the IMAP server. There don't appear to be any suspicious logs leading upto the event on the proxy server and nothing at all on the backend server. When trying to diagnose the issue, on any attempt to run ps, prstat or to HUP the syslogd process (to set the log level for imapd to debug) the command hangs and cannot be exited with Ctrl+C. Similarly, attempts to kill the master process or shut down the system (even bypassing the shutdown scripts by using reboot) do not have any effect other than hanging the shell in which the commands were issued. New shells can be opened and certain commands run, but we aren't much closer to knowing precisely what is wrong. The only way to bring the system back is to reset it via the on board console. My suspicion is somehow the behaviour in Cyrus is tickling a Solaris bug, but I wanted to check with other Cyrus admins to see if they have seen similar behaviour and had tracked it down to anything in particular. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK [1] imapd.log: Jul 8 11:04:51 tyrrell-z1.bath.ac.uk lmtpunix[984]: [ID 130975 mail.error] connect(sauber.bath.ac.uk) failed: Connection timed out exim log: 2010-07-08 11:04:51 1OWnyD-Fo-Ic == xx...@bath.ac.uk R=imap T=cyrus_lmtp defer (-46): LMTP error after end of data: 451 4.4.3 Remote server unavailable Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Building an interface to undelete for users
We have recently upgraded to Cyrus 2.3 and are making full use of the delayed delete feature, and we are considering writing an interface to allow users to undelete their own messages and mailboxes. Before I start work on this myself, I thought I'd check with people here to see if anyone has any tips or code they are willing to share. I hope we will be able to publish the product we create. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Building an interface to unexpunge for users
Bron Gondwana wrote: On Mon, Mar 29, 2010 at 11:28:50AM +0100, David Mayo wrote: We have recently upgraded to Cyrus 2.3 and are making full use of the delayed delete feature, and we are considering writing an interface to allow users to undelete their own messages and mailboxes. Before I start work on this myself, I thought I'd check with people here to see if anyone has any tips or code they are willing to share. I hope we will be able to publish the product we create. Are you planning to use the unexpunge tool behind the scenes for this, or something more hooked into the innards? (I ask because the innards are going through a big overhaul at the moment, but I hope to keep the unexpunge tool working nicely!) I haven't given this much thought as I wanted to check if someone had a working solution or some ideas they wanted to share. My thinking was that we'd have a Perl script on our web site that SSHed into our IMAP server and ran the unexpunge command with appropriate options. Also - do people care about losing their \Seen state? Because maintaining that over unexpunge is possibly viable, but might cost a bit more IO. I would hope the \Seen state isn't that important to people who are recovering emails that got deleted so I wouldn't want to implement it initially. My biggest concern is the interface will be relying on the info from unexpunge which removes all the punctuation and spaces from the fields[1] which will look naff to the users. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK [1] UID: 27636 Size: 3686 Sent: Mon Mar 22 12:00:00 2010 Recv: Mon Mar 22 18:58:33 2010 Expg: Wed Mar 24 08:17:13 2010 From: jonsmith jon.sm...@example.com To : xxx...@bath.ac.uk Cc : josephbloggs x...@bath.ac.uk Bcc : Subj: applicationforteachingfellow-pgce Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Nested INBOX folders - hard to delete
Jukka Huhta wrote: We have several users with folders like this: user.username.INBOX.INBOX.INBOXINBOX.INBOX.INBOX.INBOX.Deleted Messages (total of 76 INBOXes or something). No doubt the folders are generated by a bug in Apple Mail, but how to get rid of them? Users can't do that by themselves, and no MUA I've tried is able to handle the folders either. I wonder what Cyrus thinks of them, probably doesn't like too long names or something. Cyradm is just giving an error from Cyrus: cyradm dm user.username.INBOX.* Deleting mailbox user.username.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.Deleted Messages...Invalid mailbox name I'd really like to fix the problem without shutting any Cyrus instances down and manually dumping, editing and undumping the mailbox list. We have also seen this problem. We fixed it by running: cyradm sam user.username.INBOX.* cyrus c cyradm dm user.username.INBOX.* If you are running delayed deletes for mailboxes, you will need to then delete all the subsequent DELETED mailboxes that get generated. This problem actually generated a few errors[1] and prevented us from running the quota command on the IMAP server[2] until the mailbox was deleted. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK [1] Mar 18 03:54:05 imapserver.bath.ac.uk imap[27395]: [ID 240394 mail.error] IOERROR: opening quota file /opt/etc/imapd/quota/x/user.x.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.Deleted Messages: File name too long Mar 18 03:54:05 imapserver.bath.ac.uk imap[27395]: [ID 335833 mail.error] DBERROR: error fetching user.x.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.INBOX.Deleted Messages: cyrusdb error [2] $ quota failed building quota list for '*': System I/O error: %m Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Corruption to quotas.db (skiplist)
Hi guys, I touched on this in a recent topic about XFER however we have had a few more problems with the quotas database and it is quite worrying. We are transferring our mailboxes from our old Cyrus 2.2 IMAP server to a Cyrus 2.3 IMAP server with replication. We have 25,000 mailboxes totalling around 1.2 TB. Trying to move mailboxes in parallel ended up with serious corruption to quotas.db. We saw lots of these lines: Jan 23 04:06:47 sauber.bath.ac.uk imap[4434]: [ID 602473 mail.error] IOERROR: lock_shared /opt/etc/imapd/quotas.db: Bad file number Eventually resulting in *lots* of these lines: Jan 23 08:10:32 sauber.bath.ac.uk imap[4434]: [ID 362402 mail.error] skiplist: version mismatch: /opt/etc/imapd/quotas.db has version 2.1264205870 Jan 23 08:10:32 sauber.bath.ac.uk imap[4434]: [ID 558109 mail.error] skiplist: closed while still locked Jan 23 08:10:32 sauber.bath.ac.uk imap[4434]: [ID 729713 mail.error] DBERROR: opening /opt/etc/imapd/quotas.db: cyrusdb error Jan 23 08:10:32 sauber.bath.ac.uk imap[4434]: [ID 637875 mail.error] Fatal error: can't read quotas file Users couldn't log in, mail wasn't being delivered and I couldn't even run ctl_mboxlist -d. We had to regenerate the quotas DB from scratch and reconstruct some mailboxes that were suddenly using 3,000% of their quota! Since then we have been moving mailboxes one at a time. We transferred student mailboxes for two nights and this went fine with no errors, but when we transferred some staff mailboxes we started seeing the Bad file number errors again. We ran quota -f and fixed any corrupted quotas, and the Bad file number errors stopped appearing. Is there anything more we can do to protect ourselves from these errors? Is anyone else using skiplist as their quotas.db format? I note that quotalegacy is the default database format which is what our old Cyrus 2.2 IMAP server is using. I wondered whether problems were caused due to staff leaving their PCs switched on overnight however the logs do not show a correlation between quotas that were corrupted and mailboxes that were being checked overnight. Is the skiplist format suitably reliable for this database? It certainly seems to work OK for all the other databases. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Using xfer to migrate mailboxes to a new server
might even be doing everything wrong here. I've googled and it seems like xfer is a common headache for Cyrus admins with no easy solution. However, it also seems to be a particularly bad headache -- most old e-mails I've found about this topic seem to have gone unanswered. I don't claim to be an expert at xfer, since I've only used it to migrate to new hardware once, but hopefully it's helpful anyway. I moved approximately 2000 mailboxes, scripting against the Cyrus::IMAP perl module. About a dozen had some sort of transfer error, and these were fixable with no loss of email. If you're moving from one Cyrus server to another I think it's definitely the best way forward - especially if you're in a Murder environment, as xfer handles all the mailbox reservations too. Unfortunately the xfer code doesn't seem as robust as the rest of the Cyrus distribution so we've had several false starts in transferring our mailboxes. The Cyrus 2.3 xfer code may be better but I haven't tried it. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Building cyrus sasl on solaris 10
Egoitz, Egoitz Aurrekoetxea wrote: I'm trying to build a mail machine box with Postfix (cyrus saslauthd authentication), cyrus sasl (with saslauthd) and cyrus-imap. The main problem I'm finding for the moment is that when building cyrus sasl plugins... only static libraries are created for auth mechs We had to do some fiddling to get Cyrus SASL working under Solaris 10. It involved hacking some of the source files after running configure! ./configure \ --enable-shared \ --disable-static \ --disable-java \ --disable-krb4 \ --with-gss_impl=mit \ --with-rc4 \ --with-dblib=berkeley \ --with-saslauthd=/var/sasl2 --without-pwcheck \ --with-devrandom=/dev/urandom \ --enable-anon \ --enable-cram \ --enable-digest \ --enable-ntlm \ --enable-plain \ --enable-login \ --without-ldap \ --disable-otp \ --disable-ldapdb \ --disable-sql --without-mysql --without-pgsql --without-sqlite \ --enable-gssapi=$KERBEROSDIR \ --with-openssl=$OPENSSLDIR # don't use /usr/include/crypt.h cp saslauthd/saslauthd.h saslauthd/saslauthd.h.orig sed -e 's:^.*HAVE_CRYPT_H.*$:/* */:' saslauthd/saslauthd.h.orig saslauthd/saslauthd.h [ $? -eq 0 ] || exit 1 # fiddle to get correct dynamic linking for plugins.. # haven't found a nice way to propogate the following through # the likes of LD or LDFLAGS..so hardwiring.. cp libtool libtool.orig sed -e s:^\(LD *=.*\)\:\1 $LDFLAGS\: libtool.orig libtool [ $? -eq 0 ] || exit 1 make make install Hope this is of some help. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Delayed delete oddities
David Mayo wrote: We have just started transferring the first mailboxes onto our new Cyrus IMAP 2.3.13 server running on Solaris 10. Delayed expunge seems to work fine but I've noticed a strange phenomenon with delayed delete on mailboxes. We have set up cyrus to delete expunged emails and deleted mailboxes after 7 days: delprune cmd=cyr_expire -E 3 -D 7 -X 7 at=0400 It appears to be deleting emails 7 days after they have been expunged, so this looks fine: ... It's just the first entry I'm concerned by: Dec 11 04:00:00 imap-1.bath.ac.uk cyr_expire[29491]: [ID 956378 mail.notice] Expunged 4662 messages from DELETED.user.usera.BUCS.IT discussion list.4B2123D3 This mailbox was only deleted yesterday: Dec 10 16:37:39 sauber.bath.ac.uk imap[11321]: [ID 630590 mail.notice] Deleted mailbox user.usera.BUCS.IT discussion list The mailbox was moved to the right area of the IMAP tree but all the messages were removed after just one day. The messages in the mailbox were not marked for deletion after being moved. Worse still, it hasn't even kept the messages on the filesystem: imap-1 $ ls -la /mail/02/DELETED/user/usera/BUCS/IT\ discussion\ list/4B2123D3/ total 21441 drwx-- 2 cyruscyrus 12 Dec 11 04:00 . drwx-- 3 cyruscyrus 3 Dec 10 16:37 .. -rw--- 1 cyruscyrus 27366 Feb 17 2009 4861. -rw--- 1 cyruscyrus 3151 May 31 2009 5007. -rw--- 1 cyruscyrus 2543 Jun 15 10:04 5025. -rw--- 1 cyruscyrus 2396 Jun 15 11:45 5026. -rw--- 1 cyruscyrus 2366 Jul 29 09:46 5084. -rw--- 1 cyruscyrus 4 Dec 11 04:00 cyrus.cache -rw--- 1 cyruscyrus536 Dec 10 16:37 cyrus.expunge -rw--- 1 cyruscyrus175 Dec 10 16:37 cyrus.header -rw--- 1 cyruscyrus 96 Dec 11 04:00 cyrus.index -rw--- 1 cyruscyrus10734041 Dec 11 02:00 cyrus.squat (squatter is run at 2am, cyr_expire at 4am, which explains the size of cyrus.squat) There were emails newer and older than those ones there, so I'm not sure why those 5 were kept. Running unexpunge on the mailbox resulted in this: imap-1 $ unexpunge -l DELETED.user.usera.BUCS.IT discussion list.4B2123D3 UID: 4861 Size: 27366 Sent: Tue Feb 17 12:00:00 2009 Recv: Tue Feb 17 10:47:12 2009 Expg: Thu Dec 10 16:37:39 2009 Segmentation Fault (core dumped) [imap-1][/] # pstack /var/tmp/core.unexpunge core '/var/tmp/core.unexpunge' of 516: unexpunge -l DELETED.user.usera.BUCS.IT discussion list.4B2123D3 08058317 list_expunged (8046734, 8134898, 5, fea5) + 186 080599df main (3, 8047a5c, 8047a6c) + 941 08057f84 _start (3, 8047b58, 8047b62, 8047b65, 0, 8047b9a) + 80 Is the deleted mailbox code considered reliable? Should we be pruning these using a different method? I'm happy to supply more information if it would lead to this being better understood and/or fixed. I have been doing more testing on this by deleting some of my own mailboxes. Each time, all the messages are removed from the mailbox and most or all of them are deleted from the filesystem too by cyr_expire the first time it runs after the mailbox has been deleted. I tried setting the 'expire' annotation to '99' on the DELETED hierarchy yesterday but the emails in the mailbox were still all deleted. Tonight I have removed the -D 7 switch from cyr_expire to see if that fixes things. Is anyone else using delayed delete for mailboxes? Does cyr_expire leave the mailboxes untouched after a mailbox has been deleted? Are other people running cyr_expire from within cyrus.conf ? It would be nice to know if other people are seeing this or if it's something unique to our setup. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Tel: +44 1225 38 6046 Email: d.j.m...@bath.ac.uk Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Sieve server dropping connections after NOOP
After installing a security certificate on our new IMAP back end server, our web-based sieve script manager broke. The web script uses the Perl Net::Sieve library to connect to the IMAP server. It will always try to initiate a secure connection if STARTTLS is advertised. After it has successfully negotiated, it issues a NOOP to resynchronise the connection. The server responds to the NOOP by ending the connection[1]. When the server was not advertising STARTTLS, a NOOP was never issued by the client so never saw these problems. Looking further, typing any command that the server doesn't recognise results in the connection being dropped. We have tried this on our production 2.3.13 server and a test server running 2.3.15 (in a non-Murder environment). NOOP itself is listed as part of the MANAGESIEVE protocol so presumably this behaviour is contrary to the RFC. Additionally the server should be returning NO responses to unrecognised commands rather than ending the connection. I can work around our local problems but I wanted to know if this happens to other people and see if the developers can fix this problem. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK [1] debug output from Net::Sieve IMPLEMENTATION Cyrus timsieved (Murder) v2.3.13 SASL PLAIN GSSAPI SIEVE comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy STARTTLS OK STARTTLS\r\n OK Begin TLS negotiation now --- TLS activated here NOOP\r\n IMPLEMENTATION Cyrus timsieved (Murder) v2.3.13 SASL PLAIN GSSAPI SIEVE comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy OK Broken Pipe [2] $ telnet sauber sieve IMPLEMENTATION Cyrus timsieved (Murder) v2.3.13 SASL PLAIN GSSAPI SIEVE comparator-i;ascii-numeric fileinto reject vacation imapflags notify envelope relational regex subaddress copy STARTTLS OK NOTACOMMAND Connection closed by foreign host. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Delayed delete oddities
We have just started transferring the first mailboxes onto our new Cyrus IMAP 2.3.13 server running on Solaris 10. Delayed expunge seems to work fine but I've noticed a strange phenomenon with delayed delete on mailboxes. We have set up cyrus to delete expunged emails and deleted mailboxes after 7 days: delprune cmd=cyr_expire -E 3 -D 7 -X 7 at=0400 It appears to be deleting emails 7 days after they have been expunged, so this looks fine: Dec 11 04:00:00 imap-1.bath.ac.uk cyr_expire[29491]: [ID 956378 mail.notice] Expunged 4662 messages from DELETED.user.usera.BUCS.IT discussion list.4B2123D3 Dec 11 04:00:00 imap-1.bath.ac.uk cyr_expire[29491]: [ID 956378 mail.notice] Expunged 29 messages from user.x Dec 11 04:00:00 imap-1.bath.ac.uk cyr_expire[29491]: [ID 956378 mail.notice] Expunged 63 messages from user.x.Trash Dec 11 04:00:00 imap-1.bath.ac.uk cyr_expire[29491]: [ID 956378 mail.notice] Expunged 2 messages from user.x.aa-lists Dec 11 04:00:00 imap-1.bath.ac.uk cyr_expire[29491]: [ID 956378 mail.notice] Expunged 26 messages from user.x.spam Dec 11 04:00:03 imap-1.bath.ac.uk cyr_expire[29491]: [ID 301543 mail.info] skiplist: checkpointed /opt/etc/imapd/quotas.db (13 records, 816 bytes) in 0 seconds Dec 11 04:00:05 imap-1.bath.ac.uk cyr_expire[29491]: [ID 956378 mail.notice] Expunged 36 messages from user.usera.spam Dec 11 04:00:05 imap-1.bath.ac.uk cyr_expire[29491]: [ID 427756 mail.notice] Expunged 5039 out of 82483 messages from 232 mailboxes Dec 11 04:00:05 imap-1.bath.ac.uk cyr_expire[29491]: [ID 630590 mail.notice] Deleted mailbox DELETED.user.z.Trash.ucu.4B184D30 Dec 11 04:00:05 imap-1.bath.ac.uk cyr_expire[29491]: [ID 857756 mail.notice] skiplist: unlock while not locked Dec 11 04:00:05 imap-1.bath.ac.uk cyr_expire[29491]: [ID 489633 mail.notice] Removed 1 deleted mailbox Dec 11 04:00:05 imap-1.bath.ac.uk cyr_expire[29491]: [ID 730478 mail.notice] duplicate_prune: pruning back 3 days Dec 11 04:00:12 imap-1.bath.ac.uk cyr_expire[29491]: [ID 371192 mail.notice] duplicate_prune: purged 1659 out of 6999 entries It's just the first entry I'm concerned by: Dec 11 04:00:00 imap-1.bath.ac.uk cyr_expire[29491]: [ID 956378 mail.notice] Expunged 4662 messages from DELETED.user.usera.BUCS.IT discussion list.4B2123D3 This mailbox was only deleted yesterday: Dec 10 16:37:39 sauber.bath.ac.uk imap[11321]: [ID 630590 mail.notice] Deleted mailbox user.usera.BUCS.IT discussion list The mailbox was moved to the right area of the IMAP tree but all the messages were removed after just one day. The messages in the mailbox were not marked for deletion after being moved. Worse still, it hasn't even kept the messages on the filesystem: imap-1 $ ls -la /mail/02/DELETED/user/usera/BUCS/IT\ discussion\ list/4B2123D3/ total 21441 drwx-- 2 cyruscyrus 12 Dec 11 04:00 . drwx-- 3 cyruscyrus 3 Dec 10 16:37 .. -rw--- 1 cyruscyrus 27366 Feb 17 2009 4861. -rw--- 1 cyruscyrus 3151 May 31 2009 5007. -rw--- 1 cyruscyrus 2543 Jun 15 10:04 5025. -rw--- 1 cyruscyrus 2396 Jun 15 11:45 5026. -rw--- 1 cyruscyrus 2366 Jul 29 09:46 5084. -rw--- 1 cyruscyrus 4 Dec 11 04:00 cyrus.cache -rw--- 1 cyruscyrus536 Dec 10 16:37 cyrus.expunge -rw--- 1 cyruscyrus175 Dec 10 16:37 cyrus.header -rw--- 1 cyruscyrus 96 Dec 11 04:00 cyrus.index -rw--- 1 cyruscyrus10734041 Dec 11 02:00 cyrus.squat (squatter is run at 2am, cyr_expire at 4am, which explains the size of cyrus.squat) There were emails newer and older than those ones there, so I'm not sure why those 5 were kept. Running unexpunge on the mailbox resulted in this: imap-1 $ unexpunge -l DELETED.user.usera.BUCS.IT discussion list.4B2123D3 UID: 4861 Size: 27366 Sent: Tue Feb 17 12:00:00 2009 Recv: Tue Feb 17 10:47:12 2009 Expg: Thu Dec 10 16:37:39 2009 Segmentation Fault (core dumped) [imap-1][/] # pstack /var/tmp/core.unexpunge core '/var/tmp/core.unexpunge' of 516: unexpunge -l DELETED.user.usera.BUCS.IT discussion list.4B2123D3 08058317 list_expunged (8046734, 8134898, 5, fea5) + 186 080599df main (3, 8047a5c, 8047a6c) + 941 08057f84 _start (3, 8047b58, 8047b62, 8047b65, 0, 8047b9a) + 80 Is the deleted mailbox code considered reliable? Should we be pruning these using a different method? I'm happy to supply more information if it would lead to this being better understood and/or fixed. Regards, Dave Networks/Systems Administrator, BUCS Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: setting up replication
Johannes, Johannes Rußek wrote: i'm trying to replicate an existing imap servers over to a new one, this is basically fedora core 6 to rhel5. i've set up the replication as per the howto and it appears to basically work. however, when sync_client reaches the point of creating a mailbox, i get the following error in the log: sync_client[30485]: CREATE received BAD response: Unexpected extra arguments to Create Do you have the same mailbox partitions set up in imapd.conf on the master and the slave? Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Clients sporadically dropping connections in Murder environment
In August we introduced a Cyrus 2.2 IMAP proxy server in front of our Cyrus 2.2 backend server. This is the first part of the process to move our users' mailboxes to a Cyrus 2.3 backend server. Since the upgrade we have had sporadic reports of users' clients disconnecting from the IMAP server. One user (a fellow sysadmin) using Mulberry 4 reports that occasionally when they send an email, they get a message, The draft has been sent successfully, but an error has occurred whilst doing post-processing. The message is not saved to their IMAP sent-mail folder, as configured in Mulberry. We enabled telemetry logging on the user's account. There was no attempt to APPEND to the folder, and there were no BAD or NO responses during their session. No errors in Cyrus logs (although we're only logging upto 'info' levels on the production service). The proxyd daemon hasn't crashed[1]. Their client checks for new messages every 2 minutes so presumably this keeps the connection open. People using the same client haven't noticed this problem. Pine, Alpine and Evolution users have apparently also reported similar symptoms although I don't have any first-hand reports. No reports of problems from Outlook, Thunderbird or our Horde Webmail client which are our main supported clients. I've looked through the list archives and can't see this being discussed. Has anyone else using a Murder environment seen issues like this? We're running Solaris 10. Our front-end server is 2.2.13. Our back-end server is 2.2.12. The servers use GSSAPI to authenticate to each other and we refresh the KerberosV tickets every hour from a key tab. We aren't running IMAPS on the back-end server. Any thoughts on this gratefully received! Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK [1] proxyd has crashed a couple of times a week on the proxy server but that's a different matter! We will properly investigate it if the problems are still present when we upgrade the proxy server to Cyrus 2.3 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP proxy advertising MAILBOX-REFERRALS and confusing Pine
Andrew Morgan wrote: On Wed, 12 Aug 2009, David Mayo wrote: Does anyone have any suggestions as to how to stop the front-end proxy server from advertising MAILBOX-REFERRALS and/or for Pine to ignore such capabilities. This was originally a patch I obtained from the folks at Portland State University: http://oregonstate.edu/~morgan/cyrus/patches/imapd-disable-referrals.patch It does exactly what you want in cyrus 2.2. Thanks very much for this! We applied this patch this morning and Pine is now behaving itself. On the down-side, our quota scripts using Cyrus::IMAP::Admin have stopped working as they are not being sent MAILBOX-REFERRALS in the CAPABILITY string. This means they are ignoring the referral sent to them by the server when they issue a GETQUOTA. Reading RFC 2193 Section 3, paragraph 1 this is understandable: IMAP4 servers that support this extension MUST list the keyword MAILBOX-REFERRALS in their CAPABILITY response. No client action is needed to invoke the MAILBOX-REFERRALS capability in a server. I feel like we've picked at the thread of a jumper and it's slowly unravelling here! Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
IMAP proxy advertising MAILBOX-REFERRALS and confusing Pine
We undertook the first stage of our IMAP server upgrade yesterday and
introduced a front-end proxy running Cyrus 2.2 in front of our sole
back-end server running Cyrus 2.2.
Since the upgrade, when a Pine user sends an email, the email is sent
fine however when Pine appends the message to the sent-mail mailbox it
asks the user to reauthenticate. Looking at the telemetry, the proxy
server is advertising MAILBOX-REFERRALS[1] and Pine is trying to use
them[2] but isn't able to silently reauthenticate the user.
Looking at the man pages for imapd.conf there is a useful feature
proxyd_disable_mailbox_referrals however this was introduced in Cyrus 2.3.
Does anyone have any suggestions as to how to stop the front-end proxy
server from advertising MAILBOX-REFERRALS and/or for Pine to ignore such
capabilities.
Many thanks,
Dave.
David Mayo
Networks/Systems Administrator
University of Bath Computing Services, UK
[1]
* OK imaphost.bath.ac.uk Cyrus IMAP4 Murder v2.2.13 server ready
1 capability
* CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ MAILBOX-REFERRALS
NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND
BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE
STARTTLS AUTH=GSSAPI SASL-IR
1 OK Completed
[2]
12499847670009 RLIST INBOX.sent-mail
1249984767* LIST (\HasNoChildren) . INBOX.sent-mail
0009 OK Completed
1249984767000a APPEND INBOX.sent-mail {335}
1249984767000a NO [REFERRAL
imap://;aut...@imap-backend1.bath.ac.uk/INBOX.sent-m
ail] Remote mailbox.
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Problems running ctl_mboxlist -m on 2.2 back-end
Michael, Michael Bacon wrote: I think I may have seen exactly what you're seeing, and it's a doozy to find, but simple to fix. If it's what I saw (and it was this EXACT symptom), you just need to rebuild your binaries with the thread-safe switch. If you're using Sun Studio (we did because of its optimization on the T2 processor), you need to pass -mt to the compiler at compile time (CFLAGS), or else Solaris won't set errno properly in a multi-threaded process, and non-blocking I/O will eat itself. The painful details here: http://cyrusimap.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrusmsg=48912 The switch on GCC is different (I think it's -mthread), but just make sure you're specifying the thread-safe switch. This was it exactly - thanks!! The agrument to gcc is in fact -pthreads. Better still, I only had to apply this to the MUPDATE server, which means I don't need to recompile imapd on the live back-end server. Allow me to buy you a beer or two if you're ever in the area! Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK --On July 23, 2009 1:35:00 PM +0100 David Mayo d.j.m...@bath.ac.uk wrote: We are looking at upgrading our single 2.2 IMAP server to a Murder setup with a 2.3 back-end server. For the transition we will put the current IMAP server into the Murder and gradually transfer the mailboxes over to the new server using 'xfer'. I have just tested the first stage of the migration by dumping the list of mailboxes on the currently live server and importing that into our test 'currently live' server. The initial ctl_mboxlist -m transfer to the MUPDATE master took ~12 minutes for ~225,000 mail folders (skiplist format). Subsequent attempts to run ctl_mboxlist -m on the server do not work - there is a small flurry of activity at the start according to truss on the local machine and snoop on the MUPDATE server, then nothing happens for exactly 30 minutes and it finally gives up with couldn't do LIST command on mupdate server. I have restarted the IMAP daemons on both servers and tried converting the mboxlist_db on the back-end from skiplist to berkeley - none of these steps have made any difference. I can run mupdatetest and issue a LIST command which shows plenty of mailboxes. This shouldn't be a problem in itself as long as all the mailbox operations work as expected, however it is a bit of a worry. Both machines are running Solaris 10. The back-end server is running 2.2.12 and the front-end server is running 2.2.13. Has anyone experienced this problem and is there a way round it if this command doesn't work? Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Problems running ctl_mboxlist -m on 2.2 back-end
We are looking at upgrading our single 2.2 IMAP server to a Murder setup with a 2.3 back-end server. For the transition we will put the current IMAP server into the Murder and gradually transfer the mailboxes over to the new server using 'xfer'. I have just tested the first stage of the migration by dumping the list of mailboxes on the currently live server and importing that into our test 'currently live' server. The initial ctl_mboxlist -m transfer to the MUPDATE master took ~12 minutes for ~225,000 mail folders (skiplist format). Subsequent attempts to run ctl_mboxlist -m on the server do not work - there is a small flurry of activity at the start according to truss on the local machine and snoop on the MUPDATE server, then nothing happens for exactly 30 minutes and it finally gives up with couldn't do LIST command on mupdate server. I have restarted the IMAP daemons on both servers and tried converting the mboxlist_db on the back-end from skiplist to berkeley - none of these steps have made any difference. I can run mupdatetest and issue a LIST command which shows plenty of mailboxes. This shouldn't be a problem in itself as long as all the mailbox operations work as expected, however it is a bit of a worry. Both machines are running Solaris 10. The back-end server is running 2.2.12 and the front-end server is running 2.2.13. Has anyone experienced this problem and is there a way round it if this command doesn't work? Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: upgrading a 2.2.12 murder to 2.3.14
Andrew Morgan wrote: On Thu, 16 Jul 2009, Gavin Gray wrote: We are planning towards upgrading our existing murder. The murder has four front ends, three backends and separate mupdate and lmtp servers. We want to move from version 2.2.12 to 2.3.14 so that we can make use of delayed expunge an possible replication. 2. We should end up then with our existing murder but with three backends running 2.3.14. We then plan to upgrade the other machines in the murder to 2.3.14 in the following order: frontends then lmtp and finally the mupdate server. Does this make sense? Whatever you do, don't upgrade any of your frontends while you have older backends. The 2.3 code uses new IMAP calls that don't exist in 2.2. Quoting myself from a couple years ago: I proceeded assuming I could have a v2.3.10 frontend with older v2.2.13 backends. However, I was unable to get the APPEND command to work. With telemetry logging enabled, I discovered that a 2.3 frontend issues the IMAP command Localappend to a backend. However, my v2.2.13 backend does not recognize Localappend as a valid command (and it is not present in the source code). We are in a very similar position to the Edinburgh guys except we have a single IMAP server running 2.2.12. Our plan was to set up our new 2.3.13 back-end, use a 2.3.13 front-end, add the existing back-end to the new Murder, then transfer our mail using xfer. Would it be more sensible to use a 2.2 front-end, move our mail to the new back-end and upgrade the front-end to 2.3? Presumably there is no compatibility mode for 2.3 that would make it issue only 2.2-compatible commands? I'm surprised it can't work this out for itself. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: DBERROR with Cyrus 2.2.12
Christophe, Christophe Boyanique wrote: I've got a problem with an old Cyrus installation: it is a 2.2.12 version on RedHat AS3 server which used to work correctly. It seems that after a power failure, some mailboxes have been corrupted. On 1000 mailboxes, 5 seem to be unreachable (for imap reads or lmtp deliveries) with this message in the log: DBERROR: error fetching user.foobar: cyrusdb error I may be way off the mark here, but have you run the Berkeley DB recovery program? /etc/init.d/cyrus-imapd stop cd /var/lib/imap /path/to/berkeley/bin/db_recover ? Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services, UK Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Mailboxes with group: ACLs crashing imapd on delivery
Hi guys, I am upgrading our Cyrus installation from 2.2.12 to 2.3.14 and just started looking at ACLs. When I attempt to deliver to a mailbox with group: ACLs, LMTP crashes and will not deliver the message. Here's the ACLs for the mailbox: timaphost.bath.ac.uk lam user.exim exim lrswipkxtecda ma9djm lrs group:exim lrs This is a grab of me doing an LMTP session to the frontend: 220 timaphost.bath.ac.uk Cyrus LMTP Murder v2.3.14 server ready LHLO timaphost.bath.ac.uk 250-timaphost.bath.ac.uk 250-8BITMIME 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-SIZE 250-STARTTLS 250-AUTH EXTERNAL 250 IGNOREQUOTA MAIL FROM: 250 2.1.0 ok RCPT TO:e...@timaps.bath.ac.uk Connection to localhost closed by foreign host. Relevant imapd.log lines: May 29 15:57:53 sauber-z1.bath.ac.uk master[16505]: [ID 970914 mail.error] process 16674 exited, signaled to death by 11 May 29 15:57:53 sauber-z1.bath.ac.uk master[16505]: [ID 621917 mail.debug] service lmtp pid 16674 in BUSY state: terminated abnormally If I take the group:exim ACL off the mailbox, it delivers fine. The Unix group exists on the MUPDATE master and the backend. Relevant bits of imapd.conf: seenstate_db: skiplist duplicate_db: skiplist subscription_db: skiplist quota_db: skiplist sasl_pwcheck_method: saslauthd sasl_mech_list: plain gssapi allowanonymouslogin: no allowplaintext: yes auth_mech: unix We are using Cyrus-SASL/GSSAPI for authentication. We built imapd-2.2.12 with --with-auth=unix but configure-2.3.14 does not have this option so we've set the auth_mech option in imapd.conf as above. Do any other values need to be set in configure or imapd.conf for this to work? Can anyone suggest how to make this work and/or where to look for more information about why it's failing so horribly. Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
mupdate - GSSAPI authentication
+yblWDpQOPP7rCkMVYxsCT9FQ0cgaU7IsRT4r+jw45HG99w4cvqLhA9RHwg9cXfrg3umajngNovT13CD0deXlQjWwlO7m9bZN/zI S: YIGZBgkqhkiG9xIBAgICAG+BiTCBhqADAgEFoQMCAQ+iejB4oAMCARKicQRv6ysRnz7c5/jXdrML5GDO3yUDRd6e483bvcFFSv7Om/LcVmstU3vc7py4zljh1sI9cqP6wV0d6NKtUNJBEGaQNciHdasq+ywbgRsMvAsAM5/m7i06vByFOdRvZX2MxCdEMVW9KbAGRIHBvK6JQFxG C: S: BQQF/wAMOaMBYAcAEAAlGhxrUx+QK7vb6rg= C: BQQE/wAMELjSmQQABAB9zam/40LRAaw4zaw= S: A01 OK Authenticated Authenticated. Security strength factor: 56 C: Q01 LOGOUT Q01 OK bye-bye Connection closed. sauber $ klist Ticket cache: FILE:/tmp/krb5cc_58 Default principal: cy...@bath.ac.uk Valid starting Expires Service principal 08/05/2009 10:27:37 08/05/2009 20:27:37 krbtgt/bath.ac...@bath.ac.uk renew until 15/05/2009 10:27:37 08/05/2009 10:27:43 08/05/2009 20:27:37 mupdate/tyrrell.bath.ac...@bath.ac.uk renew until 15/05/2009 10:27:37 Relevant logs from tyrrell: May 8 10:27:42 tyrrell.bath.ac.uk mupdate[15803]: [ID 596527 mail.notice] login: sauber.bath.ac.uk [138.38.132.132] cyrus GSSAPI User logged in The *only* difference is we are using a default principal of cy...@bath.ac.uk rather than mupdate/sauber.bath.ac...@bath.ac.uk. This does not seem to make sense. Relevant lines from config files: sauber imapd.conf: admins: cyrus imap/sauber.bath.ac.uk sasl_pwcheck_method: saslauthd sasl_mech_list: plain gssapi mupdate_server: tyrrell.bath.ac.uk mupdate_config: standard mupdate_authname: mupdate/sauber.bath.ac.uk mupdate_username: cyrus tyrrell imapd.conf: admins: cyrus mupdate/sauber.bath.ac.uk sasl_pwcheck_method: saslauthd sasl_mech_list: plain gssapi We compiled cyrus-imapd-2.3.14 with the following flags: PROGDIR=/opt/packages/cyrus-imapd \ ./configure --prefix=$PROGDIR --mandir=/opt/share/man \ --sysconfdir=/opt/etc/imapd \ --enable-listext --enable-idled --with-snmp \ --enable-murder \ --enable-replication \ --enable-nntp \ --disable-gssapi \ --with-cyrus-group=cyrus \ --with-cyrus-user=cyrus \ --with-cyrus-prefix=$PROGDIR \ --with-openssl=$OPENSSLDIR \ --with-ucdsnmp=/opt/packages/net-snmp \ --with-sasl=$SASLDIR \ --with-dbdir=/opt/packages/berkeley-db \ --with-syslogfacility=MAIL We are using Cyrus SASL 2.1.22 built like this: PROGDIR=/opt/packages/cyrus-sasl \ ./configure --prefix=$PROGDIR --sysconfdir=/opt/etc/cyrus \ --with-plugindir=/opt/packages/cyrus-sasl/lib/sasl2 \ --enable-shared \ --disable-static \ --disable-java \ --with-configdir=/opt/etc/sasl2 \ --disable-krb4 \ --with-gss_impl=mit \ --with-rc4 \ --with-dblib=berkeley \ --with-saslauthd=/var/sasl2 --without-pwcheck \ --with-devrandom=/dev/urandom \ --enable-anon \ --enable-cram \ --enable-digest \ --enable-ntlm \ --enable-plain \ --enable-login \ --without-ldap \ --disable-otp \ --disable-ldapdb \ --disable-sql --without-mysql --without-pgsql --without-sqlite \ --enable-gssapi=$KERBEROSDIR \ --with-openssl=$OPENSSLDIR We are using MIT KerberosV 1.6.3 and running on Solaris 10 x86. tyrrell is actually a Solaris 'Zone' on sauber. If anyone has any ideas of what might be causing this problem we'd be very interested! Regards, Dave. David Mayo Networks/Systems Administrator University of Bath Computing Services Tel: +44 1225 38 6046 Email: d.j.m...@bath.ac.uk Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
