Archiving emails with Cyrus
Hi, we use Cyrus 2.2 on a Debian Etch Linux with Sasl and Thunderbird as a IMAP mailclient for our users. The problem is, that the mailboxes become very big, so I look for an automatic archiving software, which takes the mails out of the mailbox into a file and a searchable database. Thunderbird offers the possibility to mark a mail with an editable keyword (right mouseclick) even as a bulk operation on all selected emails. So my plan is, that the user marks all mails which can be archived with that keyword (lets say archive), or move them to a subfolder archive. There must be a process in cyrus, which copies these emails into a (zip)-file and/or into a database, to have them somehow accessable. Cyrus must do this with the administrator account, because the imap credentials of all the users are of course not known to us. Or we install an archive-useraccount which has access to all mailboxes. Does something like this exist? Where to look for? Maybe Opensource? Please give a hint Thanks Alex Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Archiving emails with Cyrus
we use Cyrus 2.2 on a Debian Etch Linux with Sasl and Thunderbird as a IMAP mailclient for our users. The problem is, that the mailboxes become very big, so I look for an automatic archiving software, which takes the mails out of the mailbox into a file and a searchable database. Are you looking to archive message for data-retention or because you need to recover capacity? Thunderbird offers the possibility to mark a mail with an editable keyword (right mouseclick) even as a bulk operation on all selected emails. Does that actually get saved to the server? Or just in TB's cache? So my plan is, that the user marks all mails which can be archived with that keyword (lets say archive), or move them to a subfolder archive. There must be a process in cyrus, which copies these emails into a (zip)-file and/or into a database, to have them somehow accessable. Cyrus must do this with the administrator account, It should be trivial to write an IMAP tool that will archive messages. I'm curious because I am currently working on a .NET/Gtk# application to do some management of Cyrus. Currently it profiles a mailstore and creates a graph relating age of messages to storage space consumed. because the imap credentials of all the users are of course not known to us. Or we install an archive-useraccount which has access to all mailboxes. Recommend the second, I try never to connect to the box as the cyrus account. Just grant appropriate access to the appropriate resources. Does something like this exist? Where to look for? Not to my knowledge. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Archiving emails with Cyrus
Hi Adam, Adam Tauno Williams schrieb: Are you looking to archive message for data-retention or because you need to recover capacity? I'll need both, sooner or later. For the moment it'll be enough to gain some more capacity. Thanks for your answer. Thunderbird offers the possibility to mark a mail with an editable Keyword (right mouseclick) even as a bulk operation on all selected emails. Does that actually get saved to the server? Or just in TB's cache? Good question. I played around a bit, and looked in the cyrus mail spool, but it seems these marks are not transferred back to the server. plication to do some management of Cyrus. Currently it profiles a mailstore and creates a graph relating age of messages to storage space consumed. I could do that with php, but if the users can't mark their messages to be archived, than it'll not be flexible enough. I can do that, just by date, but that's not what I want. cu Alex Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Archiving emails with Cyrus
Adam Tauno Williams schrieb: Are you looking to archive message for data-retention or because you need to recover capacity? I'll need both, sooner or later. For the moment it'll be enough to gain some more capacity. Thanks for your answer. Thunderbird offers the possibility to mark a mail with an editable Keyword (right mouseclick) even as a bulk operation on all selected emails. Does that actually get saved to the server? Or just in TB's cache? Good question. I played around a bit, and looked in the cyrus mail spool, but it seems these marks are not transferred back to the server. What I expected, plication to do some management of Cyrus. Currently it profiles a mailstore and creates a graph relating age of messages to storage space consumed. I could do that with php, but if the users can't mark their messages to be archived, than it'll not be flexible enough. I can do that, just by date, but that's not what I want. It's a bummer that basically no clients support annotation. Probably setting up an archive folder is the only workable solution. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Archiving emails with Cyrus
On Monday 24 November 2008 08:56:35 am Alexandros Gougousoudis wrote: There must be a process in cyrus, which copies these emails into a (zip)-file and/or into a database, to have them somehow accessable. Cyrus must do this with the administrator account, because the imap credentials of all the users are of course not known to us. Or we install an archive-useraccount which has access to all mailboxes. Here's an idea I've been toying with for an upcoming implementation... Let's say you create everyone's Inbox/Drafts/etc mailboxes on your reasonably fast (expensive/small?) storage with a relatively low mailbox quota. You then create user.username.archive on a separate Cyrus partition, perhaps residing on SATA with a relatively high mailbox quota. Inform your users that to store mail and keep their Inbox available they should move it there. You can then use Cyrus' built-in search mechanisms (squat) and have to change very little. John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Archiving emails with Cyrus
On Nov 24, 2008, at 10:36 AM, John Madden wrote: On Monday 24 November 2008 08:56:35 am Alexandros Gougousoudis wrote: There must be a process in cyrus, which copies these emails into a (zip)-file and/or into a database, to have them somehow accessable. Cyrus must do this with the administrator account, because the imap credentials of all the users are of course not known to us. Or we install an archive-useraccount which has access to all mailboxes. Here's an idea I've been toying with for an upcoming implementation... Let's say you create everyone's Inbox/Drafts/etc mailboxes on your reasonably fast (expensive/small?) storage with a relatively low mailbox quota. You then create user.username.archive on a separate Cyrus partition, perhaps residing on SATA with a relatively high mailbox quota. Inform your users that to store mail and keep their Inbox available they should move it there. You can then use Cyrus' built-in search mechanisms (squat) and have to change very little. I've been toying with the idea of replacing the cyrus routine(s?) that open/read the MESSAGE#. spool files with something that also understands gzipped data. That way I can selectively gzip messages based on some algorithm involving message size and lack of activity. I haven't yet had the chance to do the analysis on the spool to see what sort of space gains I could expect. Nik Conwell Systems Programming Boston University Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
I just wanted to follow up on this thread, rather than leaving it hanging. It seems there was a more serious issue, which ultimately lead to the failure we experienced, with our murder setup. Specifically our internal DNS servers, were having sporadic time-outs for Linux and Macintosh clients. After disabling ipv6 on both of our nameservers, it seemed the name resolution issue cleared up. The strange thing is that it has been working without any complications for the last 6 months. But nobody at Marshall can pinpoint anything that may have changed recently with regards to ipv6 networking on the internal network. What is even stranger is that the name resolution timeouts only seemed to occur on Linux and Macintosh clients, while having no harmful effects on Microsoft Windows servers and clients. I personally want to thank everyone who helped troubleshoot and resolve this issue. Wesley Craig at the University of Michigan, thank you for your input, steering me in the right direction, and clearing up misconceptions about perceived errors. Andrew Morgan at Oregon State University, thank you for the input on working around performance bottlenecks (due to a huge backlog of e-mail) during the re-synchronization to the front-ends. Also, a big thanks to Jeff Eaton at Carnegie Mellon, for his assistance in troubleshooting and his advice on what to do next. Andrew Morgan wrote: On Thu, 20 Nov 2008, Wolfe, Eric G wrote: We are using skiplist, I copied the mailboxes.db to frontends. If the frontends are updating, it is not apparent. I could not verify that either of them were synching from the master after the mupdate master synch. Which is why I copied them to the frontends to speed things up. I've tried this in the past for the same reasons, but it doesn't work. The only way I've been able to get the correct mailboxes.db on frontends is to let them sync with the mupdate master. And yes, this takes a while from scratch when you are using skiplist because writes are much slower with skiplist. Strange that it would just start causing problems now. We probably are seeing a cascading effect of failure, with the backlog though. Do the latest vanilla trees, have these patches included in them? The packages here: http://cyrusimap.web.cmu.edu/downloads.html#imap. I am somewhat reluctant to upgrade things in a fragile state. If these patches are included in latest releases, is 2.3.13 a fairly painless upgrade path from 2.2.12, or do we need to go with 2.2.13? I wouldn't upgrade until you can get a working system. Is there anything we can turn off in the cyrus.conf or imapd.conf, to work around this bottleneck? In other words, can we keep the MTA from knocking on the door for long enough to get everything running smoothly again? Yes, comment out lmtp in your cyrus.conf on the frontends. After you have successfully synced mailboxes.db, enable lmtp again and restart cyrus on the frontends. You probably should put some limits on lmtp as well. We use maxchild=50 here (3 frontends). Ok, because it sounds like a problem of connecting to the mupdate master port on 3905, to the unitiated. Hard to say, but take the steps above to simplify the problem. :) Andy -- Eric G. Wolfe, IT Associate, Sr. One John Marshall Drive Marshall University, Drinko Library 428k Huntington, WV 25755 Phone: 304.696.3428 Email: [EMAIL PROTECTED] Nobody's gonna believe that computers are intelligent until they start coming in late and lying about it. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: murder setup - mailboxes.db corruption - trouble recovering with ctl_mboxlist
I just wanted to follow up on this thread, rather than leaving it hanging. It seems there was a more serious issue, which ultimately lead to the failure we experienced, with our murder setup. Specifically our internal DNS servers, were having sporadic time-outs for Linux and Macintosh clients. After disabling ipv6 on both of our nameservers, it seemed the name resolution issue cleared up. The strange thing is that it has been working without any complications for the last 6 months. But nobody at Marshall can pinpoint anything that may have changed recently with regards to ipv6 networking on the internal network. What is even stranger is that the name resolution timeouts only seemed to occur on Linux and Macintosh clients, while having no harmful effects on Microsoft Windows servers and clients. I had a strange issue recently with name resolution (OT because it has nothing todo with cyrus-imapd). Resolvers on Linux and recent OSX seem to always try to query for IPv6 () records even if NO IPv6 is configured on the client (no, I think it only happens if the client program tries to resolve IPv6 like firefox in our case). The problem starts if the DNS servers in question don't reply correctly to those requests and reply with SERVFAIL for example. The resolver will then ask the next DNS server it find in resolv.conf and go on until no more servers to ask, which can take quite some time if you have 4 servers configured like in our case. In the end it turned out the company in question were running DNS loadbalancers and they simply replied with SERVFAIL for records. I've been told customers with Windows don't have that problem and I guess it's because their resolver doesn't ask for IPv6 adresses if no IPv6 is configured. Simon I personally want to thank everyone who helped troubleshoot and resolve this issue. Wesley Craig at the University of Michigan, thank you for your input, steering me in the right direction, and clearing up misconceptions about perceived errors. Andrew Morgan at Oregon State University, thank you for the input on working around performance bottlenecks (due to a huge backlog of e-mail) during the re-synchronization to the front-ends. Also, a big thanks to Jeff Eaton at Carnegie Mellon, for his assistance in troubleshooting and his advice on what to do next. Andrew Morgan wrote: On Thu, 20 Nov 2008, Wolfe, Eric G wrote: We are using skiplist, I copied the mailboxes.db to frontends. If the frontends are updating, it is not apparent. I could not verify that either of them were synching from the master after the mupdate master synch. Which is why I copied them to the frontends to speed things up. I've tried this in the past for the same reasons, but it doesn't work. The only way I've been able to get the correct mailboxes.db on frontends is to let them sync with the mupdate master. And yes, this takes a while from scratch when you are using skiplist because writes are much slower with skiplist. Strange that it would just start causing problems now. We probably are seeing a cascading effect of failure, with the backlog though. Do the latest vanilla trees, have these patches included in them? The packages here: http://cyrusimap.web.cmu.edu/downloads.html#imap. I am somewhat reluctant to upgrade things in a fragile state. If these patches are included in latest releases, is 2.3.13 a fairly painless upgrade path from 2.2.12, or do we need to go with 2.2.13? I wouldn't upgrade until you can get a working system. Is there anything we can turn off in the cyrus.conf or imapd.conf, to work around this bottleneck? In other words, can we keep the MTA from knocking on the door for long enough to get everything running smoothly again? Yes, comment out lmtp in your cyrus.conf on the frontends. After you have successfully synced mailboxes.db, enable lmtp again and restart cyrus on the frontends. You probably should put some limits on lmtp as well. We use maxchild=50 here (3 frontends). Ok, because it sounds like a problem of connecting to the mupdate master port on 3905, to the unitiated. Hard to say, but take the steps above to simplify the problem. :) Andy -- Eric G. Wolfe, IT Associate, Sr. One John Marshall Drive Marshall University, Drinko Library 428k Huntington, WV 25755 Phone: 304.696.3428 Email: [EMAIL PROTECTED] Nobody's gonna believe that computers are intelligent until they start coming in late and lying about it. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: sieve vacation reposnse gives no response even after end of days specified
On 24 Nov 2008, at 01:51, ram wrote: Is there a patch that fixes this value to 1 Find one attached. Is this in bugzilla? I couldn't find it... If someone puts it there and marks it as a blocker, it will be fixed in the next release. :wes sieve-min.diff Description: Binary data Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Cyrus-IMAPD Hot Standby Configuration
Hi, we're currently running a mailsystem with postfix and cyrus imapd with about 37.000 mboxes on a single server. For HA purposes (Hot Standby ist ok) I'm looking for a way to replicate the mailboxes to a second server on the fly. I found a description an the net http://cyrusimap.web.cmu.edu/imapd/install-replication.html but it's from 2005 so my question is: Is this still the recommended way for doing replication or is there something new? we are using cyrus-imapd-2.2.8-1 release from Red Hat Enterprise Linux AS release 3 (Taroon Update) Any hel / comments aprechiated Thanks Helmut Weigel DFB-Medien GmbH Co. KG Otto-Fleck-Schneise 6 60528 Frankfurt fon +49 (69) 6788-319 fax +49 (69) 6788-343 email: [EMAIL PROTECTED] Homepage: www.dfb-medien.de Hermann-Neuberger-Haus | Otto-Fleck-Schneise 6 | 60528 Frankfurt | DFB Medien GmbH Co. KG | Geschäftsführung: DFB Medien Verwaltungs-GmbH, deren Geschäftsführer: Kurt Gärtner, Tilman Walk |Vorsitzender des Aufsichtsrates: Dr. Theo Zwanziger | HRA 30550 | Registergericht: Frankfurt Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus-IMAPD Hot Standby Configuration
On 24 Nov 2008, at 13:10, Helmut Weigel wrote: I found a description an the net http://cyrusimap.web.cmu.edu/imapd/install-replication.html but it's from 2005 Definitely a bit dated, but replication is also much more mature now. so my question is: Is this still the recommended way for doing replication or is there something new? It is the recommended approach, yes. There are definitely other options, e.g., mupdate_config: replicated plus a HA disk system is another (much more expensive) method. we are using cyrus-imapd-2.2.8-1 release from Red Hat Enterprise Linux AS release 3 (Taroon Update) Replication requires 2.3.x and 2.3.13 is the best replicating cyrus imapd available. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
AW: Cyrus-IMAPD Hot Standby Configuration
Hi All Replication requires 2.3.x and 2.3.13 is the best replicating cyrus imapd available. RedHat ships 2.3.7 with RHEL Enterprise Server 5. Is that sufficient? Does anybody have experiences with that version? Thanks in advance Helmut Weigel Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
AW: Cyrus-IMAPD Hot Standby Configuration
-- Helmut Weigel [EMAIL PROTECTED] is rumored to have mumbled on 24. November 2008 22:19:58 +0100 regarding AW: Cyrus-IMAPD Hot Standby Configuration: Replication requires 2.3.x and 2.3.13 is the best replicating cyrus imapd available. RedHat ships 2.3.7 with RHEL Enterprise Server 5. Is that sufficient? Does anybody have experiences with that version? There's no need to use Red Hat's stock RPMs, because Simon Matter provides excellent more current ones. We (Universität zu Köln) have been using them successfuly for years: http://invoca.ch/pub/packages/cyrus-imapd/ Initially we staid with Red Hat's because we thought they'd give us better support if we did, but their support isn't worth much. Believe me, you're much better off with the community support you're getting here! -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum für angewandte Informatik - Universitätsweiter Service RRZK Universität zu Köln / Cologne University - Tel. +49-221-478-5587 pgpRfG62Kmsoo.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: AW: Cyrus-IMAPD Hot Standby Configuration
I must second Sabastian's comments. Simon Matter's Invoca RPMs are very well put together. I use them extensively. Best regards, -nic Sebastian Hagedorn wrote: -- Helmut Weigel [EMAIL PROTECTED] is rumored to have mumbled on 24. November 2008 22:19:58 +0100 regarding AW: Cyrus-IMAPD Hot Standby Configuration: Replication requires 2.3.x and 2.3.13 is the best replicating cyrus imapd available. RedHat ships 2.3.7 with RHEL Enterprise Server 5. Is that sufficient? Does anybody have experiences with that version? There's no need to use Red Hat's stock RPMs, because Simon Matter provides excellent more current ones. We (Universität zu Köln) have been using them successfuly for years: http://invoca.ch/pub/packages/cyrus-imapd/ Initially we staid with Red Hat's because we thought they'd give us better support if we did, but their support isn't worth much. Believe me, you're much better off with the community support you're getting here! -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum für angewandte Informatik - Universitätsweiter Service RRZK Universität zu Köln / Cologne University - Tel. +49-221-478-5587 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Nic Bernstein [EMAIL PROTECTED] Onlight llc. www.onlight.com 2266 North Prospect Avenue #610 v. 414.272.4477 Milwaukee, Wisconsin 53202-6306 f. 414.290.0335 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: AW: Cyrus-IMAPD Hot Standby Configuration
On 24 Nov 2008, at 16:19, Helmut Weigel wrote: RedHat ships 2.3.7 with RHEL Enterprise Server 5. Is that sufficient? Does anybody have experiences with that version? I haven't reviewed RH's 2.3.7. Perhaps they closely follow cyrus imapd development and back port all of the important stability, security and correctness changes that have been made between 2.3.7 and 2.3.13. But I strongly doubt it :) :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
