Re: Sieve EditHeaders and Logging
Hi David, > Is it possible to enable the "editheaders" sieve extension? if so, how? Not in 3.0, but it's available in 3.2 > Are sieve actions logged anywhere, e.g. to aid with debugging? Generally? I don't know. Maybe if you increase your syslog log level to "debug" and add "debug: yes" to your imapd.conf? 3.2 has an experimental "x-cyrus-log" extension, which adds an action that produces a log line. You probably don't want to allow this for user-supplied sieve scripts, but if you have some sort of "sieve builder" system that your users use (rather than uploading their own handcrafted sieve scripts), then, if you were running 3.2, this extension could be useful. On the master branch, and therefore in the next major release (2021), this extension will be formalised as "vnd.cyrus.log". Cheers, ellie Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Object Storage and Cyrus IMAP
On Fri, Jun 5, 2020, at 4:48 AM, Albert Shih wrote: > Le 04/06/2020 à 10:23:12+0200, Marco a écrit > > Hello, > > > >I see that Cyrus IMAP 3 can interface with some Object Storage such as > > Caringo or OpenIO. > > > > Is anyone using these solutions? > > > > I would like to know how I can find more details about these deployment, > > other than the brief description in imapd.conf man page. > > > > In particular I would like to know if these interfaces are stable and > > supported in future releases of Cyrus IMAP. > > > > Do you plan to add wider support to object storage, maybe by adopting some > > standard vendor independent? > > > > I thank you very much for every info you could provide. > > No sure if my informations are still correct, but long time ago, I ask > openio about that. They say the connector between openIO and cyrus are > maintained by openIO, it's not opensource and you need to pay a licence. > > And when Cyrus make a new release openio would make adjustment to make it > work. > > I not sure who use that, but as I ear they(openIO) got few customer use > this solution on a very large scale > 1Po. Our object storage support was contributed by one of those two vendors (i.e Caringo or OpenIO, though I don't remember which). As I understand it, they implemented support in Cyrus for both backends, to ensure that the object storage support was generalised, not specific to their own product. This might have been a condition we imposed for accepting their patches? Anyway, in theory, some third backend (whether closed- or open-source) could be similarly integrated, now that the abstract support is there. I don't know if such a third backend even exists. Looking through commit history, the last commits from the vendor developers to the object storage code were just before the release of 3.0.0. So I would expect this works okay for 3.0 deployments. I'm not sure about 3.2 -- we haven't had any specific updates, so maybe it works okay and doesn't need any? Or maybe it won't work until it's updated, and their customers are simply staying on 3.0 for the time being. I guess this is kind of a long-winded way of saying, you probably need to speak to those vendors about this. Whether Cyrus supports their backend is almost incidental compared to the larger question of whether (and how) they support their backend being used from Cyrus. Cheers, ellie Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Replication and Deleted Files
On Thu 04 Jun 2020 at 18:57:37, Michael Menge (michael.me...@zdv.uni-tuebingen.de) wrote: > you also need to run cyr_expire on the "new_server" to remove the old expunged mails and deleted folders. Obvious when you try it! Thanks so much. Expired 23 and expunged 7617 out of 289060 messages from 268 mailboxes For some reason I had decided that you only ran cyr_expire on the master, and I was quite emphatic about it some years ago: # expire old stuff: dups 7 days, keep deletions for 3 days # XXX XXX XXX expire does not run on replica, does run on master XXX XXX XXX # expire cmd="cyr_expire -E 7 -X 3 -D 3" at=0100 Thank you again,.,.I shall be back in another 25 years with another query :-) ian Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Object Storage and Cyrus IMAP
Le 04/06/2020 à 10:23:12+0200, Marco a écrit > Hello, > >I see that Cyrus IMAP 3 can interface with some Object Storage such as > Caringo or OpenIO. > > Is anyone using these solutions? > > I would like to know how I can find more details about these deployment, > other than the brief description in imapd.conf man page. > > In particular I would like to know if these interfaces are stable and > supported in future releases of Cyrus IMAP. > > Do you plan to add wider support to object storage, maybe by adopting some > standard vendor independent? > > I thank you very much for every info you could provide. No sure if my informations are still correct, but long time ago, I ask openio about that. They say the connector between openIO and cyrus are maintained by openIO, it's not opensource and you need to pay a licence. And when Cyrus make a new release openio would make adjustment to make it work. I not sure who use that, but as I ear they(openIO) got few customer use this solution on a very large scale > 1Po. Regards -- Albert SHIH DIO bâtiment 15 xmpp: j...@obspm.fr Heure local/Local time: Thu 04 Jun 2020 08:44:56 PM CEST Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: imap clients say i have 4K messages but spool has 12894 files
Interestingly, through no action on my (as admin) part, this problem seems to have resolved itself on May 31. According to my backup, on May 29 for my main inbox, user.brian, there were 13339 files on the disk but on May 31's backup there are only 4136. IMAP has always reported in the neighborhood of the 4K messages. Strange. b. signature.asc Description: This is a digitally signed message part Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Replication and Deleted Files
Hi, Quoting Ian Batten via Info-cyrus : Hi, long-time Cyrus user (25 years, I think), but stumped on this one… I have an ancient Cyrus 2.5.11 on Solaris 11 installation I am trying to migrate off. The strategy is to run rolling replication onto the new server (3.0.8-6+deb10u4 on Debian 10.4), and then point the DNS record at the new server. With Covid, this has become more protracted than I would like, as I don’t want to accidentally mess up users who are isolating, so the replication has been running for some weeks. The replication structure is old-server -> new-server -> (backup1, backup2) where backup1 and backup2 are configured as separate channels on new-server. This has been running seemingly correctly for about three months now. Today I decided to check all was well by using rsync -an to confirm that the replicas have everything that is on the master. They do, in that using rsync -anvO --size-only --exclude='cyrus.*' root@mail:/var/imap/partition1/user/ /var/imap/partition1/user where “mail” is the old server shows that there are no messages missing (—size-only because there’s some time slew in a few places, usually only of a few seconds, but up to a day in others). However, reversing it: rsync -anvO --size-only --exclude='cyrus.*' /var/imap/partition1/user/ root@mail:/var/imap/partition1/user Shows that there are a _lot_ of files on the replicas which are not on the master, some of them relating to recent deletions, but some of them seemingly quite old. I am using: delete_mode: delayed expunge_mode: delayed everywhere, running cyr_expire on the master but not on the replicas. I have enough bandwidth that sync_reset and re-sync is realistic, but I’d rather not have to do that immediately prior to a cut-over. These old files are a worry because if I ever had to reconstruct one of the mailboxes, presumably the deleted (I think) messages would all reappear. Does anyone have any suggestions? you also need to run cyr_expire on the "new_server" to remove the old expunged mails and deleted folders. I haven't used replication for backup but I suspect that cyr_expire is also required your backup servers M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Replication and Deleted Files
Hi, long-time Cyrus user (25 years, I think), but stumped on this one… I have an ancient Cyrus 2.5.11 on Solaris 11 installation I am trying to migrate off. The strategy is to run rolling replication onto the new server (3.0.8-6+deb10u4 on Debian 10.4), and then point the DNS record at the new server. With Covid, this has become more protracted than I would like, as I don’t want to accidentally mess up users who are isolating, so the replication has been running for some weeks. The replication structure is old-server -> new-server -> (backup1, backup2) where backup1 and backup2 are configured as separate channels on new-server. This has been running seemingly correctly for about three months now. Today I decided to check all was well by using rsync -an to confirm that the replicas have everything that is on the master. They do, in that using rsync -anvO --size-only --exclude='cyrus.*' root@mail:/var/imap/partition1/user/ /var/imap/partition1/user where “mail” is the old server shows that there are no messages missing (—size-only because there’s some time slew in a few places, usually only of a few seconds, but up to a day in others). However, reversing it: rsync -anvO --size-only --exclude='cyrus.*' /var/imap/partition1/user/ root@mail:/var/imap/partition1/user Shows that there are a _lot_ of files on the replicas which are not on the master, some of them relating to recent deletions, but some of them seemingly quite old. I am using: delete_mode: delayed expunge_mode: delayed everywhere, running cyr_expire on the master but not on the replicas. I have enough bandwidth that sync_reset and re-sync is realistic, but I’d rather not have to do that immediately prior to a cut-over. These old files are a worry because if I ever had to reconstruct one of the mailboxes, presumably the deleted (I think) messages would all reappear. Does anyone have any suggestions? Thanks ian Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus IMAP 3.2.1 released
On 29/05/2020 06:20, ellie timoney has written: The Cyrus team is proud to announce the immediate availability of a new version of Cyrus IMAP: 3.2.1 Hello, in Redhat EL8 I still fail these tests: ERRORS: Rename.rename_inbox Perl exception: Errors found in syslog at Cassandane/Instance.pm line 1322, line 91. FAILS: 1) Admin.imap_admins expected 'ok', got 'no' at Cassandane/Cyrus/Admin.pm line 90. 2) Caldav.supports_event Boolean assertion failed at /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13. 3) ImapTest.append-binary Boolean assertion failed at /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13. 4) ImapTest.fetch-binary-mime Boolean assertion failed at /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13. 5) ImapTest.urlauth-binary Boolean assertion failed at /usr/share/perl5/vendor_perl/Test/Unit/Exception.pm line 13. 10) Cassandane::Test::Core.core_files_* didn't match /(?^:Core files found)/ at Cassandane/Test/Core.pm line 96. The details are attached, if it could be of interest. The Cassandane commit is very recent: 68e4890346edeb68463d20f394a37f0862bef2ae I think all these failure are somewhere related to Redhat environment... I'll skip these tests in Cyrus IMAP 3.2.1 build. Regards Marco Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: imap clients say i have 4K messages but spool has 12894 files
On 6/4/20 7:45 AM, Brian J. Murrell wrote: On Wed, 2020-06-03 at 19:35 -0400, Ken Murchison wrote: Brian, Trying running 'unexpunge -l' on the mailbox in question. This avenue has already been explored earlier in this thread: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-May/041258.html To save the effort of re-reading the message: # sudo -u cyrus bash -c "/usr/lib/cyrus-imapd/unexpunge -l user.brian" [nothing returned] So this is looking more like a "bad accounting" problem than something typically operational. But how to reconcile it? It seems to me that a process of comparing what's in the index to what's on disk to account for the orphans is needed. I just don't know what that process is. I probably just don't know the toolset well enough to know which tools to apply and how. mbexamine seems a candidate but I'm not sure how to interpret it's output to this task. Or maybe there other/better tools? Any suggestions? Have you looked in some of the orphaned messages to see if they are emails you have deleted before? My thought would be to move these orphaned messages out of /var/spool/imap/b/user/brian . Then delete and expunge a few messages using your mail client and see if they are also removed from /var/spool/imap/b/user/brian Cheers, b. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus <> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: imap clients say i have 4K messages but spool has 12894 files
You could try 'reconstruct -R' which should force a re-parsing of all message files in the mailboxes directory. Note that if this works, you will have 8k new messages show up in your mailbox. Adding -n may just report what reconstruct will do rather than actually doing it. On 6/4/20 6:45 AM, Brian J. Murrell wrote: On Wed, 2020-06-03 at 19:35 -0400, Ken Murchison wrote: Brian, Trying running 'unexpunge -l' on the mailbox in question. This avenue has already been explored earlier in this thread: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-May/041258.html To save the effort of re-reading the message: # sudo -u cyrus bash -c "/usr/lib/cyrus-imapd/unexpunge -l user.brian" [nothing returned] So this is looking more like a "bad accounting" problem than something typically operational. But how to reconcile it? It seems to me that a process of comparing what's in the index to what's on disk to account for the orphans is needed. I just don't know what that process is. I probably just don't know the toolset well enough to know which tools to apply and how. mbexamine seems a candidate but I'm not sure how to interpret it's output to this task. Or maybe there other/better tools? Any suggestions? Cheers, b. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Kenneth Murchison Senior Software Developer Fastmail US LLC Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: imap clients say i have 4K messages but spool has 12894 files
On Thu, 2020-06-04 at 09:30 +1000, Ian Willis wrote: > Hi Brian, Hi Ian, > The answer to your question is that yes, UID appears to correlate > with > the message file name. Thanks. > At a guess something appears significantly awry. Indeed. > Have you tried create a separate mail user. Copy your existing > message > over via imap to the new folder. > Delete and expunge the original mailbox and recreate, recopy. I have considered that. But I suspect that is going to cause the message numbers on the disk to be recreated. Since this seems to affect many (all?) folders for many (again, all?) users, that would result in trashing the efficiency of my incremental backup. > In the longer term I would be tempted to move to a newer version of > cyrus I'm stuck with what my distro vendor supplies. That said, once the cause of this problem, or it's reproducibiilty can be confirmed, distro vendor will be getting a ticket to resolve this in some way. But the first step is identifying the issue, resolving it and seeing if it reproduces. > or if you have the patience closely monitor the file-system to > debug how this is occurring. Right. I think the first step is to figure out how to get things back to normal so that it can be monitored more closely and the discrepancy is no longer a haystack and will be more incrementally obvious when it does happen. Figuring out how to get back to normal will also provide the tools/process to monitor on an ongoing basis to see why it's happening. I'm just not sure how to get back to normal at this point. Cheers, b. signature.asc Description: This is a digitally signed message part Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: imap clients say i have 4K messages but spool has 12894 files
On Wed, 2020-06-03 at 19:35 -0400, Ken Murchison wrote: > Brian, > > Trying running 'unexpunge -l' on the mailbox in question. This avenue has already been explored earlier in this thread: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2020-May/041258.html To save the effort of re-reading the message: # sudo -u cyrus bash -c "/usr/lib/cyrus-imapd/unexpunge -l user.brian" [nothing returned] So this is looking more like a "bad accounting" problem than something typically operational. But how to reconcile it? It seems to me that a process of comparing what's in the index to what's on disk to account for the orphans is needed. I just don't know what that process is. I probably just don't know the toolset well enough to know which tools to apply and how. mbexamine seems a candidate but I'm not sure how to interpret it's output to this task. Or maybe there other/better tools? Any suggestions? Cheers, b. signature.asc Description: This is a digitally signed message part Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Object Storage and Cyrus IMAP
Hello, I see that Cyrus IMAP 3 can interface with some Object Storage such as Caringo or OpenIO. Is anyone using these solutions? I would like to know how I can find more details about these deployment, other than the brief description in imapd.conf man page. In particular I would like to know if these interfaces are stable and supported in future releases of Cyrus IMAP. Do you plan to add wider support to object storage, maybe by adopting some standard vendor independent? I thank you very much for every info you could provide. Warm Regards -- Marco Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
