Let's map tickets to milestones
A conversation has been started on when Cyrus IMAP 2.5 would be released. So far, there's little certainty about it, but we can try and make it more visible by creating tracker tickets and logging every bug, enhancement and/or task into Bugzilla. If we were to give people a week or so to do all of this, then I reckon next week we could have a meeting to review everything we want to do for 2.5. So, I've created a 2.5.0 release blocker ticket; https://bugzilla.cyrusimap.org/show_bug.cgi?id=3674 Its dependency tree[1] indicates what needs to actually happen. Its direct dependencies are considered high-level bugs, enhancements or tasks we aim to complete before we release 2.5.0. For example, ticket #3669[2] is a high-level enhancement to convert to using autofoo. While we work on this, of course we run into issues, which we log as blocking #3669. Now, all tickets the depencency tree if #3669 auto-magically also become visible in the dependency tree for the release blocker ticket. All of the enhancement specific bugs need to be resolved in order for the feature to be complete. Another example enhancement, #3343[3] (conversations support), currently has no dependencies... but I'm sure these will pop up as the work on the feature moves back to origin and closer and closer to master. If you'll allow me to wing it as we go along, please don't hesitate and make your mark, as follows: - Log your new enhancement request and set Blocks: 3674. Please use version 2.5-next and milestone 2.5-next. - Take an existing enhancement request and set Blocks: 3674 - you can leave the version untouched but you are at liberty to set the milestone to 2.5-next, even if it is currently set to 2.4-next. - We would like existing bugs to mostly remain as they are, but feel free to set Blocks: 3674 to existing bug reports. - Add yourself to CC: on tracker ticket 3674 to receive a moderate amount of traffic as new dependencies are added and existing dependencies are removed/implemented. I plan on releasing development snapshots as work on enhancements is progressing, so that people can take an interest in a specific thing (before it's merged back into master?) and find whether or not it works for them. Ideas? Questions? Comments? Gripes? Kind regards, Jeroen van Meeuwen [1] https://bugzilla.cyrusimap.org/showdependencytree.cgi?id=3674 [2] https://bugzilla.cyrusimap.org/show_bug.cgi?id=3669 [3] https://bugzilla.cyrusimap.org/show_bug.cgi?id=3343 -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 signature.asc Description: This is a digitally signed message part.
Re: Automake Support for cyrus-imapd 2.5
On 2012-04-16 3:14, Greg Banks wrote: The largest remaining issue is working out how to make upgrades from 2.3 and 2.4 as painless as possible, and how far we're willing to go to achieve that. Can we create a ticket for this? Other that that, I have on my agenda for 2.5: - some pending cleanups - improving the way 'quota -f' works (again) Can we create a ticket for this, and close off all quota (-f) related bugs in Bugzilla in favour of this final overhaul? - fix a lot of documentation Can we also please settle for a single set of documentation? I'm inclined to just have you throw it over the fence and log Yet Another Ticket in bugzilla, against the documentation component. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: "make dist" and cunit
On 2012-04-20 11:15, Дилян Палаузов wrote: Hello, Can SMakefile be deleted? If it's not needed, it can. Do we need "make snapshot" in addition to "make dist"? It'd be nice to maintain a target doing pre-releases of current GIT, as especially working towards 2.5 I plan on doing some of them. Perhaps superfluous, but xversion.h only needs to be created on make snapshot because it is the git repository and not the resulting tarball that is capable of getting it's hands on the current git commit hash for HEAD. This is not necessary for make dist though, as the version number should have been configured in configure.ac and there should be no use of a git commit hash anywhere. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Build failed in Jenkins: cyrus-imapd-master #542
On 2012-04-27 8:32, Greg Banks wrote: So it turns out there were several things wrong with the code in configure.ac which checked for LDAP. Most of these have been broken for a very long time. - we check for ldap_initialize() twice, pointlessly - if given --with-ldap i.e. with no argument, configure checks for a directory called "yes" and then generates -I/usr/local/include for LDAP_CPPFLAGS instead of an empty value - if the LDAP libraries are not detected, we get spurious non-empty values of LDAP_CPPFLAGS etc - if LDAP libraries are detected, the final Makefile gets a spurious extra copy of LDFLAGS - the recently added automake conditional never worked, as it tests for $HAVE_LDAP which is never set, because the AC_CHECK_LIB has a non-default action-if-set argument, so we compile against the LDAP headers but don't link against the libraries. So I fixed all those. Maybe *this* time it'll build... The AFSKRB checks are probably similarly broken; certainly the automake conditional is checking for a variable which is never set. But I'll let someone who cares fix that. Thanks Greg, for this fix - I had run into these a couple of times as well but had failed in my attempts to fix it. Seems you did it! Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
In preparation of Cyrus IMAP 2.5: autoconf and automake
Hello there, With many thanks to Дилян Палаузов , we would like to let you know about one particular feature now definitely included for a pending Cyrus IMAP 2.5 release. As a feature for the upcoming 2.5 release of Cyrus IMAP, though the exact schedule is yet unknown, we have merged into master the grand overhaul to using autoconf / automake. This marks a first significant milestone closing in on actually producing a 2.5 series release, but, and this is very important: NOT without your help. We would like those of you that have a need to or experience with building Cyrus IMAP from source to let us know whether the autoconf and automake (or, as I like to call it, "autofu") Works For You(TM). To this end, we encourage you to clone the GIT repository master branch and attempt a build, or, alternatively, download the following snapshot release: http://git.cyrusimap.org/cyrus-imapd/snapshot/cyrus-imapd-2.5-snapshot- autoconf-and-automake.tar.gz The canonical build process we think applies, generally speaking, is: $ autoreconf -v $ ./configure [your-options] $ make This process currently requires autoconf >= 2.67. We would appreciate you let us know whether or not such process works for you, preferrably though Bugzilla (please use product 'Cyrus IMAP' and component 'Distribution'). Thank you, in advance, On behalf of the Cyrus team, Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 signature.asc Description: This is a digitally signed message part.
Re: Libtool and Support for Shared Libraries
On 2012-05-10 17:32, Bron Gondwana wrote: On Thu, May 10, 2012 at 04:03:26PM +0200, Дилян Палаузов wrote: Hello, >While we're at it, I'd love to split things like 'mailbox.c' which are >really libraries into a separate directory from 'imapd.c' which is a >system daemon only run by master from cyrus.conf and separate again >from cyr_dbtool.c which is a tool designed to be mainly run by humans. I would suggest, that the sources of libcyrus are moved to src/lib/cyrus, the sources of libcyrus_min are moved to src/lib/cyrus_min, the sources of libimap are moved to src/lib/imap, the sources of libsieve are moved to src/lib/sieve, the sources of the services from imap/ are moved to src/service, probably the installable header files are moved to src/include, and then check what to do with the remaining files. With services I mean fud, imapd, lmtpd, pop3d, smmapd and sync_server. May I suggest the services end up in src/libexec/ perhaps? There's still things like master as well. I'd really like to move that elsewhere so the name 'master' doesn't clash with the git branch name 'master'. It makes a few commands more annoying to run. src/master or something would be better. It could still match remote 'src' branch 'master' - if we consider renaming the installed utility to cyrus-master, why not name the source file cyrus-master as well? A location could be (this is the only service process to be executed, really) src/sbin/. The utilities could live in src/bin/ (cyrus-quota, cyrus-mbpath). But - it will invalidate basically every patch out there, Yes indeed it will, and we have another (huge) pending task; fixing the indentation. so I don't want to embark on something like this until we've merged everything we're planning to merge, and everyone has advance notice of the plan. Even the automake changes so far have been quite invasive. I agree. This tidying up is going to majorly invasive - don't forget we want to merge in caldav before this happens as well. I strongly recommend any major really tidying things up is postponed to 2.6. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Libtool and Support for Shared Libraries
On 2012-05-11 2:00, Greg Banks wrote: On Thu, May 10, 2012, at 09:42 PM, Jeroen van Meeuwen (Kolab Systems) wrote: I agree. This tidying up is going to majorly invasive - don't forget we want to merge in caldav before this happens as well. I strongly recommend any major really tidying things up is postponed to 2.6. Agreed. I think we've changed enough for 2.5. It might be worth declaring 2.5 to be in feature freeze after the caldav merge, and starting a 2.6 branch. Otherwise we'll never get it released. My thoughts were to organize a meeting and look at Bugzilla tickets (everything currently targeted for 2.5). Per ticket/enhancement/issue we'd say at least two things: - Yes/No (is or is not targeted for 2.5), - Must/May/Might (is blocker, or not crucial and no worry if it's postponed) We'd want "one throat to choke" for each separate ticket (the assignee), and we want to make sure the people interested in progress are (or know that they should put themselves) in CC: on the ticket. We'd want to clarify the methodology of making sure everything's in good shape as well; - Jenkins CI should continue to succeed & Cassandane tests should be created where appropriate and possible, - Documentation (at the very least, Release Notes) If there's even the smallest issue that should not get lost it should end up in Bugzilla so it continues to pop up on someone's radar. Blocking the original (enhancement) ticket would service that. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: libzephyr and notifications
On 2012-05-22 9:40, Bron Gondwana wrote: We're having issues building zephyr with the new automake stuff, and before we spend too much time fixing it - there's a question worth asking... Does anyone actually use zephyr? I (we) don't. If not, I'd prefer to remove it and integrate worldline's notification bus work rather than having multiple competing notification systems. No objections from me (us). Andreas Osowski has successfully merged in / rebased the notification bus work from worldline's github, BTW, so if it can be given a glance-over before it's finally merged into origin/master, that'd be great. I've given Andreas commit / push access to origin so he can do the final merge and push himself on whatever is origin/master HEAD by then. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: libzephyr and notifications
On 2012-05-29 1:38, Greg Banks wrote: Ok, I'll review that, starting today or tomorrow (flu permitting). I saw Andreas' message go past but I can't seem to find it now that I need it :( so what's the git URL? He's git://github.com/aosowski/cyrus-imapd.git The branch is ticket/3605 Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: libzephyr and notifications
On 2012-05-29 11:32, Sébastien Michel wrote: Hi, I'm very sorry that Andreas done my job to rebase against the current master... No worries. I plan to do some cleanup of my code : I need to use commit 6735484f76470b02c439cc553149b0beb0e34e81 from branch dev/sieve/vacation-seconds (instead of mine 77cab79f920629a964340cfcfff14338f161a3c1), merge some commits into one to clean up the history, and finally export IMAP URL into the external form. Can you and Andreas coordinate / work together on this? If you need push access to git.cyrusimap.org as well, please send me a public ssh key. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Libtool and Support for Shared Libraries (2)
On 2012-05-29 1:25, Greg Banks wrote:
On Mon, May 28, 2012, at 09:33 PM, Dilyan Palauzov wrote:
>> [...]probably using make install DESTDIR= with
>> libtool is either wrong or implemented/handled wrong in
Automake/libtool.
>
> Well, that sucks.
Why do not you use ./configure --prefix=$(DESTDIR), so that make
install DESTDIR=somewhere is not necessary? To my understanding
installing in DESTDIR is used to create packages,
So we now generate dozens of warnings when doing a straightforward,
entirely normal, and unavoidable step in the packaging process? I
don't
see how that's acceptable.
While of course in my realm of packaging, I could use ./configure, what
I actually use is %configure. It expands to a predefined set of standard
configure options such as --prefix=/usr, --libdir=/usr/lib64, etc, along
with first exporting a bunch of variables.
When the make install DESTDIR=/some/where is issued, we point it to the
root directory of a buildroot (%{buildroot}) and expect the other
./configure options to kick in so that everything is still finally
placed in the correct directories (i.e.%{buildroot}etc/ for
--sysconfdir=%{_sysconfdir}, %{buildroot}usr/lib64 for
--libdir=%{_libdir}, etc. exec dir, libexec dir, ...) for which, if the
prefix were to be set to the buildroot root directory, we would need to
add the options for all the other --*dir= configure options as well.
Kind regards,
Jeroen van Meeuwen
--
Systems Architect, Kolab Systems AG
e: vanmeeuwen at kolabsys.com
m: +44 74 2516 3817
w: http://www.kolabsys.com
pgp: 9342 BF08
Re: Libtool and Support for Shared Libraries (3)
On 2012-05-31 18:25, Дилян Палаузов wrote: - "make install DISTDIR=" causes warnings from libtool, that state, that the libraries are not installed on the place they are intended to be installed (as configured) and the executables are not going to work (as they will not be able to find the libraries under the DESTDIR= root). This warning gives right information, on the other side Greg says, that the "make install DISTDIR=" is a normal process that shall not lead to warnings. I asked at libtool-...@gnu.org , but I if they do not answer, I can't say anything more about this. Is the "typo" DISTDIR/DESTDIR deliberate or accidental? My world uses DESTDIR, for the record. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: official dev team position regarding multiple times requested feature (global sieve)
On 2012-07-23 22:26, mailing list subscriber wrote: With all due respect, what is the development's team position regarding this feature and how do the development team see a solution that meets both requirements? Users will most likely continue to require write access to a script that allows them to set what level of spam is to be filtered to a different folder then one's INBOX, which addresses are on a whitelist, and whether or not the vacation or out of office responses is active/de-active. If an organization wishes to enforce a particular script (with or without particular settings, and with or without allowing the user some level of editing), then it is (now) mostly some or the other management solution on top of the Cyrus IMAP deployment that takes care of this level of management. I'm curious to learn what exactly would be the set of requirements that would enable a mandatory Sieve script feature to be integrated into Cyrus IMAP; - Would setting a mandatory script implicitly disallow users to write additional(?), new scripts? Would this be Yet Another Setting? Would Cyrus IMAP magically adjust any user-uploaded scripts to conform with the mandatory script policy? - Would a script (if it were read-only for the user) read settings from a location not Cyrus IMAP, such as the proverbial boolean "Yes I am out-of-office" and/or "my vacation lasts until $x"? - How many Sieve editing clients would remain compatible / could become compatible with such Sieve semantics? Please allow me to shamelessly plug some thoughts from Kolab[1] here, that relate to but are not entirely in the same realm. Kind regards, Jeroen van Meeuwen [1] https://wiki.kolab.org/KEP:14 -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Build failed in Jenkins: cyrus-imapd-master #1059
On 2013-01-02 10:16, Sébastien Michel wrote: 3 options in this case: - Install jansson library on the jenkins server - Add --disable-event-notification in the Cyrus build script - Disable event notification by default at compile time and replace --disable-event-notification by --enable-event-notification my preference is for option 1 My preference too, but this is an Enterprise Linux 5 system for which jansson (>= 2.0) is not available. So, I've updated the jenkins-build.sh script to use --disable-event-notification for now. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Build failed in Jenkins: cyrus-imapd-master #1059
On 2013-01-02 15:07, Sébastien Michel wrote: I used initially json-c that is the most common. However, only the trunk offered support to 64bit integers. This is why I switched to jansson that is also valuable. Unfortunately, it is less common than json-c, and for which the version 0.10 is now available in EPEL. Should I change back the JSON library? there is still time before releasing Cyrus 2.5 We have discussed this before, though perhaps not on the list; I have no trouble *not* having event notifications available with Cyrus IMAP 2.5 on Enterprise Linux 5, but; - We should consider upgrading our CI system to Enterprise Linux 6, - It is not very unlikely nobody will even attempt to build Cyrus IMAP 2.5 let alone ship off to production, based on Enterprise Linux 5... So, it is really up to the rest of the list (especially those with the Solaris, -UX and other UNIX systems), for which, if I recall correctly, json-c and jansson are both equally unavailable and need to be built on/for these platforms regardless. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: CI Builld #1169 has not completed for 50 days
On 2013-04-20 01:11, Дилян Палаузов wrote: Hello, ci.cyrusimap.org/job/cyrus-imapd-master/1169/ ) has not completed for over a month. Ideas? I aborted the build, so we're rolling forward with build #1170 now. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
git master notifyd/notify_external.c and *user
Hi there,
I was trying to get git master HEAD to execute an external program
(notify_external: /usr/bin/myprogram), but it complained "no recipient
sepcified" - short of the typo that I corrected, it seemed that the
check:
if (!*user) {
syslog(LOG_ERR, "ERROR: recipient not specified");
return strdup("NO external recipient not specified");
}
at [1] was causing this.
For the sake of argument, I removed the check, and voila - my program
is actually being executed.
Does anyone have any idea what the "-u USER" parameter is supposed to
be doing / does anyone actually successfully use notify_external?
Thanks.
Kind regards,
Jeroen van Meeuwen
[1]
http://git.cyrusimap.org/cyrus-imapd/tree/notifyd/notify_external.c#n78
--
Systems Architect, Kolab Systems AG
e: vanmeeuwen at kolabsys.com
m: +44 74 2516 3817
w: http://www.kolabsys.com
pgp: 9342 BF08
Re: [PATCH/FEATURE] Per-user XLIST support for cyrus-imapd
On 2013-04-30 09:20, Thomas Jarosch wrote: Hi Bron, On Tuesday, 30. April 2013 08:53:24 Bron Gondwana wrote: Hi - have you looked at all at the special-use support in mainline Cyrus? The xlist-* behaviour is planned to be removed, in favour of using the RFC 6154 mandated annotation. The implementation in git at the moment doesn't quite match the standard, because I wrote it before the RFC was released, but I have a branch which makes it fully compatible. I wrote to this list just the other day asking about it! Eduardo is innocent, we (=Intra2net) needed support for this :) We didn't upgrade to 2.4.x yet, so we needed to develop this for 2.3.16 anyway. As far as I understood it, Outlook 2013 does not implement the RFC properly. I suppose you would then use the Cyrus IMAP internal SPECIAL-USE folder flags storage(?) to represent a response to X-LIST(?) rather than supplying additional configuration that can quickly get out-of-sync with other clients (that do support SPECIAL-USE?) as well as across folders. I haven't actually looked at anything to back up this rambling with, tell me if it makes sense ;-) Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Special-Use and Cyrus 2.5 git
On 2013-04-26 16:17, Bron Gondwana wrote: Is anybody actually using cyrus 2.5 git in production other than FastMail? Yes, though admittedly I should say "production". I'm planning to do the mailboxes.db format changes before releasing 2.5, so that we don't have to support a stupid intermediate format. It's been on my todo list for AGES as one of the final blockers to 2.5 being released, and I've finally got some spare cycles away from our internal search code. But it would simplify things considerably if I don't have to write an importer that goes upstream. I'd just manually script a conversion of the FastMail servers, and then toss away the intermediate code. None of our systems with 2.5 in "production" use special-use, nor are they expected to be upgraded smoothly without manual / scripted sysadmin intervention. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: [PATCH] 2.4 branch does not build anymore
On 2013-05-18 16:34, Thomas Cataldo wrote: As conflict marks were committed. Thanks! Committed and pushed. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Changing LIST (again)
On 2013-05-02 07:29, Bron Gondwana wrote: One of my release goals for Cyrus 2.5 is to be correct in our implementation of every standard that we claim to support. This is why I emailed the list last week asking if anyone is using the intermediate SPECIALUSE representation in git. Since there was no reply, I'm assuming it will be OK :) I've just started testing it into production at FastMail, and will roll it out to all our users later this week. Assuming performance is acceptable, I'll push that upstream. Meanwhile, there's one glaring problem remaining, as evidenced by Timo's imaptest tool. Our LIST-EXTENDED implementation is broken in some places. I've already had reports of users of recent versions of some clients having problems with it. That sucks. Rob M (CC'd) and I had a good talk about it yesterday. We're looking at switching the default at FastMail to use altnamespace - it works better with Outlook 2013 and also with some phones. One problem is that you can't create subfolders of Inbox. We've looked at the standard, and we don't see any reason why we can't allow them, The clients that do create (allow) a sub-folder of the INBOX (to be created) apparently do not adhere to the \NoInferiors flag... Should you make the change to allow sub-folders (of the INBOX) to in fact be created (while using altnamespace), would that eliminate the use of \NoInferiors? Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Changing LIST (again)
On 2013-05-19 15:07, Bron Gondwana wrote: On Sun, May 19, 2013, at 08:36 PM, Jeroen van Meeuwen (Kolab Systems) wrote: Should you make the change to allow sub-folders (of the INBOX) to in fact be created (while using altnamespace), would that eliminate the use of \NoInferiors? Indeed it would :) I really, really like calling out stupid clients that do not honour \NoInferiors though... Can we please keep a sane default of (enforcing) RFC(s)-compliant behaviour? Do you suggest to remove the \NoInferiors flag for deployments with "sub-folder of INBOX w/ altnamespace allowed" type of configurations, or would you keep it around (and simply behave differently by honouring the request instead of returning an error)? I suppose the former scenario appears the most decent / smoothest? Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
SETMETADATA: missing annotation value
Hi there, I'm wondering whether it is me (us) doing something wrong, or whether Cyrus IMAP git origin master HEAD is at fault... C: A0019 SETMETADATA INBOX (/shared/vendor/kolab/folder-type mail /private/vendor/kolab/folder-type mail.inbox) S: A0019 BAD Missing metadata value C: A0020 SETMETADATA INBOX (/private/vendor/kolab/folder-type mail.inbox) S: A0020 BAD Missing metadata value or: C: . GETMETADATA INBOX "/shared/comment" S: * METADATA INBOX (/shared/comment NIL) S: . OK Completed C: . SETMETADATA INBOX (/shared/comment hey) S: . BAD Missing metadata value C: . SETMETADATA INBOX (/shared/comment 'hey') S: . BAD Missing metadata value C: . SETMETADATA INBOX /shared/comment 'hey' S: . BAD Missing metadata entry list Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Changing LIST (again)
On 2013-05-19 21:45, Bron Gondwana wrote: On Sun, May 19, 2013, at 11:21 PM, Jeroen van Meeuwen (Kolab Systems) wrote: On 2013-05-19 15:07, Bron Gondwana wrote: > On Sun, May 19, 2013, at 08:36 PM, Jeroen van Meeuwen (Kolab Systems) > wrote: >> Should you make the change to allow sub-folders (of the INBOX) to in >> fact be created (while using altnamespace), would that eliminate the >> use >> of \NoInferiors? > > Indeed it would :) I really, really like calling out stupid clients that do not honour \NoInferiors though... I guess everyone needs a hobby... *g* Can we please keep a sane default of (enforcing) RFC(s)-compliant behaviour? Default, sure. Thanks! Do you suggest to remove the \NoInferiors flag for deployments with "sub-folder of INBOX w/ altnamespace allowed" type of configurations, or would you keep it around (and simply behave differently by honouring the request instead of returning an error)? You kind of have to - since you want RFC-compliant clients to be able to create subfolders of Inbox. That's the whole point. Fine with me - I just wanted to understood the changes you were about to make correctly ;-) Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Cyrus IMAP 2.5 Discrete Murder
Hi there,
I've attempted to deploy a discrete murder topology setup using git master @
534066.
I run in to a particular issue attempting to create a mailbox, for which I
have attempted two scenarios:
a) have the frontends configured with no partition-default setting,
b) have the frontends configured with a partition-default of
/var/spool/imap/
The configuration is working fine for cyrus-imapd-2.4.17.
In case a), a ". CREATE user/someu...@domain.tld" results in a
"Unknown/invalid partition" error,
In case b), the mailbox does get created, but on the frontend, which of course
is not the intention, and while I would be able to reproduce the error message
I subsequently get, it's going to take me a while to move forward and then
backwards again.
Attached are the configuration files, for reference.
Kind regards,
Jeroen van Meeuwen
--
Systems Architect, Kolab Systems AG
e: vanmeeuwen at kolabsys.com
m: +44 74 2516 3817
w: http://www.kolabsys.com
pgp: 9342 BF08# standard standalone server implementation
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
# this is only necessary if using idled for IMAP IDLE
idled cmd="idled"
# Push out my current list of mailboxes to the mupdate server
mupdate cmd="ctl_mboxlist -m"
}
# UNIX sockets start with a slash and are put into /var/lib/imap/sockets
SERVICES {
# add or remove based on preferences
imapcmd="imapd" listen="imap" prefork=5
imaps cmd="imapd -s" listen="imaps" prefork=1
pop3cmd="pop3d" listen="pop3" prefork=3
pop3s cmd="pop3d -s" listen="pop3s" prefork=1
sieve cmd="timsieved" listen="sieve" prefork=0
lmtpunixcmd="lmtpd" listen="/srv/imap/config/socket/lmtp" prefork=1
# this is only necessary if using notifications
notify cmd="notifyd" listen="/srv/imap/config/socket/notify"
proto="udp" prefork=1
}
EVENTS {
# this is required
checkpoint cmd="ctl_cyrusdb -c" period=30
# this is only necessary if using duplicate delivery suppression,
# Sieve or NNTP
duplicateprune cmd="cyr_expire -E 3" at=0400
# Expire data older then 69 days. Two full months of 31 days
# each includes two full backup cycles, plus 1 week margin
# because we run our full backups on the first sat/sun night
# of each month.
deleteprune cmd="cyr_expire -E 4 -D 69" at=0430
expungeprune cmd="cyr_expire -E 4 -X 69" at=0445
# this is only necessary if caching TLS sessions
tlsprunecmd="tls_prune" at=0400
## Create search indexes regularly
#squattercmd="squatter -s -i" at=0530
}
# standard standalone server implementation
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
# this is only necessary if using idled for IMAP IDLE
idled cmd="idled"
}
# UNIX sockets start with a slash and are put into /var/lib/imap/sockets
SERVICES {
# add or remove based on preferences
imapcmd="proxyd" listen="imap" prefork=5
imaps cmd="proxyd -s" listen="imaps" prefork=1
pop3cmd="proxyd" listen="pop3" prefork=3
pop3s cmd="proxyd -s" listen="pop3s" prefork=1
sieve cmd="timsieved" listen="sieve" prefork=0
mupdate cmd="mupdate" listen="3905" prefork=1
}
EVENTS {
# this is required
checkpoint cmd="ctl_cyrusdb -c" period=30
# this is only necessary if caching TLS sessions
tlsprunecmd="tls_prune" at=0400
}
# standard standalone server implementation
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
}
# UNIX sockets start with a slash and are put into /var/lib/imap/sockets
SERVICES {
mupdate cmd="mupdate -m"listen=3905 prefork=1
}
EVENTS {
# this is required
checkpoint cmd="ctl_cyrusdb -c" period=30
# this is only necessary if caching TLS sessions
tlsprunecmd="tls_prune" at=0400
}
/vendor/horde/share-params,mailbox,string,backend,value.shared value.priv,a
/vendor/kolab/color,mailbox,string,backend,value.shared value.priv,a
/vendor/kolab/folder-test,mailbox,string,backend,value.shared value.priv,a
/vendor/kolab/folder-type,mailbox,string,backend,value.shared value.priv,a
/vendor/kolab/incidences-for,mailbox,string,backend,value.shared value.priv,a
/vendor/kolab/pxfb-readable-for,mailbox,string,backend,value.shared value.priv,a
/vendor/kolab/h-share-attr-desc,mailbox,string,backend,value.shared value.priv,a
/vendor/kolab/activesync,mailbox,string,backend,value.priv,r
/vendor/x-toltec/test,mailbox,string,backend,value.shared value.priv,aconfigdirectory: /var/lib/imap
admins: cyrus-admin
sievedir: /var/lib/imap/sieve
sendmail: /usr/sbin/sendmail
sasl_pwcheck_method: saslauthd
sasl_mech_list: PLAIN LOGIN
allowplaintext: no
tls_cert_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
tls_key_file: /etc/pki/cyrus-imapd/cyrus-imapd.pem
ACL Change notifications
Hi there,
as part of an exercise to make use of event notifications for the
purposes of auditing (non-syslog), I wanted to add an event notification
for ACL changes.
Please find attached a patch for your review, an aggregate of the work
in dev/acl-change-notification[1].
I have a couple of things I myself am pondering as well;
- ACL change notifications are not a part of any RFC as such (but for
[2]), and therefore the fields aclSubject / aclRights may need a
'vnd.cmu' prefix? Does the event name "AclChange" need a similar prefix?
- In relation to the previous consideration, this change (in part) could
relate to "Access Control List Changes in IMAP NOTIFY"[2].
- The event (type) could perhaps use a separate "event_groups" in
imapd.conf(5), but for now I stuffed them under "access" - the fields
themselves could also be subject to inclusion in event_extra_params
instead, perhaps.
Thanks, in advance,
Kind regards,
Jeroen van Meeuwen
[1]
http://git.cyrusimap.org/cyrus-imapd/log/?h=dev/acl-change-notification
[2] http://tools.ietf.org/html/rfc5465#section-5.9
--
Systems Architect, Kolab Systems AG
e: vanmeeuwen at kolabsys.com
m: +44 74 2516 3817
w: http://www.kolabsys.com
pgp: 9342 BF08diff --git a/imap/mboxevent.c b/imap/mboxevent.c
index db45821..2ad0fc7 100644
--- a/imap/mboxevent.c
+++ b/imap/mboxevent.c
@@ -72,7 +72,7 @@
#define MAILBOX_EVENTS (EVENT_MAILBOX_CREATE|EVENT_MAILBOX_DELETE|\
EVENT_MAILBOX_RENAME|EVENT_MAILBOX_SUBSCRIBE|\
- EVENT_MAILBOX_UNSUBSCRIBE)
+ EVENT_MAILBOX_UNSUBSCRIBE|EVENT_ACL_CHANGE)
#define QUOTA_EVENTS (EVENT_QUOTA_EXCEED|EVENT_QUOTA_WITHIN|EVENT_QUOTA_CHANGE)
@@ -120,6 +120,8 @@ static struct mboxevent event_template =
{ EVENT_FLAG_NAMES, "flagNames", EVENT_PARAM_STRING, 0, 0 },
{ EVENT_USER, "user", EVENT_PARAM_STRING, 0, 0 },
{ EVENT_MESSAGE_SIZE, "messageSize", EVENT_PARAM_INT, 0, 0 },
+{ EVENT_ACL_SUBJECT, "aclSubject", EVENT_PARAM_STRING, 0, 0 },
+{ EVENT_ACL_RIGHTS, "aclRights", EVENT_PARAM_STRING, 0, 0 },
/* always at end to let the parser to easily truncate this part */
{ EVENT_BODYSTRUCTURE, "bodyStructure", EVENT_PARAM_STRING, 0, 0 },
{ EVENT_MESSAGE_CONTENT, "messageContent", EVENT_PARAM_STRING, 0, 0 }
@@ -169,7 +171,7 @@ EXPORTED void mboxevent_init(void)
enabled_events |= FLAGS_EVENTS;
if (groups & IMAP_ENUM_EVENT_GROUPS_ACCESS)
- enabled_events |= (EVENT_LOGIN|EVENT_LOGOUT);
+ enabled_events |= (EVENT_LOGIN|EVENT_LOGOUT|EVENT_ACL_CHANGE);
if (groups & IMAP_ENUM_EVENT_GROUPS_MAILBOX)
enabled_events |= MAILBOX_EVENTS;
@@ -360,6 +362,10 @@ static int mboxevent_expected_param(enum event_type type, enum event_param param
return extra_params & IMAP_ENUM_EVENT_EXTRA_PARAMS_SERVICE;
case EVENT_TIMESTAMP:
return extra_params & IMAP_ENUM_EVENT_EXTRA_PARAMS_TIMESTAMP;
+case EVENT_ACL_SUBJECT:
+ return type & EVENT_ACL_CHANGE;
+case EVENT_ACL_RIGHTS:
+ return type & EVENT_ACL_CHANGE;
case EVENT_UIDNEXT:
if (!(extra_params & IMAP_ENUM_EVENT_EXTRA_PARAMS_UIDNEXT))
return 0;
@@ -621,6 +627,16 @@ EXPORTED void mboxevent_set_access(struct mboxevent *event,
}
}
+EXPORTED void mboxevent_set_acl(struct mboxevent *event, const char *identifier,
+const char *rights)
+{
+if (!event)
+ return;
+
+FILL_STRING_PARAM(event, EVENT_ACL_SUBJECT, xstrdup(identifier));
+FILL_STRING_PARAM(event, EVENT_ACL_RIGHTS, xstrdup(rights));
+}
+
EXPORTED void mboxevent_extract_record(struct mboxevent *event, struct mailbox *mailbox,
struct index_record *record)
{
@@ -979,6 +995,8 @@ static const char *event_to_name(enum event_type type)
return "MailboxSubscribe";
case EVENT_MAILBOX_UNSUBSCRIBE:
return "MailboxUnSubscribe";
+case EVENT_ACL_CHANGE:
+ return "AclChange";
default:
fatal("Unknown message event", EC_SOFTWARE);
}
@@ -1178,6 +1196,12 @@ EXPORTED void mboxevent_set_access(struct mboxevent *event __attribute__((unused
{
}
+EXPORTED void mboxevent_set_acl(struct mboxevent *event __attribute__((unused)),
+const char *identifier __attribute__((unused)),
+>...>...>...>...const char *rights __attribute__((unused)))
+{
+}
+
EXPORTED void mboxevent_extract_record(struct mboxevent *event __attribute__((unused)),
struct mailbox *mailbox __attribute__((unused)),
struct index_record *record __attribute__((unused)))
diff --git a/imap/mboxevent.h b/imap/mboxevent.h
index 1171bd2..9a85927 100644
--- a/imap/mboxevent.h
+++ b/imap/mboxevent.h
@@ -79,10 +79,11 @@ enum event_type {
EVENT_MAILBOX_DELETE = (1<<16),
EVENT_MAILBOX_RENAME = (1<<17),
EVENT_MAILBOX_SUBSCRIBE = (1<<18),
-EVENT_MAILBOX_UNSUBSCRIBE = (1<<19)
+EVENT_MAILBOX_UNSUBSCRIBE = (1<<19),
+EVENT_ACL_CHANGE = (1<<20),
};
-#define MAX_PARAM 21 /* messageContent number that is always the last */
+#define MAX_PARAM 23 /* messageContent number that is always the l
multi-domain/multi-rootdn for ptclient/ldap.c
Hi there,
I've previously (a long time ago, actually, too long if you ask me) made
inquiries as to who might be using ptclient/ldap.c[1,2], and in which
fashion; I got three points from the responses;
- Everything should be configurable as LDAP deployments typically vary
widely and often pre-date a Cyrus IMAP deployment[3],
- It should handle groups better[4], namely nested/recursive groups,
the 'memberOf' attribute,
- It should handle memberUrls[5].
While these are all valid points and worth working on for me as well,
today I have another aspect; the handling of multi-domain deployments,
with isolated root dns for each parent domain. A very ugly and
presumptuous patch is attached, that needs extra careful checking and a
nice cleanup.
This scenario (of multiple domains separated in to multiple, different
root DNs) is widely in use with Kolab Groupware, while canonification
nor group ACLs would work.
The scenario for such a deployment could be described as follows:
- A list of objectClass=domainRelatedObject LDAP objects exists in
cn=kolab,cn=config.
- A domain "example.org" may have a root dn of "dc=example,dc=org",
and would be an LDAP entry as follows:
dn: associatedDomain=example.org,cn=kolab,cn=config
objectClass: top
objectClass: domainRelatedObject
associatedDomain: example.org
- To translate "example.org" to "dc=example,dc=org" in this particular
case, the C equivalent of:
$root_dn = 'dc=' . implode(',dc=', explode('.', "example.org"));
can be used.
- Another domain "example.com" mayhave a root dn of "o=example,c=de",
and could be an LDAP entry as follows:
dn: associatedDomain=example.com,cn=kolab,cn=config
objectClass: top
objectClass: domainRelatedObject
objectClass: inetDomain
associatedDomain: example.com
inetDomainBaseDN: o=example,dc=de
- Here, the inetDomainBaseDN attribute should be used to translate a
user login of "lucy.me...@example.com" to a search against
"o=example,dc=de".
This leads me to believe the following items should be configurable:
- domain_base_dn
The base dn to use when searching for "domains" or "domain name
spaces".
For Kolab Groupware deployments, this is a default of
"cn=kolab,cn=config", but could of course be
"ou=Domains,dc=domain,dc=tld" as well.
- domain_filter
The filter to use.
Perhaps something like
"(&(objectClass=domainRelatedObject)(inetDomainStatus=on)(associatedDomain=%s))"
- domain_scope
The search scope. "sub", "one" or "base", with a default of "sub".
- domain_name_attribute
The attribute name for actual domain name spaces, such as
"associatedDomain".
For LDAP deployments without the Netscape schemas I suppose this
attribute name might be "cn".
For situations in which a domainRelatedObject does not contain one,
but multiple values for the domain_name_attribute, the first value
returned by LDAP is used (this typically corresponds with the relative
DN attribute value, and should be consistent).
- domain_result_attribute
Name of the attribute to look for, for example "inetDomainBaseDN".
If no such attribute exists (for the object found), fall back to the
"standard" root dn described above (the implode over explode thing).
I would appreciate your thoughts and feedback.
Thanks, in advance,
Kind regards,
Jeroen van Meeuwen
[1]
http://lists.andrew.cmu.edu/pipermail/cyrus-devel/2011-August/001923.html
[2]
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2011-August/035257.html
[3]
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2011-August/035259.html
[4]
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2011-August/035262.html
[5]
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2011-August/035294.html
--
Systems Architect, Kolab Systems AG
e: vanmeeuwen at kolabsys.com
m: +44 74 2516 3817
w: http://www.kolabsys.com
pgp: 9342 BF08diff --git a/lib/imapoptions b/lib/imapoptions
index ecb54ef..0725fd9 100644
--- a/lib/imapoptions
+++ b/lib/imapoptions
@@ -597,6 +597,21 @@ Blank lines and lines beginning with ``#'' are ignored.
{ "ldap_deref", "never", STRINGLIST("search", "find", "always", "never") }
/* Specify how aliases dereferencing is handled during search. */
+{ "ldap_domain_base_dn", "", STRING }
+/* Base DN to search for domain name spaces. */
+
+{ "ldap_domain_filter", "(&(objectclass=domainrelatedobject)(associateddomain=%s))", STRING }
+/* Filter to use searching for domains */
+
+{ "ldap_domain_name_attribute", "associateddomain", STRING }
+/* The attribute name for domains. */
+
+{ "ldap_domain_scope", "sub", STRINGLIST("sub", "one", "base") }
+/* Search scope */
+
+{ "ldap_domain_result_attribute", "inetdomainbasedn", STRING }
+/* Result attribute */
+
{ "ldap_filter", "(uid=%u)", STRING }
/* Specify a filter that searches user identifiers. The following tokens can be
used in the filter string:
diff --git a/ptclient/ldap.c b/ptclient/ldap.c
i
Re: FastMail Xapian patches
On 2014-01-18 02:08, Bron Gondwana wrote:
The version of Xapian that we build against has the following patches
applied to the upstream Debian xapian-core before building:
rm -rf .pc # remove memory of the debian quilt series
cp -a $FMPATCHES/xapian_quilt patches
QUILT_PATCHES=patches quilt push -a || exit
autoreconf || exit
I've attached a tarball with the patches in it.
Thanks for these, I've built (without *DAV) using the following patch
attached.
The conundrum is the build dependencies are not required by configure,
but are required by the code, if --enable-http is not given.
Kind regards,
Jeroen van Meeuwen
--
Systems Architect, Kolab Systems AG
e: vanmeeuwen at kolabsys.com
m: +44 74 2516 3817
w: http://www.kolabsys.com
pgp: 9342 BF08diff --git a/imap/mailbox.c b/imap/mailbox.c
index f9ae70a..3baf44e 100644
--- a/imap/mailbox.c
+++ b/imap/mailbox.c
@@ -51,7 +51,9 @@
#elif defined(HAVE_STDINT_H)
# include
#endif
+#ifdef WITH_DAV
#include
+#endif
#include
#include
#include
@@ -81,8 +83,10 @@
#include "annotate.h"
#include "assert.h"
+#ifdef WITH_DAV
#include "caldav_db.h"
#include "carddav_db.h"
+#endif
#include "crc32.h"
#include "md5.h"
#include "exitcodes.h"
@@ -2666,6 +2670,7 @@ out:
return r;
}
+#ifdef WITH_DAV
static int mailbox_update_carddav(struct mailbox *mailbox,
struct index_record *old,
struct index_record *new)
@@ -2886,6 +2891,7 @@ static int mailbox_update_dav(struct mailbox *mailbox,
return mailbox_update_caldav(mailbox, old, new);
return 0;
}
+#endif //WITH_DAV
EXPORTED int mailbox_update_conversations(struct mailbox *mailbox,
struct index_record *old,
@@ -3112,8 +3118,10 @@ static int mailbox_update_indexes(struct mailbox *mailbox,
{
int r = 0;
+#ifdef WITH_DAV
r = mailbox_update_dav(mailbox, old, new);
if (r) return r;
+#endif
r = mailbox_update_conversations(mailbox, old, new);
@@ -4238,6 +4246,7 @@ static int chkchildren(char *name,
return r;
}
+#ifdef WITH_DAV
EXPORTED int mailbox_add_dav(struct mailbox *mailbox)
{
struct index_record record;
@@ -4257,6 +4266,7 @@ EXPORTED int mailbox_add_dav(struct mailbox *mailbox)
return 0;
}
+#endif
EXPORTED int mailbox_add_conversations(struct mailbox *mailbox)
{
diff --git a/imap/mailbox.h b/imap/mailbox.h
index 5d3ed01..fe0bf91 100644
--- a/imap/mailbox.h
+++ b/imap/mailbox.h
@@ -587,6 +587,8 @@ extern int mailbox_get_xconvmodseq(struct mailbox *mailbox, modseq_t *);
extern int mailbox_update_xconvmodseq(struct mailbox *mailbox, modseq_t, int force);
extern int mailbox_has_conversations(struct mailbox *mailbox);
+#ifdef WITH_DAV
extern int mailbox_add_dav(struct mailbox *mailbox);
+#endif
#endif /* INCLUDED_MAILBOX_H */
diff --git a/imap/mboxevent.c b/imap/mboxevent.c
index eabc462..13f0945 100644
--- a/imap/mboxevent.c
+++ b/imap/mboxevent.c
@@ -53,8 +53,10 @@
#include "annotate.h"
#include "assert.h"
+#ifdef WITH_DAV
#include "caldav_db.h"
#include "carddav_db.h"
+#endif
#include "exitcodes.h"
#include "imapurl.h"
#include "libconfig.h"
@@ -131,9 +133,11 @@ static struct mboxevent event_template =
{ EVENT_USER, "user", EVENT_PARAM_STRING, 0, 0 },
{ EVENT_MESSAGE_SIZE, "messageSize", EVENT_PARAM_INT, 0, 0 },
{ EVENT_MESSAGE_CID, "vnd.fastmail.cid", EVENT_PARAM_STRING, 0, 0 },
+#ifdef WITH_DAV
{ EVENT_MBTYPE, "vnd.cmu.mbtype", EVENT_PARAM_STRING, 0, 0 },
{ EVENT_DAV_FILENAME, "vnd.cmu.davFilename", EVENT_PARAM_STRING, 0, 0 },
{ EVENT_DAV_UID, "vnd.cmu.davUid", EVENT_PARAM_STRING, 0, 0 },
+#endif
/* always at end to let the parser to easily truncate this part */
{ EVENT_ENVELOPE, "vnd.cmu.envelope", EVENT_PARAM_STRING, 0, 0 },
{ EVENT_BODYSTRUCTURE, "bodyStructure", EVENT_PARAM_STRING, 0, 0 },
@@ -374,8 +378,10 @@ static int mboxevent_expected_param(enum event_type type, enum event_param param
return extra_params & IMAP_ENUM_EVENT_EXTRA_PARAMS_VND_FASTMAIL_SESSIONID;
case EVENT_MAILBOX_ID:
return (type & MAILBOX_EVENTS);
+#ifdef WITH_DAV
case EVENT_MBTYPE:
return (type & MAILBOX_EVENTS);
+#endif
case EVENT_MAX_MESSAGES:
return type & QUOTA_EVENTS;
case EVENT_MESSAGE_CONTENT:
@@ -387,12 +393,14 @@ static int mboxevent_expected_param(enum event_type type, enum event_param param
case EVENT_MESSAGE_CID:
return (extra_params & IMAP_ENUM_EVENT_EXTRA_PARAMS_VND_FASTMAIL_CID) &&
(type & (EVENT_MESSAGE_APPEND|EVENT_MESSAGE_NEW));
+#ifdef WITH_DAV
case EVENT_DAV_FILENAME:
return (extra_params & IMAP_ENUM_EVENT_EXTRA_PARAMS_VND_CMU_DAVFILENAME) &&
(type & EVENT_CALENDAR);
case EVENT_DAV_UID:
return (extra_params & IMAP_ENUM_EVENT_EXTRA_PARAMS_VND_CMU_DAVUID) &&
(type & EVENT_CALENDAR);
+#endif
case EVENT_MESSAGES:
if (type & (EVENT_QUOTA_EXCEED|EVENT_QUOTA_WITHIN))
return 1;
@@ -738,6 +746,7 @@ EXPORTED void mboxevent_extract_record(struct mboxevent *event, struct mai
Re: Any ETA for next release?
On 2014-06-24 13:30, Ondřej Surý wrote: Hi, Debian is already shipping beta with caldav support, and we have a freeze scheduled in this fall. It would be a nice to have an official release in next stable Debian. Are there any plans to finish caldav and release it as a final version? Kolab Groupware 3.3 (^1) is shipping Cyrus IMAP 2.5 from git origin master -- hence the recent series of commits from yours sincerely. We don't have much in terms of release planning, albeit we have a wishlist; neither of those features are easily mergeable back in to master (except for few involved who have other things to do). I have to say it's a shame we have to go willy-nilly about our own itches (many trees out there have diverged a lot) and don't move forward Cyrus IMAP as a whole, collaboratively, together. ^1: To be released this August some time. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Patch for adding tls_honor_cipher_order
On 2014-10-16 19:32, Kristian Kræmmer Nielsen wrote: Hi, Patch attached. Something similar is already in cyrus-imapd-2.4: http://git.cyrusimap.org/cyrus-imapd/commit/?h=cyrus-imapd-2.4&id=4b26d2d7244eeaa481871c337e57cd393fd76dfe For master / 2.5, I have a push pending of a similar nature, while it also addresses some client vs. server certificate chain configuration options (i.e. Internet-facing public CA, verify client certificates against private CA, offer client certificates between Cyrus IMAP servers, and allow requiring certs to be set to "optional" or "required"). Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Pull Request: Sieve imap4flags extension
On 2014-10-20 13:11, James Cassell wrote:
Hello,
After over a year since I began, I have finished implementing the
imap4flags extension. It is ready for 2.5.
Hi James,
I've pushed your enhancement, all individual commits nicely rebased
against the then current master HEAD, nicely done!
Just in time for 2.5.0 -- slated to be alpha'd and beta'd one of these
next few weeks, Bron/Ken/Dave are having face-to-face sessions about it.
Could you please make sure a full build of 531b18a works exactly the way
you intended? I've verified (quickly, dirty) things build and run, but
not the functionality.
If you're interested, I have my autogen.sh script attached representing
the way I also build RPM packages -- makes it nice and reproducible ;-)
Kind regards,
Jeroen van Meeuwen
--
Systems Architect, Kolab Systems AG
e: vanmeeuwen at kolabsys.com
m: +41 79 951 9003
w: http://www.kolabsys.com
pgp: 9342 BF08#!/bin/bash -x
git_branch=$(git symbolic-ref HEAD | sed -e 's|refs/heads/||g')
git_branch_friendly=$(echo ${git_branch} | sed -e 's|/|_|g')
git_remote=$(git config branch.${git_branch}.remote)
(
automake --add-missing
libtoolize
autoconf -v
autoreconf -v --install
export CPPFLAGS='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/include/et -I/usr/include/kerberosIV'
export CFLAGS='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC'
export CXXFLAGS='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC'
export CCDLFLAGS=-rdynamic
export LDFLAGS=' -pie'
export CFLAGS='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC'
export CXXFLAGS='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC'
export FFLAGS='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -I/usr/lib64/gfortran/modules'
./configure \
--build=x86_64-redhat-linux-gnu \
--host=x86_64-redhat-linux-gnu \
--target=x86_64-redhat-linux-gnu \
--program-prefix= \
--prefix=/usr \
--exec-prefix=/usr \
--bindir=/usr/bin \
--sbindir=/usr/sbin \
--sysconfdir=/etc \
--datadir=/usr/share \
--includedir=/usr/include \
--libdir=/usr/lib64 \
--libexecdir=/usr/libexec \
--localstatedir=/var \
--sharedstatedir=/var/lib \
--mandir=/usr/share/man \
--infodir=/usr/share/info \
--enable-autocreate \
--enable-event-notification \
--enable-gssapi \
--enable-idled \
--enable-maintainer-mode \
--enable-murder \
--enable-netscapehack \
--enable-nntp \
--enable-replication \
--enable-unit-tests \
--enable-xapian \
--without-bdb \
--with-cyrus-prefix=/usr/lib/cyrus-imapd \
--with-extraident=Kolab-2.5-0.1.dev20140802.el6 \
--without-wrap \
--with-krbimpl=mit \
--with-ldap=/usr \
--with-perl=/usr/bin/perl \
--with-service-path=/usr/lib/cyrus-imapd \
--with-snmp \
--with-syslogfacility=MAIL
make clean
make -B -j4
make check
) 2>&1 | tee ${TMPDIR:-/tmp}/cyrus-imapd-${git_remote}.${git_branch_friendly}-$(date +'%s').log
Re: POLL: per-domain shared folder/sieve/etc
On 2014-10-22 23:02, Bron Gondwana wrote: Yes, that means a massive change, instead of internally: example.com!user.foo.bar <=> user/foo/b...@example.com (which is a million ways of bogus) we would have: user.foo@example^com.bar <=> user/f...@example.com/bar Or in alt namspace: Other Users/f...@example.com/bar This means we will finally be able to share things across domains. It creates a single consistent way to access everything. The "domain" used to be used as an "authorization realm", where a user j...@example.com would only see "Other Users/foo/bar" -- without the domain. How would this translate to the new way? If the external name (the new default) uses unix hierarchy separators, would it be reasonable to update the internal format to that as well, and translate "user/foo/b...@example.org" or "user/f...@example.org/bar" back to using the netnews hierarchy separator if so configured? The problem is, it means you can't set quotas per domain, you can't have sieve scripts per domain, and most of all - you can't have shared folders in a domain. example.com!shared.stuff worked fine, but shared.example^com.stuff would be weird. It's just a folder, and wouldn't be treated specially in any way. The domain would have no special meaning. This is now shared/st...@example.org, I suppose a hierarchy of such folders would lead to shared/st...@example.org/something? This is all, obviously, Cyrus 3.0 stuff. In the multi-domain environments we typically run, while sharing between domains is indeed an often requested feature, we love the inability to share cross-realm -- preventing accidentally sharing your @company.com content with @competitor.com people. If the new way of doing things is to allow cross-realm sharing, I would like to ensure some sort mandatory access policy is in place, where one has to specify @something can in fact share with @else. On 2014-10-24 02:59, Bron Gondwana wrote: No, the per-user namespace is still fine - users can still share with other users in their own domain - just currently it is technically impossible to share with users in other domains right now - because the mailbox naming is not RFC compliant, so it's not compatible with real IMAP client, only with Cyrus management tools. You mentioned in another post (quote above) that the current naming of mailboxes is not necessarily RFC-compliant, and that only the Cyrus tooling is compatible. I may be misunderstanding what this means, because only an administrator without a realm (as part of its login username) is currently able to view lists of mailboxes across realms -- bear with me while I transcribe from the top of my head: Settings: virtdomains: userid admins: cyrus-admin ad...@example.org cyrus-admin: C: . LIST "" "*" S: * user/j...@company.com S: * user/j...@example.org S: * user/m...@example.org ad...@example.org: C: . LIST "" "*" S: * user/jane S: * user/max j...@example.org: C: . LIST "" "*" S: * INBOX S: * Other Users/max Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: http://www.kolabsys.com pgp: 9342 BF08
Re: POLL: per-domain shared folder/sieve/etc
On 2014-11-01 21:29, Bron Gondwana wrote: We already have one at FastMail to stop users setting an 'anyone' ACL. I think this may already be in upstream, unless you're talking about a different implementation/solution? http://git.cyrusimap.org/cyrus-imapd/tree/lib/imapoptions#n179 > This is all, obviously, Cyrus 3.0 stuff. > In the multi-domain environments we typically run, while sharing between domains is indeed an often requested feature, we love the inability to share cross-realm -- preventing accidentally sharing your @company.com content with @competitor.com people. Yes, that's pretty dangerous, isn't it. If the new way of doing things is to allow cross-realm sharing, I would like to ensure some sort mandatory access policy is in place, where one has to specify @something can in fact share with @else. This is tricky, particularly for FastMail where multiple companies can have addresses in the shared domains (e.g. fastmail.com) as well. It sounds like the right general approach is to allow an external daemon to be queried for policy responses. I suppose the level of complexity depends on the level of flexibility / features required. Suppose that the default is to not have any realm be allowed to use any other realm (no other realm's mailboxes are visible, no ACL subject identifiers validate). This, I would say, represents the current situation most accurately. Suppose a list of source realms (the authenticated user is in this realm) is used as a lookup key, and for any other realm that realm must have presence in the lookup value) -- admittedly a very simplistic view: company.com: subsidiary.com partner.com subsidiary.com: company.com partner.com: company.com competitor.com competitor.com: partner.com Suppose this means that @company.com people (source, lookup key) can share @company.com mailboxes (source, lookup key, must match logged in account?) with @subsidiary.com and with @partner.com ACL subject identifiers. Suppose this means @partner.com (target lookup value) can therefore see @company.com mailboxes -- but cannot share with @company.com because of the first rule, but because of the third rule. I do not suppose there is any use-case to nesting such rules to the tune to, in the above list, allow subsidiary to partner descending to company authorizations (or any other way). I suppose in the case of FastMail, you would list fastmail.com as a lookup key and perhaps use a regular expression .* to ensure @fastmail.com mailboxes are visible to all other realms? Of course, to a certain degree this is trying to make a technical solution to a human problem. If it's that vital that they can't see each other, they should be on separate Cyrus instances at the very least, if not entirely separate infrastructure. There's nothing to stop mall...@example.org just adding a sieve rule to forward a copy of everything to j...@company.com, or handing over credentials, or any number of things. I agree with the general sentiment, but disagree with such separation on the infrastructure level being that kind of a must (for that level of vital). There are other considerations that could require an organization to have infrastructure isolated from a multi-customer "hosted" environment, but such are in the gut-feeling-more-so-than-technical-literacy, compliance and telco regulatory domains. While "to share or not" is certainly a social problem, and "to enable sharing or not" probably is so too, "to allow sharing or not" is definitely a more technical one if the implementation thereof leaves behind a Free-for-All. For Sieve rules forwarding content, cross-realm ACLs are rather irrelevant. One could (define to) forward content using Sieve regardless, unless Cyrus IMAP's Sieve implementation is extended to allow a similar level of access control. The current methodology to prevent this from happening lays in limiting the user interfaces, not including Sieve extensions, MTA configuration and Data-Loss Prevention systems -- neither of which are helped or negated by introducing cross-realm ACLs, and neither of which requires a given customer to run off of completely separate infrastructure. Should sharing across domains be allowed, but without mandatory access control, however, then you move Cyrus IMAP from the "perfect for hosted environments with multiple customers" universe to the "it's such a Free-for-All you require separate infrastructure for each customer". On 2014-10-24 02:59, Bron Gondwana wrote: > No, the per-user namespace is still fine - users can still share with > other users in their own domain - just currently it is technically > impossible to share with users in other domains right now - because the > mailbox naming is not RFC compliant, so it's not compatible with real > IMAP client, only with Cyrus management tools. > You mentioned in another post (quote above) that the current naming of mailboxes is not necessarily
Mailbox URI format in Event Notifications
Hello there, In relation to event notifications that include a mailbox uri (almost all of them), I have the following questions; - For a user-prefix, the owner of the personal namespace is used, if any, such that a user john doing something to a user jane mailbox, ends up in the URI as follows: imap://j...@imap.example.org/Other Users/jane while it was actually not 'jane' doing anything. - Like in the aforementioned URI, the "external" mailbox name seems to be used, from the perspective of the user triggering the action. It seems to me that j...@imap.example.org though has no "Other Users/jane", this would rather be the INBOX for her. For those of you using event notifications, I'm wondering how you make other software interpret these things -- our "other" software looks at everything from the administrative perspective, and so we'd opt for a format of imap://imap.example.org/user/j...@example.org -- but I'm suspecting this may have implications I'm unaware of. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Mailbox URI format in Event Notifications
On 2014-11-05 11:35, Bron Gondwana wrote: On Wed, Nov 5, 2014, at 09:28 PM, Jeroen van Meeuwen (Kolab Systems) wrote: For those of you using event notifications, I'm wondering how you make other software interpret these things -- our "other" software looks at everything from the administrative perspective, and so we'd opt for a format of imap://imap.example.org/user/j...@example.org -- but I'm suspecting this may have implications I'm unaware of. Totally agree, that's how it should be. I guess we aren't using altnamespace, so we didn't notice - but I'd agree with both the location of the domain (notice how I'm wanting that elsewhere too) and of course the use of non-alt namespace for administrative things like events. Could you tell me more about how you are using event notifications? We use event notifications for the sake of audit trails, in that client applications do not currently "consume" the event notifications nor their payload, so what URI is being used in the notification payload is preferably (for us) a consistent URI (i.e. the one from a "global", not domain-specific administrator). Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Mailbox URI format in Event Notifications
On 2014-11-05 12:27, Bron Gondwana wrote: So we do a ton of stuff with them now :) Most importantly, they feed into the EventSource pipeline for web browser clients to get immediate updates, and likewise the Apple and Google push notification channels if you have logged in with our app on those platforms. We also have alarms from the calendar as events. It's kind of our go-to hammer ;) Does the payload for and/or format of the uri parameter matter anywhere? Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Mailbox URI format in Event Notifications
On 2014-11-05 15:16, Sébastien Michel wrote: For those of you using event notifications, I'm wondering how you make other software interpret these things -- our "other"software looks at everything from the administrative perspective, and so we'd opt for a format of imap://imap.example.org/user/j...@example.org -- but I'm suspecting this may have implications I'm unaware of. Not sure that 'imap://imap.example.org/user/j...@example.org' is right because the format defined in the RFC 5092 - IMAP URL : imap:///[][][][] The component common to all types of absolute IMAP URLs has the following syntax expressed in ABNF [ABNF]: [iuserinfo "@"] host [ ":" port ] If I understand this correctly, the uri here then is defined to have to be a uri that a client could interpret and follow and end up OK with? This makes me wonder why it would be the mailbox owner being in the spot of , because imap://j...@imap.example.org/Other Users/jane will not likely be understood correctly by clients -- as "jane", "Other Users/jane" will not exist. As not jane, the notification is not necessarily invalid nor irrelevant. Neither would, I suppose, imap://j...@imap.example.org/INBOX make all that much sense -- not unless a client is taught to understand "jane@'s INBOX is Other Users/jane". The jane@ part should go away or be substituted for the actual user that had the perspective of Jane's INBOX being at "Other Users/jane" when triggering the event notification. If user john is doing something to a user jane mailbox, the notification should contain the user john in the field "user" and the URI imap://j...@imap.example.org/INBOX in the fields "uri" and "mailboxID". This is how I understand the RFC 5423. Many of the event notifications have turned out to not contain the "user" triggering the event at all, and I suppose I'll need to double-check if it is in fact the authenticated user id being used, or the mailbox owner. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Possible TLS dir option name discrepancy?
On 2015-01-12 13:23, Patrick Goetz wrote: The 2.5 documentation here (http://www.cyrusimap.org/~vanmeeuwen/imap/release-notes/2.5.0.html) states that some of the TLS options will change in 2.5, namely tls_client_ca_dir (was: tls_ca_dir) However, there is no tls_ca_dir option given here (https://cyrusimap.org/docs/cyrus-imapd/2.4.17/man/imapd.conf.5.php). I've been using tls_ca_path as per the 2.4.17 man page. Am I missing something, or is this just a typo in the 2.5 documentation? Typo. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Meeting Notes Feb 20
On 2015-02-20 12:00, Matt wrote: On 20/02/15 10:25, Robert Norris wrote: I'm happy to help out with infrastructure support and other random tech crap if necessary - especially for the busy programmers and documenters and testers that have better things to do that figuring out how to install random $server. Just yell on this list or privately or in #cyrus or whatever. Bron, you own my time anyway, so volunteer me for stuff if you like :) I have a DevOps background, so I can do infrastructure stuff and code stuff as well; between Rob and myself we could make reasonably sure there is someone available at most times (I'm in the UK so we'd have most of the clock covered). This sort of builds like a team, and every team needs one throat to choke -- pardon my French -- and of course some consensus on what is a standard install vs. the odd ones out. I've proposed some tooling around Cyrus IMAP and Cyrus SASL development that we're going to try and test to see if it would work for the development process we're employing. Whether or not it is going to be Phabricator notwithstanding, others use this for their operations as well, the tooling chosen preferably includes the ops team, right? Steering an environment from within some source code management system seems to imply a level of configuration management. I myself am a particular fan of Puppet, but I can see it disappear in the (near) future, and it is likely already surpassed by (at least) Ansible. Then of course there is always the question of the favorite distro to make the de-facto standard. I do not want to start a discussion on these (often controversial) topics, but I do want to suggest that the aforementioned throat to choke sets the tone ;-) Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Test installation of Phabricator
Hi there, I have a test installation of Phabricator with Cyrus IMAP and SASL projects and GIT repositories imported, and so it is time to start stabbing at what tooling and work-flows would look like if we were to roll with this tooling. At the moment, I have created various user accounts for people (Bron, Ken, Robert, Matt) to get some key positions filled, and partly because of that I've suspended email delivery for this system (otherwise the placeholder accounts on these key posititions would have gotten spammed over the course of this weekend). This means that everyone needs to contact me -- I require a full name and email address for an account, preferably over IRC (kanarip on FreeNode, I'm in the #cyrus channel in case you can't find me). I'd love to get some more hands on the system and see how hard we can poke at it before we decide whether or not to actually run with it in full force, or ditch it altogether / explore other tooling, but from what I'm seeing I'm liking what we have. Big fat note: if you're willing to entertain a self-signed cert, substitute the scheme for the URLs below with https; SHA-256 fingerprint is: 34:C6:3D:65:17:E9:61:C7:FA:9E:3E:4F:46:A7:16:CE:81:AD:03:34:1A:C1:F0:64:15:86:2E:1E:3D:17:BA:CE That said, don't expect "Arcanist" to work (the cert ain't trusted, see T4, T5, T6 and T7), and therefore just in general -- don't use any sort of valuable information anywhere near this system. For now, this installation needs to be understood as demonstrative, such that; - We've figured out how to get a project (IMAP, SASL, Documentation, System Engineering) going without; 1) Closing off joining such project only to those who have commit access, yet 2) Provide direct commit access to only those who are "blessed" developers, and 3) Yet allow "the general public" to propose changes under an established process, without 4) involuntarily cluttering up the kanboards of "blessed" developers, in that 5) those that are "blessed" developers can voluntarily join the group of "reviewers" to assert themselves in to the loop for proposed changes from "the general public". all of which I think is pretty fruitful for a weekend. - T4 (at http://95.128.36.13/T4) Illustrates how a task squarly on Bron's plate might block other tasks on my plate (T5), which in turn may block other (yet unassigned) tasks (T6, T7). - T11 (at http://95.128.36.13/T11) is not actually a goal, albeit it may become one at some point, but illustrates how a kanboard Task item can depend on proposed changes (that contain code to backup the proposed changes) that are subject to review by "blessed" developers (who's opted in to review of contributions from the "general public", which to them then also become kanboard items). - Wiki pages like http://95.128.36.13/w/arcanist_workflow/ (which is not to say these do not need any further work), - Herald (the pre- and post-commit rule engine) can trigger an Audit^1 of certain changes, where an "Audit" needs to be understood as a non-blocking "Review", whereas a "Review" normally is "blocking". There's much more I can't really go in to right now -- my family needs feeding too -- so I hope you pop in to #cyrus and start asking questions. Kind regards, Jeroen van Meeuwen ^1: For example, changes to lib/imapoptions in cyrus-imapd.git need an "Audit" from the Documentation team -- not because they want to be saying yay or nay, but because documentation may require updating. -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Test installation of Phabricator
On 2015-02-22 16:52, Jeroen van Meeuwen (Kolab Systems) wrote: Big fat note: if you're willing to entertain a self-signed cert, substitute the scheme for the URLs below with https; SHA-256 fingerprint is: 34:C6:3D:65:17:E9:61:C7:FA:9E:3E:4F:46:A7:16:CE:81:AD:03:34:1A:C1:F0:64:15:86:2E:1E:3D:17:BA:CE Correction, Phabricator does not use // for its assets and will start throwing all sorts of errors unless you hit it over https:// first and accept the certificate. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Minutes 23 Feb
On 2015-02-23 14:31, Matt wrote: On 23/02/15 13:19, Bron Gondwana wrote: [snip snip] Matt: * splitting lib/imap code into sensible directories and heirarchy * configuration parsing * potentially SNMP/statistics code (I have an email from Greg somewhere detailing what's needed here) [snip snip] I could be wrong here but I think maybe Jeroen was doing work on the config system; I'm happy to defer to him or whatever makes it easiest to get it done. I see my role as supporting the main coders; so although I have preferences I have no agenda and I'm happy to do whatever people feel helps them the most. I'll take it as a compliment, but I'm not a main coder :P My configuration work set out to achieve two things; - More consistent naming convention for options (so that all autocreate functional options indeed start with autocreate_, etc.), - Correct the naming for new options introduced (such as the separate server cert+key for inbound connections, a separate server CA for outbound connections (murder/proxy), and separate settings for checking client certificates for authentication). This resulted in coding an "obsoleted setting warning" system, but that as far as I've needed to go. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Test installation of Phabricator
On 2015-02-22 16:52, Jeroen van Meeuwen (Kolab Systems) wrote: Big fat note: if you're willing to entertain a self-signed cert, substitute the scheme for the URLs below with https; SHA-256 fingerprint is: 34:C6:3D:65:17:E9:61:C7:FA:9E:3E:4F:46:A7:16:CE:81:AD:03:34:1A:C1:F0:64:15:86:2E:1E:3D:17:BA:CE Now that the name-servers for the zone have been updated, you can start using https://git.cyrus.foundation/ Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Mailbox deletion race: folders and files are never deleted
On 2015-02-26 16:51, Samir Aguiar wrote: Proposed solution: - Patch the mailbox_close() function to reload the index before trying to clean the files (and merge the current index changes with the ones found) - Because the above would still be skipped when shutting down Cyrus: expand cyr_expire to find mailboxes with the deleted flag set (by looping through the filesystem, and not by using the database) and try to remove them. This could then be run periodically by administrators. What do you think? Please do not take my word for it, but I think Cyrus IMAP 2.5 introduces a state flag of MBTYPE_DELETE for mailboxes [1] in the actual mboxlist, so the related functions would still be able to find it after the facts of the case. IIRC, I ran in to this (particular MBTYPE) because it purges ACLs, and rendered (discrete) murder topologies unable to ctl_mboxlist -m (as well as made ctl_mboxlist -d segfault). I think the intention is exactly what you describe is the scenario, but if it is I'm not certain it also properly addresses it. Kind regards, Jeroen van Meeuwen [1] https://git.cyrus.foundation/diffusion/I/browse/master/imap/mboxlist.h;4571c101e542f169cfd3b70347b47eaea8f2deea$70 -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Meeting notes 26/2
On 2015-02-27 13:17, Bron Gondwana wrote: Jeroen: * Phabricator up. * git.cyrus.foundation DNS set up. Getting SSL key. Done. * Can set up build integration with CI system. Done, but the callbacks are wonky. Will see what we can achieve together with Chris. * Need to create accounts (Bron especially) and get people access Done (for Bron especially), and registration is now open, no admin approval required. * Will activate email sending soon - can get quite spammy Done, and messages should be sent out with a nicely filterable From: phabricator@cyrus.foundation. * Question: do we want to use the full workflow built into Phabricator? Conclusion: yes, let's try it for at least a few weeks. Ellie has gone through "Arcanist" motions once, but it resulted in me stealing a commit from her. This should now be resolved, and the reviewer should land the revision. As much is now documented in the workflow documentation [1]. * Magic switchover point to git.c.f being the primary git commit point? Next week. * Will move everything, including legacy branches As much is already in. Some questions remain about some cyrus-sasl-* tags in the cyrus-imapd repository, and whether or not we should push these, but I haven't examined them yet and so there's no qualified opinion on the matter from my side. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Meeting notes 26/2
On 2015-02-27 21:50, Jeroen van Meeuwen (Kolab Systems) wrote: On 2015-02-27 13:17, Bron Gondwana wrote: * Question: do we want to use the full workflow built into Phabricator? Conclusion: yes, let's try it for at least a few weeks. Ellie has gone through "Arcanist" motions once, but it resulted in me stealing a commit from her. This should now be resolved, and the reviewer should land the revision. As much is now documented in the workflow documentation [1]. Sorry, where [1] is https://git.cyrus.foundation/w/arcanist_workflow/ It should be noted that going through this process is open for any registered user, and not just people with commit access. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Status on Cyrus IMAP 2.5.0 Release
Hi there,
This is a short status message on the pending 2.5.0 release, which is
lagging behind our schedule of March 1st.
While Bron was up late last night, and I got involved this morning, we
discovered that our shift to the otherwise awesome autofoo left some
important files excluded from a tarball release if composed with "make
dist".
We had a choice to make in either supporting the downloading of a
tarball and requiring autofoo to get a "./configure" file built, or only
"./configure; make; sudo make install".
The choice became obvious as soon as we figured that docs/ tends to be
published on www.cyrusimap.org and we did not want to void publishing
that with 2.5.0 quite yet.
From our first attempt at setting a version number of 2.5.0, the
following summarizes the amount of changes needed somewhat accurately,
to what I believe enables us to continue to use "make dist" for the
release tarball:
$ git diff 777a1b4624d92bf657f4fde1fdff988114023251 --stat
.gitignore| 2 +
Makefile.am | 238
+---
configure.ac | 44 -
imap/mbdump.c | 2 +
4 files changed, 236 insertions(+), 50 deletions(-)
All in all, while I want Bron to review and approve the changes tomorrow
morning, before we finally push the button, it could use some feedback
from you as well.
The following segment is what I have used to build and sanity-check what
is in the tarball:
autoreconf -vi && \
./configure --enable-maintainer-mode && \
make && \
make dist && \
rm -rf cyrus-imapd-2.5.0/ && \
tar zxvf cyrus-imapd-2.5.0.tar.gz && \
pushd cyrus-imapd-2.5.0/ && \
./configure && \
make && \
popd && \
for file in `git ls-files`; do
if [ ! -f cyrus-imapd-2.5.0/$file ]; then
echo "File cyrus-imapd-2.5.0/$file is missing"
fi
done
The initial ./configure option of --enable-maintainer-mode is needed for
(among others, probably) imap/rfc822_header.{c,h} to be created, and we
want a user of the tarball to "./configure; make; sudo make install".
A quick sanity check would be appreciated.
Kind regards,
Jeroen van Meeuwen
--
Systems Architect, Kolab Systems AG
e: vanmeeuwen at kolabsys.com
m: +41 79 951 9003
w: https://kolabsystems.com
pgp: 9342 BF08
Re: Status on Cyrus IMAP 2.5.0 Release
On 2015-03-03 15:53, Jeroen van Meeuwen (Kolab Systems) wrote: A quick sanity check would be appreciated. FWIW, this applies to the release notes as well: https://docs.cyrus.foundation/imap/release-notes/2.5-current.html Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Website & Documentation
On 2015-03-05 07:26, Chris Davies wrote: Posting to the mailing list as there is currently no documentation section in Bugzilla: *Cyrus Documentation:* * The "Introduction to Cyrus documentation contains an tag with no text explaining what AMS is. This results in some web browsers displaying a '?' when the cursor is over the AMS text: It's the Andrew Mail System as mentioned but you're correct in that the abbreviation should be defined in the docs as well. I've fixed this in my local working copy, so it shouldn't be long for that to end up on docs.cyrus.foundation. * The Debian & OpenSUSE install guides are missing: Yes, they are looking for volunteers. * The build-dependencies page links to http://asg.web.cmu.edu/sasl/sasl-library.html for cyrus-sasl-devel. This is not the most recent version. That page also says it was last modified in 1999 but contains a build from 2003. * The DIY page (/cyrus-docs-master/build/html/imap/installation/diy.html) points to http://www.oracle.com/database/berkeley-db/. This returns a 404. I've used the `rpm -qi` URLs for all these packages, some indeed may need corrections. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Docker Images for Testing
Hi there, As part of a more participatory and therefore hopefully better Quality Assurance, I have created a series of Docker images for some 13 platforms (in codenames); * squeeze through sid * santiago through maipo * heisenbug through rawhide * precise through vivid These Docker images' purpose is purely Development and Quality Assurance for now, meaning that their entrypoints have been set up to run the most tests thinkable; * ./configure with everything enabled, * "relaxed" vs. "strict" make, * make check, * a Cassandane run [1]. That said, tests are built to fail, and so I've documented the process including the steps to get an interactive shell: https://docs.cyrus.foundation/imap/developer/docker.html I believe this could also aid our developers in enabling them to easily run the target platform (for a certain bug or issue). If you have Docker running or could easily fire it up, please consider humouring me and executing your distro of interest, and giving it a star; https://registry.hub.docker.com/repos/cyrusimapd/ and giving me a task for the distro of your interest not included; https://git.cyrus.foundation/ I'm also specifically interested in feedback from our Solaris and *BSD users -- whether they are able to run Docker and whether we could target using the same set of scripts to execute builds and continuous integration on those platforms in a similar manner. I'm hoping for all of this to become the next generation of continuous integration for the Cyrus project, with Jenkins / Travis CI becoming a distant second/third. Anyone else interested in Docker is free to join the project I've created for it; https://git.cyrus.foundation/project/view/12/ Some of the things we'll be figuring out as we go may include; * Running a (certain) version of Cassandane tests against a (certain) version of Cyrus IMAP, * Exporting/uploading legible/parseable test results to Phabricator/Harbormaster/DryDock, * Whatever tickles your fancy. Enjoy! Kind regards, Jeroen van Meeuwen [1]: For those of you unaware of what Cassandane is -- it is the functional test suite for Cyrus IMAP. -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Documentation on "Other" Linux Distros
Hi there, Please pardon the cross-posting, but I'm going to need as broad an audience I can get. My personal corner of the universe is squarely within the RPM(4) based systems on this globe, and as such I've written up the bare necessities to get from a "yum install" to a successful IMAP login: https://docs.cyrus.foundation/imap/installation/distributions/centos.html Some of you will have noticed that Linux distros like Debian, Ubuntu and openSUSE have not (at all) been documented to the same "standard" (I use this term loosely). The same, in fact, goes for the DIY installation guide [1] (where all dependency names are RPM(4)-based platform package names) and the custom version guide(s) for each platform -- as well as the hints for packagers for that matter. I would like to not be as ignorant as I seem, and therefore I'm asking for your help in composing the notes on what is necessary to get from an apt-get or zypper install to a successful IMAP login for your favourite platform -- including those not currently listed. I'll take anything from a few scribbles on the mailing-list to a patch review in Phabricator, and I think the CentOS/RHEL guide is quite that bare minimum effort we're looking to document (based on the current codebase). Thanks in advance, Kind regards, Jeroen van Meeuwen [1] https://docs.cyrus.foundation/imap/installation/diy.html -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Minutes 9 March
On 2015-03-09 14:05, Bron Gondwana wrote: And we're up to date! Present: Ken, Ellie, Jeroen, Bron - small group today! New changes: * BDB removed from tree entirely - YAY As a matter of process, our ace needed not to do the work, rather just approve the proposed change. There's mere mortals like myself that can run in a couple of circles and do the various tidbits. It's easy to build docker images just given a COMMIT (actually ref) - so we can do integration tests on the 2.5 tree too. Rephrase: Run Docker containers based on the images we distribute. Environment variables are the key, and they're documented at: https://docs.cyrus.foundation/imap/developer/docker.html This makes it *WAY EASY* for RPM aficionado's to run APT* systems, and vice-versa, including unit and functional tests. I hope the documentation is clear but I love it being Docker containers -- no going through an installer any longer, no nurturing any VMs you want to keep around because you've fiddled with them to get them to work. Will be easier with committer privs - Ellie given privs to write directly to git.c.f trees. In general, be liberal with commit bit, and slap people who do bad things with it. We can always revert code. Side-note(s): * All committers are allowed to add new members to the committers groups they are a member of -- this way, it's a "by invitation" system of anyone already "blessed", and we can open up the generic "IMAP" project for *everyone* to join (and get the news and progress and alerts and such). * Committers should consider opting in to Reviewers. If you don't know what groups there are: https://git.cyrus.foundation/project/query/all/ "Reviewers" are the ones being spammed with approval requests like the ones submitted to "Differential" [1]. * Even though one may be a "committer", the use of "Differential" [1] is still available -> "D6" [2] is one such example. Docs: * all manpages need attention - verbiage used. I've submitted a "Pholio Mock" [3] in which I (attempt to) show how man-pages could potentially be generated from the online documentation. I'm afraid I have not yet found out how to generate online documentation from the (manually maintained by developers) man/* pages in the Cyrus IMAP git repository. * lib/imapoptions - generated imapd.conf.5 which includes all options, even those not compiled in. -> again, I want to fix that by just not allowing so much conditional compilation dammit. Go get the libs already. Otherwise online docs are a pain, and users get confused if their distro didn't compile in something they want. It is often not libs that are the problem, as may also be illustrated in the first "important" segment of the automatic mailbox creation feature [4]. That is to say, that this is not solely limited to `--with-ldap=/usr` building in ptclient/ldap.c, or even just the availability of an --enable-autocreate. In general, we need to move to "what is available (on the system / in the build root), we'll use". But the conundrum is in suppressing lib/imapoptions resulting in all its options included in imapd.conf(5), along with those things that are not built in -- just like there's a conundrum in having "make check" skip any BDB tests if "./configure --with-bdb=no" was issued -- I know, BDB is the wrong example by now :P Next meeting is same time UTC and Melbourne: 2015-03-11T22:00:00Z (9am Thursday in Melbourne, 6pm now in US East) This is going to be getting late for Europe, so we'll talk again about what to do when Australia moves in a few weeks. We both move positively toward one another for the northern hemisphere's summer -- your DST and ours gets you some 8 instead of some 10 hours ahead of us. Kind regards, Jeroen van Meeuwen [1] https://git.cyrus.foundation/differential/ [2] https://git.cyrus.foundation/D6 [3] https://git.cyrus.foundation/M1 [4] https://docs.cyrus.foundation/imap/features/automatic-creation-of-mailboxes.html -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
RE: What would it take for FastMail to run murder
On 2015-03-13 23:54, Dave McMurtrie wrote: From my phone, so excuse brevity and top-posting, but Fastmail running murder would be a huge bonus. I not-so-fondly recall the intimate relationship I developed with gdb debugging murder issues when we upgraded from 2.3 to 2.4 :) You won't have to for 2.5 (as much) because we're running it at supported customer sites, and I'm to blame for the alleged "fixes" ;-) Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: What would it take for FastMail to run murder
On 2015-03-13 23:50, Bron Gondwana wrote: So I've been doing a lot of thinking about Cyrus clustering, with the underlying question being "what would it take to make FastMail run a murder". We've written a fair bit about our infrastructure - we use nginx as a frontend proxy to direct traffic to backend servers, and have no interdependencies between the backends, so that we can scale indefinitely. With murder as it exists now, we would be pushing the limits of the system already - particularly with the globally distributed datacentres. Why would FastMail consider running murder, given our existing nice system? a) we support folder sharing within businesses, so at the moment we are limited by the size of a single slot. Some businesses already push that limit. How, though, do you "ensure" that a mailbox for a new user in such business is created on the same backend as all the other users of said business? Here are our deal-breaker requirements: 1) unified murder - we don't want to run both a frontend AND a backend imapd process for every single connection. We already have nginx, which is non-blocking, for the initial connection and auth handling. There's one particular "problem" with using NGINX as the IMAP proxy -- it requires that external service that responds with the address to proxy to. I say "problem" in quotes to emphasize I use the term "problem" very loosely -- whether it be a functioning backend+mupdate+frontend or a functioning backend+mupdate+frontend+nginx+service is a rather futile distinction, relatively speaking. 2) no table scans - anything that requires a parse and ACL lookup for every single row of mailboxes.db is going to be a non- starter when you multiply the existing mailboxes.db size by hundreds. I don't understand how this is an established problem already -- or not as much as I probably should. If 72k users can be happy on a murder topology, surely 4 times as many could also be happen -- inefficiencies notwithstanding, they're "only" a vertical scaling limitation. That said of course I understand it has it's upper limit, but getting updated lookup tables in-memory pushed there when an update happens would seem to resolve the problem, no? 3) no single-point-of-failure - having one mupdate master which can stop the entire cluster working if it's offline, no thanks. This is not necessarily what a failed mupdate server does though -- new folders and folder renames (includes deletions!) and folder transfers won't work, but the cluster remains functional under both the SMTP-to-backend and LMTP-proxy-via-frontend topology -- autocreate for Sieve fileinto notwithstanding, and mailbox hierarchies distributed over multiple backends when also using the SMTP-to-backend topoplogy notwithstanding. Thankfully, the state of the art in distributed databases has moved a long way since mupdate was written. I have also written a one-or-two line patch that enables backends that replicate, to both be a part of the same murder topology, to prevent the replica "slave" from bailing out on the initial creation of a mailbox -- consulting mupdate and finding that it would already exist. Along with this, we need a reverse lookup for ACLs, so that any one user doesn't ever need to scan the entire mailboxes.db. This might be hooked into the distributed DB as well, or calculated locally on each node. I reckon this may be the "rebuild more efficient lookup trees in-memory or otherwise" I may have referred to just now just not in so many words. And that's pretty much it. There are some interesting factors around replication, and I suspect the answer here is to have either multi- value support or embed the backend name into the mailboxes.db key (postfix) such that you wind up listing the same mailbox multiple times. In a scenario where only one backend is considered "active" for the given (set of) mailbox(es), and the other is "passive", this has been more of a one-line patch in mupdate plus the proper infrastructure in DNS/keepalived type of failover service IP addresses than it has been about allowing duplicates and suppressing them. Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: What would it take for FastMail to run murder
On 2015-03-14 22:48, Bron Gondwana wrote:
On Sun, Mar 15, 2015, at 07:18 AM, Jeroen van Meeuwen (Kolab Systems)
wrote:
How, though, do you "ensure" that a mailbox for a new user in such
business is created on the same backend as all the other users of said
business?
If the business already exists, the create user code will fetch the
server name
from the business database table and make that the creation server.
There's a cron job which runs every hour and looks for users who aren't
on
the right server, so if we import a user to the business, they get
moved.
Right -- so you seem to "agree" that "one business" is limited to "one
backend server", which is precisely what the larger businesses that are
our customers need to work around, when the number of mailboxes is
typically "tens of thousands", and the mechanism you describe "stops
working".
There's one particular "problem" with using NGINX as the IMAP proxy --
it requires that external service that responds with the address to
proxy to.
T108
I say "problem" in quotes to emphasize I use the term "problem" very
loosely -- whether it be a functioning backend+mupdate+frontend or a
functioning backend+mupdate+frontend+nginx+service is a rather futile
distinction, relatively speaking.
Sure, but backend+distributed mailbox service+nginx would be a much
simpler setup.
Yes, T108 here ;-)
I don't understand how this is an established problem already -- or
not
as much as I probably should. If 72k users can be happy on a murder
topology, surely 4 times as many could also be happen --
inefficiencies
notwithstanding, they're "only" a vertical scaling limitation.
"happy" is a relative term. You can get most of the benefit from using
foolstupidclients, but otherwise you're paying O(N) for the number of
users - and taking 4 times as long to do every list command is not
ideal.
Sure -- the majority of the processing delays seem to lay on the client
side taking off the wire what is being dumped on it, however.
You're far better entitled to speak to what is in a mailboxes.db and/or
its in-memory representation by the time you get to scanning the
complete list for items to which a user might have access, I just have
to say we've not found this particular part to be as problematic for
tens of thousands of users (yet).
That said of course I understand it has it's upper limit, but getting
updated lookup tables in-memory pushed there when an update happens
would seem to resolve the problem, no?
Solving the problem is having some kind of index/lookup table indeed.
Whether this is done all in-memory by some sort of LIST service which
scans the mailboxes.db at startup time and then gets updates from
mupdate.
For frontends specifically ("discrete murder"), we're able to use tmpfs
for mailboxes.db (and some other stuff of course) solving a bit of the
I/O constraints, but it's still a list of folders with parameters
containing whether the user has access, and what I meant was perhaps the
list can (in addition) be inverted to be a list of users with folders
(and rights?).
This is not necessarily what a failed mupdate server does though --
new
folders and folder renames (includes deletions!) and folder transfers
won't work, but the cluster remains functional under both the
SMTP-to-backend and LMTP-proxy-via-frontend topology -- autocreate for
Sieve fileinto notwithstanding, and mailbox hierarchies distributed
over
multiple backends when also using the SMTP-to-backend topoplogy
notwithstanding.
Yeah, until you start up the mupdate server again or configure a new
one.
Again, you get user visible failures (folder create, etc) while the
server is
down. The reason I want to shave off all these edge cases is that in a
big enough system over a long enough time, you will hit every one of
them.
We promote a standby frontend not otherwise used, to become the new
mupdate server. The interruption is a matter of seconds this way, unless
of course you're in the typical stalemate.
> Thankfully, the state of the art in distributed databases has moved a
> long way since mupdate was written.
I have also written a one-or-two line patch that enables backends that
replicate, to both be a part of the same murder topology, to prevent
the
replica "slave" from bailing out on the initial creation of a mailbox
--
consulting mupdate and finding that it would already exist.
Interesting. Does it also handle the case where the same mailbox gets
accidentally created on two servers which aren't replica pairs though?
Or do you get a mailbox fork?
The race condition is not addressed with it, like it is not addressed
currently.
It solely makes the MUPDATE server not reject the reservation request
from a server that use
Re: What would it take for FastMail to run murder
On 2015-03-18 01:51, Bron Gondwana wrote: On Wed, Mar 18, 2015, at 09:00 AM, Jeroen van Meeuwen (Kolab Systems) wrote: We promote a standby frontend not otherwise used, to become the new mupdate server. The interruption is a matter of seconds this way, unless of course you're in the typical stalemate. Hmm so maybe it's affordable. It scales up with number-of-servers as well though. Making sure it's up to date costs at least O(number of backends). I suppose in your specific case, which I'm not at all too familiar with, perhaps enhancing murder/mupdate to allow cascading and/or (geo-based) replication and/or multi-master would serve your deployment yet even better? I'm suggesting so because I would be concerned with the round-trip times between datacenters if there were only one mupdate master across all -- and perhaps the replicas are faster in issuing the cmd_set() than the mupdate master is(?). > Interesting. Does it also handle the case where the same mailbox > gets accidentally created on two servers which aren't replica pairs > though? Or do you get a mailbox fork? > The race condition is not addressed with it, like it is not addressed currently. I'm not 100% happy living with unaddressed race conditions. Addressing this would be an important part of making FastMail happy to run it. Ghe, neither am I, but c'est la vie. That said, in ~5 years and dozens of murder deployments, I have yet to encounter a situation or even a support case in which one mailbox is -- accidentally or otherwise -- created in two locations without the second failing / being rejected. It solely makes the MUPDATE server not reject the reservation request from a server that uses the same "servername" if it already has an entry for the same "servername!partition", so that the replica successfully creates the local copy -- after which replication is happy. Yeah, that makes sense. Of course, the backend should probably not be "reserving" so much. There are two things conflated here: 1) I'm running cmd_create in an IMAPd and I want to see if this folder already exists. 2) I'm a replica backend getting a copy of an existing folder (or indeed, a backend which already has a folder) and I'm informing mupdate of the fact. Those two should be treated differently. The first is "does this already exist", which is a legitimate question to ask. The second should always succeed. MUPDATE is a representation of facts, and the backends are the masters of those facts. With two-way replication safety however (and in your case, channelled as well, right?), which end of the replication (just in case things end up load-balanced across replicas?) gets to submit the original cmd_set() is up in the air, no? So this would build a scenario in which: "pair-1-replica-1.example.org" and "pair-1-replica-2.example.org" present themselves as "pair-1.example.org" A DNS IN A RR is created for the fail-over address(es) for "pair- 1.example.org" and attached to whichever replica in the pair is considered the active node. Both replicas would be configured to replicate to one another, which works in a PoC scenario but may seem to require lmtpd/AF_INET delivery. So they both have the same server name in mupdate. Yes, and frontends proxy the connections for mailboxes on the backend to the same fake server address. My plan is that they have different server names in mupdate. There's a separate channel somehow to say which is the primary out of those servers, which can be switched however (failover tooling) based on which servers are up, but the murder has the facts about where the mailbox really exists. It may even have statuscache. Man, how awesome would distributed statuscache be. So there are multiple records for the same mailbox, with different server names, in the murder DB. Would this not open back up a route to entertaining a variety of race conditions (that would need to be addressed somehow) though? Should then one of the duplicate mailboxes be marked as the primary? A scenario that comes up often is the geographically close-yet-distant secondary site for disaster recovery, where a set of backends on the primary site replicate to a set of backends on the secondary site. While initially this succeeds perfectly fine, and the backends on the secondary site can participate in a local-to-site murder, transferring mailboxes from one backend to another on the primary site will fail to replicate to the secondary site's backends (because of their participation in the murder). This is in part because it is not the XFER being replicated as such, but the target backend's CREATE/cmd_set(), which will fail because the mailbox already resides on another backend. I suppose a scenario in which the mupda
Re: Docker/Cyrus user
On 2015-03-19 05:29, Chris Davies wrote: I think it's because the cyrus user isn’t being created. Perl exception: No user named 'cyrus' Has anyone else got the docker images working? Am I missing a step or command somewhere? Are you sure you have the latest version of the GIT repo? https://git.cyrus.foundation/diffusion/ID/browse/master/santiago;6c242b5541d257fb6464f86b99c9ff35b7bcc959$75 I'm new to docker, any attempt to get command line access to my container is failing the the run script getting rerun, which takes another 5+ minutes Any suggestions? To get an interactive shell, specify the entrypoint (and a -s at the end): $ docker run -it --entrypoint="/bin/bash" cyrusimapd/santiago -s Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Docker/Cyrus user
On 2015-03-20 07:39, Chris Davies wrote: Not sure if you saw my previous IRC messages. The IRC server rejected some of them because I used copy & paste which broke each line down to individual messages. I'm currently rerunning the run command '*docker run -t -i cyrus- imapd-3.0.santiago*'. It failed multiple times but continued running tests. Unfortunately my terminal only shows the last 512 lines, so I’ve updated it to show unlimited lines so I can see what actually failed first. Should the tests stop once they hit a failure? or capture that and output it at the end? It should do its best to run as many tests if possible. So, if a "relaxed" make succeeds but a "strict" make fails, it can continue doing unit tests and cassandane based on a relaxed build. All output should be available on your stdout/stderr, so you might as well: $ docker run -it cyrusimapd/santiago 2>&1 | tee santiago.log Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: uniqueid based paths
On 2015-03-23 13:50, Gerd v. Egidy wrote: Tech support is another use case here. People often call and say they can't find a message in this and that folder or that some Kolab data is not up to date. We ssh into the machine and then use f.e. midnight commander to browse the folders of the user. Yes, that is a common use case. Being able to grep, du, copy or otherwise script access to all folders of a user really speeds up support. You can immediately see what the user has done with your natural tools on the commandline. With expunge_mode: delayed we use unexpunge -l user/john.doe/some/fol...@example.org today -- iow, no need to know the uniqueid there. If something does require us to 'cd' in to the mailbox or 'grep' some hierarchy, we use the mbpath utility (despite the outcome being predicable for a default installation, it is not so much with fulldirhash enabled). With delete_mode: delayed we use kolab list-deleted-mailboxes (giving you the original path and a humanly readable datetime of the deletion). Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Intended response for LIST "" %?
Hi there, In working on event notifications to include the correct folder path to shared folder mail delivery, I had accidentally broken the response to LIST "" %. See [1] for my commit breaking it and [2] for the revert. Naturally most of the output listed here depends on unixhierarchysep, altnamespace, sharedprefix, userprefix ;-) For the hierarchy stated below, the current code shows a response of: From ctl_mboxlist -d: example.org!user.jane^doe example.org!user.john^doe example.org!user.john^doe.Archive example.org!user.john^doe.Lists.Topic C: . LIST "" % S: * LIST (\Noinferiors \HasNoChildren) "/" INBOX S: * LIST (\HasNoChildren) "/" Archive S: * LIST (\Noselect \HasChildren) "/" Lists S: * LIST (\Noselect \HasChildren) "/" "Other Users" The code tests for whether a mailbox has children which obliges it to list it in the "toplevel" hierarchy, which is very applicable to an other users namespace, but this doesn't apply to shared folders -- each of them may be prefixed with virtually anything other than "user": From ctl_mboxlist -d: example.org!lists.kolab^org.devel example.org!shared.info example.org!shared.sales example.org!user.jane^doe example.org!user.john^doe example.org!user.john^doe.Archive example.org!user.john^doe.Lists.Topic A regular 'LIST "" *': C: . LIST "" * S: * LIST (\Noinferiors \HasNoChildren) "/" INBOX S: * LIST (\HasNoChildren) "/" Archive S: * LIST (\HasChildren) "/" "Other Users/jane.doe" S: * LIST (\HasNoChildren) "/" "Shared Folders/lists/kolab.org/devel" S: * LIST (\HasNoChildren) "/" "Shared Folders/shared/info" S: * LIST (\HasNoChildren) "/" "Shared Folders/shared/sales" S: . OK Completed (0.000 secs 35 calls) But 'LIST "" %' doesn't change: C: . LIST "" % S: * LIST (\Noinferiors \HasNoChildren) "/" INBOX S: * LIST (\HasNoChildren) "/" Archive S: * LIST (\Noselect \HasChildren) "/" Lists S: * LIST (\Noselect \HasChildren) "/" "Other Users" The question is now whether '. LIST "" %' is supposed to also include: S: * LIST (\Noselect \HasChildren) "/" "Shared Folders" Kind regards, Jeroen van Meeuwen [1] https://git.cyrus.foundation/rI44e8e7f7d0e16cc6005441709fc7c02e7032d6b3 [2] https://git.cyrus.foundation/rI10c621a954ce976e419201037666146fde1814b9 -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Meeting Minutes 2014-04-01/02
Hi there, a few quick notes on the meeting we had tonight; In attendance; - Anthony, - Jeroen, - Chris Topics; Chris and Jeroen are going to be sitting together for hackery purposes, in an attempt to make a new CI environment hosted by Fastmail work with the Phabricator instance hosted by Kolab Systems, possibly with the assistance of some Docker images (but who knows). Date/Time/Space: Wednesday 08:30 Melbourne, way-too-late Tuesday Europe, not-sure Americas ;-) Anthony to wrap up and push a branch for, so that we can all do and assist with, rebases and/or merges of, his netbsd and autofoo related work. Licensing but more importantly copyright. I think we drew the conclusion of letting more managerial people figure it out among themselves, perhaps with help of experts in the field of Open Source/Free Software/Licensing/Copyright/Trademark, such as Mr. Greve -- founder of the Free Software Foundation Europe and also the CEO of Kolab Systems. Furthermore, we discussed in general terms, the intricacies of an Easter weekend. This may have taken us about 20 minutes. Later, one topic discussed involves the choosing of a reference platform... While the attendees agreed choosing something like "Debian Wheezy" seemed rather reasonable, we can't make a decision without a broader consensus. For those of you simply reading the minutes, this choosing of anything simply means that *development* be focused around such platform's capabilities, and everything else can be considered a proverbial "afterthought". This is solely the space of active development and volatile code (under active development). We further deliberated there will likely not be a meeting on Easter Monday (April 6th). Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
Re: Harbormaster notes from today's conference call - 29 June 2015.
On 2015-06-30 01:11, Bron Gondwana wrote: This is Jeroen. Jeroen, we need you to look at this - it's holding people up. clamav-devel has been installed, but the builds are still failing: https://git.cyrus.foundation/harbormaster/build/1049/ Sphinx and docutils have been upgraded as well; [root@phab10 ~]# pip show sphinx --- Name: Sphinx Version: 1.3.1 Location: /usr/lib64/python2.7/site-packages Requires: sphinx-rtd-theme, Jinja2, alabaster, babel, six, Pygments, snowballstemmer, docutils [root@phab10 ~]# pip show docutils --- Name: docutils Version: 0.12 Location: /usr/lib/python2.7/site-packages Requires: Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
JMAP Proxy Docker Image
Hi there, For those of you who are willing to entertain a little JMAP in their client application implementations (rather than a lot of IMAP), I have pushed a Docker image earlier today. I hope it provides some significant simplification to get started, but of course I'm eager to hear your feedback. For more information, please see https://kanarip.wordpress.com/2015/10/08/jmap-proxy-docker-image/ Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08
