Re: Unifying the Cyrus World
Hi - Having just recently struggled with a still-not-100%-clear-how-it-happened mailboxes.db corruption issue that Bron ended up having to help me fix, I have some thoughts on this. And the recent bout of banging my head against the wall over this issue for days has left me motivated to write up some of this documentation myself, assuming I can find the information necessary. Here is my list 1. Suppose the /var/imap folder (the one containing the critical *.db files) disappears completely: A step by step tutorial on how to get your mail system back up and running. Such a tutorial alone would go a long way towards clearing up the mysteries of how indexing is related user mail folders. This tutorial should include explanations of what can't be recovered (e.g. which messages have been viewed, possibly sieve scripts, etc.). 2. Documentation on safely backing up and restoring a cyrus mail system. There was a discussion about this on the list a year ago: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2014-December/037849.html (Further up the thread) https://lists.andrew.cmu.edu/pipermail/info-cyrus/2014-December/037799.html Which I consider inconclusive. While Nic's suggestion to use replication is probably the safest approach, there still needs to be something for people who run a single mail server and want to keep it that way. There's something to be said for simplicity. Also, the process of setting up a replication server (with and without murder) needs to be documented better. In the thread above, Nic suggested that I look at this documentation: http://cyrusimap.web.cmu.edu/docs/cyrus-imapd/2.4.17/ag.php which I found to be mostly unhelpful. Tutorials are good even when they dont match your exact use case. Too much abstract hand-waving is just confusing to most people. I'd rather read a common use case tutorial and then extrapolate to my own situation from there. Way too many critical details end up getting left out when you talk about these issues absractly. 3. The creation, use, and maintenance of sieve scripts. I dug into this earlier this month and found almost nothing available online. I did get some helpful responses on the cyrus list: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2015-December/038701.html but this should be in the documentation; including a tutorial on how to work with sieve scripts and pointers to 3rd party tools, with perhaps even a description of how to set up and use them. 4. Organization of the documentation. It looks like the worst case scenario of having docs spread out over 2 domains with completely different organizational schemes, style, etc.. is being addressed, but some thought should nevertheless be given to this very difficult task. My suggestion is to look at the organization of the documentation from different perspectives: - a new or potentially new cyrus user who knows absolutely nothing - a somewhat experienced cyrus admin looking to quickly get an answer to a specific question - a cyrus admin who wants to implement some new functionality, say moving from a single server to a murder, or start using sieve scripts I use both cyrus and dovecot (the first by choice, the latter because I got outvoted in a larger organization). Arch, Red Hat, and Ubuntu all package dovecot as the default IMAP server; on Arch I have to install cyrus from the AUR. I'm convinced that the one and only one reason for this is that the dovecot guy has had pretty good and thorough documentation from the very beginning. (That, and the cyrus project was dormant for a while.) Technically, cyrus is cleaner, faster, more scalable, etc.. Soapbox: having worked with open source software for decades, it's very clear to me that open source projects become successful not on the quality of the code, but rather the quality of the documentation. Good projects with poor documentation generally die if there is an alternative, poorly conceived projects with good documentation succeed. Django is (in my opinion) pretty crappy software compared to say, bottle.py (which most people haven't even heard of). Why is Django so popular? Because they took the time to write good documentation. In fact, if you want to use an MVC python web framework, the trick is to read the Django documentation first, and then use something like bottle.py or flask for your web app. Most developers don't bother to follow up with step 2. They read the Django documentation ... and then use Django. On 12/30/2015 03:02 AM, Nicola Nye via Cyrus-devel wrote: Hi Patrick, Just before Christmas we got all the access sorted out and there's now a copy of docs.cyrus.foundation on cyrusimap.org/new, and it's auto-updating. It all took longer than we had hoped given FastMail's exciting few weeks with the DDoS situation. When I'm back in the office after the New Year I'll look at what's the best way to go moving the old si
Re: cyrus-imapd 3.0.0 next beta and release plan?
The main motivation was that I/we wanted this completely contained within Cyrus rather than using an external service. A second, and minor, motivation was that I wanted to prove that I could do it. A lot of the code needed was already in place from the other services. On 12/29/2015 02:38 PM, Thomas Jarosch wrote: Hi Ken, Am 29.12.2015 um 16:32 schrieb Ken Murchison: If all that can be done safely in the signal handler is setting a global variable, I might just scrap the heartbeat functionality. The alternative is that the actual method processin code will have periodically check the status of the global variable. Or do you have a more creative idea? checking a global variable is how it's done usually. Or if the HTTP server is using some kind of event driven framework, set up a timer instead of SIGARLM. On linux there's signalfd(), but that's non-portable, so a no no for cyrus. The global variable should be of type "volatile sig_atomic_t". Random side note: What was the main motivation for implementing an own HTTP server including authentication stuff / chunked encoding etc? Was implementing an Apache httpd module in C considered without redoing the whole HTTP protocol layer? Just wondering, we are doing our stuff at dayjob as a C++ apache module. You just register a handler for URIs / mime-types you are interested in. Other HTTP server like ngix would probably work, too. Cheers, Thomas -- Kenneth Murchison Principal Systems Software Engineeer Carnegie Mellon University
Re: Unifying the Cyrus World
Hi Patrick, Just before Christmas we got all the access sorted out and there's now a copy of docs.cyrus.foundation on cyrusimap.org/new, and it's auto-updating. It all took longer than we had hoped given FastMail's exciting few weeks with the DDoS situation. When I'm back in the office after the New Year I'll look at what's the best way to go moving the old site out of the way and whether we maintain any redirects so any old bookmarks still work for common entry points. Then we can start on the larger task of integrating the docs source back into the cyrus code source. If there's areas that you think need immediate and obvious updating, do let me know (especially if you can provide pointers to where I can get the information from, even if it's just links to list threads); I'd rather ensure that my time is spent easing definite pain points. Cheers, Nicola On Mon, Dec 28, 2015, at 01:45 AM, Patrick Goetz via Cyrus-devel wrote: > Was this issue ever resolved? I'm still finding the documentation split > between cyrus.foundation and cyrusimap.org, and it's incomplete in both > places. > > On 09/16/2015 08:50 PM, Nicola Nye wrote: > > Dear Cyrus folks, > > After nominally completing the migration of the docs from the wiki at > > cyrusimap.org onto the sphinx repository at docs.cyrus.foundation, I > > worked out there was a whole bunch of existing services at cyrusimap.org > > that were non-trivial to relocate. Which, after some discussion at the > > Cyrus hangout, we felt that retaining cyrusimap.org as our primary (and > > sole) presence made more sense. > > Equally, there are some documentation issues that are affected by our > > current straddling of two worlds. Read on for a discussion of the issues > > and down to the options and recommendation. > > I'd love to have your input (particularly Nic Bernstein on the docs > > angle!) if you agree or disagree so we can make for a more unified Cyrus > > presence. > > *Issues* > > 1) Branding > > We are currently split across two internet presences: cyrus.foundation > > and cyrusimap.org. This is confusing. We should only have one. > > The cyrus.foundation domain name is not of real use as there is a > > pre-existing entity (The Cyrus Foundation) of the same name. > > Swapping our internet domain name presence from cyrusimap.org to > > cyrus.foundation is no longer a significant driver as a result, > > especially as cyrusimap.org is already indexed and linked to around the > > net and is unambiguous. (Yes, Cyrus does so much more than just imap, > > but it's the primary association) > > Currently services are split across the two domains - git repos, > > phabricator and authoritative online docs on cyrus.foundation, while > > downloads, bugzilla and generated docs are on cyrusimap.org. > > 2) Docs logistics > > The docs repo is separate to source which is causing friction keeping > > man pages updated. > > We want man pages in two formats: *nix for actual man pages to ship and > > install with cyrus, and html for web presentation. > > Nice to have: ship a snapshot of all current docs (not just man pages) > > along with source distribution. > > 3) Sphinx vs wiki > > We have been working towards making the content on cyrus.foundation to > > be the most up to date. This is using Sphinx, which allows us to > > generate, from the same source, man pages as well as web pages. (And > > even pdf or single page html). So we get nicer output format options. > > Clearer history in the git log than you get from wiki. > > Cyrusimap docs are held in mediawiki. Not so great for man pages. But > > it's much easier for third party contributor to pitch in and help out: > > they don't need to learn rst, they don't need a phabricator account, > > they don't need to navigate git. > > *Possible solutions* > > In all cases it seems clear that we should stop using the > > cyrus.foundation domain and ideally services that are hosted there be > > shifted to live alongside cyrusimap.org for ease of access. > > There are a number of @cyrus.foundation email accounts that would need > > to be dealt with for Chris Davies' testing. > > Option 1: Single domain, unify git repos, use sphinx everywhere > > > > Moving the git repo to cyrusimap (or anywhere) isn't hard. (a job for Bron) > > Get rid of separate cyrus-docs repo. Put cyrus-docs stuff back inside > > cyrus-imapd/docs for easier man page generation and tagging of docs > > versions with source, and shipping of current docs with source. > > Put sphinx on cyrusimap to replace the wiki. This requires either: > > > > 1. > > working out how to set up the existing generated docs for use with > > sphinx, > > 2. > > or make all the stuff in cyrus-imapd/doc into rst so it works with > > sphinx. We can still ship the built html. > > > > Option 2: Single domain, unify git repos, use wiki for docs > > ---
