Re: When is imapproxy usefull?
Dilyan, In my own experience, an IMAP proxy is helpfulfor webmail clients, in that the proxy can maintain a persistent connection to the IMAP server, thus saving the time & CPU cycles which would otherwise go in to bringing up and tearing down the TCP/IP sessions. This isespecially important for web clients which are using frames (like SquirrelMail) or would otherwise spawn multiple connections to the IMAP server. The best performance may be had by placing the proxy on the web server, so the bulk of the connections can happen via Unix sockets, or localhost TCP/IP connections. Then just a handful of IMAP connections happen between the web and IMAP servers. I've used the old imapproxyd, as well as Perdition, for this. I have not tried nginx, so cannot speak to that. I hope this is useful information, -nic On 3/9/20 12:10 PM, Дилян Палаузов wrote: Hello, the documentation on performance of Horde/Imp: https://www.horde.org/apps/imp/docs/PERFORMANCE recommends installing an imap proxy , e.g. the one from https://squirrelmail.org/download.php which ceased deveopment decades ago. The idea is that horde/imp/the-webmail-client connects to the imap-proxy and this should make everything faster. Does such an imap-proxy make things faster for a webmail client? Is nginx-imap-proxy or anything else better than squirrelmail-imap-proxy? Greetings Дилян -- Nic Bernstein n...@nicbernstein.com mobile: +1 414 807 1734 snail: N Astor St Apt A5, Milwaukee, WI 53202-3319 https://www.nicbernstein.com https://www.linkedin.com/in/nic-b-26577a178/
Re: Question for upgrading
On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote:
I was trying to upgrade part of our Cyrus imap installation,
concretely that one consisting in still 2.3. I was planning to set up
Cyrus 3.0. I have seen all works properly except for the unexpunge
command because as someone stated here, a reconstruct -V max was
needed.The problem is that this reconstruct command, takes ages and
I'm not able to keep the service offline so many time. So I have been
thinking in the following scenario :
Egoitz,
As long as you've followed all of the various steps needed to account
for version changes, as outlined in the Release Notes for /all/
intermediary releases, then the last step should be the updating of the
indexes. Rather than running "reconstruct -V max" on all mailboxes at
once, simply run it on subsets. We've done this on large installations
without ill effect. You can have the service on line during the
reconstruct, and, as you note, have most all functionality present.
We build a list of users (mboxlist.txt), and then run a cron job, during
off-peak hours, using the 'parallel' command like so (from 'crontab -e
-u cyrus'):
### ## Start a batch of recursive reconstruct jobs at 7PM and
discontinue ## at 6:53AM. 0 19 * * * parallel -j 5 --resume --joblog
/var/lib/imap/reconstruct.log cyrus reconstruct -G -V max -r -u {} <
/var/lib/imap/mboxlist.txt 53 06 * * * kill -TERM `ps ax | grep
[p]arallel | awk '{print $1}'`
Cheers,
-nic
--
Nic Bernstein n...@onlight.com
Onlight Inc. www.onlight.com
6525 W Bluemound Rd., Ste 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: backup replication: sync_try_imap
On 09/04/2018 09:56 PM, ellie timoney wrote: Can you please point out where in the docs csync is described as deprecated/outdated? Those docs are probably bad and need reviewing! Ellie, The reference to deprecated operation in /docsrc/imap/reference/admin/sop/replication.rst, (line 172 in my checkout): 2. If using the deprecated sync_server scheme, add the following line to the ``/etc/services`` file. Note that the port number is arbitrary as long as its not being used by any other services on the network. :: csync 2005/tcp 3. If using the deprecated sync_server scheme, add a line similar to the following in the SERVICES section of :cyrusman:`cyrus.conf(5)`: :: syncserver cmd="/usr/cyrus/bin/sync_server" listen="csync" This is in the general discussion on configuration of a replica server. I think this is correct for a typical replica, just not for a backup server, whose configuration is discussed elsewhere, /docsrc/imap/reference/admin/backups.rst. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight Inc. www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: Internal error: assertion failed imap/message.c: 4246: !message_need(m, M_RECORD)
Bron, Answering my own question, I now see that the fix you refer to is in cfb3054, imap/mailbox.c on 1/2/2017, which is in both master and 3.0... However, I've still got the problem, so same failed assertion, but different bug? -nic On 07/24/2018 12:10 PM, Nic Bernstein wrote: Bron, et al., Was this change ever cherry-picked to 3.0? I am seeing the same issue with recent 3.0 HEAD, but slightly different location: user.masked: updating sync_crc 521983118 => 503807715 fatal error: Internal error: assertion failed: imap/message.c: 4286: !message_need(m, M_RECORD) A git log of imap/message.c doesn't show a commit from 1/2/2017, and nothing affecting imap/message.c around that time seems to line up with this. Please advise, -nic On 01/02/2017 07:13 AM, Bron Gondwana via Cyrus-devel wrote: Thanks for the data. It was 8 bytes of zeros across a UID and INTERNALDATE in the cyrus.index file. I now have a fixed reconstruct which can detect and repair this rather than aborting, pushed to master. I also have a Cassandane testcase for this and a couple of other things that reconstruct does :) Bron. On Thu, 29 Dec 2016, at 09:45, Bron Gondwana via Cyrus-devel wrote: Wow, interesting. Are you willing to send me a tarball containing the spool folder including cyrus.index and cyrus.cache files as well as the email files themselves? I'll need your imapd.conf file as well :) Cheers, Bron. On Thu, 29 Dec 2016, at 00:28, Thomas Cataldo via Cyrus-devel wrote: Hi, Running a build of 3.0.0-beta6 I hit the following assertion on one of my test mailboxes after playing a bit with the replication stuff : root@bm1604:~# /usr/lib/cyrus/sbin/sync_client -n eclipse -o -u t...@ex2016.vmw Fatal error: Internal error: assertion failed: imap/message.c: 4246: !message_need(m, M_RECORD) root@bm1604:~# cyradm -u admin0 localhost Password: localhost> version name : Cyrus IMAPD version : 3.0.0-beta6-3-gf721e5b vendor : Project Cyrus support-url: http://www.cyrusimap.org os : Linux os-version : 4.4.0-57-generic environment: Built w/Cyrus SASL 2.1.26 Running w/Cyrus SASL 2.1.26 Built w/OpenSSL 1.0.2g 1 Mar 2016 Running w/OpenSSL 1.0.2g 1 Mar 2016 Built w/zlib 1.2.8 Running w/zlib 1.2.8 CMU Sieve 2.4 mmap = shared lock = fcntl nonblock = ioctl idle = idled root@bm1604:~# telnet localhost 1143 Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR] server ready . login t...@ex2016.vmw xx . OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATEMORE ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] User logged in SESSIONID= . select inbox * BYE Fatal error: Internal error: assertion failed: imap/message.c: 4246: !message_need(m, M_RECORD) Connection closed by foreign host. Trying to reconstruct the mailbox does not help : root@bm1604:~# /usr/lib/cyrus/sbin/reconstruct -rfxGROU t...@ex2016.vmw t...@ex2016.vmw The error is still here after that. Any idea ? Regards, Thomas. -- Bron Gondwana br...@fastmail.fm -- Bron Gondwana br...@fastmail.fm -- Nic bernstein...@onlight.com Onlight, Inc.www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 <>
Re: Internal error: assertion failed imap/message.c: 4246: !message_need(m, M_RECORD)
Anthony, I think this is a different error, just hitting the same assertion tripwire. I'm getting a problem in reconstruct(8), but your is showing up in imapd. -nic On 07/26/2018 04:16 AM, Anthony Prades via Cyrus-devel wrote: Hi, Same problem here. Context: Upgrade cyrus from 2.4.20 to Cyrus 3.0.7. As our spool is very big, we plan to run reconstruct -V max in a second time as it seems to be possible from 2.5 (https://www.cyrusimap.org/2.5/imap/release-notes/2.5/x/2.5.0.html). After upgrade binaries, mailbox are available until we change a flag. After changing a flag, mailbox is unavailable with message: Jul 25 12:39:28 bluemind-debian cyrus/imap[9875]: Fatal error: Internal error: assertion failed: imap/message.c: 4286: !message_need(m, M_RECORD) Jul 25 12:39:28 bluemind-debian cyrus/master[8463]: process type:SERVICE name:imap path:/usr/lib/cyrus/imapd age:0.138s pid:9875 exited, status 75 To reproduce: - create a new mailbox using Cyrus 2.4.20: user/ad...@apr-vmnet.loc - put at least 2 messages - using LMTP: # ls -l /var/spool/cyrus/data/mail/domain/a/apr-vmnet.loc/a/user/admin/ total 12 -rw--- 1 cyrus mail 1423 Jul 25 12:31 1. -rw--- 1 cyrus mail 1425 Jul 25 12:31 2. - upgrade to cyrus 3.0.7 - update flags on mail 1: # telnet localhost 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE AUTH=PLAIN SASL-IR] server ready A1 LOGIN ad...@apr-vmnet.loc admin A1 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATEMORE ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION URLAUTH URLAUTH=BINARY LOGINDISABLED X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] User logged in SESSIONID= A2 SELECT INBOX * 2 EXISTS * 0 RECENT * FLAGS (\Answered \Flagged \Draft \Deleted \Seen) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok * OK [UNSEEN 1] Ok * OK [UIDVALIDITY 1532514648] Ok * OK [UIDNEXT 3] Ok * OK [HIGHESTMODSEQ 3] Ok * OK [URLMECH INTERNAL] Ok * OK [ANNOTATIONS 65536] Ok A2 OK [READ-WRITE] Completed A3 STORE 1 +FLAGS \Flagged * 1 FETCH (FLAGS (\Flagged)) A3 OK Completed A4 LOGOUT * BYE LOGOUT received A4 OK Completed Connection closed by foreign host. - select INBOX: # telnet localhost 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE AUTH=PLAIN SASL-IR] server ready A1 LOGIN ad...@apr-vmnet.loc admin A1 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATEMORE ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION URLAUTH URLAUTH=BINARY LOGINDISABLED X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] User logged in SESSIONID= A2 SELECT INBOX * BYE Fatal error: Internal error: assertion failed: imap/message.c: 4286: !message_need(m, M_RECORD) Connection closed by foreign host. Running reconstruct -V max on this mailbox fix the problem but we loose mail 2 flags as it is rediscovered: # reconstruct -V max user/ad...@apr-vmnet.loc apr-vmnet.loc!user.admin: update uniqueid from header (null) => 5d0161495b585158 apr-vmnet.loc!user.admin uid 2 rediscovered - appending apr-vmnet.loc!user.admin updating quota_mailbox_used: 4273 => 2848 apr-vmnet.loc!user.admin: updating exists 3 => 2 apr-vmnet.loc!user.admin: updating sync_crc 3520663294 => 4247682740 user/ad...@apr-vmnet.loc Repacked user/ad...@apr-vmnet.loc to version 13 # ls -l /var/spool/cyrus/data/mail/domain/a/apr-vmnet.loc/a/user/admin/ total 16 -rwxr-x--- 1 cyrus mail 1423 Jul 25 12:31 1. -rwxr-x--- 1 cyrus mail 1425 Jul 25 12:31 3. Anthony On 07/24/2018 07:10 PM, Nic Bernstein wrote: Bron, et al., Was this change ever cherry-picked to 3.0? I am seeing the same issue with recent 3.0 HEAD, but slightly different location: user.masked: updating sync_crc 521983118 => 503807715 fatal error: Internal error: assertion failed: imap/message.c: 4286: !message_need(m, M_RECORD) A git log of imap/message.c doesn't show a commit from 1/2/2017, and nothing affecting imap/message.c around that time seems to line up with this. Please advise, -nic On 01/02/2017 07:13 AM, Bron Gondwana via Cyrus-deve
Re: Internal error: assertion failed imap/message.c: 4246: !message_need(m, M_RECORD)
Bron, et al., Was this change ever cherry-picked to 3.0? I am seeing the same issue with recent 3.0 HEAD, but slightly different location: user.masked: updating sync_crc 521983118 => 503807715 fatal error: Internal error: assertion failed: imap/message.c: 4286: !message_need(m, M_RECORD) A git log of imap/message.c doesn't show a commit from 1/2/2017, and nothing affecting imap/message.c around that time seems to line up with this. Please advise, -nic On 01/02/2017 07:13 AM, Bron Gondwana via Cyrus-devel wrote: Thanks for the data. It was 8 bytes of zeros across a UID and INTERNALDATE in the cyrus.index file. I now have a fixed reconstruct which can detect and repair this rather than aborting, pushed to master. I also have a Cassandane testcase for this and a couple of other things that reconstruct does :) Bron. On Thu, 29 Dec 2016, at 09:45, Bron Gondwana via Cyrus-devel wrote: Wow, interesting. Are you willing to send me a tarball containing the spool folder including cyrus.index and cyrus.cache files as well as the email files themselves? I'll need your imapd.conf file as well :) Cheers, Bron. On Thu, 29 Dec 2016, at 00:28, Thomas Cataldo via Cyrus-devel wrote: Hi, Running a build of 3.0.0-beta6 I hit the following assertion on one of my test mailboxes after playing a bit with the replication stuff : root@bm1604:~# /usr/lib/cyrus/sbin/sync_client -n eclipse -o -u t...@ex2016.vmw Fatal error: Internal error: assertion failed: imap/message.c: 4246: !message_need(m, M_RECORD) root@bm1604:~# cyradm -u admin0 localhost Password: localhost> version name : Cyrus IMAPD version : 3.0.0-beta6-3-gf721e5b vendor : Project Cyrus support-url: http://www.cyrusimap.org os : Linux os-version : 4.4.0-57-generic environment: Built w/Cyrus SASL 2.1.26 Running w/Cyrus SASL 2.1.26 Built w/OpenSSL 1.0.2g 1 Mar 2016 Running w/OpenSSL 1.0.2g 1 Mar 2016 Built w/zlib 1.2.8 Running w/zlib 1.2.8 CMU Sieve 2.4 mmap = shared lock = fcntl nonblock = ioctl idle = idled root@bm1604:~# telnet localhost 1143 Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR] server ready . login t...@ex2016.vmw xx . OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATEMORE ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] User logged in SESSIONID= . select inbox * BYE Fatal error: Internal error: assertion failed: imap/message.c: 4246: !message_need(m, M_RECORD) Connection closed by foreign host. Trying to reconstruct the mailbox does not help : root@bm1604:~# /usr/lib/cyrus/sbin/reconstruct -rfxGROU t...@ex2016.vmw t...@ex2016.vmw The error is still here after that. Any idea ? Regards, Thomas. -- Bron Gondwana br...@fastmail.fm -- Bron Gondwana br...@fastmail.fm -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 <>
Using LDAP with Cyrus [was: Re: [Diffusion] [Committed] rI0b8b7ab02b36: Documentated several saslauthd ldap options.]
Dan, I am trying for the first time to set up Cyrus (3.0.4 & 3.0.5) with ptloader, sasl auxprop, etc. Even though I've used LDAP for many years, I've only ever used saslauthd with mech=ldap or mech=pam, and a fairly simple configuration. For example: ldap_servers: ldapi://%2fvar%2frun%2fopenldap%2fldapi ldap_bind_dn: cn=proxyUser,ou=systems,dc=example,dc=com ldap_bind_pw: secret ldap_filter: (|(&(|(uid=%u)(mail=%u)(mailRoutingAddress=%u))(objectClass=person))(&(cn=%u)(objectClass=organizationalRole))) ldap_search_base: dc=example,dc=com When I search my archive of the cyrus-devel list, the only references to ldap in the subjects are you making some commits to the old Phabricator system. Unfortunately all of the associated tracking from that era is gone. Could you perhaps provide some guidance on this? (see below) I've looked in the modern-day equivalent to the affected documents listed below, but don't see many notes on LDAP. I was hoping to write up some comprehensive documentation on using LDAP with Cyrus, as there is currently nothing beyond the imapd.conf(5) man page. Any help you could provide would be most welcome. The only cogent examples I find online are all from you, but are many years old, so I have no frame of reference as to how accurate they still are. If you would prefer to discuss this off-list, or via phone, please advise. Specifically, I am trying to configure so that users may authenticate with either just UID (i.e. "nic") or email address (i.e. "n...@onlight.com"). The saslauthd example shown above does just this, but Cyrus still only works with the simple user ID, not the email address, which is what leads me to trying ptloader and auxprop. Anyone else, I would welcome working LDAP configuration examples from any and all, just remember to obfuscate or remove any security information. Thanks in advance, -nic On 03/14/2016 02:52 AM, Phabricator wrote: Dan White <dwh...@olp.net> committed rI0b8b7ab02b36: Documentated several saslauthd ldap options. (authored by Dan White <dwh...@olp.net>). Herald added auditors: Documentation. Documentated several saslauthd ldap options. AFFECTED FILES /doc/Administrator_Guide/en-US/Administrator_Guide.xml /doc/Administrator_Guide/en-US/appe-Mailbox_Distribution.xml /doc/Administrator_Guide/en-US/part-Configuration_Reference.xml /doc/Deployment_Guide/Makefile /doc/Deployment_Guide/en-US/Deployment_Guide.xml /doc/Deployment_Guide/en-US/Deployment_Scenarios.xml /doc/Deployment_Guide/en-US/Performance_Recommendations.xml USERS Documentation (Auditor) COMMIT https://git.cyrus.foundation/rI0b8b7ab02b36 EMAIL PREFERENCES https://git.cyrus.foundation/settings/panel/emailpreferences/ To: davies, nicolan, onlight, amor, admin, vanmeeuwen -- Nic bernstein...@onlight.com Onlight Inc.www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335 <>
Re: Xapian partition definition when using archiving?
Ellie,
Thanks for weighing in on this, and for doing so on a weekend! :-)
I've just pushed some documentation edits in PR #2224 as a first attempt
at clarifying this stuff.
Cheers,
-nic
On 12/28/2017 09:22 PM, ellie timoney wrote:
> It's really the default search tier to use, and has no direct
relationship to the "default" partition.
Yep, or at least, this is my understanding too.
> In other words, this is just another circumstance where seemingly
obvious partition names (like default-default) get us into
documentation trouble. Right?
We trip over this all the time... and it's hard to document because
the trivial case and the complicated cases are so divergent. Even at
FM, where our setup is pretty advanced, it's still relatively simple,
in that we only have a single partition-/name/ entry ("default") per
Cyrus instance. So all of our /foo/partition-/name/ settings are
/foo/partition-default!
I guess what I'm getting at is there's a few dimensions of complexity
here. I think the "multiple-named-partitions" dimension of complexity
might be reasonably well documented, but then each additional
dimension (e.g. archive partitions, search tiers, etc) is only
documented for the single-named-default-partition case.
It'd be great if the documentation could move away from the "single
partition is named 'default'" scheme -- I think that would ease a LOT
of confusion for people trying to understand the different dimensions
all at once. But I don't really have any ideas for what might be a
better name for it. I haven't seen real-world multi-partition Cyrus
instances, so I don't know the use cases for having multiple
partitions, so it's hard to suggest a good name for someone's first
and maybe only partition.
Cheers,
ellie
On Fri, Dec 29, 2017, at 4:47 AM, Nic Bernstein wrote:
Bron, et al.,
Okay, one more time around on this. When I try a version of the
partition layout listed below, in my original post, and run "cyr_info
conf-lint" against it, I get all of the "/yadda/searchtier: blah"
lines thrown back at me. I had assumed that "/default/searchtier:
blah" was to specify to which tier to index the default partition,
but that's wrong, isn't it? It's really the default search tier to
use, and has no direct relationship to the "default" partition.
Am I correct in this new interpretation? It is implied by the man
page for imapd.conf (derived from lib/imapoptions):
defaultsearchtier:
Name of the default tier that messages will be indexed to.
Search indexes can be organized in tiers to allow index
storage in different directories and physical media. See the man
page of squatter for details. The default search
tier also requires the definition of an according
searchtierpartition-name entry.
This option MUST be specified for xapian search.
...
searchtierpartition-name:
The pathname where to store the xapian search indexes of
searchtier for mailboxes of partition name. This must be
configured for the defaultsearchtier and any additional search
tier (see squatter for details).
For example: if defaultpartition is defined as part1 and
defaultsearchtier as tier1 then the configuration must
contain an entry tier1partitionname-part1 that defines the
path where to store this tier1's search index for the
part1 partition.
This option MUST be specified for xapian search.
This is all so muddled, due to the use of the word "default" as an
embedded string within so many settings, some of which refer to a
default value and some of which refer to a partition called
"default". I really think we need to go through the documentation
from top to bottom and weed out such confusing language. The same is
true for the various uses of the word "archive" and some others.
This sort of thing is clearly leading to some folks just giving up on
complex Cyrus features, the implementation of which depend upon
piercing the veil of confusion surround this language (he says in
frustration).
Oh, and that imapd.conf(5) comment "(see sqautter for details)" is
useless, as the squatter(8) man page says nothing about this (damn
manpage writers!).
Thanks in advance,
-nic
On 12/20/2017 04:04 PM, Bron Gondwana wrote:
Totally! The name "archive" is overused. It could be called
something else easily enough.
Bron.
On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote:
Bron,
Thanks for the swift reply. So if I understand this correctly, the
"archivesearchpartition-default" is named such because it's for the
archive location of Xapian search data, not because it's Xapian
search data from an Archive partition. Is that correct? In other
words, this
Re: Xapian partition definition when using archiving?
Bron, et al., Okay, one more time around on this. When I try a version of the partition layout listed below, in my original post, and run "cyr_info conf-lint" against it, I get all of the "/yadda/searchtier: blah" lines thrown back at me. I had assumed that "/default/searchtier: blah" was to specify to which tier to index the default partition, but that's wrong, isn't it? It's really the default search tier to use, and has no direct relationship to the "default" partition. Am I correct in this new interpretation? It is implied by the man page for imapd.conf (derived from lib/imapoptions): defaultsearchtier: Name of the default tier that messages will be indexed to. Search indexes can be organized in tiers to allow index storage in different directories and physical media. See the man page of squatter for details. The default search tier also requires the definition of an according searchtierpartition-name entry. This option MUST be specified for xapian search. ... searchtierpartition-name: The pathname where to store the xapian search indexes of searchtier for mailboxes of partition name. This must be configured for the defaultsearchtier and any additional search tier (see squatter for details). For example: if defaultpartition is defined as part1 and defaultsearchtier as tier1 then the configuration must contain an entry tier1partitionname-part1 that defines the path where to store this tier1's search index for the part1 partition. This option MUST be specified for xapian search. This is all so muddled, due to the use of the word "default" as an embedded string within so many settings, some of which refer to a default value and some of which refer to a partition called "default". I really think we need to go through the documentation from top to bottom and weed out such confusing language. The same is true for the various uses of the word "archive" and some others. This sort of thing is clearly leading to some folks just giving up on complex Cyrus features, the implementation of which depend upon piercing the veil of confusion surround this language (he says in frustration). Oh, and that imapd.conf(5) comment "(see sqautter for details)" is useless, as the squatter(8) man page says nothing about this (damn manpage writers!). Thanks in advance, -nic On 12/20/2017 04:04 PM, Bron Gondwana wrote: Totally! The name "archive" is overused. It could be called something else easily enough. Bron. On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote: Bron, Thanks for the swift reply. So if I understand this correctly, the "archivesearchpartition-default" is named such because it's for the archive location of Xapian search data, not because it's Xapian search data from an Archive partition. Is that correct? In other words, this is just another circumstance where seemingly obvious partition names (like default-default) get us into documentation trouble. Right? Thanks again, -nic On 12/20/2017 05:39 AM, Bron Gondwana wrote: Hi Nic, The Xapian partitions are entirely separate from archive partitions. The indexing code will find the file in the correct location if it needs to read it. Here's an example from one of our servers: defaultpartition: default defaultsearchtier: temp partition-default: /mnt/ssd21d2/sloti21d2t40/store254/spool tempsearchpartition-default: /tmpfs-search/sloti21d2t40 metasearchpartition-default: /mnt/ssd21d2/sloti21d2t40/store254/search datasearchpartition-default: /mnt/i21d2search/sloti21d2t40/store254/search archivesearchpartition-default: /mnt/i21d2search/sloti21d2t40/store254/search-archive archivepartition-default: /mnt/i21d2t40/sloti21d2t40/store254/spool-archive (clearly auto-generated!) Bron. On Wed, 20 Dec 2017, at 07:32, Nic Bernstein wrote: Bron, et al., We're about to set up a whole new bunch of partitions in support of Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will be introducing archive functionality, too. How do archive partitions and Xapian partitions interact? For example, the server currently has the following in imapd.conf: defaultpartition: default partition-default: /var/mailstores/default metapartition-default: /var/imapmeta/default partition-1: /var/mailstores/1 partition-2: /var/mailstores/2 partition-3: /var/mailstores/3 partition-4: /var/mailstores/4 ... partition-29: /var/mailstores/29 partition-30: /var/mailstores/30 partition-100: /var/mailstores/100 partition-temp: /var/mailstores/temp ... # non-default metapartitions metapartition-1: /var/imapmeta/1 metapartition-2: /var/imapmeta/2 metapartition-3: /var/imapmeta/3 metapa
Xapian partition definition when using archiving?
Bron, et al., We're about to set up a whole new bunch of partitions in support of Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will be introducing archive functionality, too. How do archive partitions and Xapian partitions interact? For example, the server currently has the following in imapd.conf: defaultpartition: default partition-default: /var/mailstores/default metapartition-default: /var/imapmeta/default partition-1: /var/mailstores/1 partition-2: /var/mailstores/2 partition-3: /var/mailstores/3 partition-4: /var/mailstores/4 ... partition-29: /var/mailstores/29 partition-30: /var/mailstores/30 partition-100: /var/mailstores/100 partition-temp: /var/mailstores/temp ... # non-default metapartitions metapartition-1: /var/imapmeta/1 metapartition-2: /var/imapmeta/2 metapartition-3: /var/imapmeta/3 metapartition-4: /var/imapmeta/4 ... metapartition-29: /var/imapmeta/29 metapartition-30: /var/imapmeta/30 metapartition-100: /var/imapmeta/100 metapartition-temp: /var/imapmeta/temp Going by the documentation, which I wrote with help from you good folk at Fastmail, the Archive partition scheme might look something like this: https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html archivepartition-default: /var/mailarchives/default archivepartition-1: /var/mailarchives/1 archivepartition-2: /var/mailarchives/2 archivepartition-3: /var/mailarchives/3 archiveartition-4: /var/mailarchives/4 ... archivepartition-29: /var/mailarchives/29 archivepartition-30: /var/mailarchives/30 archivepartition-100: /var/mailarchives/100 archivepartition-temp: /var/mailarchives/temp And the Xapian partition structure to mate with this would look something like this (again, from the docs): https://www.cyrusimap.org/imap/developer/install-xapian.html defaultpartition: default partition-default: /var/mailstores/default search_engine:xapian search_index_headers: no search_batchsize: 8192 defaultsearchtier: t1 1searchtier: t1 2searchtier: t1 3searchtier: t1 4searchtier: t1 ... 29searchtier: t1 30searchtier: t1 100searchtier: t1 tempsearchtier: t1 ... t1searchpartition-default: /var/search/default t1searchpartition-1: /var/search/1 t1searchpartition-2: /var/search/2 t1searchpartition-3: /var/search/3 t1searchpartition-4: /var/search/4 ... t1searchpartition-29: /var/search/29 t1searchpartition-30: /var/search/30 t1searchpartition-100: /var/search/100 t1searchpartition-temp: /var/search/temp First question, since there's no examples to work from; Is this Xapian layout correct? Do I need to define & create Xapian partitions for the metadata partitions, as is indirectly implied in Bron's original email on this topic: https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html Also, how do these Xapian and Archive, interact? Do I need to add a separate Xapian partition for each Archive partition, or will the Archive partition be treated like a child of the non-Archive partition? (again, implied but not directly addressed in that email). Any guidance gladly accepted, and whatever I learn will be repackaged into more complete documentation on same. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 <>
Re: Notes 27/11
Nicola, Thanks for this write-up. My comments interspersed, below... On 11/27/2017 06:43 PM, Nicola Nye wrote: Summary of feedback from the Write the Docs doc sprint last Friday. http://www.writethedocs.org/conf/au/2017/ As mentioned, I had two users I was talking with. One is a technical writer and used to dealing with subject-specific terminology but had no experience with mail systems. The other is an engineer who currently runs Dovecot for a side project and thinks he might have used Cyrus in the past. *Front page* Wall of text and acronyms. Intimidating. I'm looking at the website (https://www.cyrusimap.org/) and I'm not seeing a lot of acronyms. Were you referring to some other location? * Pull in content from the github readme that explains in simpler language what Cyrus is Yes! * Some images to break up the text and reinforce that Cyrus supports not just mail, but calendaring and contacts on mobile too. Yes! * Showcase places that Cyrus is being used. (If you're happy for your organization name to appear in the docs, please let me know! Optional: send me a logo and a link) This helps distinguish Cyrus as mail software, not a mail hosting service. * If there's anyone who does professional services support for Cyrus, make sure they're listed. (anyone? anyone?) We do, we do! Onlight, Inc. are an Internet consultancy, and Cyrus has been a focus of ours for over 20 years. http://www.onlight.com/products/email-groupware/ * Explain target profile. "You want to run Cyrus if you want these features, and you have this many mailboxes/users and you know what you're doing with running/supporting a mail system. You do not want to run Cyrus if: ..." I think we've developed some verbiage on this, which we'll be happy to share. I'll dig it up when I've got a chance. *Navigation* The top navigation bar confused users. They were expecting dropdown sub menu, but then there's also the left menu. * The top menu bar no longer needs to be there. It's a hangover from when the left hand menu bar was a maze of twisty passages all alike and I wanted a way to quickly help users find common entry points. The search box was overlooked. * I might make this bigger and/or put it in the top menu bar as the only thing. Confused by left hand menu ordering. * They both expected the Overview to come first, followed by Quickstart, then Download, then Setup. * They did like the categorization of the top level menu hierarchy. Easy to drill down one level. * Second level and beyond menu items were intimidating in many places. Again, technical overwhelm. I agree with all of this. Reminds me of the late discussions we had whilst I was in Melbourne. *Content* * Cyrus is not a complete solution on its own: you need other system components (MTA: sendmail/postfix/exim/..., possibly webmail interface). * Both users went looking for system requirements and couldn't find a succinct list. I'll need to write this. (including content from the above point) * They wanted to know if it could be run on vmware (or equivalent) or cloud hosted. * Overview was deemed too technical. o The Overview/Features section was just okay o The Overview/Concepts section was more of a reference manual than an explanation of ideas: need to review and split this apart. The documentation has plenty of ways to evolve! If anyone is interested in helping out, do let me know. I should be able to get back into this in the new year. I've just completed an extraordinary season of travel, and have three Cyrus upgrades planned in the next month. Come January, I should have a lot to say about upgrading and new feature deployment. And if you know of anyone who * does professional Cyrus support, or Ahem, * uses Cyrus and is happy to have that listed on our site drop me a line and I'll put you in. Will see if any clients would not object to such a listing. Cheers, -nic Cheers, Nicola On Mon, Nov 27, 2017, at 10:26 PM, Bron Gondwana wrote: Present: Bron, Ken, Nicola, ellie Bron: * Lots of Xapian issues last week * including seqset bug * incorrect UID issue for per-user-data only updates with calendar events. * index_snippets is keeping the mailbox locked while reading from disk and generating snippets, which can be slow and block out other actions. We should find a solution to this. Ken: * working on docs for internals changes - sync, rename safety etc * SASL - looks like Alexey is correct, and people were relying on buggy behaviour * found a logic error with Bearer Token authentication. * short week this week Nicola: * went to write-the-docs on Friday, single day conference * didn't fix any doc bugs * had people with mail server knowledge - used them as guinea-pigs to look at our existing docs - have notes which will write up and send to
Won't make conference call...
Cyrus friends, Having just returned late from travels, I shan't make the meeting to commence in a mere 7hrs. Please have productive fun in my absence. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight Inc. www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: [cyrusimap/cyrus-imapd] man cyradm: document getmd and setmd (#1965)
Dillian, Please read the very next sentence in the documentation: "To set annotations for another user you must *authorize* as that user." (emphasis added) If the user invokes cyradm without a specific --authz parameter, they will automatically *authorize* with the same identity as they have *authenticated*. Cheers, -nic PS - Please direct queries related to Cyrus directly to the list and not to my company email (which this is). On 05/18/2017 11:23 AM, Дилян Палаузов wrote: Hello Nic, Note, too, that "private" annotations are private to the user currently authenticated as, not necessarily the owner of the mailbox. is this really the user who is authenticated, or the user who is authorized? In terms of cyradm --user A --authz B, will setdm --private Spam "\\Junk" will set the special use for A or for B? Greetings Dilian On 05/18/17 18:02, Nic Bernstein wrote: This is addressed in PR #1977 <https://github.com/cyrusimap/cyrus-imapd/pull/1977> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/cyrusimap/cyrus-imapd/issues/1965#issuecomment-302450965>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEwvsyktvWzEBcOYh98_-61B42jGbYypks5r7GukgaJpZM4NV59T>. <>
How to prime user calendars in Cyrus 3?
Ken, I'm back to the project of trying to migrate my own Cal/CardDav users from DaviCal to Cyrus 3. Problem I'm faced with is that other than myself, my users do not have #calendars or #addressbooks in the Cyrus mail store. Thus my migration script fails, as destination collections do not exist. I do have "|caldav_create_default:| 1" set, but when pointing a browser to "http://newjiji:8008/dav/calendars/user/" for users who don't already have the collections created, I get a 404 error "Mailbox does not exist." So my question is, how does one get these users primed? My migration scripts use the administrative user to connect and perform the work, so such connection is not happening as the user themselves. If my only recourse is to make each user connect a calendar client, then I'll do that, but I don't think that scales well. My concern here is dealing with sites which, like our own, have already deployed a CalDav solution and are looking to migrate to the inbuilt solution on Cyrus. Any thoughts? Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 <>
Re: RPM packaging help sought
Pavel, Could you send us a copy of your spec file, so we can get this into our testing rig? Please send to myself or Nicola. -nic On 04/06/2017 03:31 PM, Pavel Zhukov wrote: Hi, I'm working on update in Fedora. https://bugzilla.redhat.com/show_bug.cgi?id=1435623 It should land in rawhide soon and hopefully in will be included in next Fedora release (F27). Nic Bernstein <n...@onlight.com> writes: Cyrus IMAP Friends, We're preparing a small set of "official" Cyrus IMAP packages for the purposes of bootstrapping adoption of 3.0 within the community of people loathe to install from source. We've got packages for Ubuntu (Xenial) and Debian (Jessie) but would appreciate if someone with recent RPM building experience would be willing to step up and prepare the appropriate spec file for modern Fedora/RHEL/CentOS and related distros. It's not our goal to supplant vendor-supplied packages. We merely want to leapfrog the current situation where we want folks to be using Cyrus IMAP 3.0.0 as soon as possible, but there are no packages available to start with. Please respond to the list, or to myself or Nicola Nye <nico...@fastmail.com>. Thanks in advance, -nic -- Nic Bernstein n...@onlight.com Onlight Inc. www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: cyrus 2.5, 3.0 and DAV
Dilia, Thanks for your notes. I'm forwarding this to the cyrus-devel list, as I don't have answers for all of this. Please submit any further documentation notes to the mailing list for action, or open an issue on github: https://github.com/cyrusimap/cyrus-imapd/issues Cheers, -nic On 04/04/2017 05:05 AM, Дилян Палаузов wrote: Some other DAV related questions: https://cyrusimap.org/imap/rfc-support.html should not mention RFC 2445, as it is obsoleted by RFC 5545. https://cyrusimap.org/imap/rfc-support.html mentions draft-daboo-calendar-availability and the latter is superseded by RFC7953. If Cyrus does support the draft, but not the RFC, please make this clear at the abovementioned address. Likewise for RFC7953 and draft-ietf-calext-availability . 2.5/docs/specs.html and https://cyrusimap.org/imap/rfc-support.html mention "RFC 6231 xCal: The XML Format for Icalendar", but the xcal rfc is 6321. Why does https://cyrusimap.org/imap/rfc-support.html end with: RFC Wishlist RFC 6851 Internet Message Access Protocol (IMAP) - MOVE Extension New in version 2.5.0. Is RFC6851 in the wishlist, or is implemented in 2.5? Do RFC6715, RFC 7986, RFC 6474, RFC 6473 require any changes on the server side, in terms of grammar parsing, in order to be supported? Can you document under "Features' what is supported at Dav/Caldav/Carddav level and what is not supported? In particular CardDav 4.0, WebDav (CardDav + CalDav) resource sharing, as discussed at https://evertpot.com/webdav-caldav-carddav-sharing/ and the mentioned drafts E.g. I know that in 2.5 the non-standard apple calendar colors can be set and CardDav 4.0 is not supported, but this is nowhere documented. As for Carddav 4.0, which is RFC 6350, it is mentioned under 2.5/docs/specs.html, but CardBook does not work, if I specify, that my address book is 4.0, it works when I enter 3.0. My experience with CardBook is that it works reliably, so I guess the 4.0 problem is on cyrus side. Greetings Dilia On 04/03/2017 08:19 PM, Дилян Палаузов wrote: Hello, docsrc/imap/concepts/features/dav-collection-mgmt.rst is rendered at https://cyrusimap.org/imap/concepts/features/dav-collection-mgmt.html as "https:///dav/calendars/user/%3Cusername%3E". Yo mean probably and an obligatory terminating slash. This happens also for the addressbooks URL. At the header of the page is written: "Calendars and addressbooks are maintained as “Collections” within the Cyrus mail store. They appear as mailboxes within the heirarchy, as set by the calendarprefix: option in imapd.conf(5) (default is #calendars), but should rarely be directly manipulated using either cyradm(8) or other mailbox-centric tools." If claendarprefix is mentioned, so must be addressbookprefix. Greetings Dilian -- Nic Bernstein n...@onlight.com Onlight Inc. www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: Typo in quota.h or imapoptions?
Thanks, will update accordingly.
-nic
On 01/19/2017 05:43 PM, Bron Gondwana via Cyrus-devel wrote:
The code is correct, so quotas.db.
Bron.
On Fri, 20 Jan 2017, at 09:16, Nic Bernstein via Cyrus-devel wrote:
Folks,
cyrus-imapd/lib/imapoptions (line 1655) says this:
{ "quota_db_path", NULL, STRING }
/* The absolute path for the quota database (if you choose a single-file
quota DB type - or the base path if you choose quotalegacy). If
not specified will be confdir/*quota.db* or confdir/quota/ */
but cyrus-imapd/imap/quota.h (line 55) says this:
#define FNAME_QUOTADB "/*quotas.db*"
Which should it be?
-nic
--
Nic bernstein...@onlight.com <mailto:n...@onlight.com>
Onlight, Inc.www.onlight.com <http://www.onlight.com>
6525 W Bluemound Road, Suite 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073
Email had 1 attachment:
*
|nic.vcf|
1k (text/x-vcard)
--
Bron Gondwana
br...@fastmail.fm
--
Nic Bernstein n...@onlight.com
Onlight Inc. www.onlight.com
6525 W Bluemound Rd., Ste 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Quota namespace questions
Friends,
I'm working on documentation for various things Quota related, and find
that I'm a little confused about the "quota namespace." It appears
that, due to the second argument in
"mboxname_init_namespace(_namespace, 1)" (quota.c:176) that quota
operations don't use 'altnamespace':
namespace->isadmin = isadmin;
namespace->hier_sep =
config_getswitch(IMAPOPT_UNIXHIERARCHYSEP) ? '/' : '.';
namespace->isalt = !isadmin && config_getswitch(IMAPOPT_ALTNAMESPACE);
namespace->accessible[NAMESPACE_INBOX] = 1;
namespace->accessible[NAMESPACE_USER] =
!config_getswitch(IMAPOPT_DISABLE_USER_NAMESPACE);
namespace->accessible[NAMESPACE_SHARED] =
!config_getswitch(IMAPOPT_DISABLE_SHARED_NAMESPACE);
if (namespace->isalt) {
...
}
else {
/* standard namespace */
sprintf(namespace->prefix[NAMESPACE_INBOX], "%s%c",
"INBOX", namespace->hier_sep);
sprintf(namespace->prefix[NAMESPACE_USER], "%s%c",
"user", namespace->hier_sep);
strcpy(namespace->prefix[NAMESPACE_SHARED], "");
}
return 0;
}
Am I reading that right (hint: not a C programmer)? But it also looks
to me like it should still use '/' as the hierarchy separator, right?
That's not what I'm seeing in my quota_db. I started with a system with
no quotas. The old configuration used quotalegacy, so when I added some
quotas, they ended up in /var/lib/imap/quota...:
# cyradm -U cyrus localhost
Password:
localhost> sq user/nic STORAGE 2000
localhost> sq user/tim STORAGE 2000
localhost> sq user/crystal STORAGE 2000
localhost> sq user/nic X-NUM-FOLDERS 1000
localhost> quit
# head /var/lib/imap/quota/*/*
==> /var/lib/imap/quota/H/user.crystal <==
%(S (1190064060 2000) M (13742) AS (3) NF (15))
==> /var/lib/imap/quota/K/user.tim <==
%(S (401903846 2000) M (7308) AS (2) NF (8))
==> /var/lib/imap/quota/N/user.nic <==
%(S (1484883490) M (36930) AS (18) NF (42 1000))
I then used cvt_cyrusdb to convert from quotalegacy to twoskip, and I
still am seeing the netnews separator, rather than unix, both in the
quota_db and in the output from "quota -f":
# strings /tmp/quota.db
twoskip file
user.crystal%(S (1190064060 2000) M (13742) AS (3) NF (15))
user.nic%(S (1484883490 2000) M (36930) AS (18) NF (42))
user.tim%(S (401903846 2000) M (7308) AS (2) NF (8))$
# su cyrus -c "cyrus quota -f"
Quota % Used Used Resource Root
20005 1162171 STORAGE user.crystal
13742 MESSAGE user.crystal
0 X-ANNOTATION-STORAGE user.crystal
15X-NUM-FOLDERS user.crystal
1450081 STORAGE user.nic
36930 MESSAGE user.nic
0 X-ANNOTATION-STORAGE user.nic
10004 42X-NUM-FOLDERS user.nic
20001 392484 STORAGE user.tim
7308 MESSAGE user.tim
0 X-ANNOTATION-STORAGE user.tim
8X-NUM-FOLDERS user.tim
Now to be clear, I have no problem with this if it works, but I'm
concerned about confusing administrators.
We already have the task of teaching existing Cyrus admins about all of
the ramifications of converting to 'altnamespace: on' and
'unixhierarchysep: on' as the new defaults. This brings a new "but not
here" context on top of it.
Similarly, for new folks, who don't know or care about historical
legacies, we need to explain that while they're used to using slash "/"
they won't see that from "quota" runs, or when they go poking around to
repair quota problems.
I think I just need a good dose of cluefullness to proceed. :-)
Cheers,
-nic
PS - perl/imap/IMAP/Shell.pm POD info still says "The only I
understood by B is C." This needs updating (I'm happy
to do this.)
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073
<>
Update to cyr_info(8) to reflect "reid" command
On Friday Jan 13, at 09:12AM CST, Onlight wrote: brong: or whomever, cyr_info appears to have a command "reid" which is not explained in either the usage or man page. The purpose seems to be to create a new UID for a given mailbox. Is that correct? Shall I add that to the usage and man page? On Sunday Jan 15, at 10:52PM CST, Bron wrote: onlight: yeah, it creates a new uniqueid for the mailbox. It's slightly buggy it should be documented though. It does indeed just give the mailbox a new uniqueid. It doesn't fix up seen state for other people or anything like that This is fixed in the following pull request #1798 <https://github.com/cyrusimap/cyrus-imapd/pull/1798> Add "reid" command information to cyr_info #1798 Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 <>
Re: State of Perl scripts in tools, esp. in re: hash
After much git heroism (new desktop) a new version of translatesieve is available here: https://github.com/cyrusimap/cyrus-imapd/pull/1792 Comments please, or merge. -nic On 01/13/2017 12:39 PM, Nic Bernstein via Cyrus-devel wrote: On 01/13/2017 07:00 AM, Nic Bernstein via Cyrus-devel wrote: I can (and, honestly, already have) fix translatesieve, and a similar fix can be added to upgradesieve or any other of these scripts. I think rehash should be left alone, as anyone who's already used hashing has what they have, which means they may already have upper-case directories. Let's not break that. I'll have something committed in an hour or two. Okay, that was a bogus claim. Turns out translatesieve assumes that one needs adjust for both altnamespace AND unixhierarchysep. If you have already been using altnamespace, it borks up the sieve scripts mercilessly. This requires a complete rewrite, which I've undertaken, so it won't be getting committed anytime soon. Hopefully this weekend :-) Cheers, -nic -- Nic bernstein...@onlight.com Onlight Inc.www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: State of Perl scripts in tools, esp. in re: hash
On 01/13/2017 07:00 AM, Nic Bernstein via Cyrus-devel wrote: I can (and, honestly, already have) fix translatesieve, and a similar fix can be added to upgradesieve or any other of these scripts. I think rehash should be left alone, as anyone who's already used hashing has what they have, which means they may already have upper-case directories. Let's not break that. I'll have something committed in an hour or two. Okay, that was a bogus claim. Turns out translatesieve assumes that one needs adjust for both altnamespace AND unixhierarchysep. If you have already been using altnamespace, it borks up the sieve scripts mercilessly. This requires a complete rewrite, which I've undertaken, so it won't be getting committed anytime soon. Hopefully this weekend :-) Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 <>
Re: State of Perl scripts in tools, esp. in re: hash
Bron,
I'll agree that there are good reasons to keep this part (i.e. systems
sans shebang):
exec perl -x -S $0 ${1+"$@"} # -*-perl-*-
Or even to keep the second chunk, which searches for a Perl 5 interpreter:
if ($] !~ /^5\..*/) {
# uh-oh. this isn't perl 5.
foreach (split(/:/, $ENV{PATH})) { # try to find "perl5".
exec("$_/perl5", "-x", "-S", $0, @ARGV) if (-x "$_/perl5");
}
# we failed. bail.
die "Your perl is too old; I need perl 5.\n";
}
But my argument in re: Perl5 being over 20 years old bears on that.
The ugly, gnarly part is the final eval blob to protect potential Perl 4
users.
# load the real script. this is isolated in an 'eval' so perl4 won't
# choke on the perl5-isms.
eval join("\n", );
if ($@) { die "$@"; }
__END__
And it's unnecessary given that the next line is:
require 5;
That eval's the thing which bullocks up the syntax highlighters and
tends to confuse idle readers.
How about we keep the former and ditch the latter?
-nic
On 01/12/2017 10:54 PM, Bron Gondwana via Cyrus-devel wrote:
It's there because lots of people on non-Linux systems have their perl
in somewhere other than /usr/bin. It is annoying for syntax
highlighting, though we could work around that with some build magic.
It works fine, there's no reason to remove it.
The rehash stuff no the other hand, yeah - those scripts should be
fixed. I don't have time to do it today before the rc though!
Bron.
On Fri, 13 Jan 2017, at 10:30, ellie timoney via Cyrus-devel wrote:
I'm 100% in favour of discarding the "exec perl" hack -- solely
because it confuses syntax highlighting something fierce :P
But I have no particular familiarity with its historical context, so
can't reliably evaluate whether it's still needed for some obscure reason
On Fri, Jan 13, 2017, at 06:08 AM, Nic Bernstein via Cyrus-devel wrote:
Folks,
I've been working on an upgrade from 2.4.18 to 3.0.0rc1, and
simultaneously adding to Nicola's fine Upgrading document. Along
the way, however, I've encountered several anomalies with the wide
assortment of Perl scripts accumulated in cyrus-imapd/tools.
There's a lot of them, they often overlap and sometimes don't
inter-operate.
For example, 'rehash' understands the 'fulldirhash' setting, but
'dohash' does not. Furthermore, that setting, when processed by
'rehash', results in directories named "A" - "Z", but most of the
other tools, such as 'translatesieve' or 'upgradesieve' (why so
many?) explicitly expect these to be lower cased, "a" - "z":
foreach $i ("a".."z") {
...
and so ignore the fulldirhash-ed directories.
Also, most of the Perl scripts still have this sort of baggage:
exec perl -x -S $0 ${1+"$@"} # -*-perl-*-
...
if ($] !~ /^5\..*/) {
# uh-oh. this isn't perl 5.
foreach (split(/:/, $ENV{PATH})) { # try to find "perl5".
exec("$_/perl5", "-x", "-S", $0, @ARGV) if (-x "$_/perl5");
}
# we failed. bail.
die "Your perl is too old; I need perl 5.\n";
}
# load the real script. this is isolated in an 'eval' so perl4 won't
# choke on the perl5-isms.
eval join("\n", );
if ($@) { die "$@"; }
__END__
require 5;
...
Given that Perl5 was released over twenty years ago, do we really
need this, as opposed to say, "#!/usr/bin/perl -w"? Hell, we could
even parameterize that and substitute with @PERL@?
Just asking, because one needs a tool such as translatesieve to
handle the transition to ``unixhierarchysep: yes'', and as it sits,
that tool won't work. I'm happy to fix it, but would like guidance
on how far to go.
Cheers,
-nic
--
Nic bernstein...@onlight.com <mailto:n...@onlight.com>
Onlight, Inc.www.onlight.com <http://www.onlight.com>
6525 W Bluemound Road, Suite 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073
Email had 1 attachment:
*
|nic.vcf|
1k (text/x-vcard)
--
Bron Gondwana
br...@fastmail.fm
--
Nic Bernstein n...@onlight.com
Onlight Inc. www.onlight.com
6525 W Bluemound Rd., Ste 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: State of Perl scripts in tools, esp. in re: hash
I can (and, honestly, already have) fix translatesieve, and a similar
fix can be added to upgradesieve or any other of these scripts. I think
rehash should be left alone, as anyone who's already used hashing has
what they have, which means they may already have upper-case
directories. Let's not break that.
I'll have something committed in an hour or two.
-nic
On 01/12/2017 10:54 PM, Bron Gondwana via Cyrus-devel wrote:
It's there because lots of people on non-Linux systems have their perl
in somewhere other than /usr/bin. It is annoying for syntax
highlighting, though we could work around that with some build magic.
It works fine, there's no reason to remove it.
The rehash stuff no the other hand, yeah - those scripts should be
fixed. I don't have time to do it today before the rc though!
Bron.
On Fri, 13 Jan 2017, at 10:30, ellie timoney via Cyrus-devel wrote:
I'm 100% in favour of discarding the "exec perl" hack -- solely
because it confuses syntax highlighting something fierce :P
But I have no particular familiarity with its historical context, so
can't reliably evaluate whether it's still needed for some obscure reason
On Fri, Jan 13, 2017, at 06:08 AM, Nic Bernstein via Cyrus-devel wrote:
Folks,
I've been working on an upgrade from 2.4.18 to 3.0.0rc1, and
simultaneously adding to Nicola's fine Upgrading document. Along
the way, however, I've encountered several anomalies with the wide
assortment of Perl scripts accumulated in cyrus-imapd/tools.
There's a lot of them, they often overlap and sometimes don't
inter-operate.
For example, 'rehash' understands the 'fulldirhash' setting, but
'dohash' does not. Furthermore, that setting, when processed by
'rehash', results in directories named "A" - "Z", but most of the
other tools, such as 'translatesieve' or 'upgradesieve' (why so
many?) explicitly expect these to be lower cased, "a" - "z":
foreach $i ("a".."z") {
...
and so ignore the fulldirhash-ed directories.
Also, most of the Perl scripts still have this sort of baggage:
exec perl -x -S $0 ${1+"$@"} # -*-perl-*-
...
if ($] !~ /^5\..*/) {
# uh-oh. this isn't perl 5.
foreach (split(/:/, $ENV{PATH})) { # try to find "perl5".
exec("$_/perl5", "-x", "-S", $0, @ARGV) if (-x "$_/perl5");
}
# we failed. bail.
die "Your perl is too old; I need perl 5.\n";
}
# load the real script. this is isolated in an 'eval' so perl4 won't
# choke on the perl5-isms.
eval join("\n", );
if ($@) { die "$@"; }
__END__
require 5;
...
Given that Perl5 was released over twenty years ago, do we really
need this, as opposed to say, "#!/usr/bin/perl -w"? Hell, we could
even parameterize that and substitute with @PERL@?
Just asking, because one needs a tool such as translatesieve to
handle the transition to ``unixhierarchysep: yes'', and as it sits,
that tool won't work. I'm happy to fix it, but would like guidance
on how far to go.
Cheers,
-nic
--
Nic bernstein...@onlight.com <mailto:n...@onlight.com>
Onlight, Inc.www.onlight.com <http://www.onlight.com>
6525 W Bluemound Road, Suite 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073
Email had 1 attachment:
*
|nic.vcf|
1k (text/x-vcard)
--
Bron Gondwana
br...@fastmail.fm
--
Nic Bernstein n...@onlight.com
Onlight Inc. www.onlight.com
6525 W Bluemound Rd., Ste 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Missing Upgrade docs [was Re: Release plan blog post]
On 12/27/2016 06:04 PM, Bron Gondwana via Cyrus-devel wrote:
Hi,
Sorry for the delay in responding to this - I left it over Christmas
so I could sit down without distraction and reply when I was back in
the office.
On Sat, 24 Dec 2016, at 17:09, Anatoli via Cyrus-devel wrote:
*3. New deployments* (vs ongoing upgrades/maintenance). How easy and
straightforward it is to setup a new deployment (possibly migrating
from other email servers). Here I'm referring to both the initial
configuration, tools and documentation.
Yeah, we know about this one. I'm not going to create a specific bug
for it, because it's kind of spread out over lots of different
things. Nicola is working on improving our documentation, but again
the best people to give advice are people who've recently done it. I
haven't really "installed Cyrus from scratch" for 12 years, certainly
not without the FastMail configuration and build systems. Except for
the test environment, which has its own special magics.
Hey, on a related note, I've just noticed that
"cyrus-imapd/doc/legacy/install-upgrade.html" is not up on cyrusimap.org
and there doesn't appear to be any equivalent. Is there a plan for this?
I'll confess that I'm a little perplexed by how the stuff at
cyrusimap.org is updated or maintained. The document tree doesn't
seem to match what I see in git.
I'm not trying to be whiny here; Hell, I'll even help with it. I
just don't know what the general plan is, or where things are
supposed to go.
Seeing as I'm in the midst of a 2.3.16 => 2.5.10 upgrade, I can probably
help, but I also need to find the proper information.
Cheers,
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073
<>
Re: Where did old Tasks go? [was Re: git.cyrus.foundation deprecated]
On 12/21/2016 05:55 PM, Nicola Nye wrote: So it looks like the release notes going to 2.5 need to understand the implications of using the dot separator. Please go ahead and update the release notes for 2.5.X accordingly! Of course, if you're going to move to the (soon to be released) 3.0, we ditch netnews in favour of unixhs anyway. (Now more fully documented http://cyrusimap.org/dev/imap/concepts/features/namespaces.html) But we do need to check that the docs don't contain any more references to poor Phabricator as it's now defunct. Long may it RIP. Nicola, Actually, that snippet from Jeroen's commit had nothing at all to do with T16, as I had previously reported. He must have typo'd that. However, Bron helpfully mined the Internet archive to find the original T16, which included this text: When upgrading a Cyrus IMAP Murder (discrete), 2.4 backends do not advertise the MOVE capability, but 2.5 frontends will -- and happily proxy the UID MOVE command despite the fact that the backends do not support RFC 6851 <https://web.archive.org/web/20150406213645/https://tools.ietf.org/html/rfc6851> So the fix is to disable MOVE on the frontend by suppressing the capability. But, what users have reported on the list is that the approach of using "suppress_capabilities" may not do the trick. So, I have updated the 2.5.0 Release Notes thus: Cyrus IMAP Murder Topologies Environments that run a Cyrus IMAP Murder topology will want to upgrade their backends before they upgrade their frontends: When upgrading a Cyrus IMAP Murder (discrete), 2.4 backends do not advertise the |MOVE| capability, but 2.5 frontends will – and happily proxy the |UID MOVE| command despite the fact that the backends do not support *RFC 6851* <https://tools.ietf.org/html/rfc6851.html> So the fix is to disable |MOVE| on the frontend by suppressing the capability. However, subsequent to these notes, users in the field have found the use of |suppress_capabilities| on frontends may not be a suitable fix in all situations. It is unclear to me how to effect this change to github, however, as the file referenced on the active website doesn't appear in the same place in my checkouts. So, I am attaching it herein for you to figure out how to apply. :-) Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 :tocdepth: 3 == Cyrus IMAP 2.5.0 Release Notes == .. IMPORTANT:: Make sure to read the :ref:`imap-relnotes-2.5.0-upgrading` notes (all of them). * :ref:`relnotes-2.5.0-new-features` * :ref:`relnotes-2.5.0-config-option-changes` * :ref:`relnotes-2.5.0-development-autoconf` * :ref:`relnotes-2.5.0-development-phabricator` * :ref:`relnotes-2.5.0-development-documentation` Download via HTTP: * http://www.cyrusimap.org/releases/old/cyrus-imapd-2.5.0.tar.gz * http://www.cyrusimap.org/releases/old/cyrus-imapd-2.5.0.tar.gz.sig Download via FTP: * ftp://ftp.cyrusimap.org/cyrus-imapd/OLD-VERSIONS/cyrus-imapd-2.5.0.tar.gz * ftp://ftp.cyrusimap.org/cyrus-imapd/OLD-VERSIONS/cyrus-imapd-2.5.0.tar.gz.sig .. _imap-relnotes-2.5.0-upgrading: Upgrading to Cyrus IMAP 2.5.0 = It is strongly recommended to shut down the Cyrus IMAP services before performing the upgrade, as the newer binaries will end up writing to ``mailboxes.db`` in a way that is not compatible with the older binaries (that would otherwise still be running). You can start the server immediately after upgrading, however. Underscores in ``cmd`` Names in :cyrusman:`cyrus.conf(5)` - Underscores (the ``_`` character) are no longer allowed in the ``START``, ``SERVICES`` and ``EVENTS`` sections of :cyrusman:`cyrus.conf(5)`, as they interfere with configuration options in :cyrusman:`imapd.conf(5)` being prefixed by service names and an underscore (``_``) character. Database Format Upgrade for Mailboxes - Upgrading to Cyrus IMAP 2.5.0 recommends the upgrade of database formats, for which the reconstruct utility has been modified allowing administrators to run: .. parsed-literal:: # :command:`/usr/lib/cyrus-imapd/reconstruct -V max` [options] This command can be run at the administrator's convenience, and while running the database format upgrade process is not strictly required, it is certainly strongly recommended. Releases prior to 2.5.0, namely the 2.4 series, upgraded ``cyrus.index`` database formats in-line (i.e. as the mailbox in question as being used). This may have caused an I/O storm in some situations, which 2.5.0 aims to address. The minor versio
Where did old Tasks go? [was Re: git.cyrus.foundation deprecated]
Friends,
Where did the Maniphest Tasks go when the move to github was made?
I ask because the release notes for 2.5.X say:
Make sure to read the Upgrading to Cyrus IMAP 2.5.0
<https://cyrusimap.org/imap/release-notes/2.5/x/2.5.0.html#imap-relnotes-2-5-0-upgrading>
notes (all of them).
and those notes contain this:
Cyrus IMAP Murder Topologies
Environments that run a Cyrus IMAP Murder topology will want to
upgrade their backends before they upgrade their frontends. See Task
#16 <https://git.cyrus.foundation/T16> for details.
But the link for Task #16 is dead, still pointing to git.cyrus.foundation:
glop:master$ grep -iR foundation cyrus-imapd/docsrc/
cyrus-imapd/docsrc/imap/download/release-notes/2.5/x/2.5.0.rst:Please
see https://git.cyrus.foundation/.
cyrus-imapd/docsrc/conf.py:
'task':('https://git.cyrus.foundation/T%s', 'Task #'),
I'd be happy to fix that, but am unclear as to where this information
now lives.
Cheers,
-nic
On 06/23/2016 06:37 PM, Bron Gondwana via Cyrus-devel wrote:
Hi all,
Phabricator is being deprecated. The project is moving to github at:
https://github.com/cyrusimap/
I'm working on migrating all the bugs from bugzilla and phabricator to be
github issues. For now the website is still hosted at CMU.
One thing I'm doing is applying a "kill commit" to the top of the git
repositories at the other locations. This doesn't lose any history, you can just reset
--hard HEAD^ to get back to the latest master commit - but it does discourage any further
pushes of commits that get lost, because they won't merge or rebase very happily on top
of the kill commit!
The kill commit contains a small README.txt which tells you how to update your
git repository to point to the new location.
Cheers,
Bron.
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073
<>
Re: Q about gs2 plugin
I suspect you should post this to the Cyrus-SASL mailing list, as this is a SASL issue. More info here: http://cyrusimap.org/feedback-mailing-lists.html Cheers, -nic On 03/10/2016 11:05 AM, Jan Parcel via Cyrus-devel wrote: In 2.1.26, configure --help does not offer a way to enable the gs2 plugin. Does that mean it's severely under-utilized and under-tested, i.e. immature? Is it being tested with imap? Just wanted to get a handle on how experimental it is. Thanks Jan Parcel Senior Software Engineer Oracle -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 <>
Re: Meeting minutes 15 Feb [was Re: Meeting minutes 8 Feb]
On 02/15/2016 05:16 PM, ellie timoney via Cyrus-devel wrote: But the really fiddly thing here is always going to be the fact that we need to build most of the man pages from rst files, but we need to build some of them (and thus their corresponding web pages...) from imapd source files. Taken broadly, i.e. treating "man pages" as a homogenous collection, that's a cyclical dependency. Squishing it all together (options 1, 2, 4) makes that relatively easy to resolve just with autoconf, but option 3 will need tooling/process work. Ellie, Nicola, et alia, As far as I'm aware, the only man page (RST file) which depends upon imapd source files is cyrus-docs/source/imap/admin/configs/imapd.conf, which derives from cyrus-imapd/lib/imapoptions by way of the cyrus-imapd/tools/config2rst perl script -- the same as the old cyrus-imapd/man/imapd.conf.5 was created. This is handled by the top-level Makefile (from Makefile.am): man/imapd.conf.5: $(top_srcdir)/tools/config2man $(top_srcdir)/lib/imapoptions @echo creating man/imapd.conf.5 @$(MKDIR_P) man $(AM_V_GEN)$(top_srcdir)/tools/config2man $(top_srcdir)/lib/imapoptions > $@ doc/rst/imapd.conf.rst: $(top_srcdir)/tools/config2rst $(top_srcdir)/lib/imapoptions @echo creating man/imapd.conf.rst @$(MKDIR_P) doc/rst $(AM_V_GEN)$(top_srcdir)/tools/config2rst $(top_srcdir)/lib/imapoptions > $@ So as long as Sphinx's "make man" target depends on doc/rst/imapd.conf.rst -- which is the same as /imap/admin/configs/imapd.conf.rst, or would be in a post-merge world -- then we should be fine with our existing autoconf stuff. There's also some HTML pages which depend upon doc/rst/imapd.conf.rst, but again, if Sphinx's "make html" target depends on /imap/admin/configs/imapd.conf.rst that should sort itself out, too. What's missing, of course, is some super sweet autoconf goodness on the cyrus-docs side of things, which would wrap this all up. Just my tupence... -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 <>
Re: Meeting minutes 15 Feb [was Re: Meeting minutes 8 Feb]
Oops! My bad! I was looking at the cyrus-sasl repo, which has it's own doc branch, and took that to be the cyrus-sasl documentation. Upon further review, that seems to be mostly the supporting RFC data. My apologies for the snap-answer. I've spent so much time buried in cyrus-docs, I somehow managed to miss this. -nic On 02/15/2016 07:14 PM, ellie timoney via Cyrus-devel wrote: But the cyrus-docs repository appears (at a superficial glance) to contain SASL docs? So merging it wholesale into the cyrus-imapd repository, especially in a way that obsoletes or removes the cyrus-docs repository, could have ramifications On Tue, Feb 16, 2016, at 12:08 PM, Nic Bernstein via Cyrus-devel wrote: The cyrus-sasl <https://git.cyrus.foundation/diffusion/S/> software and documentation is in a wholly separate repository, and should not be in any way affected by what happens with the cyrus-docs <https://git.cyrus.foundation/diffusion/D/> and cyrus-imapd <https://git.cyrus.foundation/diffusion/I/> repositories. -nic On 02/15/2016 06:53 PM, ellie timoney via Cyrus-devel wrote: Hmm. I don't know if/how this will affect SASL. On Tue, Feb 16, 2016, at 11:44 AM, Jan Parcel via Cyrus-devel wrote: As a consumer, I'd like to say a couple of things. 1. I really REALLY don't want to see it become difficult, on my end, to get the right versions of the libsasl2 docs on download. Apparently quite a few pages disappeared between 1.5.28 and 2.1.26, but those pages still appear in other places (including Solaris, possibly-corrected for 2.N.N) I don't view them as "part of" imap. 2. I've got some severe formatting problems displaying saslauthd(8) , which I hope is unique to that file and not a trend. saslauthd.8 appears to be pre-roff'd, it looks like the others are not. -- Nic bernstein...@onlight.com <mailto:n...@onlight.com> Onlight Inc.www.onlight.com <http://www.onlight.com> 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335 -- Nic Bernstein n...@onlight.com Onlight Inc. www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: Meeting minutes 15 Feb [was Re: Meeting minutes 8 Feb]
The cyrus-sasl <https://git.cyrus.foundation/diffusion/S/> software and documentation is in a wholly separate repository, and should not be in any way affected by what happens with the cyrus-docs <https://git.cyrus.foundation/diffusion/D/> and cyrus-imapd <https://git.cyrus.foundation/diffusion/I/> repositories. -nic On 02/15/2016 06:53 PM, ellie timoney via Cyrus-devel wrote: Hmm. I don't know if/how this will affect SASL. On Tue, Feb 16, 2016, at 11:44 AM, Jan Parcel via Cyrus-devel wrote: As a consumer, I'd like to say a couple of things. 1. I really REALLY don't want to see it become difficult, on my end, to get the right versions of the libsasl2 docs on download. Apparently quite a few pages disappeared between 1.5.28 and 2.1.26, but those pages still appear in other places (including Solaris, possibly-corrected for 2.N.N) I don't view them as "part of" imap. 2. I've got some severe formatting problems displaying saslauthd(8) , which I hope is unique to that file and not a trend. saslauthd.8 appears to be pre-roff'd, it looks like the others are not. -- Nic Bernstein n...@onlight.com Onlight Inc. www.onlight.com 6525 W Bluemound Rd., Ste 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: Meeting today (Jan 4)
On 01/03/2016 06:21 PM, Bron Gondwana via Cyrus-devel wrote:
Happy New Year all in Cyrus Land
We're going to have a meeting today - it's been a while, and I've missed the
last couple of timeslots due to moving house and not being set up for it.
The meeting will be a the usual time: 11am GMT (10pm Melbourne, 6am New York,
etc)
That's the brutally early hour of 5:00CST, so I'll miss this one. I
would like to get a thread going, however, about documentation again,
and the grand reunification with the source tree, as a recently
rejuvenated discussion ("Unifying the Cyrus World") touched on in this list.
I know I've been out of the loop a bit lately, but think I could start
to pick things up a bit. In particular, I think Patrick Goetz made an
interesting point in a recent post in the Unification thread:
On 12/30/2015 07:01 AM, Patrick Goetz wrote:
Tutorials are good even when they dont match your exact use case.
Too much abstract hand-waving is just confusing to most people.
I'd rather read a common use case tutorial and then extrapolate to
my own situation from there. Way too many critical details end up
getting left out when you talk about these issues absractly.
...
My suggestion is to look at the organization of the documentation
from different perspectives:
- a new or potentially new cyrus user who knows absolutely nothing
- a somewhat experienced cyrus admin looking to quickly get an
answer to a specific question
- a cyrus admin who wants to implement some new functionality,
say moving from a single server to a murder, or start using sieve
scripts
I think Patrick's got some good ideas there, although his citations (in
the unquoted portions of the message) indicate that his comments and
complaints have mostly to do with the older cyrusimap.org documentation,
and not the newer work that Jeroen, Nicola and myself have worked on.
In any event, if it's at all possible, could someone post minutes or
notes from the meeting? That was handy back when Bron (who has too much
time on his hands ;-) was doing so.
Cheers,
-nic
--
Nic Bernstein n...@onlight.com
Onlight Inc. www.onlight.com
6525 W Bluemound Rd., Ste 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
Re: Unifying the Cyrus World
Nicola, et alia., Firstly, I apologize that I haven't made a hang-out in ages -- for reasons too long, boring and personal to get into -- but I'll stick my head above the ramparts long enough to cause some trouble... On 09/16/2015 08:50 PM, Nicola Nye wrote: Equally, there are some documentation issues that are affected by our current straddling of two worlds. Read on for a discussion of the issues and down to the options and recommendation. I'd love to have your input (particularly Nic Bernstein on the docs angle!) if you agree or disagree so we can make for a more unified Cyrus presence. *Issues* 1) Branding 2) Docs logistics The docs repo is separate to source which is causing friction keeping man pages updated. Yes! We need to get cyrus-docs into cyrus-imapd, and eliminate the problems caused by the split repos. Bron and I discussed this some time back, when I had tackled config2rst, and needed to find a target home for the resultant doc/rst/imapd.conf.rst We want man pages in two formats: *nix for actual man pages to ship and install with cyrus, and html for web presentation. Nice to have: ship a snapshot of all current docs (not just man pages) along with source distribution. 3) Sphinx vs wiki We have been working towards making the content on cyrus.foundation to be the most up to date. This is using Sphinx, which allows us to generate, from the same source, man pages as well as web pages. (And even pdf or single page html). So we get nicer output format options. Clearer history in the git log than you get from wiki. Cyrusimap docs are held in mediawiki. Not so great for man pages. But it's much easier for third party contributor to pitch in and help out: they don't need to learn rst, they don't need a phabricator account, they don't need to navigate git. I like the git/sphinx combo for the audit trail it provides (blame) and the easy ability to back out errant changes. I am not a MediaWiki expert, barely a novice, so if it offers the same capabilities... But perhaps more importantly, the existing docs on cyrus.foundation make use of the nifty macros and such in Sphinx, in order that they can include references to, or portions of, man pages. The aforementioned config2rst, for example, was purposefully written such that it adds useful delimiters into the resultant imapd.config.rst to facilitate this, thus eliminating duplication of information. For an example of this, check out the page on automatic creation of mailboxes, which makes heavy use of this to present the actual imapd.conf settings to the user, in context: https://docs.cyrus.foundation/imap/features/automatic-creation-of-mailboxes.html Which is produced by the simple code here: https://docs.cyrus.foundation/_sources/imap/features/automatic-creation-of-mailboxes.txt This ensures that the docs for the website (or package inclusion, why not?) will always accurately reflect the actual settings in lib/imapoptions, tracking changes as they're made. Is there any parallel in MediaWiki? For all of these reasons, I vote for solution 1, below. Cheers, -nic (ducking back below the wall...) *Possible solutions* Option 1: Single domain, unify git repos, use sphinx everywhere Moving the git repo to cyrusimap (or anywhere) isn't hard. (a job for Bron) Get rid of separate cyrus-docs repo. Put cyrus-docs stuff back inside cyrus-imapd/docs for easier man page generation and tagging of docs versions with source, and shipping of current docs with source. Put sphinx on cyrusimap to replace the wiki. This requires either: 1. working out how to set up the existing generated docs for use with sphinx, 2. or make all the stuff in cyrus-imapd/doc into rst so it works with sphinx. We can still ship the built html. Option 2: Single domain, unify git repos, use wiki for docs - Put sphinx on cyrusimap.org just for generating man page html. Leave the existing wiki where it is for documentation. (Requires porting the page updates I made from cyrus.foundation onto the wiki). This means the only thing left in the cyrus-docs git repo would be the man pages, at which point it makes more sense to put them back into the cyrus-source repo. We can still ship docs with the release if we tie in a wiki export into the release-building process. (A job for Ellie and I) Option 3: Single domain, separate git repos, use sphinx for docs -- Move all services to cyrusimap.org, but still leaves us with all the docs logistics and sphinx and wiki chafe points. Not recommended. *Recommendation* I am leaning towards suggesting option 2. Anything that makes documentation support easier is a good! But we'd still like to retain the usefulness of sphinx for generating
Re: Update to Murder docs (D69) and question on style.
On 08/21/2015 02:58 AM, Michael Menge wrote: Hi, Quoting Nic Bernstein n...@onlight.com: snip I can write this up, I just wasn't sure if it was still needed. I put a big ol' Note: in the replication page saying: Important Within a Cyrus /Murder/ https://docs.cyrus.foundation/imap/developer/architecture.html#architecture-murder environment, replicas must *not* be configured to invoke ctl_mboxlist(8) http://docs.cyrus.foundation/imap/admin/commands/ctl_mboxlist.html on startup (pushing the local mailbox list to the *Mupdate Master*). This may only be done on the Master instance. That's the only real gotcha I know of, but, having said that, I did write up a brief set of instructions about this very topic not that long ago (IIRC) for user mailing list. I figured I could start with that. For the initial configuration I second this. But there are IHMO things to consider on failover. 1. ctl_mboxlist must be used with -m and -a Option on failover 2. on big installations updating all entries in mailbox.db on the mupdate server can take some time, on our setup we switch the IP address of master and replic on failover And on 21 August at 06:12AM CDT, Ken Murchison wrote: Right. We don't even have our replicas as part of our Murder. They replicate their backend as if it were a standalone server. Yes indeed. We use the following in /etc/imapd.conf on our servers: ## # Only one of these should be uncommented @include: /etc/imapd-master.conf #@include: /etc/imapd-replica.conf And then comment/uncomment as needed. The difference between these being the following (sanitized for your protection): root@mailbox:~# diff -uwb /etc/imapd-master.conf /etc/imapd-replica.conf --- /etc/imapd-master.conf 2014-11-24 23:06:49.830675999 + +++ /etc/imapd-replica.conf 2014-11-24 23:06:49.834675999 + -servername: mailbox.example.com +servername: mailbox.wi.example.com -## -# Auth credentials -mupdate_server: postman.example.com -mupdate_username: postman -mupdate_authname: postman -mupdate_password: secret - -## -# Replication support -# This is how the BACKEND for this host is defined -sync_host: mailbox.ia.example.com -sync_authname: mailproxy -sync_password: secret -sync_compress: true -sync_log: true -sync_repeat_interval: 5 -sync_shutdown_file: /var/run/cyrus/sync_stop +## Auth credentials +# The credentials below must match the account listed in lmtp_admins +# on the backend servers. +proxy_authname: mailproxy +proxy_password: secret +serverlist: mailbox mailbox.wi So the replica has no clue about the Murder. We switch DNS between the two hosts during failover, so no IP address change [the servers are in different data centers, so that wouldn't be practical in any event]. I doubt we actually need that last blob of stuff on the replica, but it doesn't seem to have hurt. As for /etc/cyrus.conf, we do something similar, in regards to commenting/uncommenting START entries for ctl_mboxlist and sync_client, versus an SERVICES entry for sync_server. It's not the cleanest process in failover, but a damn sight better than nothing. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 attachment: nic.vcf
Clarification needed on replication frequency settings
Mostly to Bron, but if anyone else wishes to jump in...
There seems to be a conflict between documentation and code in regards
to the settings for replication frequency. Specifically,
cyrus-imapd/lib/imapoptions has this:
{ sync_repeat_interval,*1*, INT }
/* Minimum interval (in seconds) between replication runs in rolling
replication mode. If a replication run takes longer than this
time, we repeat immediately.
Prefix with a channel name to only apply for that channel */
which would seem to indicate that the default is 1 second. Meanwhile,
sync_client(8) has this:
.. option:: -d delay
Minimum delay between replication runs in rolling replication mode.
Larger values provide better efficiency as transactions can be
merged. Smaller values mean that the replica system is more up to
date and that you don't end up with large blocks of replication
transactions as a single group. Default:*3 seconds*.
and the code, sync_client.c, has this:
int timeout = 600;
int min_delta = 0;
const char *channel = NULL;
...
case 't':
timeout = atoi(optarg);
break;
case 'd':
min_delta = atoi(optarg);
break;
...
else {
/* rolling replication */
if (!sync_shutdown_file)
sync_shutdown_file = get_config(channel,
sync_shutdown_file);
if (!min_delta)
min_delta = get_intconfig(channel, sync_repeat_interval);
do_daemon(channel, sync_shutdown_file, timeout, min_delta);
}
So I'm thinking that the actual default is 1, not 3. Am I right, or
confused?
Cheers,
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
6525 W Bluemound Road, Suite 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073
attachment: nic.vcf
Update to Murder docs (D69) and question on style.
Nicola, I've edited your recent commit of Murder docs, and added some stuff therein. Please take a look at D69 and send me any feedback. If you've not got a problem with it, then one or the other of us should land it. Just let me know. Based on your commit comment recently, something like Rewording install guide to third person. I've been trying to do the same for the reference docs we're preparing. Was this your intent, too; to aim for third person for all docs? I'd like to see that, although I'll admit that there are times where second person allows for more expeditious conveyance of information. Whatever you think is best, I'll gladly follow. Similarly, we have a mix of, for example, frontend and front end in the Murder docs. I think we should standardize, but have no strong preference as to which is better. The former seems to better serve as a noun, but the latter lends itself to list usages, such as both front and back ends should have ``servername`` set... Again, you lead, I'll follow. But whatever the decision is, let's effect a bulk replace to cope with it. Lastly, Ellie had suggested, in IRC: i'm thinking, maybe the murder docs should include specifics on replication in a murder environment (assuming you probably want the one if you've got the other), and the replication-only doc should just be like if you're using a murder, see the other doc instead Given the current state of both documents, which is quite different than that Ellie was commenting upon, do we still think this would be a Good Thing®? I'm ambivalent, but will be glad to do whatever needs doing for this. Your thoughts? Anyone else (there's a reason this is posted to the list, too)? Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 6525 W Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, Wisconsin 53213-4073 attachment: nic.vcf
Re: Docs and the :manpage: tag
On 08/13/2015 12:15 AM, Nicola Nye wrote: Delicious victory is mine! We now have a :cyrusman: sphinx option which generates urls into our docs.cyrus.foundation tree, performing string munging magic to match the generated url to our directory and filename structure. Now, to look at updating all the references in our existing docs so that it uses the new tag... Nicola, Doesn't work for me. I pulled your changes, and then ran the following script to replace all :manpage: references with :cyrusman: $ for file in `grep -lR :manpage: source/imap`; do sed -i $file -e 's/:manpage:/:cyrusman:/g'; done Then I ran a build: $ make man html sphinx-build -b cyrman -d build/doctrees source build/man Running Sphinx v1.2.2 Initializing cyrusman plugin loading pickled environment... done building [cyrman]: all manpages updating environment: [extensions changed] 274 added, 18 changed, 0 removed reading sources... [ 3%] imap/admin/access-control/rights-reference Exception occurred: File /home/nic/Checkouts/cyrus.foundation/cyrus-docs/source/exts/sphinxlocal/writers/cyrusman.py, line 49, in man_role manpage_num = m.group(2) AttributeError: 'NoneType' object has no attribute 'group' The full traceback has been saved in /tmp/sphinx-err-yWaXf3.log, if you want to report the issue to the developers. Full traceback is attached. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335 # Sphinx version: 1.2.2 # Python version: 2.7.6 # Docutils version: 0.11 release # Jinja2 version: 2.7.2 # Loaded extensions: # sphinx.ext.graphviz from /usr/lib/python2.7/dist-packages/sphinx/ext/graphviz.pyc # sphinx.ext.mathjax from /usr/lib/python2.7/dist-packages/sphinx/ext/mathjax.pyc # sphinx.ext.extlinks from /usr/lib/python2.7/dist-packages/sphinx/ext/extlinks.pyc # sphinxlocal.builders.manpage from /home/nic/Checkouts/cyrus.foundation/cyrus-docs/source/exts/sphinxlocal/builders/manpage.pyc # sphinx.ext.coverage from /usr/lib/python2.7/dist-packages/sphinx/ext/coverage.pyc # sphinx.ext.todo from /usr/lib/python2.7/dist-packages/sphinx/ext/todo.pyc # sphinxlocal.writers.cyrusman from /home/nic/Checkouts/cyrus.foundation/cyrus-docs/source/exts/sphinxlocal/writers/cyrusman.py # sphinx.ext.ifconfig from /usr/lib/python2.7/dist-packages/sphinx/ext/ifconfig.pyc # sphinx.ext.oldcmarkup from /usr/lib/python2.7/dist-packages/sphinx/ext/oldcmarkup.pyc Traceback (most recent call last): File /usr/lib/python2.7/dist-packages/sphinx/cmdline.py, line 254, in main app.build(force_all, filenames) File /usr/lib/python2.7/dist-packages/sphinx/application.py, line 212, in build self.builder.build_update() File /usr/lib/python2.7/dist-packages/sphinx/builders/__init__.py, line 209, in build_update self.build(['__all__'], to_build) File /usr/lib/python2.7/dist-packages/sphinx/builders/__init__.py, line 234, in build purple, length): File /usr/lib/python2.7/dist-packages/sphinx/builders/__init__.py, line 134, in status_iterator for item in iterable: File /usr/lib/python2.7/dist-packages/sphinx/environment.py, line 477, in update_generator self.read_doc(docname, app=app) File /usr/lib/python2.7/dist-packages/sphinx/environment.py, line 624, in read_doc pub.publish() File /usr/lib/python2.7/dist-packages/docutils/core.py, line 217, in publish self.settings) File /usr/lib/python2.7/dist-packages/docutils/readers/__init__.py, line 72, in read self.parse() File /usr/lib/python2.7/dist-packages/docutils/readers/__init__.py, line 78, in parse self.parser.parse(self.input, document) File /usr/lib/python2.7/dist-packages/docutils/parsers/rst/__init__.py, line 172, in parse self.statemachine.run(inputlines, document, inliner=self.inliner) File /usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py, line 171, in run input_source=document['source']) File /usr/lib/python2.7/dist-packages/docutils/statemachine.py, line 239, in run context, state, transitions) File /usr/lib/python2.7/dist-packages/docutils/statemachine.py, line 460, in check_line return method(match, context, next_state) File /usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py, line 2962, in text self.section(title.lstrip(), source, style, lineno + 1, messages) File /usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py, line 328, in section self.new_subsection(title, lineno, messages) File /usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py, line 396, in new_subsection node=section_node, match_titles=True) File /usr/lib/python2.7/dist-packages/docutils/parsers/rst/states.py, line 283, in nested_parse node=node, match_titles=match_titles) File /usr/lib/python2.7/dist-packages
Re: Patch: forcing SSL before auth
Carlos, Could you add a stanza to /lib/imapoptions describing any configuration options you've added? Don't worry if it's perfect, just get something in there so we documentation folk can make sure they get into the man pages. Thanks! -nic On 08/09/2015 09:17 AM, Carlos Velasco wrote: Version 2 patch. Including timsieved. Also in the patch is some code for Serverinfo switch in timsieved to not disclose name and/or version info in IMPLEMENTATION if Serverinfo is Off or Min. Regards, Carlos Velasco Original Message Subject: Patch: forcing SSL before auth From: Carlos Velasco carlos.vela...@nimastelecom.com To: cyrus-devel@lists.andrew.cmu.edu Date: 9/8/2015 12:18:36 Hi, Right now, allowplaintext option disallow using a plain authentication if session is not protected by TLS. However, this setting still allows a client to make MD5 or SHA1 auth without session being protected by TLS. This can lead to not data confidentiality when using not plain auth. There are several admins now requesting to force TLS for all sessions, and although this can be done using allowplaintext and removing all mechs but Plain, it would be right to be able to provide another layer of security and use TLS+SHA1 or so... Attached is a patch with a new imapd.conf option: forcetlsauth: 0 | 1. Default 0 If enabled all authentications require a TLS session negotiated before. Patch also hides AUTH and other authentication commands that are not allowed before TLS, in Capabilites commands. Patched in imapd, pop3d, nntpd, httpd. This patch does not break cyradm functionality at all, however I attach another patch for the cyradm perl part to allow --cafile option (got tired of certificate validation warnings) and also fixed a minor bug when requesting capabilities to server without the callback. Please, consider committing this to mainstream. Regards, Carlos Velasco -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
Re: Today's pop quiz: replication
On 07/24/2015 12:07 AM, ellie timoney wrote: - a single cyrus instance may be the primary server for some users but a replica server for other users Are you sure about that? I'm not sure about any of this. But you make a good point: I thought I could see a way that this was possible, but now I don't think I can. I think I can, again, though I guess not in a murder. But in a non-murder setup, I believe the only difference between a primary vs replica is which one the user interacts with? Which, in a non-murder setup, one is presumably managing in some way external to cyrus (e.g. database, proxies, and some glue)... So it seems like it should be plausible to have two (or more) cyrus instances (hosted wherever), each running a sync_server plus a channel/sync_client for each of the others, and whatever happens on any replicates to the others. You'd be trusting your outer, non-cyrus layer to make sure not to proxy any individual user to the wrong instance (at least, not while there's pending replication transactions for them). And I guess shared folders would be thorny. But I don't (yet) see a reason from a replication standpoint why this wouldn't work. So: - a single cyrus instance may* be the primary server for some users but a replica server for other users [* with caveats and requiring custom tooling, and specifically not in a murder] Yikes, well I guess we can do pretty much anything*. :-0 But, I think, in the context of Nicola's original post, it's safer to say that this is not a standard or supported configuration. Nicola is writing documentation, so an understanding of typical usage trumps speculative possibilities. :-) I do think, however, that a more thorough description of the roles of sync_client(s) and sync_server are in order, and if I have a chance will write something up soon. Cheers, -nic *PS - Imagine the sync storm resulting from such an arrangement run amok! ellie On Fri, Jul 24, 2015, at 10:01 AM, ellie timoney wrote: Okay, I'll bite. Here's what a bit of a sync_log looks like: Thanks! So anything processing a sync_log (sync_client, squatter, etc) needs to look at an actual copy of the user/mailbox in order to determine its current state, and needs to look at both copies to work out what to replicate between them. - a single cyrus instance may be the primary server for some users but a replica server for other users Are you sure about that? I'm not sure about any of this. But you make a good point: I thought I could see a way that this was possible, but now I don't think I can. Cheers, ellie On Thu, Jul 23, 2015, at 11:42 PM, Nic Bernstein wrote: On 07/23/2015 01:14 AM, ellie timoney wrote: Is the file format of the sync log defined anywhere? I assume it correlates with a set of commands. (Not that this is important to a user: it may as well be opaque, but it made me wonder!) I'm a bit confused about this myself. Each time I go digging into the code my understanding flips back the opposite way. I think, either: * the sync log contains all the information needed to reproduce what's happened (e.g. if a message has arrived, the sync log will contain the message itself); OR * the sync log contains just enough to identify things that have changed (e.g. if a message has arrived, the sync log contains a message id of some sort), and the sync_client processing the log just uses the log to discover which things to sync, but then uses the actual mailbox to construct the changes to send to the replica. Either way I haven't seen any documentation on the sync log format. I suspect it's either the raw sync protocol or some subset thereof? Okay, I'll bite. Here's what a bit of a sync_log looks like: MAILBOX user.newjersey MAILBOX user.support USER onlight USER nic USER admin USER randy MAILBOX user.randy.Trash USER lynn MAILBOX user.lynn.Trash It's basically a list of either users or mailboxes which have been altered. When sync_log is enabled, all of the daemons which might alter a mailbox or user will write a line to this log each time they do so. That means the obvious suspects -- imapd, pop3d, timsieved, lmtpd, etc. -- but also cyr_expire and friends (as in the USER...MAILBOX couplets at the end of the sample). - a single cyrus instance may be the primary server for some users but a replica server for other users Are you sure about that? How does one specify the users for which such an instance would play each role? A single HOST may run separate instances, which may perform these different roles, but I cannot fathom how to configure a single instance to do both at once for different user cohorts. This raises potential problems when one deploys replication within a murder (Cyrus aggregation). Only one server may claim ownership of any given mailbox, via a mupdate call, so an instance which is a replica MAY NOT push updates
Re: Harbormaster notes from today's conference call - 29 June 2015.
Nicola, [back at computer now, so can send proper reply] This was fixed in D58, committed and closed. Take a look at build 2119: https://git.cyrus.foundation/B2119 Harbormaster shows all clean builds since 2118: https://git.cyrus.foundation/harbormaster/ Cheers, -nic On 07/01/2015 07:32 PM, Nicola Nye wrote: Nic, I think the build is in your camp. Log shows: *** No rule to make target `tools/config2rst', needed by `doc/rst/imapd.conf.rst'. Stop. Thanks Jeroen for updating the build environment for us! On Wed, Jul 1, 2015, at 07:39 PM, Jeroen van Meeuwen (Kolab Systems) wrote: On 2015-06-30 01:11, Bron Gondwana wrote: This is Jeroen. Jeroen, we need you to look at this - it's holding people up. clamav-devel has been installed, but the builds are still failing: https://git.cyrus.foundation/harbormaster/build/1049/ Sphinx and docutils have been upgraded as well; [root@phab10 ~]# pip show sphinx --- Name: Sphinx Version: 1.3.1 Location: /usr/lib64/python2.7/site-packages Requires: sphinx-rtd-theme, Jinja2, alabaster, babel, six, Pygments, snowballstemmer, docutils [root@phab10 ~]# pip show docutils --- Name: docutils Version: 0.12 Location: /usr/lib/python2.7/site-packages Requires: Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +41 79 951 9003 w: https://kolabsystems.com pgp: 9342 BF08 -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
Harbormaster notes from today's conference call - 29 June 2015.
Friends, As discussed at today's meeting, there are two things we need for harbormaster to properly do its job: * Install ClamAV development package (needed by cyrus-imapd build) o sudo apt-get install libclamav-dev * Install newer Docutils and Sphinx packages (needed by cyrus-docs build) o sudo apt-get install python-pip o sudo pip install --upgrade docutils sphinx Could whomever has keys to the castle get this sorted? Thanks in advance! -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 1442 N Farwell Ave., Suite 600v. 414.272.4477 Milwaukee, Wisconsin 53202 attachment: nic.vcf
Documentation notes from conference call 6/29/2015
Friends,
Here's my notes from today's discussion about documentation.
The man pages have all been converted (ported, what have you) from their
original troff format to the new ReStructuredText (rst) format.
This work was done in the new cyrus-docs (rD) repo, which is separate
from the cyrus-imapd (rI) repo. The singular exception to this is
imapd.conf.5, which is generated by software from
cyrus-imapd/lib/imapoptions.
I produced a now tool, cyrus-imapd/tools/config2rst, cribbed from
config2man, which achieves this goal (with small complaints from
Sphinx). Following the conference call Bron landed my commits for this,
and it passed harbormaster review (after dancing clamav jig).
The need to include suitably troff formatted manpages in
cyrus-imapd/man, but the presence of most of them in
cyrus-docs/source/imap/admin/{configs|commands} or
cyrus-docs/source/imap/developer/libraries (and ultimately in troff
format in cyrus-docs/build/man), seems awkward. Especially when the
case of imapd.conf.5 is added to the mix.
We decided that there is probably no really compelling reason to have
the manpages be in a separate repo from the sources. For now, however,
the only nod to such a change was to have cyrus-imapd/Makefile put
imapd.conf.rst into a newly created cyrus-imapd/doc/rst directory.
Ultimately, either all of the docs should be moved back into the
cyrus-imapd repo, or at least the manpages should. This was left as a
matter for further discussion, as, quite likely, the ultimate
disposition of cyrus-imapd/doc will be.
Cheers,
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
1442 N Farwell Ave., Suite 600v. 414.272.4477
Milwaukee, Wisconsin 53202
attachment: nic.vcf
Re: Doc build broken
We documentation folk don't have access to the underlying machine on
which the build tests run, so we can't do anything like that, to the
best of my knowledge.
The solution I came up with works, for now, and Nicola has requested an
update of the python stuff on the build host.
-nic
On 06/27/2015 06:36 AM, John wrote:
Hi,
I use virtualenv to avoid this kind of pain. It allows the use of
python package versions independent of the underlying distribution. Is
there a reason that virtualenv could not be used in this case?
John
On 26/06/15 22:50, Nic Bernstein wrote:
Nicola,
No rush. I put on my big-boy pants and got this sorted. In effect I
took every change between docutils v0.8.1 v0.11.3, and between
sphinx v1.1.2 v1.2.3 and incorporated them verbatim in the bottom
of our cyrus-docs/source/exts/sphinxlocal/writes/manpage.py
https://git.cyrus.foundation/diffusion/D/browse/master/source/exts/sphinxlocal/writers/manpage.py.
This is a total, absolute and unrepentant hack, but it does work, it
builds on Wheezy, and even harbormaster is happy (see B2110
https://git.cyrus.foundation/B2110). https://docs.cyrus.foundation/
is now up to date and has all of your changes as well as mine.
I commented the start of the ugly hack quite clearly, so we can lop
it off if and when we ever do get modern versions of the toolset
installed.
Cheers,
-nic
On 06/26/2015 04:40 PM, Nicola Nye wrote:
Hi Nic,
Yes! Spot on. I remember looking into the Sphinx versions when I first
started and then promptly forgot about it.
I totally agree about your proposed versions. I'll chase down Jeroen who
I think has the keys to that box and see if we can't get it brought up
to date. Or at least out of the dark ages.
Cheers,
Nicola
On Sat, Jun 27, 2015, at 05:28 AM, Nic Bernstein wrote:
On 06/25/2015 11:01 PM, Nicola Nye wrote:
Hi Nic,
After some false starts using arc, I finally got a bunch of my changes
committed through onto the cyrus docs tree. And then I noticed the
website wasn't updating.
After some help from ellie, it turns out our doc build broke a while ago
and we hadn't noticed!
I think it has to do with the new man page builder you wrote.
Developer's lament: it works for me (and for you!) but for some reason
it looks like it's falling over on the server.
https://git.cyrus.foundation/harbormaster/build/964/
School holidays are here for the next couple of weeks so I'm on reduced
hours but I'll be checking in to see if I can also work out what's going
on.
Cheers,
Nicola
Nicola,
Note sure what to do about this. I've spent the day bringing up a
Wheezy VM to play with this stuff, and made a version of
writer/manpage.py which works, but there's lots of other problems with
the old python-docutils which I could spend the next month trying to
work around.
Wheezy uses Sphinx v1.1.3 (2012-03-10) and docutils v0.8.1, which dates
from 2011-08-31. There's been a lot of changes since then, some of
which had to do with basic functionality. For example, in
docutils-0.8.1, in writer/manpage.py, is this definition:
def visit_Text(self, node):
text = node.astext()
text = text.replace('\\','\\e')
replace_pairs = [
(u'-', ur'\-'),
(u'\'', ur'\(aq'),
(u'´', ur'\''),
(u'`', ur'\(ga'),
]
for (in_char, out_markup) in replace_pairs:
text = text.replace(in_char, out_markup)
# unicode
text = self.deunicode(text)
if self._in_literal:
# prevent interpretation of . at line start
if text[0] == '.':
# BUG
text = '\\' + text
text = text.replace('\n.', '\n\\.')
self.body.append(text)
The line if text[0] == '.': is a flat out bug, since if 'text' is ever
empty, it fails. This bug was fixed in release 0.11 (2013-07-22), with
this line:
if text.startswith('.'):
So in order to get manpages to build _at all_, I need to redefine
visit_Text in our custom writer/manpage.py. I've done so, but that's
just the start of the problems.
I would like to propose that we get a newer version of python-docutils
and python-sphinx installed on harbormaster and declare that certain
minimum versions of both packages are required to build documentation
from source. For the record, I further propose that these be:
* python-docutils-0.11.3
* python-sphinx-1.2.2
By the way, those are the versions which are supported by Ubuntu Trusty
(14.04.2) and Utopic (14.10); Debian Jessie and Sid; Fedora 20 21;
etc. This is not bleeding edge stuff, but at least it doesn't suffer
from long-ago fixed bugs.
Building Cyrus documentation from source is a different proposition from
building the server from source. It's my understanding that the
manpages, in particular, are to be built
Re: Doc build broken
On 06/25/2015 11:01 PM, Nicola Nye wrote:
Hi Nic,
After some false starts using arc, I finally got a bunch of my changes
committed through onto the cyrus docs tree. And then I noticed the
website wasn't updating.
After some help from ellie, it turns out our doc build broke a while ago
and we hadn't noticed!
I think it has to do with the new man page builder you wrote.
Developer's lament: it works for me (and for you!) but for some reason
it looks like it's falling over on the server.
https://git.cyrus.foundation/harbormaster/build/964/
School holidays are here for the next couple of weeks so I'm on reduced
hours but I'll be checking in to see if I can also work out what's going
on.
Cheers,
Nicola
Nicola,
Note sure what to do about this. I've spent the day bringing up a
Wheezy VM to play with this stuff, and made a version of
writer/manpage.py which works, but there's lots of other problems with
the old python-docutils which I could spend the next month trying to
work around.
Wheezy uses Sphinx v1.1.3 (2012-03-10) and docutils v0.8.1, which dates
from 2011-08-31. There's been a lot of changes since then, some of
which had to do with basic functionality. For example, in
docutils-0.8.1, in writer/manpage.py, is this definition:
def visit_Text(self, node):
text = node.astext()
text = text.replace('\\','\\e')
replace_pairs = [
(u'-', ur'\-'),
(u'\'', ur'\(aq'),
(u'´', ur'\''),
(u'`', ur'\(ga'),
]
for (in_char, out_markup) in replace_pairs:
text = text.replace(in_char, out_markup)
# unicode
text = self.deunicode(text)
if self._in_literal:
# prevent interpretation of . at line start
if text[0] == '.': #
BUG
text = '\\' + text
text = text.replace('\n.', '\n\\.')
self.body.append(text)
The line if text[0] == '.': is a flat out bug, since if 'text' is ever
empty, it fails. This bug was fixed in release 0.11 (2013-07-22), with
this line:
if text.startswith('.'):
So in order to get manpages to build _at all_, I need to redefine
visit_Text in our custom writer/manpage.py. I've done so, but that's
just the start of the problems.
I would like to propose that we get a newer version of python-docutils
and python-sphinx installed on harbormaster and declare that certain
minimum versions of both packages are required to build documentation
from source. For the record, I further propose that these be:
* python-docutils-0.11.3
* python-sphinx-1.2.2
By the way, those are the versions which are supported by Ubuntu Trusty
(14.04.2) and Utopic (14.10); Debian Jessie and Sid; Fedora 20 21;
etc. This is not bleeding edge stuff, but at least it doesn't suffer
from long-ago fixed bugs.
Building Cyrus documentation from source is a different proposition from
building the server from source. It's my understanding that the
manpages, in particular, are to be built and included, pre-built, in the
cyrus-imapd distribution package. So as long as _we_ can build the
manpages, we're golden (as they say).
That's why I feel comfortable recommending that we ask that a newer
version of these tools be installed on harbormaster.
Your thoughts?
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
1442 N Farwell Ave., Suite 600v. 414.272.4477
Milwaukee, Wisconsin 53202
attachment: nic.vcf
Re: Doc build broken
Nicola,
No rush. I put on my big-boy pants and got this sorted. In effect I
took every change between docutils v0.8.1 v0.11.3, and between sphinx
v1.1.2 v1.2.3 and incorporated them verbatim in the bottom of our
cyrus-docs/source/exts/sphinxlocal/writes/manpage.py
https://git.cyrus.foundation/diffusion/D/browse/master/source/exts/sphinxlocal/writers/manpage.py.
This is a total, absolute and unrepentant hack, but it does work, it
builds on Wheezy, and even harbormaster is happy (see B2110
https://git.cyrus.foundation/B2110). https://docs.cyrus.foundation/ is
now up to date and has all of your changes as well as mine.
I commented the start of the ugly hack quite clearly, so we can lop it
off if and when we ever do get modern versions of the toolset installed.
Cheers,
-nic
On 06/26/2015 04:40 PM, Nicola Nye wrote:
Hi Nic,
Yes! Spot on. I remember looking into the Sphinx versions when I first
started and then promptly forgot about it.
I totally agree about your proposed versions. I'll chase down Jeroen who
I think has the keys to that box and see if we can't get it brought up
to date. Or at least out of the dark ages.
Cheers,
Nicola
On Sat, Jun 27, 2015, at 05:28 AM, Nic Bernstein wrote:
On 06/25/2015 11:01 PM, Nicola Nye wrote:
Hi Nic,
After some false starts using arc, I finally got a bunch of my changes
committed through onto the cyrus docs tree. And then I noticed the
website wasn't updating.
After some help from ellie, it turns out our doc build broke a while ago
and we hadn't noticed!
I think it has to do with the new man page builder you wrote.
Developer's lament: it works for me (and for you!) but for some reason
it looks like it's falling over on the server.
https://git.cyrus.foundation/harbormaster/build/964/
School holidays are here for the next couple of weeks so I'm on reduced
hours but I'll be checking in to see if I can also work out what's going
on.
Cheers,
Nicola
Nicola,
Note sure what to do about this. I've spent the day bringing up a
Wheezy VM to play with this stuff, and made a version of
writer/manpage.py which works, but there's lots of other problems with
the old python-docutils which I could spend the next month trying to
work around.
Wheezy uses Sphinx v1.1.3 (2012-03-10) and docutils v0.8.1, which dates
from 2011-08-31. There's been a lot of changes since then, some of
which had to do with basic functionality. For example, in
docutils-0.8.1, in writer/manpage.py, is this definition:
def visit_Text(self, node):
text = node.astext()
text = text.replace('\\','\\e')
replace_pairs = [
(u'-', ur'\-'),
(u'\'', ur'\(aq'),
(u'´', ur'\''),
(u'`', ur'\(ga'),
]
for (in_char, out_markup) in replace_pairs:
text = text.replace(in_char, out_markup)
# unicode
text = self.deunicode(text)
if self._in_literal:
# prevent interpretation of . at line start
if text[0] == '.':
# BUG
text = '\\' + text
text = text.replace('\n.', '\n\\.')
self.body.append(text)
The line if text[0] == '.': is a flat out bug, since if 'text' is ever
empty, it fails. This bug was fixed in release 0.11 (2013-07-22), with
this line:
if text.startswith('.'):
So in order to get manpages to build _at all_, I need to redefine
visit_Text in our custom writer/manpage.py. I've done so, but that's
just the start of the problems.
I would like to propose that we get a newer version of python-docutils
and python-sphinx installed on harbormaster and declare that certain
minimum versions of both packages are required to build documentation
from source. For the record, I further propose that these be:
* python-docutils-0.11.3
* python-sphinx-1.2.2
By the way, those are the versions which are supported by Ubuntu Trusty
(14.04.2) and Utopic (14.10); Debian Jessie and Sid; Fedora 20 21;
etc. This is not bleeding edge stuff, but at least it doesn't suffer
from long-ago fixed bugs.
Building Cyrus documentation from source is a different proposition from
building the server from source. It's my understanding that the
manpages, in particular, are to be built and included, pre-built, in the
cyrus-imapd distribution package. So as long as _we_ can build the
manpages, we're golden (as they say).
That's why I feel comfortable recommending that we ask that a newer
version of these tools be installed on harbormaster.
Your thoughts?
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
1442 N Farwell Ave., Suite 600v. 414.272.4477
Milwaukee, Wisconsin 53202
Email had 1 attachment:
+ nic.vcf
1k (text/x-vcard)
--
Nic Bernstein
Re: Stripping ALL tabs from the code
On 06/14/2015 07:48 PM, Bron Gondwana wrote: On Tue, Jun 9, 2015, at 08:42 PM, Bron Gondwana wrote: The big news from yesterday's meeting is that we are going to check in a single massive commit which strips all leading tabs from the code, everywhere. The new coding standard is 4 spaces for each level of indent, and spaces for all alignment within code as well. I've pushed a small perl script to master which performs the necessary surgery. it has the smarts to know when a tab isn't on a multiple-of-8 boundary and substitute fewer spaces. The code still builds and passes tests afterwards, so I'm pretty sure it's good. I will be running the on master on June 14th (Sunday) night Melbourne time. That's weekend for everyone, everywhere in the world. I will immediately push the result back to master (after running the tests again of course). If you have a patch series lying around that you want to apply, it might be a good idea to do it before then! Alternatively, you can patch with -l or --ignore-whitespace to apply your patches to the tree, and then run tools/remove-tabs.pl on the resulting tree to remove the tabs again. Enjoy your new, simpler coding style with no trailing whitespace left! This has been done - I didn't quite finish the instructions last night, so I did the work this morning. Bron, Maybe I'm doing something wrong here, but this morning I figured I'd start over clean, so deleted my existing cyrus-docs checkout and grabbed a new one: $ git clone ssh://git@git.cyrus.foundation/diffusion/I/cyrus-docs.git What I ended up with looked suspiciously like the cyrus-imapd tree. So I checked that by doing a diff of the two: $ diff -Nuwb cyrus-docs cyrus-imapd Common subdirectories: cyrus-docs/cmulocal and cyrus-imapd/cmulocal Common subdirectories: cyrus-docs/com_err and cyrus-imapd/com_err Common subdirectories: cyrus-docs/contrib and cyrus-imapd/contrib Common subdirectories: cyrus-docs/cunit and cyrus-imapd/cunit Common subdirectories: cyrus-docs/depot and cyrus-imapd/depot Common subdirectories: cyrus-docs/doc and cyrus-imapd/doc Common subdirectories: cyrus-docs/.git and cyrus-imapd/.git Common subdirectories: cyrus-docs/imap and cyrus-imapd/imap Common subdirectories: cyrus-docs/imtest and cyrus-imapd/imtest Common subdirectories: cyrus-docs/lib and cyrus-imapd/lib Common subdirectories: cyrus-docs/man and cyrus-imapd/man Common subdirectories: cyrus-docs/master and cyrus-imapd/master Common subdirectories: cyrus-docs/netnews and cyrus-imapd/netnews Common subdirectories: cyrus-docs/notifyd and cyrus-imapd/notifyd Common subdirectories: cyrus-docs/perl and cyrus-imapd/perl Common subdirectories: cyrus-docs/ptclient and cyrus-imapd/ptclient Common subdirectories: cyrus-docs/sieve and cyrus-imapd/sieve Common subdirectories: cyrus-docs/snmp and cyrus-imapd/snmp Common subdirectories: cyrus-docs/timsieved and cyrus-imapd/timsieved Common subdirectories: cyrus-docs/tools and cyrus-imapd/tools Could something have gone awry with this whitespace cleanup? Concerned, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 1442 N Farwell Ave., Suite 600v. 414.272.4477 Milwaukee, Wisconsin 53202 attachment: nic.vcf
Re: LinkChecker
On 06/14/2015 10:59 PM, Chris Davies wrote: I've just installed LinkChecker http://wummel.github.io/linkchecker/on the Jenkins server. It's currently running once an hour against http://www.cyrusimap.org/ and will report any broken links to https://ci.cyrus.foundation/job/LinkChecker%20-%20CyrusImap.org/ If you click the most recent job (under build history), then click 'Console Output' you can see the most recent report. Over time it would be good if we could fix the broken links or I can add ignore rules for anything we don't plan on fixing. Cheers, Chris Chris, Just to clarify, is this installed on http://www.cyrusimap.org/ only, or also on https://*.cyrus.foundation/ as well? I ask because Bron's meeting notes say the latter. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
Re: [Differential] [Updated, 2,615 lines] D44: Update reStrusctured Text versions of man pages
On 06/11/2015 05:03 AM, Nicola Nye wrote: For what it's worth, Bron and I were discussing whether we should look at migrating the website from Sphinx to some wiki variant in order to make it easier for contributors to add to the documentation. Obviously this then allows greater flexibility in building the man pages: we can leave them off the online docs and just leave them to ship with the cyrus package, or we find another way to include them online (which might just mean we end up with a different, but still slightly annoying process, such as wikihtml2man!). Something to discuss at today's meeting... https://plus.google.com/hangouts/_/g4xnqjjb5zvomzeb4kqvja3fz4a Or shoot through your ideas on the list! Nicola, For my part, I'd just as soon we don't go changing horses in midstream. I've got a lot of hours invested in the current suite of solutions, and to change to some new, different, set of annoyances at this point would just mean more work. As for whether other parts of the website are in Wiki or Sphinx, that's a different matter. I don't see any reason why we can't use a mix of the two. Using something like Sphinx for material which needs flexibility in presentation makes sense. Using a Wiki for community contributed material makes sense. Let's do both. Being in US Central time, while you all are nearing the end of your day, I've only just stared mine -- no caffine yet -- so the meetup is out of reach. Ta! -nic -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
Re: Documentation Unification and Awesomeification
On 06/08/2015 07:43 PM, Nicola Nye wrote: Nice sleuthing work! Thanks! :-) The only alternative I have to offer is that much as Sphinx allows you to write custom themes to change the look and feel of the html, you can also write your own builder (or extend an existing one). http://sphinx-doc.org/extdev/index.html#dev-extensions That would let us override sphinx's builders/manpage.py and therefore sphinx's writers/manpage.py. If we had more than one manpage issue, this is probably a better way forwards, but for just tweaking this bold issue, I'm totally on board with updating the Makefile. Go ye forth and fix! Nicola I've gone ahead and created an extension, a customized version of builders/manpage.py and writers/manpage.py, and am now re-writing the already updated manpage .rst files to work as expected with this new version, rather than using workarounds. In addition to the problems with fonts, there was gratuitous indentation on all literal_block entries, which made writing examples quite vexing. I will try to get at least an interim commit up sometime soon so others can start to poke at this with great big sticks. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 1442 N Farwell Ave., Suite 600v. 414.272.4477 Milwaukee, Wisconsin 53202 attachment: nic.vcf
Re: [Differential] [Updated, 2,615 lines] D44: Update reStrusctured Text versions of man pages
Nicola, I've just made a big commit in D44, and would appreciate another set of eyes on it prior to accepting it. In this commit are the new Sphinx extension for manpage building, as well as a load of said man pages. My specific concerns are to verify that the Sphinx extension works for others as well as for me. Also, as I am not a Python person, I think we'd really benefit from someone who is looking at the extent of my subclassing in the new extension. I am not really sure if I've managed to do as little as possible there (which is what I think we'd prefer). Please take a look and let me know. Cheers, -nic On 06/10/2015 05:45 PM, onlight (Nic Bernstein) wrote: onlight updated this revision to Diff 96. onlight added a comment. More man page updates and cleanup. New Sphinx extension. - Fix building of man pages and port more over - More updates to man pages from cyrus-imapd/man REPOSITORY rD cyrus-docs CHANGES SINCE LAST UPDATE https://git.cyrus.foundation/D44?vs=90id=96 BRANCH dev/update-command-man-pages REVISION DETAIL https://git.cyrus.foundation/D44 -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 1442 N Farwell Ave., Suite 600v. 414.272.4477 Milwaukee, Wisconsin 53202 attachment: nic.vcf
Re: Documentation Unification and Awesomeification
On 06/09/2015 04:59 AM, Sebastian Hagedorn wrote: --On 8. Juni 2015 17:14:21 -0500 Nic Bernstein n...@onlight.com wrote: So here's my quandary: • I've found it completely impossible to produce standards-compliant man pages using ReStructuredText with the Sphinx/Docutils tool suite, without running into this problem. • There's just no way I can find to work around this other than to either: • Alter the docutils/writers/manpage.py code, as shown above, or • Grep or sed out the gratuitous .ft commands from the resultant output. I think the best way forward with this is to opt for the latter and modify cyrus-docs/Makefile to strip the resulting manpages from cyrus-docs/build/man, like so: ... BUILDDIR = build SED = /bin/sed snip man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man $(SED) -i -e 's/^\.ft.*//' $(BUILDDIR)/man/*.[0-9]* @echo @echo Build finished. The manual pages are in $(BUILDDIR)/man. ... Thoughts? How about reporting this as a bug upstream? I fully intend to. My quandary, perhaps incompletely summarized, had to do with the exigencies of producing usable manpages, in a repeatable manner, given the time frame of releases. I think I'll opt for Nicola's suggestion of an extension for Sphinx, if that will do the job. For the record, however, the problem here is with a component of docutils, not Sphinx, so it's unclear to me if a Sphinx extension can fix it. I'll look into that. I'm not a Python expert, so... Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
Re: Documentation Unification and Awesomeification
On 05/19/2015 07:22 AM, Nic Bernstein wrote:
# The Sphinx man page output can be unpredictable
* Man output can sometimes shift to *bold* and never come back to
normal.
o See cyrus-docs/source/imap/admin/commands/unexpunge.rst for an
example
Nicola, et al.,
I've tracked down the problem with Sphinx formatting of manpage output.
When using the parsed-literal role, like so:
.. parsed-literal::
**arbitron -d** *14*
Normal short list format for the past 14 days.
The docutils manpage writer will produce this output (note my red
highlights):
.INDENT 3.5
.sp
.nf
.ft C
\fBarbitron \-d\fP \fI14\fP
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Normal short list format for the past 14 days.
.INDENT 0.0
But this is bogus output. It's popping fonts which have never been
installed in the font stack, so renders unreliably:
*arbitron -d* _14_
_Normal_ _short_ _list_ _format_ _for_ _the_ _past_ _14_ _days._
It should look like this:
*arbitron -d* _14_
Normal short list format for the past_14_ days.
Also, it relies on the availability of the font C (being Courier)
which is a bad assumption on many systems. Furthermore, since almost
all manpage output we're concerned with is processed with nroff, not
groff, and Fonts have limited meaning in nroff. The font used is a
constant-width font. If you specify bold, characters are overstruck in
printing. Italic is interpreted as an underline. Other fonts have no
meaning.
I've modified my local copy of (on Ubuntu Trusty)
/usr/lib/python2.7/dist-packages/docutils/writers/manpage.py to remove
these gratuitous .ft Cft P couplets, and now it works perfectly,
producing this:
.INDENT 3.5
.sp
.nf
\fBarbitron \-d\fP \fI14\fP
.fi
.UNINDENT
.UNINDENT
.sp
Normal short list format for the past 14 days.
.INDENT 0.0
Here's a diff fragment of my changes:
--- /usr/lib/python2.7/dist-packages/docutils/writers/manpage.py
2015-06-08 16:34:27.602186607 -0500
+++ /usr/local/lib/python2.7/dist-packages/docutils/writers/manpage.py
2015-05-26 12:14:01.0 -0500
@@ -213,7 +213,7 @@
'definition_list_item' : ('.TP', ''),
'field_name' : ('.TP\n.B ', '\n'),
'literal' : ('\\fB', '\\fP'),
-'literal_block' : ('.sp\n.nf\n', '\n.fi\n'),
+'literal_block' : ('.sp\n.nf\n.ft C\n', '\n.ft P\n.fi\n'),
'option_list_item' : ('.TP\n', ''),
So here's my quandary:
* I've found it completely impossible to produce standards-compliant
man pages using ReStructuredText with the Sphinx/Docutils tool
suite, without running into this problem.
* There's just no way I can find to work around this other than to either:
o Alter the docutils/writers/manpage.py code, as shown above, or
o Grep or sed out the gratuitous .ft commands from the
resultant output.
I think the best way forward with this is to opt for the latter and
modify cyrus-docs/Makefile to strip the resulting manpages from
cyrus-docs/build/man, like so:
...
BUILDDIR = build
SED = /bin/sed
snip
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
$(SED) -i -e 's/^\.ft.*//' $(BUILDDIR)/man/*.[0-9]*
@echo
@echo Build finished. The manual pages are in $(BUILDDIR)/man.
...
Thoughts?
Cheers,
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
1442 N Farwell Ave., Suite 600v. 414.272.4477
Milwaukee, Wisconsin 53202
attachment: nic.vcf
tls_prune dying on startup
Folks,
I'm seeing a problem on 2.5.2 with creation of the tls_sessions.db file:
cyrus/tls_prune[31096]: DBERROR: opening /run/cyrus/tls_sessions.db:
cyrusdb error
It appears that tls_prune(8) can't create the file if it doesn't exist,
and by exiting with an error, can cause master to die. Since this file
is supposed to be able to live (and die) on non-permanent storage,
shouldn't tls_prune be able to cope with its absence?
To be clear, I have a tls_prune in START in cyrus.conf.
START {
recovercmd=/usr/sbin/cyrus ctl_cyrusdb -r
delprunecmd=/usr/sbin/cyrus expire -E 3
tlsprunecmd=/usr/sbin/cyrus tls_prune
}
Thanks!
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
1442 N Farwell Ave., Suite 600v. 414.272.4477
Milwaukee, Wisconsin 53202
attachment: nic.vcf
Re: Documentation Unification and Awesomeification
Nicola, Thanks for the follow up. I've not had any time for documentation lately, but have found some issues which need repair on the work I have done. I'll get that committed once it's clean. Given what you've said, below, I will stop changing the man pages in the cyrus-imapd repo. So far I've already been replicating all changes made there into cyrus-docs anyway. Once I've got the commands pages doing what I want, I'll get a documentation best practices or style guide put together. Something where I can note how the manpages are being done, and you can add whatever else. Okay? Cheers, -nic On 05/20/2015 10:34 PM, Nicola Nye wrote: Hi Nic, I am not (yet!) aware of much of the internals of imapd or sasl, so I think we have a good breakdown between us of input to add to the Cyrus docs. If you're able to unify and get the commands/config docs up to date, that is awesome. Bron added you as a committer to the cyrus-docs repo, so you're free to update in there. My plan is to ignore cyrus-imapd repository. Just concentrate on getting cyrus-docs current so we can declare the old cyrus-imapd docs area obsolete. Trying to update both is just resulting in double the work for little gain. To that end, I suggest you continue declaring your conventions on handling things in cyrus-docs (such as handling the revision data, along with the stylistic conventions) and throw up a note in the Phabricator wiki about how it's now all done, much as you did with the Documentation Desktop Tools. For providing fancier web based documentation vs man page documentation, a quick rummage around the Sphinx docs shows that you can specify to only include content which contains certain tags. Perhaps we can set up some tags for html only output. http://sphinx-doc.org/markup/misc.html#directive-only I also wonder if this might be used to give us simple versioning, if we want to generate a separate doctree for each imap release. (Instead of having a single doctree with inline notes). Probably a bridge to cross later! Cheers, Nicola On Tue, May 19, 2015, at 10:22 PM, Nic Bernstein wrote: Nicola, et alia, [Apologies for top-posting, but it seemed appropriate here] Glad to see you jumping on board, and welcome! This all sounds good, and much needed. Something brought home to me by my recent push to harmonize man pages (between the two branches you've mentioned) is that the current bifurcation between cyrus-imapd and cyrus-docs repositories presents some problems. Since there /is/ still documentation, in the form of man pages, at least, in the cyrus-imapd repository, any proper document review and update ends up touching both repos, and thus requires separate commits (or diffs, in arcanist lingo) and separate review and landing tasks. whinge I submitted D31 over a month ago and it sat unattended until about a week ago, when it was reviewed and cleared for landing. But, as I have insufficient rights, I cannot land it. That's in the cyrus-imapd repo so thus requires someone with rights in that branch to land them. Similarly, I've more recently submitted D43, D44 D45, which still await review, etc. /whinge I'm not asking for such rights, but I am hoping that someone can make some proper sense out of this split repo. In an IRC exchange, Jeroen wrote: [2015-05-06 15:37:44] kanarip_ *fwiw, i also plan to generate very much larger man-pages by using the rst* [2015-05-06 15:42:36] kanarip_ *so, this for example: https://docs.cyrus.foundation/imap/admin/commands/chk_cyrus.html* [2015-05-06 15:42:44] kanarip_ *can also become chk_cyrus.8* [2015-05-06 15:43:04] kanarip_ *but there's just much more real-estate in the web page than there is in the man page, right?* [2015-05-06 15:43:14] kanarip_ *and much more markup to be used and abused* [2015-05-06 15:43:43] kanarip_ *so i would prefer that when we aim to make the documentation more complete if you will* [2015-05-06 15:44:26] kanarip_ *barring that the actual man page when used as a man page does not become completely illegible, it only needs to state the text even if in a slightly less legible format IMHO* [2015-05-06 15:44:34] onlight /Ok, but how shall we harmonize the two different versions? Shall we deprecate the old-style stuff in cyrus-imap/man in preference/ [2015-05-06 15:45:17] onlight /for the newer RST-based stuff?.../ [2015-05-06 15:45:43] kanarip_*i would like to, and then be able to create a git submodule for the docs repo, and then generate the docs, and then pull those docs back in as man-pages* [2015-05-06 15:45:56] kanarip_ *with the exception of man/imapd.conf.5 of course* I am by no means a git expert, so have no idea what's involved with realizing this goal, but I do think it would help tremendously to be able to abandon the old stuff in cyrus-imapd/man for converted pages from cyrus-docs
Re: Documentation Unification and Awesomeification
* There are stylistic differences which should/must get sorted o Traditional man pages, such as those in cyrus-imapd/man, alternate *bold* and /italic/ like so: + # *command --option* /value/ *-D -b -C* //etc/imapd-new.conf/ *-X* ... o Man page output produced by Sphinx, however, does not follow this convention. o In the existing collection of reStructured Text files, there is a mix of ``literal`` and **bold** for the same thing, such as command name. + The convention is **bold**, but whatever is used, we should unify this + For the record, this does not appear to make a significant difference in HTML output, but does affect man page rendering (at least in my experience). Okay, I'll get off of my soapbox for now. I just wanted to take this opportunity to get some of these issues out there. Cheers, -nic On 05/18/2015 10:57 PM, Nicola Nye wrote: Hi folks, I'm a tech writer with FastMail and I'm here to help. Here's my plan to organise the (many and flavourful) varieties of documentation surrounding the Cyrus imap universe. Please speak up if you have thoughts/ideas/objections! *The Goal* Make the content on docs.cyrus.foundation the authoritative source for all things Cyrus Imap/Sasl. (The server/websitename will be changing at some point, but we still need the content in one spot) *The Plan* Migrate content from cyrusimap mediawiki, where appropriate. Update as necessary. Absorb relevant content lurking on git.cyrus.foundation's wiki (specifically the developer setup/install guide) Update cyrusimap.org/cyrussasl.org http://cyrusimap.org/cyrussasl.org to just point people at docs.cyrus.foundation *Doc structure* So what should the new docs.cyrus.foundation look like? Much as it does now, just with more! Home * What is Cyrus * About the Cyrus Foundation (what, who, history, bylaws) * Latest news * Features (roadmap) Download * Beta, Stable, Old * Pointers to Docs/Install guides Documentation * Contributor (Set up your dev env, prerequisites, obtaining source and libraries, building, verifying, gotchas, faq) * Administrator (installation, verification, customisation, operation, faq, man pages) Support * Contact (irc, mailing list) * Bug reports And generally making it more pretty - cyrusimap has an icon and colour scheme which presumably could be brought over to docs.cyrus.foundation which currently looks stylistically sparse. Nic (onlight) seems to have a good handle on bringing the man pages up to date and ensuring they're current at docs.cyrus.foundation, which is great. (cyrus-docs/source/imap/admin/commands and in cyrus-imap/man both contain man pages) *Questions* Is someone able to upgrade my privileges so I can commit changes to the cyrus-docs tree? (/Jeroen?/) Does the new site need copyright content across it (a la cyrusimap which refers to CMU). If so, what do we need? Currently it just says copyright The Cyrus Team. What's the process for pushing a new batch of docs onto docs.cyrus.foundation once they're written? What else am I missing? Cheers, Nicoloa -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
Re: Minutes 11 May
On 05/11/2015 08:52 AM, Bron Gondwana wrote:
Speaking of naming - cyrusfoundation ==http://www.cyrusfoundation.org/
== not us. We need a new name. Cyrus IMAP Foundation - two imappy.
Cyrus Mail Foundation - not enough cal/carddav. Cyrus Connect -
doesn't someone already own * Connect? But maybe. reCyrus /
librecyrus / recycle. Hmm... Great names greatly appreciated. Make
your mark here, but don't make it a trademark that somebody else has
already... yeah, that.*sigh*. All the good ones are taken or insane
(or both).
How about something which recognizes the impending JMAP aspect, like
cyrusjmap.{com|org|foundation}?
-nic
--
Nic Bernstein n...@onlight.com
Onlight llc. www.onlight.com
219 N. Milwaukee St., Ste. 2A v. 414.272.4477
Milwaukee, Wisconsin 53202 f. 414.290.0335
Re: #documentation_reviewers
On 05/14/2015 08:42 PM, Chris Davies wrote: Does anyone know how I join #documentation_reviewers on phabricator? I tried to give myself permission but got: You do not have permission to edit this object. Users with the Can Edit capability: * Administrators can take this action. I assume it sends people in the #documentation_reviewers group an email whenever the documentation changes? If so I would like to join this group. If you have admin permission, could you please add 'davies' to the group? Chris, I think the best way to achieve the goal of being notified of all changes to documentation is to become a watcher on the Documentation project in Phabricator. To do so, go to the project's page, https://git.cyrus.foundation/project/profile/8/, and click Watch Project on the right side of the screen: Watching a project will let you monitor it closely. You will receive email and notifications about changes to every object associated with projects you watch. I think that will give you what you want. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
Re: Cyrus Documentation tools
On 05/13/2015 04:33 AM, Bron Gondwana wrote: Doxygen would be fine. On Wed, May 13, 2015, at 09:24 AM, Chris Davies wrote: Adding the Cyrus mailing list as I can't remember who the other documentation person is. Do you think something like Doxygen http://www.stack.nl/%7Edimitri/doxygen/index.html would be of use for the Cyrus project? it can generate HTML docs and man pages. I haven't downloaded it yet but it claims to do what we need. Chris, The current documentation, in the cyrus-docs branch, is in reStructured Text http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html. There's a page up on the wiki which describes what I have used to get up and working with this existing collection of files: https://git.cyrus.foundation/w/documentation_desktop_tools/ On Wed, 13 May 2015, at 05:08 PM, Chris Davies wrote: Just trying to figure out how Cyrus works and exploring the admin tools: https://docs.cyrus.foundation/imap/admin.html The ctl_deliver program outputs a list of files and/or directories that it expects to exist, but that in fact do not. The ctl_mboxlist program outputs a list of files and/or directories that it expects to exist, but that in fact do not. The cvt_cyrusdb program outputs a list of files and/or directories that it expects to exist, but that in fact do not. The cyr_dbtool program outputs a list of files and/or directories that it expects to exist, but that in fact do not. Yeah, maybe not. To be fair, these are all in the Work-in-progress section of the page, which includes this disclaimer: *For the following parts of the documentation, while they are a work-in- progress, you may already have better documentation on your system, in the form of actual man-pages.* We are in the process of fixing all of this. For example, a recent commit, D31 https://git.cyrus.foundation/D31, is ready to land, and contains updates to the ctl_cyrusdb(8), chk_cyrus(8) and unexpunge(8) man pages (these being in cyrus-imap branch, in the /man/ directory). Another issue with which we're contending, and which was discussed recently on IRC, is that there's two versions (at least) of some of this stuff, in cyrus-docs/source/imap/admin/commands and in cyrus-imap/man. I'm currently working to harmonize these two versions so we can dump one or the other. One of the reasons to get access control working again on the old site, www.cyrusimap.org, is to at least allow that to serve as an intermediary resource until the new site gets up to snuff. All of the commands you list, above, are fully documented here: http://www.cyrusimap.org/docs/cyrus-imapd/2.4.17/man.php But, we know that the branches for newer releases (2.5*) there are broken, so for now stick with the 2.4.17 until we can get the rest fixed. Cheers, -nic (on IRC as 'onlight') Bron, Who was the other person working on documentation? Simon Amor. Who is on this list I'm pretty sure. He's ^Simon^ in IRC. Bron. -- Bron Gondwana br...@fastmail.fm -- Nic Bernstein n...@onlight.com Onlight, Inc. www.onlight.com 1442 N Farwell Ave., Suite 600v. 414.272.4477 Milwaukee, Wisconsin 53202 attachment: nic.vcf
Re: Cyrus-devel Digest, Vol 115, Issue 7
On 05/12/2015 08:36 PM, Chris Davies wrote: [quoting me wihout reference...] I've already spoken up about trying to do something about this, but to no avail. I'll volunteer to at the very least uncrap things a bit. However, the current site has some problems with the authentication system. I have credentials on the site, but had lost/forgotten the password. I used the password reset facility, but now when I log in I am asked to change my password, and that process always results in this message: There seems to be a problem with your login session; this action has been canceled as a precaution against session hijacking. Go back to the previous page, reload that page and then try again. So I'm stuck. If someone can fix this, fine; please do so. If not, I can register again, but then need someone with authority to authorize my new account, and I'll probably be asked to update my password, and... (wash, rinse, repeat). Have you tried a different web browser? Yes, Firefox, Epiphany and Chromium on three different Linux hosts. This is not a browser issue, or if it is, it's a pervasive one. Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
Re: Minutes 11 May
On 05/11/2015 03:53 PM, Nic Bernstein wrote:
lirecyrus.{io|com|org} are all available
Opps, typo. I meant to write that librecyrus.{io|com|org} are all
available.
-nic
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
1442 N Farwell Ave., Suite 600v. 414.272.4477
Milwaukee, Wisconsin 53202
attachment: nic.vcf
Re: Minutes 11 May
On 05/11/2015 08:52 AM, Bron Gondwana wrote:
Website - the situation with cyrusimap.org and cyrus.foundation is
awfully confusing. Somebody needs to take ownership for uncrapping this.
Nobody put their hand up yet, so we'll see if it happens by Thursday or
I might start appointing people;)
I've already spoken up about trying to do something about this, but to
no avail. I'll volunteer to at the very least uncrap things a bit.
However, the current site has some problems with the authentication
system. I have credentials on the site, but had lost/forgotten the
password. I used the password reset facility, but now when I log in I
am asked to change my password, and that process always results in this
message:
There seems to be a problem with your login session; this action has
been canceled as a precaution against session hijacking. Go back to
the previous page, reload that page and then try again.
So I'm stuck. If someone can fix this, fine; please do so. If not, I
can register again, but then need someone with authority to authorize my
new account, and I'll probably be asked to update my password, and...
(wash, rinse, repeat).
Speaking of naming - cyrusfoundation ==http://www.cyrusfoundation.org/
== not us. We need a new name. Cyrus IMAP Foundation - two imappy.
Cyrus Mail Foundation - not enough cal/carddav. Cyrus Connect -
doesn't someone already own * Connect?
FWIW, both cyrus-connect.com and cyrus-connect.org are available, as are
the unhyphenated versions, cyrusconnect.{com|org}.
But maybe. reCyrus /
recyrus.{io|com|org} are all available
librecyrus
lirecyrus.{io|com|org} are all available
Okay, putting aside simple whois searches and getting back to work now.
-nic
PS - Great to meet folks at OpenSuSE and Kolab conferences last week. :-)
/ recycle. Hmm... Great names greatly appreciated. Make
your mark here, but don't make it a trademark that somebody else has
already... yeah, that.*sigh*. All the good ones are taken or insane
(or both).
--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
1442 N Farwell Ave., Suite 600v. 414.272.4477
Milwaukee, Wisconsin 53202
attachment: nic.vcf
Re: Meeting 2015-04-23T12:00:00Z
On 04/23/2015 04:50 AM, Bron Gondwana wrote: This time NEXT week, I'll be in the air still, on my way to Amsterdam for Kolab Summit, where I'm talking about Cyrus: https://conference.kolab.org/kolab-summit/sessions/cyrus-imapd-past-current-and-future I'll send slides around soon and stuff. I'll be arriving in The Hague on 30/4 for the Summit. Anyone else getting there before 2/5 who might be interested in some sort of meet-up before or during the summit? If so, please respond and I'll see if I can put something together. That's in addition to Bron's highly anticipated session, of course. :-) Cheers, -nic -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
Re: uniqueid based paths
On 03/23/2015 06:09 AM, Thomas Jarosch wrote: With uniqueid based paths, it won't be easy to use unix tools to grep the message base of a single user only. You first need to filter the list of folders and then limit your view to that folder list. A tool that lists the user folder - uniqueid based path in a machine parsable way (=scriptable) might help here. This sounds like an updated (uniqueid aware) version of mbpath(8) to me. -nic -- Nic Bernstein n...@onlight.com Onlight llc. www.onlight.com 219 N. Milwaukee St., Ste. 2A v. 414.272.4477 Milwaukee, Wisconsin 53202 f. 414.290.0335
