Re: cyrus-imap buildin: sse extention
Ellie, I also had the doubt about this feature, though I'd already seen a mention that the result of the hw implementation is incompatible (and before your last mail completely forgot about it). Maybe it makes sense to remove its mention (and detection) from configure altogether, until it becomes useful? Just to not confuse those building it from sources. Regards, Anatoli On 28/6/20 21:29, ellie timoney wrote: > Hi Sergey, > >> Hardware support: >>SSE4.2: yes > > This is detected for a hardware implementation of the CRC32c algorithm. > Cyrus doesn't actually use it though, because it's not compatible with the > existing CRC32 algorithm: i.e. for the same input, it produces a different > checksum, which would break everything on a system with pre-existing data. > > If you _want_ to use the hardware CRC32c algorithm on a brand new deployment > (and know what you are doing), I believe at this stage you would need to > patch Cyrus to use it -- the code is there, but it is never called. > >> Can Cyrus-IMAP be running on systems without SSE4 at this case? > > Yep, it'll work just fine. The hardware CRC32c code will simply not be > compiled, which, since it isn't used anyway, will have no effect. > >> If no, can I set limit to SSE2 ? > > There's currently no configurability for this at all. I don't know if it's > even possible to implement the same algorithm with SSE2. At the moment I > assume that, since it's looking for SSE42 specifically, then that means that > the hardware feature it needs is probably only available in SSE42. > > Cheers, > > ellie > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Just-Enuf Cyrus IMAP for standalone Card-/CalDAV?
You can have a complete Cyrus install but only use the DAV features. You just don't open the IMAP ports in firewall and that's it. On 16/12/19 01:12, PGNet Dev wrote: > I run my own smtp, imap & card/cal-dav services. > > Atm, none are using Cyrus. > > I'm well aware of Cyrus Imap's Cal/Card-dav support; particularly as the > basis for FastMail's excellent services. > > My current card/cal-dav is Radicale. To my read, the project's crumbling. > > I need to implement/deploy an alternative. > > But, for **now**, ONLY for the *Dav pieces; I need to leave mail services as > they are, at least 'til mid-2020. > > For my domain, 'example.com' -- with smtp/imap @ host 'mx.example.com', I'd > like to deploy a standalone Dav server elsewhere, @ host 'dav.example.com'. > > I'd like it to be Cyrus based. > > IIUC, Cyrus' *Dav are built on top of its IMAP install. > > Can Cyrus be deployed in a standalone *Dav server mode like this? > > Is there a doc/how-to/etc that address a Just-Enough-Cyrus-IMAP install for > this kind of setup? > > Thx! > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus webdav with Joplin
Hi Johan, In RFC 7231 (HTTP 1.1) section 3.1.1.5 (https://tools.ietf.org/html/rfc7231#section-3.1.1.5) it says that CT header SHOULD be present, otherwise the recipient may interpret it the way it wants, so IMO no problem on the Cyrus side here. For application/json for example it MUST be present, application/xml doesn't demand that, but not sending it IMO is not a good behavior for interoperability. For collection that exists, does the user that makes the request have the rights to overwrite the collection? If not, 403 is the correct SC (status code). 405 should be used when the specified method is not allowed at all on the specified path, independently of the current server state, which is not the case here. So, again IMO no problem on the Cyrus side here, but if the user has sufficient rights, instead of 403 I'd use "409 Conflict" which is the recommended SC when a record with specified ID/name already exists. Regards, Anatoli On 28/11/19 04:40, Johan Hattne wrote: > Dear all; > > I’m trying to get Joplin (https://joplinapp.org) to work with Cyrus’s webdav > module, and I’ve run into two issues: > > (1) When attempting to MKCOL a collection that already exists, Cyrus is > responding with a 403, rather than a 405, which is what Joplin expects. > > (2) Cyrus returns an error if the Content-type isn’t set where additional > XML-formatted information is required in a POST to complete a request. > > My skimming of the relevant RFC:s now lead me to believe that Cyrus is right > on both counts; however, I don’t know enough about this to say for sure. Can > anyone here confirm, or are these genuine Cyrus bugs? > > // Best wishes; Johan > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: strange sort order of these emails
Hi Gabriele, I suggest you send the actual headers of both emails here as plain text in the body of the email, as people may not want to open attachments from unknown sources. Also it's easier to analyze and reply this way. Regards, Anatoli On 26/9/19 11:52, Gabriele Bulfon wrote: > Hello, > someone received these two emails in the inbox, but strangely sorting by > date descending was showing them swapped (they have same date but > different hour). > I did not beleive my eyes, so I downloaded the two eml files and > uploaded them into a cyrus folder of mine, on another server. > Same effect: the SORT command with descending DATE, will always returns > indexes in swapped order, so the one at 3am is before the one at 9am. > > I looked at all the headers but didn't find anything strange. > Cyrus is version 2.5.11. > Any idea?! > Thanks! > Gabriele > > > *Sonicle S.r.l. *: http://www.sonicle.com <http://www.sonicle.com/> > *Music: *http://www.gabrielebulfon.com <http://www.gabrielebulfon.com/> > *Quantum Mechanics : *http://www.cdbaby.com/cd/gabrielebulfon > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: sieve filter based on utf-8 encoded text part of address
Hi Pete,
I guess the 'address' test command matches only the actual address, not
the description. In your example it would match "my@address".
I suggest you check
https://www.cyrusimap.org/imap/reference/admin/sieve.html and
https://thsmi.github.io/sieve-reference/en/index.html.
Regards,
Anatoli
On 30/9/19 08:12, Hans-Peter Jansen wrote:
> Hi,
>
> I try to filter based on the text part of an utf-8 encoded address, but the
> string matches neither decoded nor encoded.
>
> This is a periodic mail from one of a couple of different routers, where only
> the text part defines a useful origin:
>
> From: "=?UTF-8?B?RlJJVFohQm94IDc0OTAgU0ZK?="
>
> which decodes to:
>
> FRITZ!Box 7490 SFJ
>
> sieve filter script except:
>
> ; my requirements
> require ["fileinto", "reject", "regex", "vacation"];
>
> ; the filter rule
> if address :contains "From" [
> "FRITZ!Box 7490 SFJ",
> "=?UTF-8?B?RlJJVFohQm94IDc0OTAgU0ZK?="
> ] {
> fileinto "INBOX.some.folder";
> stop;
> }
>
> How do I do this correctly?
>
> Thanks in advance,
> Pete
>
>
>
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
>
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyrus-imapd build dependencies
Ken, Ellie, Thanks for the information. Ellie, I see that #2100 has everything to be merged, hasn't it? What's blocking it? Regards, Anatoli *From:* Ellie Timoney *Sent:* Tuesday, March 19, 2019 03:29 *To:* Info-cyrus *Subject:* Re: cyrus-imapd build dependencies On Tue, Mar 19, 2019, at 3:39 PM, Anatoli via Info-cyrus wrote: > The Cyrus httpd provides DAV services (which use the HTTP protocol). If you want the Cyrus httpd to support HTTP/2, you will need libnghttp2. Otherwise it will only support HTTP/1. Always wanted to ask what the nghttp2 dependency was for. From what you say I infer that it's only needed for HTTP/2. But what DAV service could benefit from this? Are there DAV clients that know HTTP/2? No idea, but it's there if you want it! Speculating wildly, it might be useful for JMAP? And speaking about the SNMP agent, are there any plans to complete the transfer of its code from the master process to an independent daemon, issue #1765 <https://github.com/cyrusimap/cyrus-imapd/issues/1765>? (It needs to be moved out to implement efficient chroot) It's more likely to disappear entirely (see https://github.com/cyrusimap/cyrus-imapd/pull/2100) in favour of Prometheus (which is more powerful, more flexible, more human-readable, and is actually used by Fastmail -- and therefore more tested). But it won't disappear from a stable branch, so it won't be a surprise when it does. Cheers, ellie Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: cyrus-imapd build dependencies
Hi Ellie, > The Cyrus httpd provides DAV services (which use the HTTP protocol). If you want the Cyrus httpd to support HTTP/2, you will need libnghttp2. Otherwise it will only support HTTP/1. Always wanted to ask what the nghttp2 dependency was for. From what you say I infer that it's only needed for HTTP/2. But what DAV service could benefit from this? Are there DAV clients that know HTTP/2? And speaking about the SNMP agent, are there any plans to complete the transfer of its code from the master process to an independent daemon, issue #1765 <https://github.com/cyrusimap/cyrus-imapd/issues/1765>? (It needs to be moved out to implement efficient chroot) Regards, Anatoli *From:* Ellie Timoney *Sent:* Monday, March 18, 2019 21:55 *To:* Info-cyrus *Subject:* Re: cyrus-imapd build dependencies Hi Patrick, On Mon, Mar 18, 2019, at 11:33 PM, Patrick Goetz wrote: This page on compiling cyrus-imapd: https://www.cyrusimap.org/imap/developer/compiling.html This page is in the developer section, so its context is for people who are Cyrus developers (especially for new contributors needing to get rolling quickly). Expect a certain amount of detail to be glossed over on the assumption that it's already known and/or reasonably documented elsewhere. shows a number of build dependencies; however I was just able to compile cyrus-imapd without these installed: gperf libbsd Are these actually necessary? Probably depends on which features you enable. If you run './configure' without arguments, a number of large features won't be enabled, so any libraries they depend on won't be used. Some of these features are important enough that we (developers) kind of think of them as being probably-always-included even if they default to not. Later in the page, under "Alternate database formats" it shows the configure flags to use in order to use mysql/mariadb as a backend for cyrus databases. I think this is needed if one plans to use virtual domains, but I couldn't get a confirmation on this. These are literally just "alternate database formats" -- maybe you already have extensive expertise in some other database and would rather use that than one of the builtin ones. It has nothing to do with virtual domains. Documentation about the databases used by Cyrus are here: https://www.cyrusimap.org/imap/concepts/deployment/databases.html In any case, the configure options are given as --with-mysql, --with-mysql-incdir, --with-mysql-libdir with no clear indication of what each of these does. For example, is the --with-mysql all inclusive, or does one need to set all 3? The canonical source of information on configure options is the output from './configure --help'. It's kind of assumed that a developer will look there to find this information. Finally a couple of items in the "Other" category are a real head scratcher. For example, what is the purpose of net-snmp? You can click on any of those package names to go to the website for that package and get a description of what it does. For example, Simple Network Management Protocol (SNMP) is a widely used protocol for monitoring the health and welfare of network equipment (eg. routers), computer equipment and even devices like UPSs. Net-SNMP is a suite of applications used to implement SNMP v1, SNMP v2c and SNMP v3 using both IPv4 and IPv6. libnghttp2 is listed as needed for "HTTP/2 support for httpd" -- what's using httpd? Is this to faciliate CalDAV/CardDAV? The Cyrus httpd provides DAV services (which use the HTTP protocol). If you want the Cyrus httpd to support HTTP/2, you will need libnghttp2. Otherwise it will only support HTTP/1. Hope this helps :) ellie Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: WebDAV folders internally have hundreds of copies of the same few files
> What are you using as the mailbox name? #drive/shared/test2 Ken, do you know why cyr_expire -u and -p don't work on shared DAV mailboxes? E.g. cyr_expire -X 3 -u "#drive/shared/test2" doesn't clean the folder and with -v shows 0 for everything, then unexpunge -l "#drive/shared/test2" shows a lot of items with Expg date older that 3 days, but a general cyr_expire -X 3 (i.e. without the -u or -p) does clean the entire store, including the shared DAV folders. *From:* Ken Murchison *Sent:* Tuesday, November 20, 2018 10:22 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files What are you using as the mailbox name? On 11/20/18 1:01 AM, Anatoli wrote: > Enable the imapmagicplus option in imapd.conf With this option I could login with imtest specifying "+DAV" for the user, both as the admin and as a regular user, but then when trying to select a folder, I got "NO Permission denied" for the admin and "NO Mailbox does not exist" for a common user. This looks like undocumented magic and mbexamine actually provides all needed information, so I will get file details with it. Thanks for you help, Ken. *From:* Ken Murchison *Sent:* Friday, November 16, 2018 13:17 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files On 11/16/18 1:19 AM, Anatoli wrote: Ken, > If you login as an admin, you should be able to SELECT the mailbox and use normal IMAP commands. Alternatively, if you add "+DAV" to a regular userid, this will also allow SELECTing of DAV collection mailboxes. When I do imtest -a mailadmin and then issue a002 select #drive/folder, I get: a002 NO Invalid mailbox type It appears that just logging in as an admin won't do the trick. When I do imtest -a mailadmin+DAV, I get (XX replaces the real auth value): C: A01 AUTHENTICATE PLAIN XXX S: A01 NO authentication failure Authentication failed. generic failure I've also tried these combinations with the same result: "u...@domain.tld+DAV" and "user+...@domain.tld". Am I doing it wrong? Enable the imapmagicplus option in imapd.conf Independently of this, is there a way to obtain the details about the flags for each message in a folder directly from the db files? My idea is to rsync the Cyrus store where the WebDAV is located to some other folder (probably on another server), purge there the files marked as deleted and backup the remaining files, all this without altering the Cyrus store. Is it possible? > If the append of the resource into the mailbox, or updating the DAV db entry fails, the operation should be reverted, with partial saving done. Which version of Cyrus are you using? Currently it is 3.0.5, but I can upgrade if there were any related changes in the later versions. What I was observing is that if a PC (LibreOffice Calc on Ubuntu 16.04, the folder mounted with davfs2) crashes in the middle of saving a file on the server, the result was a damaged file: some 400 bytes instead of 3-4Mb of a real file. I was thinking that maybe when the TCP stream hangs in the middle, Cyrus would interpret it as an end of data and write a file, but I've just tried to make some tests killing mount.davfs2 process and/or blocking the stream with iptables in the middle of file transmission and Cyrus responded correctly. For killing mount.davfs2 it showed in the corresponding log file the bytestream followed by: HTTP/1.1 400 Bad Request Unable to read body data And for blocking the stream with iptables it was showing the bytestream ending where the transmission stopped, without the 400 Bad Request, but in both cases the original file was not modified by the partial upload. So it appears the problem is elsewhere. Regards, Anatoli *From:* Ken Murchison *Sent:* Thursday, November 15, 2018 14:09 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files On 11/15/18 1:48 AM, Anatoli wrote: Hi Ken, Thanks a lot for the clarification, everything makes sense now. How can I list the files marked for deletion and those that are not marked? unexpunge can provide the list of files marked for deletion, is there any better way to list them, directly reading the DB? How to list those that are not marked? If you login as an admin, you should be able to SELECT the mailbox and use normal IMAP commands. Alternatively, if you add "+DAV" to a regular userid, this will also allow SELECTing of DAV collection mailboxes. One more question, related: we got a client who's PC was crashing exactly during the modify operation (some issue with the PC hardware triggered by Excel save operation, probably a RAM spike touching some bad blocks). As a result, the file in Cyrus was becoming damaged, i.e. partially saved. Is it expecte
Re: WebDAV folders internally have hundreds of copies of the same few files
Hi Vladislav, Yes, I don't actually have the expunge_mode option in the configs, so the default "delayed" is applied. I've just tried some experiments with cyr_expire, the -X n option alone clears everything, including the shared DAV folders, but I can't find how to clear only some shared DAV folder. The -p and -u options seem not to work as intented on the shared DAV folders (user-owned DAV folders are cleared with -p param). Any idea how to accomplish this? Regards, Anatoli *From:* Vladislav Kurz *Sent:* Monday, November 19, 2018 05:50 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files On 11/14/18 2:54 PM, Ken Murchison wrote: On 11/13/18 10:15 PM, Anatoli wrote: Hi, I'm not sure this is due to some configuration option, bug or feature, but I'm observing some folders on Cyrus HTTP WebDAV server having hundreds (995 at this moment to be precise) internal files in the format "NNN." that correspond to the same file but different versions in time. There are 2-3 files (xls) in the folder that are edited constantly during the day and it looks like each save operation creates a new file. The files are of some 3-5Mb each. In the explorer/web view there are only a couple of files with a total size of 17.5Mb, but the reported disk usage for the folder is of 1.6Gb. Could someone please shed some light on what's going on and how to make each file visible to the users to be stored in only one internal file? Thanks, Anatoli Because *DAV is layered on top of an IMAP store, we have to abide by IMAP semantics in which messages (in this case DAV resources) are immutable. Therefore, we can NOT overwrite an existing message in the mailbox. Each change MUST result in a new message. However, the server does mark the previous version(s) as deleted and expunged, which means that they will eventually be removed by cyr_expire. If you aren't running cyr_expire, you should consider adding an event to cyrus.conf to remove expunged messages (see -X option). Hello, you probably have "expunge_mode: delayed". That's why deleted mails (and *dav files) stay in place, until cleaned by cyr_expire as mentioned above. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: WebDAV folders internally have hundreds of copies of the same few files
> Enable the imapmagicplus option in imapd.conf With this option I could login with imtest specifying "+DAV" for the user, both as the admin and as a regular user, but then when trying to select a folder, I got "NO Permission denied" for the admin and "NO Mailbox does not exist" for a common user. This looks like undocumented magic and mbexamine actually provides all needed information, so I will get file details with it. Thanks for you help, Ken. *From:* Ken Murchison *Sent:* Friday, November 16, 2018 13:17 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files On 11/16/18 1:19 AM, Anatoli wrote: Ken, > If you login as an admin, you should be able to SELECT the mailbox and use normal IMAP commands. Alternatively, if you add "+DAV" to a regular userid, this will also allow SELECTing of DAV collection mailboxes. When I do imtest -a mailadmin and then issue a002 select #drive/folder, I get: a002 NO Invalid mailbox type It appears that just logging in as an admin won't do the trick. When I do imtest -a mailadmin+DAV, I get (XX replaces the real auth value): C: A01 AUTHENTICATE PLAIN XXX S: A01 NO authentication failure Authentication failed. generic failure I've also tried these combinations with the same result: "u...@domain.tld+DAV" and "user+...@domain.tld". Am I doing it wrong? Enable the imapmagicplus option in imapd.conf Independently of this, is there a way to obtain the details about the flags for each message in a folder directly from the db files? My idea is to rsync the Cyrus store where the WebDAV is located to some other folder (probably on another server), purge there the files marked as deleted and backup the remaining files, all this without altering the Cyrus store. Is it possible? > If the append of the resource into the mailbox, or updating the DAV db entry fails, the operation should be reverted, with partial saving done. Which version of Cyrus are you using? Currently it is 3.0.5, but I can upgrade if there were any related changes in the later versions. What I was observing is that if a PC (LibreOffice Calc on Ubuntu 16.04, the folder mounted with davfs2) crashes in the middle of saving a file on the server, the result was a damaged file: some 400 bytes instead of 3-4Mb of a real file. I was thinking that maybe when the TCP stream hangs in the middle, Cyrus would interpret it as an end of data and write a file, but I've just tried to make some tests killing mount.davfs2 process and/or blocking the stream with iptables in the middle of file transmission and Cyrus responded correctly. For killing mount.davfs2 it showed in the corresponding log file the bytestream followed by: HTTP/1.1 400 Bad Request Unable to read body data And for blocking the stream with iptables it was showing the bytestream ending where the transmission stopped, without the 400 Bad Request, but in both cases the original file was not modified by the partial upload. So it appears the problem is elsewhere. Regards, Anatoli *From:* Ken Murchison *Sent:* Thursday, November 15, 2018 14:09 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files On 11/15/18 1:48 AM, Anatoli wrote: Hi Ken, Thanks a lot for the clarification, everything makes sense now. How can I list the files marked for deletion and those that are not marked? unexpunge can provide the list of files marked for deletion, is there any better way to list them, directly reading the DB? How to list those that are not marked? If you login as an admin, you should be able to SELECT the mailbox and use normal IMAP commands. Alternatively, if you add "+DAV" to a regular userid, this will also allow SELECTing of DAV collection mailboxes. One more question, related: we got a client who's PC was crashing exactly during the modify operation (some issue with the PC hardware triggered by Excel save operation, probably a RAM spike touching some bad blocks). As a result, the file in Cyrus was becoming damaged, i.e. partially saved. Is it expected? Shouldn't Cyrus update the db with the pointer to the new file (a new message in the store) only if the operation completes successfully (e.g. the WebDAV messages exchange completes and the connection is closed at the right time or something similar)? If the append of the resource into the mailbox, or updating the DAV db entry fails, the operation should be reverted, with partial saving done. Which version of Cyrus are you using? *From:* Ken Murchison *Sent:* Wednesday, November 14, 2018 10:54 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files On 11/13/18 10:15 PM, Anatoli wrote: Hi, I'm not sure this is due to some configuration option, bug or feature, b
Re: WebDAV folders internally have hundreds of copies of the same few files
Hi Sebastian. I've already tried it but it looks like it can't examine the structure of a folder that is a copy of the Cyrus store outside of it. Though maybe I could create a custom config file pointing to the new dir and instruct mbexamine to use it with -C param? On the other hand, I could just copy the relevant files without rsync'ing and then processing the entire spool. Thanks for the tip! *From:* Sebastian Hagedorn *Sent:* Friday, November 16, 2018 03:53 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files Hi, Independently of this, is there a way to obtain the details about the flags for each message in a folder directly from the db files? you could try mbexamine for that. -- Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 Regionales Rechenzentrum (RRZK) Universität zu Köln / Cologne University - Tel. +49-221-470-89578 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: WebDAV folders internally have hundreds of copies of the same few files
Ken, > If you login as an admin, you should be able to SELECT the mailbox and use normal IMAP commands. Alternatively, if you add "+DAV" to a regular userid, this will also allow SELECTing of DAV collection mailboxes. When I do imtest -a mailadmin and then issue a002 select #drive/folder, I get: a002 NO Invalid mailbox type When I do imtest -a mailadmin+DAV, I get (XX replaces the real auth value): C: A01 AUTHENTICATE PLAIN XXX S: A01 NO authentication failure Authentication failed. generic failure I've also tried these combinations with the same result: "u...@domain.tld+DAV" and "user+...@domain.tld". Am I doing it wrong? Independently of this, is there a way to obtain the details about the flags for each message in a folder directly from the db files? My idea is to rsync the Cyrus store where the WebDAV is located to some other folder (probably on another server), purge there the files marked as deleted and backup the remaining files, all this without altering the Cyrus store. Is it possible? > If the append of the resource into the mailbox, or updating the DAV db entry fails, the operation should be reverted, with partial saving done. Which version of Cyrus are you using? Currently it is 3.0.5, but I can upgrade if there were any related changes in the later versions. What I was observing is that if a PC (LibreOffice Calc on Ubuntu 16.04, the folder mounted with davfs2) crashes in the middle of saving a file on the server, the result was a damaged file: some 400 bytes instead of 3-4Mb of a real file. I was thinking that maybe when the TCP stream hangs in the middle, Cyrus would interpret it as an end of data and write a file, but I've just tried to make some tests killing mount.davfs2 process and/or blocking the stream with iptables in the middle of file transmission and Cyrus responded correctly. For killing mount.davfs2 it showed in the corresponding log file the bytestream followed by: HTTP/1.1 400 Bad Request Unable to read body data And for blocking the stream with iptables it was showing the bytestream ending where the transmission stopped, without the 400 Bad Request, but in both cases the original file was not modified by the partial upload. So it appears the problem is elsewhere. Regards, Anatoli *From:* Ken Murchison *Sent:* Thursday, November 15, 2018 14:09 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files On 11/15/18 1:48 AM, Anatoli wrote: Hi Ken, Thanks a lot for the clarification, everything makes sense now. How can I list the files marked for deletion and those that are not marked? unexpunge can provide the list of files marked for deletion, is there any better way to list them, directly reading the DB? How to list those that are not marked? If you login as an admin, you should be able to SELECT the mailbox and use normal IMAP commands. Alternatively, if you add "+DAV" to a regular userid, this will also allow SELECTing of DAV collection mailboxes. One more question, related: we got a client who's PC was crashing exactly during the modify operation (some issue with the PC hardware triggered by Excel save operation, probably a RAM spike touching some bad blocks). As a result, the file in Cyrus was becoming damaged, i.e. partially saved. Is it expected? Shouldn't Cyrus update the db with the pointer to the new file (a new message in the store) only if the operation completes successfully (e.g. the WebDAV messages exchange completes and the connection is closed at the right time or something similar)? If the append of the resource into the mailbox, or updating the DAV db entry fails, the operation should be reverted, with partial saving done. Which version of Cyrus are you using? *From:* Ken Murchison *Sent:* Wednesday, November 14, 2018 10:54 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files On 11/13/18 10:15 PM, Anatoli wrote: Hi, I'm not sure this is due to some configuration option, bug or feature, but I'm observing some folders on Cyrus HTTP WebDAV server having hundreds (995 at this moment to be precise) internal files in the format "NNN." that correspond to the same file but different versions in time. There are 2-3 files (xls) in the folder that are edited constantly during the day and it looks like each save operation creates a new file. The files are of some 3-5Mb each. In the explorer/web view there are only a couple of files with a total size of 17.5Mb, but the reported disk usage for the folder is of 1.6Gb. Could someone please shed some light on what's going on and how to make each file visible to the users to be stored in only one internal file? Thanks, Anatoli Because *DAV is layered on top of an IMAP store, we have to abide by IMAP semantics in which messages (in this case DAV res
Re: WebDAV folders internally have hundreds of copies of the same few files
Hi Ken, Thanks a lot for the clarification, everything makes sense now. How can I list the files marked for deletion and those that are not marked? unexpunge can provide the list of files marked for deletion, is there any better way to list them, directly reading the DB? How to list those that are not marked? One more question, related: we got a client who's PC was crashing exactly during the modify operation (some issue with the PC hardware triggered by Excel save operation, probably a RAM spike touching some bad blocks). As a result, the file in Cyrus was becoming damaged, i.e. partially saved. Is it expected? Shouldn't Cyrus update the db with the pointer to the new file (a new message in the store) only if the operation completes successfully (e.g. the WebDAV messages exchange completes and the connection is closed at the right time or something similar)? Regards, Anatoli *From:* Ken Murchison *Sent:* Wednesday, November 14, 2018 10:54 *To:* Info-cyrus *Subject:* Re: WebDAV folders internally have hundreds of copies of the same few files On 11/13/18 10:15 PM, Anatoli wrote: Hi, I'm not sure this is due to some configuration option, bug or feature, but I'm observing some folders on Cyrus HTTP WebDAV server having hundreds (995 at this moment to be precise) internal files in the format "NNN." that correspond to the same file but different versions in time. There are 2-3 files (xls) in the folder that are edited constantly during the day and it looks like each save operation creates a new file. The files are of some 3-5Mb each. In the explorer/web view there are only a couple of files with a total size of 17.5Mb, but the reported disk usage for the folder is of 1.6Gb. Could someone please shed some light on what's going on and how to make each file visible to the users to be stored in only one internal file? Thanks, Anatoli Because *DAV is layered on top of an IMAP store, we have to abide by IMAP semantics in which messages (in this case DAV resources) are immutable. Therefore, we can NOT overwrite an existing message in the mailbox. Each change MUST result in a new message. However, the server does mark the previous version(s) as deleted and expunged, which means that they will eventually be removed by cyr_expire. If you aren't running cyr_expire, you should consider adding an event to cyrus.conf to remove expunged messages (see -X option). -- Ken Murchison Cyrus Development Team FastMail US LLC Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: XLIST, special-use mailboxes
Paul, To add on top of what Bron said, xlist was removed in 2.5 but added as a (somewhat?) new implementation in 3.0, so you should install the newest version of Cyrus to take advantage of this feature. In 3.0 xlist is part of autocreate functionality, i.e. an appropriate flag is set on each newly created folder once a new mailbox is created (AFAIK, it won't work on existing folders). > What about client-support? Is it much used in clients? From my experience, it's rather well supported. Some clients from time to time (depending on the vendors, versions, locales, etc.) fail to apply these flags to some of the folders, but most of the time it's very useful. Regards, Anatoli *From:* Bron Gondwana *Sent:* Friday, June 15, 2018 03:23 *To:* Info-cyrus *Subject:* Re: XLIST, special-use mailboxes Hi Paul, In newer versions the support is more accurately to spec, where you create with (USE \Foo) and then it's stored alongside that mailbox, even if it's renamed. The old 2.4 version was just a hack for XLIST before the standards support was added. Cheers, Bron. On Fri, Jun 15, 2018, at 01:03, Paul van der Vlis wrote: Hello, Does Cyrus IMAP support RFC 6154 about special-use mailboxes? https://tools.ietf.org/html/rfc6154 I read that it only wass supported in Cyrus version 2.4 ? I am using version 2.5.10 (from Debian 9). Can I put this in imapd.conf for all users? xlist-archive: Archives xlist-drafts: Drafts xlist-sent: Sent xlist-spam: Spam xlist-trash: Trash Is it only used on new created folders or always? Do I need "specialusealways: 1" or something like that? What about client-support? Is it much used in clients? With regards, Paul van der Vlis -- Paul van der Vlis Linux systeembeheer Groningen https://www.vandervlis.nl/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: setting acl on autocreate folders
Ellie, Thanks for your feedback. I've just created a new feature request issue for this: https://github.com/cyrusimap/cyrus-imapd/issues/2372. I don't have time now to implement it myself, though I'd definitely prefer to spend time on expanding Cyrus than writing custom scripts if I had the same need as 4 years ago. Maybe some new Cyrus user would have time to make it happen, or maybe I'd find some time later. So the issue is to document the intention and to have defined some implementation details. Probably, it could have the "help wanted" tag attached. Regards, Anatoli *From:* Ellie Timoney *Sent:* Tuesday, May 15, 2018 00:46 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: setting acl on autocreate folders Something like: autocreate_acl(multiple autocreate_acl entries could be specified) That's roughly what I'd expect such a feature to look like (without having thought about it in much depth). It seems like it would be very useful for admins who don't already have their own provisioning infrastructure. Ellie, do you think this is something of low complexity? In my opinion, any new feature for ACL's is inherently high complexity -- even if it's just a two line patch -- just because of the amount of work involved in checking for side effects, error handling, and making sure the documentation is up to scratch (so that people don't accidentally configure it wrong and get into trouble). That said, the code for reading config settings already exists, as does the code for parsing and applying ACL's. So in theory it should be a "simple" matter of bolting these bits together in the right place(s). I'd be happy to review/advise on a pull request along these lines! Cheers, ellie On Sat, May 12, 2018, at 7:40 AM, Anatoli wrote: > I think it's good that you have to explicitly set "anyone p", because otherwise people would be able to send plus+adressed mails to any mailbox whose name they can guess. As the default behavior, I agree. I've just made a couple of tests: remove "anyone p" then add "postman p" or add postman as "admins: postman" but none of these worked for plus+addressing (P+A), so the /postman/ user appears to be some hardcoded way of dealing with LMTP delivery and has nothing in common with the normal users and operations. If my assumptions are correct, I guess what Chen (OP) was asking would be useful, i.e. to be able to define "anyone p" (either as a toggle aimed at P+A or as a free-form for any user/ACL) for some auto-created folders along the other auto-configuration features (autocreate_XXX, x-list, etc.). The idea is to be able to setup most common settings for new users without any external scripts talking to cyradm or imtest. In my case the "anyone p" permission is the only thing pending. Something like: autocreate_acl(multiple autocreate_acl entries could be specified) Ellie, do you think this is something of low complexity? *From:* Sebastian Hagedorn *Sent:* Friday, May 11, 2018 04:36 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: setting acl on autocreate folders So what I'm observing in practice is that the "-a" option is not enough to deliver plus+addressed mails without the "anyone p" ACL permission in the folder, which makes me think that the user for "-a" option is not from the admins group, though it probably should be, right? I.e. lmtpd -a should be delivering plus+addressed mails without the "anyone p" ACL permission? I think it's good that you have to explicitly set "anyone p", because otherwise people would be able to send plus-adressed mails to any mailbox whose name they can guess. -- Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 Regionales Rechenzentrum (RRZK) Universität zu Köln / Cologne University - Tel. +49-221-470-89578 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: setting acl on autocreate folders
> I think it's good that you have to explicitly set "anyone p", because otherwise people would be able to send plus+adressed mails to any mailbox whose name they can guess. As the default behavior, I agree. I've just made a couple of tests: remove "anyone p" then add "postman p" or add postman as "admins: postman" but none of these worked for plus+addressing (P+A), so the /postman/ user appears to be some hardcoded way of dealing with LMTP delivery and has nothing in common with the normal users and operations. If my assumptions are correct, I guess what Chen (OP) was asking would be useful, i.e. to be able to define "anyone p" (either as a toggle aimed at P+A or as a free-form for any user/ACL) for some auto-created folders along the other auto-configuration features (autocreate_XXX, x-list, etc.). The idea is to be able to setup most common settings for new users without any external scripts talking to cyradm or imtest. In my case the "anyone p" permission is the only thing pending. Something like: autocreate_acl(multiple autocreate_acl entries could be specified) Ellie, do you think this is something of low complexity? *From:* Sebastian Hagedorn *Sent:* Friday, May 11, 2018 04:36 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: setting acl on autocreate folders So what I'm observing in practice is that the "-a" option is not enough to deliver plus+addressed mails without the "anyone p" ACL permission in the folder, which makes me think that the user for "-a" option is not from the admins group, though it probably should be, right? I.e. lmtpd -a should be delivering plus+addressed mails without the "anyone p" ACL permission? I think it's good that you have to explicitly set "anyone p", because otherwise people would be able to send plus-adressed mails to any mailbox whose name they can guess. -- Sebastian Hagedorn - Weyertal 121, Zimmer 2.02 Regionales Rechenzentrum (RRZK) Universität zu Köln / Cologne University - Tel. +49-221-470-89578 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: setting acl on autocreate folders
Ellie, Thanks for checking. My doubt came from another documentation (https://www.cyrusimap.org/docs/cyrus-imapd/2.5.9/faq.php): plus addressing - Plus addressing allows direct delivery to a particular mailbox (other than an INBOX). This is done in two ways. The first way allows delivery to a subfolder of a specific user's INBOX. This is done via an address of the form: username+mailfolder@domain, which will deliver to the user's INBOX.mailfolder folder (or altnamespace equivalent). *This submailbox must allow the posting user the 'p' right (generally, this means 'anyone' must have the 'p' right), otherwise the message will just be filed into the user's INBOX.* So what I'm observing in practice is that the "-a" option is not enough to deliver plus+addressed mails without the "anyone p" ACL permission in the folder, which makes me think that the user for "-a" option is not from the admins group, though it probably should be, right? I.e. lmtpd -a should be delivering plus+addressed mails without the "anyone p" ACL permission? *From:* Ellie Timoney *Sent:* Friday, May 11, 2018 02:33 *To:* Info-cyrus *Subject:* Re: setting acl on autocreate folders Looks like "postman" from a skim of the source, and I believe this is the same user as when connecting via a UNIX socket: https://github.com/cyrusimap/cyrus-imapd/blob/15c812df6a020414a2e8863fe1afdfa3273a7bad/imap/lmtpengine.c#L993-L1005 But I would welcome correction from someone who knows, I'm just looking at the code. Cheers, ellie On Fri, May 11, 2018, at 3:20 PM, Anatoli wrote: Hi Ellie, Chen's question made me recheck the docs and now I have a doubt. Could you please clarify under what user the LMTP-delivered mails enters Cyrus when "-a" option is used over TCP with lmtpd (i.e. lmtp cmd="lmtpd -a" listen="127.0.0.1:2004")? The documentation (https://cyrusimap.org/imap/concepts/overview_and_concepts.html#local-mail-transfer-protocol-lmtp) says: For final delivery via /LMTP over a TCP socket, it is necessary to use LMTP AUTH/. This is accomplished using SASL to authenticate the delivering user. If your mail server is performing delivery via LMTP AUTH (that is, using a SASL mechanism), you will want their authentication id to be an LMTP admins (either via the admins imapd.conf option or via the _admins option, typically lmtp_admins). Alternatively you may deliver via /LMTP to a unix domain socket/, and /the connection will be preauthenticated as an administrative user/ (and access control is accomplished by controlling access to the socket). But it doesn't say anything about the "-a:/Preauthorize connections initiated on an internet socket/, instead of requiring LMTP AUTH." (https://www.cyrusimap.org/imap/reference/manpages/systemcommands/lmtpd.html#cmdoption-lmtpd-a). Thanks, Anatoli *From:* Ellie Timoney *Sent:* Friday, May 11, 2018 00:46 *To:* Info-cyrus *Subject:* Re: setting acl on autocreate folders Hi Chen, So, the question : is it possible to set specific ACLs on autocreated folders ? (i.e., ACLs, different from those defined by defaultacl in imapd.conf). I believe the autocreate mechanism has no particular knowledge of ACLs all all. It just uses the standard Cyrus policy for assigning them, with no way to override it. Cheers, ellie On Wed, May 9, 2018, at 6:37 PM, Chentao Credungtao via Info-cyrus wrote: Hello, This question has been asked twice before by different users, but no answer has ever be given. In 2012 :https://www.spinics.net/lists/info-cyrus/msg14612.html In 2016 :https://www.spinics.net/lists/info-cyrus/msg17385.html I guess the answer is NO, but just the same I thought i'd asked again to be sure. So, the question : is it possible to set specific ACLs on autocreated folders ? (i.e., ACLs, different from those defined by defaultacl in imapd.conf). Thanks, Chen Cyrus Home Page:http://www.cyrusimap.org/ List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page:http://www.cyrusimap.org/ List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Backup methods
Andrew, > For a small system with a few hundred mailboxes, a simple unix filesystem backup is sufficient. You can dump the Cyrus mailboxes.db to a flat file every hour with cron (keep a few days worth). Backup everything with your regular backup system (tar, rsync, etc). > If you suffer a complete loss of the system and have to restore from the backup, you won't care much about a few database file inconsistencies, which can be repaired with Cyrus' reconstruct tool. You would recover the whole backup, recover mailboxes.db from the most recent flat file export, and then run reconstruct on every mailbox. Yepp, this is how I was (and is) doing it (hourly), so if one backup has something unrecoverable, I can check a previous backup (-1hr) and luckily it'll be in a better shape. So on the one hand this is something that "works", yes. On the other, recently I've started using Cyrus xDAV functionality that permits to store files, calendars and contacts (BTW, some minor issues apart, it works great!). All this information, if inconsistent, is more difficult to deal with. It's more fragile than mails. Also, changes to this data are more important and happen with higher frequency (I have an accounting client where 4 users make a couple of hundreds of changes to a single xls file per day over Cyrus WebDAV). It's in pre-production state in my deployments right now, but I suspect that to bear some inconsistencies or restore a -1hr backup would not be an acceptable policy for this type of data. Regards, Anatoli *From:* Andrew Morgan *Sent:* Friday, May 11, 2018 02:05 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: Backup methods On Fri, 11 May 2018, Anatoli wrote: There may be an argument that could be made for 2 backup stratagies That's the point. In the context of SME environments (Small and Medium-sized Enterprises, i.e. from 5 to 50 employees normally, up to 250 in some countries) that we were talking about, a replication is an overkill, IMO. But for large enterprises like MNCs, large universities, public mail providers (Fastmail) of course multiple masters and backups via replication is the way to go. For large deployments there are good backup solutions in Cyrus, but for the small businesses admins I don't know any to recommend. Anatoli, I think you're making this harder than it needs to be... For a small system with a few hundred mailboxes, a simple unix filesystem backup is sufficient. You can dump the Cyrus mailboxes.db to a flat file every hour with cron (keep a few days worth). Backup everything with your regular backup system (tar, rsync, etc). If you suffer a complete loss of the system and have to restore from the backup, you won't care much about a few database file inconsistencies, which can be repaired with Cyrus' reconstruct tool. You would recover the whole backup, recover mailboxes.db from the most recent flat file export, and then run reconstruct on every mailbox. If you need to recover some messages or mailboxes that were deleted by a user, then just recover those individual files or directories from you backup. Run reconstruct -rf on the mailbox. Naturally, delayed expunge and delayed delete are fantastic ways to avoid all this work. Purge them only after a few weeks or a month has passed. It is much easier to restore using those delayed delete/expunge features. Thanks, Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: setting acl on autocreate folders
Hi Ellie, Chen's question made me recheck the docs and now I have a doubt. Could you please clarify under what user the LMTP-delivered mails enters Cyrus when "-a" option is used over TCP with lmtpd (i.e. lmtp cmd="lmtpd -a" listen="127.0.0.1:2004")? The documentation (https://cyrusimap.org/imap/concepts/overview_and_concepts.html#local-mail-transfer-protocol-lmtp) says: For final delivery via /LMTP over a TCP socket, it is necessary to use LMTP AUTH/. This is accomplished using SASL to authenticate the delivering user. If your mail server is performing delivery via LMTP AUTH (that is, using a SASL mechanism), you will want their authentication id to be an LMTP admins (either via the admins imapd.conf option or via the _admins option, typically lmtp_admins). Alternatively you may deliver via /LMTP to a unix domain socket/, and /the connection will be preauthenticated as an administrative user/ (and access control is accomplished by controlling access to the socket). But it doesn't say anything about the "-a: /Preauthorize connections initiated on an internet socket/, instead of requiring LMTP AUTH." (https://www.cyrusimap.org/imap/reference/manpages/systemcommands/lmtpd.html#cmdoption-lmtpd-a). Thanks, Anatoli *From:* Ellie Timoney *Sent:* Friday, May 11, 2018 00:46 *To:* Info-cyrus *Subject:* Re: setting acl on autocreate folders Hi Chen, So, the question : is it possible to set specific ACLs on autocreated folders ? (i.e., ACLs, different from those defined by defaultacl in imapd.conf). I believe the autocreate mechanism has no particular knowledge of ACLs all all. It just uses the standard Cyrus policy for assigning them, with no way to override it. Cheers, ellie On Wed, May 9, 2018, at 6:37 PM, Chentao Credungtao via Info-cyrus wrote: Hello, This question has been asked twice before by different users, but no answer has ever be given. In 2012 : https://www.spinics.net/lists/info-cyrus/msg14612.html In 2016 : https://www.spinics.net/lists/info-cyrus/msg17385.html I guess the answer is NO, but just the same I thought i'd asked again to be sure. So, the question : is it possible to set specific ACLs on autocreated folders ? (i.e., ACLs, different from those defined by defaultacl in imapd.conf). Thanks, Chen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Backup methods
> There may be an argument that could be made for 2 backup stratagies That's the point. In the context of SME environments (Small and Medium-sized Enterprises, i.e. from 5 to 50 employees normally, up to 250 in some countries) that we were talking about, a replication is an overkill, IMO. But for large enterprises like MNCs, large universities, public mail providers (Fastmail) of course multiple masters and backups via replication is the way to go. For large deployments there are good backup solutions in Cyrus, but for the small businesses admins I don't know any to recommend. > If the mailboxes are on something like EXT4 you can do an LVM snapshot bacause of the built in auto checkpointing and for something like xfs there is freeze. Yepp, this is the idea. But the data on disk should be in a consistent state. Something like "FLUSH TABLES WITH READ LOCK" is what is needed actually, i.e. to consistently write to disk the data from the current instance and lock. > The trouble is that read operations can alter a files state so it may not be just a simple matter of a write lock. If you mean things like SEEN state that are changed when the user reads something, the implementation v1 could block it, v2 could allow it by queuing in memory such state-change pending operations. Anyway these are just implementation details that don't change the general logic and, taking into account the supposed duration of the lock, IMO don't even matter much. The significant work that blocks this feature is the global lock across the entire Cyrus. Once it's implemented, it would be much easier to introduced specific improvements here and there. > Cyrus has multiple databases that would also need to be frozen and flushed before the snapshot is taken. > If you spread your mailstorage and metadata storage over multiple file systems trying to co-ordinate snapshots becomes more complex. Sure, Cyrus would have to lock all the databases and where to store them would be up to the admin. Actually, fs snapshots is just the most obvious use-case. There could be others... up to the admin to decide. *From:* Alvin Starr *Sent:* Thursday, May 10, 2018 23:55 *To:* Info-cyrus *Subject:* Re: Backup methods On 05/10/2018 06:29 PM, Anatoli wrote: Actually, mysqldump performs a lock on the records it's dumping. If its for a MyISAM db, the entire table is locked. If it's for InnoDB and similar, an internal snapshot is created and only the records the dump is reading are unavailable for writing. Mysql provides table consistency by locking but that is only table consistency. Multiple updates across multiple tables could easily result in an inconsistent database even though the tables are individually consistent. With some tables that take hours to dump locking the tables is problematic. This is why some people use LVM snapshots combined with "FLUSH TABLES WITH READ LOCK". Cyrus could also implement a per-user lock, but in reality it doesn't need that complex syncro mechanisms, a simple global write lock would be enough (reading would not be affected, son only I, not I/O, and not to stop it but just to suspend). After all, the *write* lock would last only a second or so, the fs snapshots are almost instantaneous. If you can't tolerate a 1 second delay for writing in Cyrus, you are probably not a SME. The trouble is that read operations can alter a files state so it may not be just a simple matter of a write lock. If the mailboxes are on something like EXT4 you can do an LVM snapshot bacause of the built in auto checkpointing and for something like xfs there is freeze. Cyrus has multiple databases that would also need to be frozen and flushed before the snapshot is taken. If you spread your mailstorage and metadata storage over multiple file systems trying to co-ordinate snapshots becomes more complex. And you don't need to hold the data to transfer it. You can dump it directly to a nfs share or pass it as stdout to ssh: mysqldump | xz -9 | ssh remote_server "cat > /bkp/`date +%y%m%d_%H%M`.sql.xz". With a couple of pipes more you can encrypt the data on the fly so it's secure to store it in a cheap VPS overseas... or you could upload it to dropbox. There may be an argument that could be made for 2 backup stratagies. 1) where the mailstoreage and metadata can exist on a single volume and flushing the various databases is a short duration event. Then an LVM snapshot could be used 2) for distribted large scale mail systems where only an online live backup system can be used. Backup for 100 users has different requirements than backup for 10 users so why not support a few different backup strategies. *From:* Jason L Tibbitts Iii *Sent:* Thursday, May 10, 2018 18:41 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: Backup methods "A" == Anatoli<m...@anatoli.ws> writes: A> What about mysqldump >
Re: Backup methods
Actually, mysqldump performs a lock on the records it's dumping. If its for a MyISAM db, the entire table is locked. If it's for InnoDB and similar, an internal snapshot is created and only the records the dump is reading are unavailable for writing. Cyrus could also implement a per-user lock, but in reality it doesn't need that complex syncro mechanisms, a simple global write lock would be enough (reading would not be affected, son only I, not I/O, and not to stop it but just to suspend). After all, the *write* lock would last only a second or so, the fs snapshots are almost instantaneous. If you can't tolerate a 1 second delay for writing in Cyrus, you are probably not a SME. And you don't need to hold the data to transfer it. You can dump it directly to a nfs share or pass it as stdout to ssh: mysqldump | xz -9 | ssh remote_server "cat > /bkp/`date +%y%m%d_%H%M`.sql.xz". With a couple of pipes more you can encrypt the data on the fly so it's secure to store it in a cheap VPS overseas... or you could upload it to dropbox. *From:* Jason L Tibbitts Iii *Sent:* Thursday, May 10, 2018 18:41 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: Backup methods "A" == Anatoli <m...@anatoli.ws> writes: A> What about mysqldump > dump.sql, then mysql < dump.sql? Also a wrong A> way and didn't have to be implemented? No, that's exactly my point. Thanks for making it for me! The analog to the way you indicated that you would like it to work would be having the mysql server stop IO so that you can take a filesystem snapshot while the database is in a consistent state. But instead, the database (like cyrus) implements a backup method which you can use to extract the data. And it also requires disk space to hold the backup until you can transfer it to your backup medium. - J< Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Backup methods
> Well, sort of. It is a method that is actually focused around doing backups. It happens to make use of the replication protocol because that is actually the smart way to do it. I did detail the differences in my message. I suggest you try to use it in your deployments and then share with us your real-world experience, like how reliable it is, how well the compression works, how easy it is to recover something if both master and the backup instances become unaccessible (disk failure in both or both servers are stolen (this is a SME office, not a tier 4 datacenter) and the backups from an external location should be brought in), what data is missing (if at all) after a backup recovery, how incremental backups are done, etc. I tried it in a /real deployment/ a year ago when it was just released and my conclusion was that it was not well-suited for a SME environment (at least at that moment). > Honestly I believe that's the wrong way to go about it What about mysqldump > dump.sql, then mysql < dump.sql? Also a wrong way and didn't have to be implemented? I bet this is the most deployed method for DB backups in the real SME world (like cron mysqldump --routines --all-databases | xz -9 > /bu/`date +%y%m%d_%H%M`_full.sql.xz), though there are replication solutions available too. The Unix way is about minimalist, modular software. *From:* Jason L Tibbitts Iii *Sent:* Thursday, May 10, 2018 16:38 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: Backup methods "A" == Anatoli <m...@anatoli.ws> writes: A> What you mention is highly related to the replication backup A> we were talking about in the previous mails. Well, sort of. It is a method that is actually focused around doing backups. It happens to make use of the replication protocol because that is actually the smart way to do it. I did detail the differences in my message. A> In both cases, a copy of the master data is made, which requires A> twice the space of real usage (Cyrus Backups tries to apply A> compression on stored data, not sure how well it works). As I mentioned, the documentation discusses this. A> What is really needed, IMO, for SME environments is the ability for A> Cyrus to sync to disk all data, so one can take a hot copy of that A> data with standard UNIX tools and then handle it accordingly. Once a A> recovery is needed, one just copies a backup to the Cyrus dir and A> starts the service. Honestly I believe that's the wrong way to go about it, but it's certainly one way to do things if you have no backup solution integrated into the software. But hey, it's your data. I only wanted to mention that there really is an existing backup solution which wasn't being discussed. - J< Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Backup methods
> For me, if I put a replica in place it's get the role of backup. Meanning I will put two replica and do not make another backup. A replica is not a replacement for a backup. You may have your specific needs, but replica per-se mostly serves to cover for master's hardware failures. You are not protected with a replica from accidental or intentional deletions/changes of the data. If a user deletes some of his/her mails and discovers it after the expunge period, you won't be able to recover them as replica would also have them deleted. > Using ZFS, do no need to do that Sure, if you're using ZFS :) The solution I've described serves for any *nix OS and fs. > So if I stop the postfix on the cyrus_server You just don't need to stop it. If you expect to stop Cyrus frequently, just configure the cyrus_sever Postfix retry interval to something like 1 min. *From:* Albert Shih *Sent:* Thursday, May 10, 2018 17:32 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: Backup methods Le 10/05/2018 à 10:38:28-0300, Anatoli a écrit Not very sure to understand that. It's always true isn't ? If you have XTo of data and you want n backups you will need X*(n+1) To ? The replication as it is designed means that you create an additional (replica) instance of Cyrus that will be in sync with the master instance, so when you need to make a backup, you turn of the replica, take a backup from its data, then turn it on again so it comes in sync with the master. In this case there's no interruption to the service, you just stop a replica. But the replica will use the same amount of space as your master, so without even making a backup, you'll use 2x space. + you have to understand how the replication works, then set it up, control that the sync process is always working and the replica has the same information as the master... That's a great solution for ISP-level or public mail service operations, but IMO an absolute overkill for small deployments. For me, if I put a replica in place it's get the role of backup. Meanning I will put two replica and do not make another backup. When it comes to making a backup, the best policy IMO is to make incremental backups. In this case you only store the new mails + binary indexes. Once in a while (e.g. every month) you make a full backup, then, say, once a week a level 1 backup (that stores changes from the previous week, reset at lower level backup, i.e. every month), then daily level 2 backups and hourly level 3. This way you can restore up to hourly changes without using excessive amount of space. Of course you can compress them too (xz -9 gives a pretty good ratio). Using ZFS, do no need to do that. Just use zfs snapshot and he going to keep the differential at block level (much better than file level). Same as compression. Just need to activate compression on the dataset. Uhh don't do that. Your Postfix has no problem in retaining mails if Cyrus is not reachable, then attempt their delivery again. I was referring to that, depending on the configuration of your incoming MTA, the next delivery attempt may be in, say, 15 minutes, so you postpone incoming mail for that time if you turn off Cyrus to take a backup. If you turn off your incoming MTA, the source MTA may have issues with delivery at all (you don't control it, you don't know how it's configured, when the next delivery attempt will occur, etc.), never turn off your incoming MTA. Don't be a problem, I've got 2 public incoming MTA, 4 privates and the postfix on the cyrus-server. So incoming mail, let's say gmail.com going from gmail.com_MX to our MX, then send to cyrus-server. So if I stop the postfix on the cyrus_server, the incoming mail going to stay on the our MX. -- Albert SHIH DIO bâtiment 15 Observatoire de Paris xmpp: j...@obspm.fr Heure local/Local time: Thu May 10 22:27:22 CEST 2018 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Backup methods
Jason, What you mention is highly related to the replication backup we were talking about in the previous mails. The idea is the same, they replicate from master. Then, in a pure replica solution, the replica is stopped, a copy of its files (in the native format) is made and it's started again to continue replication. The Cyrus Backups also replicates all data from master, but stores it in a different format and has some additional functionality to assist in backing up the files, but AFAIK it's not yet complete, e.g. I'm not sure it's possible to make incremental backups with it, not sure about the SEEN state, the recovery process appears not trivial, etc. In both cases, a copy of the master data is made, which requires twice the space of real usage (Cyrus Backups tries to apply compression on stored data, not sure how well it works). What is really needed, IMO, for SME environments is the ability for Cyrus to sync to disk all data, so one can take a hot copy of that data with standard UNIX tools and then handle it accordingly. Once a recovery is needed, one just copies a backup to the Cyrus dir and starts the service. The data would be in the exact same state as when the backup took place. This is discussed in the github issue mentioned in the previous mail. *From:* Jason L Tibbitts Iii *Sent:* Thursday, May 10, 2018 14:10 *To:* Arnaldo Viegas De Lima *Cc:* Info-cyrus *Subject:* Re: Backup methods Cyrus does have an integrated backup system (see https://cyrusimap.org/imap/reference/admin/backups.html) which I'm not sure has been mentioned in this thread. But you still have to have enough space to keep the compressed backups on disk in order to move them to tape or whatever archival storage you're using. There is discussion of the storage requirements in the documentation. I don't think any of it is particularly unreasonable, but I haven't actually tried it myself. Technically I don't think you need a separate machine (though that's simpler); it may just be possible to have a second cyrus server listening on different ports to act as the replication target. I probably wouldn't do it that way anyway; old hardware with some cheap disk would suffice to stage the backups until they're sent to tape or wherever. As for it all being marked "experimental", I'm sure that if bugs were found (and reported), they would be fixed. It probably just needs more testing and back and forth with the devs to flesh out the documentation and add any missing functionality. - J< Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Backup methods
> Not very sure to understand that. It's always true isn't ? If you have XTo of data and you want n backups you will need X*(n+1) To ? The replication as it is designed means that you create an additional (replica) instance of Cyrus that will be in sync with the master instance, so when you need to make a backup, you turn of the replica, take a backup from /its data/, then turn it on again so it comes in sync with the master. In this case there's no interruption to the service, you just stop a replica. But the replica will use the same amount of space as your master, so without even making a backup, you'll use 2x space. + you have to understand how the replication works, then set it up, control that the sync process is always working and the replica has the same information as the master... That's a great solution for ISP-level or public mail service operations, but IMO an absolute overkill for small deployments. > I don't see how you can avoid that, of course you can activate heavy compression on the backup but beside of that When it comes to making a backup, the best policy IMO is to make incremental backups. In this case you only store the new mails + binary indexes. Once in a while (e.g. every month) you make a full backup, then, say, once a week a level 1 backup (that stores changes from the previous week, reset at lower level backup, i.e. every month), then daily level 2 backups and hourly level 3. This way you can restore up to hourly changes without using excessive amount of space. Of course you can compress them too (xz -9 gives a pretty good ratio). > Well that's is easy to avoid, you just have to stop postfix before stopping the VM, when postfix is stop all incoming messages will stay on the parent smtp server, so no loosing incoming mail. Uhh don't do that. Your Postfix has no problem in retaining mails if Cyrus is not reachable, then attempt their delivery again. I was referring to that, depending on the configuration of your incoming MTA, the next delivery attempt may be in, say, 15 minutes, so you postpone incoming mail for that time if you turn off Cyrus to take a backup. If you turn off your /incoming MTA/, the source MTA may have issues with delivery at all (you don't control it, you don't know how it's configured, when the next delivery attempt will occur, etc.), never turn off your incoming MTA. *From:* Albert Shih *Sent:* Thursday, May 10, 2018 04:14 *To:* Anatoli *Cc:* Info-cyrus *Subject:* Re: Backup methods Le 10/05/2018 à 02:44:18-0300, Anatoli a écrit Hi, The replication is reasonable only if you have more than one server in your deployment (and both servers with the same level of security, if not you risk to compromise the user data) or "spool size/available disk space" is low, otherwise you'd need to dedicate 2 times more space than needed to store user data, only to take a periodic backup (+ the space needed to store the backup itself). Not very sure to understand that. It's always true isn't ? If you have XTo of data and you want n backups you will need X*(n+1) To ? I don't see how you can avoid that, of course you can activate heavy compression on the backup but beside of that I suggest you take a look at this issue: https://github.com/cyrusimap/ cyrus-imapd/issues/1763, where backups for small deployments were already Thanks for the link I will read that. Answering the OP's question, I'm using Cyrus for 4 years now and I don't know about any reliable and reasonable strategy for backups of Cyrus data in SME environments. Summing it up: • FS snapshots without stopping the server: a possibility of a corrupted backup. • FS snapshots after stopping the server: service downtime, breaking open connections, delivery issues for incoming MTAs, etc. - reasonable for daily Well that's is easy to avoid, you just have to stop postfix before stopping the VM, when postfix is stop all incoming messages will stay on the parent smtp server, so no loosing incoming mail. backups in a 8/5 office, unreasonable for 24/7 deployments (e.g. users distributed in different time zones) or for intra-day backups. I check, stopping postfix, stopping the VM, take a snapshot, starting the VM, take about 10-15 secondes. So I agree with you it's not a very good solution because user still can loose the connection, but I think without replication it's acceptable. • Replication: unreasonable requirements for disk space, setup overkill. For the setup the overkill is for me a small price vs loosing dataand as for the disk space that's not a issue at all for me. Currently I run dovecot and have 2 backups, so when I say to my boss « we got X To of mail » I already got 3 * X To of disk. Say in other way, if I can afford X To, I will say I can give you X/3 To of mail. Regards. -- Albert SHIH DIO bâtiment 15 Observatoire de Paris xmpp: j...@obspm.fr Heure local/Local time: Thu May 10
Re: Backup methods
Hi, On larger systems with VMs i take a ZFS or LVM snapshot and mount it externally to "fetch" a full (incremental) filesystem backup of the mail spool and imap spool and cyrus db on a daily base. After the backup run i destroy the snapshot. The problem with this is that you can't be sure the data on disk is in sync. Depending on how heavy the load is during the "backup" (+ your luck), you may find unpleasant surprises when you have to restore it. And you'd only know that the backup is corrupted when you try to restore it. Beside this and depending from your needs you may take a look at cyrus replication features to build a "backup" or just use standard filesystem backup tools like tar, dumpfs etc. The replication is reasonable only if you have more than one server in your deployment (and both servers with the same level of security, if not you risk to compromise the user data) or "spool size/available disk space" is low, otherwise you'd need to dedicate 2 times more space than needed to store user data, only to take a periodic backup (+ the space needed to store the backup itself). I suggest you take a look at this issue: https://github.com/cyrusimap/cyrus-imapd/issues/1763, where backups for small deployments were already discussed in detail. Though, no idea if there are plans to implement it. Answering the OP's question, I'm using Cyrus for 4 years now and I don't know about any reliable and reasonable strategy for backups of Cyrus data in SME environments. Summing it up: * FS snapshots without stopping the server: a possibility of a corrupted backup. * FS snapshots after stopping the server: service downtime, breaking open connections, delivery issues for incoming MTAs, etc. - reasonable for daily backups in a 8/5 office, unreasonable for 24/7 deployments (e.g. users distributed in different time zones) or for intra-day backups. * Replication: unreasonable requirements for disk space, setup overkill. Regards, Anatoli *From:* Niels Dettenbach Via Info-cyrus *Sent:* Wednesday, May 09, 2018 06:42 *To:* Info-cyrus *Subject:* Re: Backup methods Am Mittwoch, 9. Mai 2018, 11:19:54 CEST schrieb Albert Shih: I would like to know what's kind of backup method are recommended for cyrus-imapd. My cyrus-imapd host (only one currently) are running under FreeBSD jail (something like systemd-nspawn, lxc) & ZFS so I'm intend to use this method : stop the vm take a zfs snapshot start the vm send the zfs snapshot on a backup server. This is relatively inefficient, but a working option if anything from cyrus data is on that VM - i.e. the complete mail spool and the database files (possibly plus sieve files). We do similiar on relatively small systems or to get "intraday backups" only. On larger systems with VMs i take a ZFS or LVM snapshot and mount it externally to "fetch" a full (incremental) filesystem backup of the mail spool and imap spool and cyrus db on a daily base. After the backup run i destroy the snapshot. Beside this and depending from your needs you may take a look at cyrus replication features to build a "backup" or just use standard filesystem backup tools like tar, dumpfs etc. On a file base you have to backup the mail spool and the cyrus database files. If you use SIEVE, backup the SIEVE file pool too. You can restore by just replacing the files and start cyrus. To get the common database files "interoperable" it may makes sense to dump then into a machine independent format for backup if they are in a machine dependent format. If your restore such a filesystem based backup to a new system which has other hardware / arch specs or newer / incompatible DB subsystem (instead of skiplist) you may have to "recreate" indizes and database data. reconstruct - f may be your friend to "clean" up the transfer / restore. There are several strategies for backup cyrus - this are just a few. hth a bit. good luck, Niels. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xDAV and shared namespaces
Hi Ken, Below goes the telemetry for the relevant part. What I see here is that Cyrus correctly answers to the "principal?" question and then the iOS asks for calendars of the principal (telemetry included in the attached file) and there Cyrus only returns the personal calendars, but IMO the shared folder should be somehow taken into account, either responding differently to the "principal?" question or including the shared calendar when queried for available calendars (IMO the best solution (no need to create separate accounts on the phone with different URLs). Please let me know if you need additional details. Regards, Anatoli -- t...@domain.com Sun Mar 25 06:05:36 2018 <1521968736<*PROPFIND* */dav/calendars/shared/* HTTP/1.1 Accept: */* Content-type: text/xml Connection: keep-alive Content-length: 181 Host: mail.domain.com User-agent: iOS/11.2.6 (15D100) accountsd/1.0 Prefer: return=minimal Depth: 0 Brief: t Accept-language: en-us Authorization: Basic ... Accept-encoding: br, gzip, deflate <1521968736< >1521968736>HTTP/1.1 207 Multi-Status Date: Sun, 25 Mar 2018 09:05:36 GMT Strict-Transport-Security: max-age=600 Vary: Accept-Encoding, Brief, Prefer Preference-Applied: return=minimal Content-Type: application/xml; charset=utf-8 Content-Length: 546 */dav/calendars/shared/* */dav/principals/user/t...@domain.com/* HTTP/1.1 200 OK <15219687361521968736>HTTP/1.1 200 OK Date: Sun, 25 Mar 2018 09:05:36 GMT Strict-Transport-Security: max-age=600 Cache-Control: no-cache Link: ; rel="server-info"; token="80769c2c66d340ecd178710db26d56b9c4699e3e" DAV: 1, 2, 3, access-control, extended-mkcol, resource-sharing DAV: calendar-access, calendar-auto-schedule DAV: calendar-query-extended, calendar-availability, calendar-managed-attachments DAV: calendarserver-sharing, inbox-availability DAV: addressbook Allow: OPTIONS, GET, HEAD Allow: PROPFIND, REPORT, COPY Content-Length: 0 *From:* Ken Murchison *Sent:* Monday, March 19, 2018 09:24 *To:* Info-cyrus *Subject:* Re: xDAV and shared namespaces Can you capture telemetry for the PROPFIND and OPTIONS requests on the shared calendar? On 03/19/2018 02:55 AM, Anatoli wrote: Hi Ken, Basically, the shared calendar is working fine in Thunderbird, where I can specify the exact URL. But I can't make it work on iPhone as it only takes the server and later resolves the exact URL via /.well-known/caldav which points to the personal calendar for the same user. Though, if I specify on iPhone the exact URL in the settings, it queries it: Mar 19 03:27:42 l https[18199]: [10.1.1.100] with "iOS/11.2.6 (15D100) dataaccessd/1.0"; "PROPFIND /dav/calendars/shared/ HTTP/1.1" (depth=0) => "HTTP/1.1 401 Unauthorized" (error=Must authenticate to access the specified target) Mar 19 03:27:42 l https[18199]: login: [10.1.1.100] t...@xxx.com Basic+TLS User logged in SESSIONID= Mar 19 03:27:42 l https[18199]: [10.1.1.100] as "t...@xxx.com" with "iOS/11.2.6 (15D100) dataaccessd/1.0"; "PROPFIND /dav/calendars/shared/ HTTP/1.1" (type=prop; depth=0) => "HTTP/1.1 207 Multi-Status" Mar 19 03:27:42 l https[18199]: login: [10.1.1.100] t...@xxx.com Basic+TLS User logged in SESSIONID= Mar 19 03:27:42 l https[18199]: [10.1.1.100] as "t...@xxx.com" with "iOS/11.2.6 (15D100) dataaccessd/1.0"; "OPTIONS /dav/calendars/shared/ HTTP/1.1" => "HTTP/1.1 200 OK" But finally, iPhone Calendar app doesn't show events from the shared calendar and in the Accounts the URL resets to the naked domain (i.e. without the /dav/calendars/shared/ part). What could be done to make it work? Thanks, Anatoli *From:* Ken Murchison *Sent:* Tuesday, March 13, 2018 14:09 *To:* Info-cyrus *Subject:* Re: xDAV and shared namespaces On 03/13/2018 12:50 PM, Anatoli wrote: Ken, Thanks a lot! After creating the shared folder under the root hierarchy (with imtest -a admin and the command you provided), setting the corresponding ACLs for the users that have to have access to the folder (with cyradm sam) and creating a new connection in each WebDAV client for https://domain.com/dav/drive/shared/, everything worked fine! I'll try to test the same with CalDAV / CardDAV and report it here. Its entirely possible that the URL parsing code for CalDAV and CardDAV will not resolve to a shared folder. Let me know. Regards, Anatoli *From:* Ken Murchison *Sent:* Tuesday, March 13, 2018 12:55 *To:* Info-cyrus *Subject:* Re: xDAV and shared namespaces On 03/13/2018 11:48 AM, Anatoli wrote: Hi Ken, Thanks for your quick reply. Yes, I'm willing to test it and if needed I can also apply patches (I build Cyrus from sources). > x CREATE shared (TYPE COLLECT
Re: xDAV and shared namespaces
Hi Ken,
Basically, the shared calendar is working fine in Thunderbird, where I
can specify the exact URL. But I can't make it work on iPhone as it only
takes the server and later resolves the exact URL via
/.well-known/caldav which points to the personal calendar for the same
user. Though, if I specify on iPhone the exact URL in the settings, it
queries it:
Mar 19 03:27:42 l https[18199]: [10.1.1.100] with "iOS/11.2.6 (15D100)
dataaccessd/1.0"; "PROPFIND /dav/calendars/shared/ HTTP/1.1" (depth=0)
=> "HTTP/1.1 401 Unauthorized" (error=Must authenticate to access the
specified target)
Mar 19 03:27:42 l https[18199]: login: [10.1.1.100] t...@xxx.com Basic+TLS
User logged in SESSIONID=
Mar 19 03:27:42 l https[18199]: [10.1.1.100] as "t...@xxx.com" with
"iOS/11.2.6 (15D100) dataaccessd/1.0"; "PROPFIND /dav/calendars/shared/
HTTP/1.1" (type=prop; depth=0) => "HTTP/1.1 207 Multi-Status"
Mar 19 03:27:42 l https[18199]: login: [10.1.1.100] t...@xxx.com Basic+TLS
User logged in SESSIONID=
Mar 19 03:27:42 l https[18199]: [10.1.1.100] as "t...@xxx.com" with
"iOS/11.2.6 (15D100) dataaccessd/1.0"; "OPTIONS /dav/calendars/shared/
HTTP/1.1" => "HTTP/1.1 200 OK"
But finally, iPhone Calendar app doesn't show events from the shared
calendar and in the Accounts the URL resets to the naked domain (i.e.
without the /dav/calendars/shared/ part). What could be done to make it
work?
Thanks,
Anatoli
*From:* Ken Murchison
*Sent:* Tuesday, March 13, 2018 14:09
*To:* Info-cyrus
*Subject:* Re: xDAV and shared namespaces
On 03/13/2018 12:50 PM, Anatoli wrote:
Ken,
Thanks a lot! After creating the shared folder under the root
hierarchy (with imtest -a admin and the command you provided), setting
the corresponding ACLs for the users that have to have access to the
folder (with cyradm sam) and creating a new connection in each WebDAV
client for https://domain.com/dav/drive/shared/, everything worked fine!
I'll try to test the same with CalDAV / CardDAV and report it here.
Its entirely possible that the URL parsing code for CalDAV and CardDAV
will not resolve to a shared folder. Let me know.
Regards,
Anatoli
*From:* Ken Murchison
*Sent:* Tuesday, March 13, 2018 12:55
*To:* Info-cyrus
*Subject:* Re: xDAV and shared namespaces
On 03/13/2018 11:48 AM, Anatoli wrote:
Hi Ken,
Thanks for your quick reply. Yes, I'm willing to test it and if
needed I can also apply patches (I build Cyrus from sources).
> x CREATE shared (TYPE COLLECTION)
What does this command is supposed to do and under what user should I
execute it (I mean imtest -a )? I tried to run it under admin
user (it creates a "shared" folder at the root hierarchy) and under a
normal user that already has a shared folder (in this case the folder
is created under the user hierarchy), but in both cases the folder
stays empty (with only cyrus.header and cyrus.index inside) and is
not visible under a WebDAV client. Am I missing something?
Definitely do this as an admin (e.g. 'cyrus') that DOES NOT have their
own user hierarchy.
Its been so long since I wrote and played with the WebDAV module that
I forgot how it works. Try:
x CREATE #drive/shared (TYPE COLLECTION)
Use whatever you have set for the davdriveprefix option and the proper
hierarchy ('.' if you have disabled unixhierarchysep)
Regards,
Anatoli
*From:* Ken Murchison
*Sent:* Tuesday, March 13, 2018 12:18
*To:* Info-cyrus
*Subject:* Re: xDAV and shared namespaces
xDAV hasn't had much development or testing in the shared namespace
and there isn't any tooling to create such mailboxes.
If you wanted to do some testing, you'd have to create the mailboxes
by hand and set the mailbox type accordingly. To do so, you can
connect to the server as an admin using imtest and enter the
following command:
x CREATE shared (TYPE COLLECTION)
to create a WebDAV collection. Or set the TYPE to CALENDAR or
ADDRESSBOOK
On 03/13/2018 11:08 AM, Anatoli wrote:
Hi All,
I'm experimenting with xDav (CalDAV, CardDAV and WebDAV) in Cyrus
3.0.5. All xDAV services in user namespaces work as expected, but I
can't figure out how to setup them for shared namespaces.
The most important service for me to setup for shared namespaces is
WebDAV. I'd like to have a shared folder accessible to multiple
users based on their ACLs. I know how to setup a shared folder for IMAP:
cyradm> cm shar...@domain.com
cyradm> sam shar...@domain.com us...@domain.com write
In the mail client I see the shared1 folder under "shared folders".
But I can't see it in a WebDAV client. At the server the shared1
folder doesn't have the autocreated xDAV structure as it normally
autocreates for individual users, and I don't know how to create it
manually.
Does anybody know how to configure WebDAV shared folders? I'm also
interested in creating sha
Re: xDAV and shared namespaces
Ken,
Thanks a lot! After creating the shared folder under the root hierarchy
(with imtest -a admin and the command you provided), setting the
corresponding ACLs for the users that have to have access to the folder
(with cyradm sam) and creating a new connection in each WebDAV client
for https://domain.com/dav/drive/shared/, everything worked fine!
I'll try to test the same with CalDAV / CardDAV and report it here.
Regards,
Anatoli
*From:* Ken Murchison
*Sent:* Tuesday, March 13, 2018 12:55
*To:* Info-cyrus
*Subject:* Re: xDAV and shared namespaces
On 03/13/2018 11:48 AM, Anatoli wrote:
Hi Ken,
Thanks for your quick reply. Yes, I'm willing to test it and if needed
I can also apply patches (I build Cyrus from sources).
> x CREATE shared (TYPE COLLECTION)
What does this command is supposed to do and under what user should I
execute it (I mean imtest -a )? I tried to run it under admin
user (it creates a "shared" folder at the root hierarchy) and under a
normal user that already has a shared folder (in this case the folder
is created under the user hierarchy), but in both cases the folder
stays empty (with only cyrus.header and cyrus.index inside) and is not
visible under a WebDAV client. Am I missing something?
Definitely do this as an admin (e.g. 'cyrus') that DOES NOT have their
own user hierarchy.
Its been so long since I wrote and played with the WebDAV module that I
forgot how it works. Try:
x CREATE #drive/shared (TYPE COLLECTION)
Use whatever you have set for the davdriveprefix option and the proper
hierarchy ('.' if you have disabled unixhierarchysep)
Regards,
Anatoli
*From:* Ken Murchison
*Sent:* Tuesday, March 13, 2018 12:18
*To:* Info-cyrus
*Subject:* Re: xDAV and shared namespaces
xDAV hasn't had much development or testing in the shared namespace
and there isn't any tooling to create such mailboxes.
If you wanted to do some testing, you'd have to create the mailboxes
by hand and set the mailbox type accordingly. To do so, you can
connect to the server as an admin using imtest and enter the following
command:
x CREATE shared (TYPE COLLECTION)
to create a WebDAV collection. Or set the TYPE to CALENDAR or ADDRESSBOOK
On 03/13/2018 11:08 AM, Anatoli wrote:
Hi All,
I'm experimenting with xDav (CalDAV, CardDAV and WebDAV) in Cyrus
3.0.5. All xDAV services in user namespaces work as expected, but I
can't figure out how to setup them for shared namespaces.
The most important service for me to setup for shared namespaces is
WebDAV. I'd like to have a shared folder accessible to multiple users
based on their ACLs. I know how to setup a shared folder for IMAP:
cyradm> cm shar...@domain.com
cyradm> sam shar...@domain.com us...@domain.com write
In the mail client I see the shared1 folder under "shared folders".
But I can't see it in a WebDAV client. At the server the shared1
folder doesn't have the autocreated xDAV structure as it normally
autocreates for individual users, and I don't know how to create it
manually.
Does anybody know how to configure WebDAV shared folders? I'm also
interested in creating shared calendars (CalDAV) and addressbooks
(CardDAV), but I suppose they are managed the same way as WebDAV.
Thanks in advance,
Anatoli
Cyrus Home Page:http://www.cyrusimap.org/
List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Ken Murchison
Cyrus Development Team
FastMail US LLC
Cyrus Home Page:http://www.cyrusimap.org/
List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus Home Page:http://www.cyrusimap.org/
List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Ken Murchison
Cyrus Development Team
FastMail US LLC
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xDAV and shared namespaces
Hi Ken, Thanks for your quick reply. Yes, I'm willing to test it and if needed I can also apply patches (I build Cyrus from sources). > x CREATE shared (TYPE COLLECTION) What does this command is supposed to do and under what user should I execute it (I mean imtest -a )? I tried to run it under admin user (it creates a "shared" folder at the root hierarchy) and under a normal user that already has a shared folder (in this case the folder is created under the user hierarchy), but in both cases the folder stays empty (with only cyrus.header and cyrus.index inside) and is not visible under a WebDAV client. Am I missing something? Regards, Anatoli *From:* Ken Murchison *Sent:* Tuesday, March 13, 2018 12:18 *To:* Info-cyrus *Subject:* Re: xDAV and shared namespaces xDAV hasn't had much development or testing in the shared namespace and there isn't any tooling to create such mailboxes. If you wanted to do some testing, you'd have to create the mailboxes by hand and set the mailbox type accordingly. To do so, you can connect to the server as an admin using imtest and enter the following command: x CREATE shared (TYPE COLLECTION) to create a WebDAV collection. Or set the TYPE to CALENDAR or ADDRESSBOOK On 03/13/2018 11:08 AM, Anatoli wrote: Hi All, I'm experimenting with xDav (CalDAV, CardDAV and WebDAV) in Cyrus 3.0.5. All xDAV services in user namespaces work as expected, but I can't figure out how to setup them for shared namespaces. The most important service for me to setup for shared namespaces is WebDAV. I'd like to have a shared folder accessible to multiple users based on their ACLs. I know how to setup a shared folder for IMAP: cyradm> cm shar...@domain.com cyradm> sam shar...@domain.com us...@domain.com write In the mail client I see the shared1 folder under "shared folders". But I can't see it in a WebDAV client. At the server the shared1 folder doesn't have the autocreated xDAV structure as it normally autocreates for individual users, and I don't know how to create it manually. Does anybody know how to configure WebDAV shared folders? I'm also interested in creating shared calendars (CalDAV) and addressbooks (CardDAV), but I suppose they are managed the same way as WebDAV. Thanks in advance, Anatoli Cyrus Home Page:http://www.cyrusimap.org/ List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus -- Ken Murchison Cyrus Development Team FastMail US LLC Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
xDAV and shared namespaces
Hi All, I'm experimenting with xDav (CalDAV, CardDAV and WebDAV) in Cyrus 3.0.5. All xDAV services in user namespaces work as expected, but I can't figure out how to setup them for shared namespaces. The most important service for me to setup for shared namespaces is WebDAV. I'd like to have a shared folder accessible to multiple users based on their ACLs. I know how to setup a shared folder for IMAP: cyradm> cm shar...@domain.com cyradm> sam shar...@domain.com us...@domain.com write In the mail client I see the shared1 folder under "shared folders". But I can't see it in a WebDAV client. At the server the shared1 folder doesn't have the autocreated xDAV structure as it normally autocreates for individual users, and I don't know how to create it manually. Does anybody know how to configure WebDAV shared folders? I'm also interested in creating shared calendars (CalDAV) and addressbooks (CardDAV), but I suppose they are managed the same way as WebDAV. Thanks in advance, Anatoli Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
