Re: [RFC PATCH v2] imapd.c: imapoptions: implement idle timeout
> Back in imapd.c, cmd_idle() only reads from the client socket if > IDLE_INPUT is flagged. Which it's not after a select timeout, so if > tcp_keepalive has detected a connection dropout, cyrus still doesn't > know about it. So tcp_keepalive and select() timeout are unrelated. It appears the point of a select() timeout is just a "defensive" measure, as the comment that sets up the timeout notes: /* maximum possible timeout before we double-check anyway */ timeout.tv_sec = idle_timeout; timeout.tv_usec = 0; So if the timeout is hit, we fall out of imap_idle(), do an index check and alert check, and if nothing has changed, go back into imap_idle(). In theory, this timeout shouldn't actually be needed really. But that's unreleated to tcp_keepalive. In the tcp_keepalive case, if the connection has been "cut", then eventually the keepalive will detect that, and mark the socket as EOF. In that case, select() will return with "read available on client fd" case, because: --- select(2) ... Those listed in readfds will be watched to see if characters become available for reading (more precisely, to see if a read will not block; in particular, a file descriptor is also ready on end-of-file) --- Which will go into this path: if (r > 0 && otherfd >= 0 && FD_ISSET(otherfd, )) flags |= IDLE_INPUT; So it should return IDLE_INPUT flags, which should catch the EOF condition up a layer. So I'm not sure how an IDLE connection doesn't eventually hang up if tcp_keepalive has been enabled. -- Rob Mueller r...@fastmail.fm
Re: [RFC PATCH v2] imapd.c: imapoptions: implement idle timeout
> The thing is, during IDLE, we're not blocked on a read, we're blocked on > a select (in idle_wait()). We don't even try to read from the client > until the select tells us it's readable. If the client has dropped out, > select never tells us it's readable, so we never try to read, so the > timeout doesn't get checked, so we stay there indefinitely. No, the select() call itself has a timeout, and idle_wait() will return after that timeout gets hit. > timeout. Comms between idled and imapd were handled with signals. Yes, and it was buggy and evil. The pipe system that replaced it is much saner. -- Rob Mueller r...@fastmail.fm
Re: [RFC PATCH v2] imapd.c: imapoptions: implement idle timeout
> If the client's connection has dropped out, no data will ever appear on > the socket, so select will never flag it as readable, so we will never > try to read from it, so we will never receive the read error even though > tcp_keepalive detected the dropout. And if this client was idling with > a low-activity mailbox selected (such as Drafts or Sent), it might be a > very long time before any activity prompts us to write to the socket, so > we also don't receive the write error. And so even though the socket > itself knows there's no connection anymore thanks to tcp_keepalive, we > don't know that, because we haven't tried to interact with it. And so > the connection/process doesn't get cleaned up. This is actually one of the main points of tcp_keepalive. If you're trying to read() from a socket where the other end has "died", the read blocks *forever*. However with tcp_keepalive enabled, the server eventually pings the other side, and after it doesn't get a response (after the configured backoff and retry times), it flags the connection as EOF, causing the read() to return with 0 bytes as per standard EOF/shutdown semantics. Anyway I'm not sure why this is required now. I'm pretty sure (from memory) in older versions of cyrus, the IDLE timeout was the standard inactivity timeout (30 minutes default), and after that time we'd always drop the connection. IDLE RFC even mentions that clients should DONE and re-IDLE every 29 minutes. https://tools.ietf.org/html/rfc3501 5.4.Autologout Timer If a server has an inactivity autologout timer, the duration of that timer MUST be at least 30 minutes. The receipt of ANY command from the client during that interval SHOULD suffice to reset the autologout timer. https://tools.ietf.org/html/rfc2177 The server MAY consider a client inactive if it has an IDLE command running, and if such a server has an inactivity timeout it MAY log the client off implicitly at the end of its timeout period. Because of that, clients using IDLE are advised to terminate the IDLE and re-issue it at least every 29 minutes to avoid being logged off. This still allows a client to receive immediate mailbox updates even though it need only "poll" at half hour intervals. Is there a reason to have a separate idle timeout separate to the standard inactivity timeout? timeout: 32 The length of the IMAP server's inactivity autologout timer, in minutes. The minimum value is 30, the default. The default is 32 to allow a bit of leeway for clients that try to NOOP every 30 minutes. -- Rob Mueller r...@fastmail.fm
Re: Problems with subscriptions & Thunderbird on 2.5.7
FYI a user of ours encountered this recently and we came to the conclusion it's a Thunderbird bug. An edited conversion I had with Bron is below. Checking the Thunderbird bugzilla, I see multiple reports of this. In particular a bug that was supposedly fixed in 2010 seems to have recently re-appeared for some people. https://bugzilla.mozilla.org/show_bug.cgi?id=592633 See comments near the bottom of that bug. Someone has opened a new bug. https://bugzilla.mozilla.org/show_bug.cgi?id=1089351 I see an old bug similarly related to this. https://bugzilla.mozilla.org/show_bug.cgi?id=529618 There's this comment near the bottom: --- Workaround was fortunatly found accidentally : Double click of IMAP folder. When IMAP folder was double clicked at folder pane, expand button automatically appeared and subfolder was automatically shown when subfolder exists. --- Can you check if that works for you? Rob - me: It appears LSUB and LIST (SUBSCRIBED) are only returning \HasChildren if there's a subscribed child mailbox, not if the actual mailbox has children. That seems wrong... Example in a users mailbox. . list INBOX.x.y.2015 * * LIST (\HasChildren) "." INBOX.x.y.2015 ... many more sub-folders ... . lsub INBOX.x.y.2015 * * LSUB () "." INBOX.x.y.2015 . OK Completed (0.010 secs 2 calls) . list (subscribed) INBOX.x.y.2015 * * LIST (\Subscribed) "." INBOX.x.y.2015 . OK Completed (0.010 secs 2 calls) bron: My reading of those is that they're correct. HasChildren actually only needs to be calculated in two cases: 1) it's been explicitly asked for with a RETURN option in LIST-EXTENDED 2) the expression matches a non-existent intermediate folder, but not its existent children. Otherwise it's nice to do, but not required. In particular, both LSUB and RETURN SUBSCRIBED only iterate the subscriptions database - so they don't know about children. We don't calculate them unless explicitly requested. I really do think it's a Thunderbird bug. me: . list INBOX.x.y.2015 * return (children) * LIST (\HasChildren) "." INBOX.x.y.2015 . list (subscribed) INBOX.x.y.2015 * return (children) * LIST (\Subscribed \HasChildren) "." INBOX.x.y.2015 Yeah, I think you might be right, the lack of both \HasChildren or \HasNoChildren shouldn't imply anything. On Wed, 13 Apr 2016, at 01:05, Giles Malet via Cyrus-devel wrote: > Before I head off to read RFCs, and try to understand what Cyrus is doing > and Thunderbird, and what is going wrong, I just wanted to ask if anyone > already knows the answers. Specifically, using LIST-EXTENDED, should a > mail client be able to determine which folders are subscribed to, and if > so why is Cyrus apparently not returning that information? Conceivably TB > is doing it wrong though. And if TB is trying to set a \Subscribed flag > on a folder, should that be updating Cyrus's list of subscriptions? > > Anyhow, I'll see if I can get some traces of what's going on, and see if > that's what should be going on. > > Thanks, > g -- Rob Mueller r...@fastmail.fm
Re: Changing LIST (again)
Alt External Non-Alt External Internal Inbox INBOX user.foo Inbox.sub INBOX.Inbox.subuser.foo.Inbox.sub INBOX.sub INBOX.INBOX.subuser.foo.INBOX.sub illegal INBOX.INBOXuser.foo.INBOX illegal INBOX.Inboxuser.foo.Inbox Drafts INBOX.Drafts user.foo.Drafts Trash INBOX.Trashuser.foo.Trash Inbox-2010 INBOX.Inbox-2010 user.foo.Inbox-2010 We would normalise the _display_ output in altnamespace mode to be Title Case, so Inbox, and in non-altnamespace mode to be INBOX, so the full listing output for those folders would look like this: Given I helped devise this plan, I approve and like it :) A couple of things: 1. Should we block creation of INBOX.sub in the alt namespace mode? In theory, only the top level [Ii][Nn][Bb][Oo][Xx] is special, but it *might* confuse some clients that INBOX.sub and Inbox.sub are different folders. 2. Should it be possible to block creation of the illegal folders and to help with (1), INBOX.[...everything except Inbox...] in the Non-Alt namespace mode? If we did that, we could create a script that went through and renamed all the existing bogus ones in some way sanely, and know that no invisible or confusing folders will ever be created in the future... Rob
Re: RFC5703 support (Sieve Email Filtering)
I want to check how many people out there would be interested on RFC5703 support in Cyrus IMAP, that is, Sieve Email Filtering: MIME Part Tests, Iteration, Extraction, Replacement, and Enclosure. See: http://tools.ietf.org/html/rfc5703 Any improvements to cyrus sieve support would be highly welcomed! Apart from some charset improvements that fell out of other work, I believe there really hasn't been any work on cyrus sieve recently, and it would be great to kick start that. Cheers Rob
Re: Cyrus CalDAV design decision
(Resending to include cyrus-devel) At the moment, the storage format in use is iCalendar, being stored as RFC5322 messages. That sounds very much like what Kolab did in version 1. After trying to make this interoperate for several years it was given up in favor of the Kolab XML format because iCalendar vCard looked good on paper but suffered from severe interoperability issues between implementations. While I like the idea of the Kolab format, I'd like to point out that it's not all quite as rosy as you suggest. Most the criticisms you have of iCalendar and vCard, the Kolab format has equal amount of or more. Having had a look at the spec and putting together comments from several people: 1. The format never seems to have ever made it to the finished state. It's been stuck in RC states for several years. I don't think that gives strong confidence to implementers that it's actually stable. http://kolab.org/doc/kolabformat-2.0rc7-html/index.html 2. The description is that's it's in XML, but there's no DTD or ABNF, just a pseudo english format description. So there's no way to clearly test that you can create/read conforming XML blobs in a client implementation. 3. The document is very incomplete in places, for example this node: http://kolab.org/doc/kolabformat-2.0rc7-html/x180.html 4. It's also inconsistent in places. Some attributes being described as both MAY be present and mandatory: http://kolab.org/doc/kolabformat-2.0rc7-html/c188.html 5. Some parts are underspecified. The sensitivity field is inadequately described: http://kolab.org/doc/kolabformat-2.0rc7-html/c188.html. 6. For that matter, there's no complete set of examples (an mbox full of example Events, Tasks, Contacts, etc) or a test suite that clients should be able to read and interoperate with. 7. The Datetime format seems arbitrarily made up, but is actually the ISO8601 combined extended UTC format with second precision, but you don't reference ISO8601 or RFC3339 which also defines Internet time the same way. Likewise Date. 8. Recurrence seems primitive. Recurrence across DST boundaries seems to be as broken as every other format. There doesn't seem to be a way to specify public holiday exclusions. 9. The docs talk about ANNOTATEMORE more everywhere, which really needs updating to include METADATA and the final RFC 10. The /vendor/kolab/folder-type annotation should be updated now that SPECIALUSE has been made an RFC When you add these things together, I don't think it makes it easy to create a client or server for the Kolab format. And I think that goes part of the way to explaining why the quality of alternate clients out there hasn't exactly been great. I've tried Bynari (Outlook) and SyncKolab (Thunderbird), and their implementations are pretty bad. Having said a bunch of negative things, let me finish on how I think we can actually fix these and take these forward: 1. The Kolab Format spec needs to be completely finished. Edge cases cleaned up, todo parts finished, and unspecified bits specified completely. 2. It needs to be marked as a true 2.0 version, not an RC. You can add 2.1 later if you want, but I think the eternal RC status just leaves it feeling abandoned. 3. We need some sort of test suite/code. That should consist of at least two main things: a) A large bunch of emails that test all the different edge cases of the spec that clients should be able to read provided in a simple mbox format (separate one for each folder type). That way at least client implementers can quickly test their reading code against a bunch of cases b) Some standalone C/C++ executable that should be able to read these emails, check they conform to the right format, and dump a memory structure of the parsed result. That will allow implementers to quickly check that they're generating correctly formatted data structures. Yes, that would require a bit of work from someone, but I totally think it's worth it, and I think it would go a long way to helping more implementers work on the Kolab format in the future. What do you think the chances are of dedicating a developer or two for a few months to actually getting the spec up to scratch, and building the above 3a and 3b testing sets/tools? Cheers Rob Rob Mueller r...@fastmail.fm
Re: QRESYNC (was: Re: [POLL] Development guidance)
1) Pro: * simple to calculate - only updated by cyr_expire simple seems good :) 1) Con: * Higher bandwidth use in the stale client case. If you're not syncing that often, then there's probably enough other stuff that needs downloading that it'll swamp the bandwidth anyway. 2) Con: * Extra bookkeeping - need to update a different record when removing an expired record. * Replication implications - cyr_expire and replication is already going to be tricky - but probably means arbitrary bumping of modseq on the previous record to guarantee it replicates. More spurious modseq bumps. These seem a bit scary to me. I'd lean towards (1). Simpler to implement, less side effects, less chance of odd edge case bugs or replication issues. Slightly more bandwidth if you don't sync regularly. Rob
Re: Cyrus IMAP4 to Cyrus IMAP accidental change of greeting?
This assumes that the ability of setting this special annotation also means the server will honor it. Probably a save thing to do with the current implementations. But this gives this detection two small drawbacks: This is still way better than your current solution which was to look for Cyrus IMAP4 in the opening banner! This is way less specific and less broken than that approach. Rob r...@fastmail.fm Sign up at http://fastmail.fm for fast, ad free, IMAP accessible email
Re: skiplist db corruption
When restarting (e.g. after the crash) the transaction is not rolled back in every case -- corruption. Yes, we've seen that. It's annoying, though not deadly so far since it's only been seen state databases. Workaround: Switch to BDB. I trust BDB a lot less than the skiplist db. I've seen sufficiently many cases of BDB screwing up with just the deliver db that I wouldn't trust it. The worst is seeing hundreds of processes get stuck of the course of a few minutes on some lock that leaves them in a busy loop and brings the machine to it's knees. Nasty. Rob
