Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Looks like there was a bug in here: http://hg.dovecot.org/dovecot-2.2/rev/98195220a0f7 On 8.11.2012, at 0.08, Michael M Slusarz wrote: > Quoting Timo Sirainen : > >> On 6.11.2012, at 3.49, Michael J Rubinsky wrote: >> >> These defines in mail-transaction-log-private.h anyway can be changed to >> make it much less likely to see your problem: >> >> /* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ >> #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) >> /* If log is larger than MAX_SIZE, rotate regardless of the time */ >> #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) >> #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5) >> >> /* Delete .log.2 files older than this many seconds. Don't be too eager, >> older files are useful for QRESYNC and dsync. */ >> #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2) >> >> Maybe the defaults could be changed.. > > I'm not sure changing the defaults is a good idea. But if someone does want > to use a particular dovecot server as the backend for activesync clients, for > example, it would probably make sense to allow these values to be tweaked via > the config files. (I can see an organization having a "normal" IMAP server > and a "activesync" IMAP server that differ in these details, and also in > things like IDLE timeouts). > > michael >
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Timo Sirainen : On 8.11.2012, at 0.34, Michael M Slusarz wrote: Quoting Michael M Slusarz : I see your point, but the problem is that is not intuitive when reading the RFC. One part of the RFC defines the behavior of VANISHED (EARLIER) as only returning changes since the mod-sequence given. And you are correct that another part of the RFC says that, essentially, a server is allowed to break this required response. I'm thinking that this is more of an issue with the way the RFC is written. I'll move this over to the imap protocol list to get further input. Sigh. Never mind. For some reason, I completely ignored (missed?) this part of the RFC: Note: A server that receives a mod-sequence smaller than , where is the value of the smallest expunged mod-sequence it remembers minus one, MUST behave as if it was requested to report all expunged messages from the provided UID set parameter. So you are right, I was wrong, and the world is good. I wonder how much would it help if you a) Used the uidset/seqset parameters with SELECT command We *do* use this information. However, this is not (necessarily) useful for the activesync query that was the genesis of this thread. A bit of background on our MUA design is necessary. For Horde/IMP, all IMAP server configuration is done through the IMP application. As part of this configuration, a cache backend can be configured. There are multiple potential users of this IMAP object. Within IMP itself, multiple sessions can be open at any one time. Additionally, several views of IMP, our dynamic view and our smartmobile view, have another cache of messages kept on the browser side. Finally, the ActiveSync library also uses the IMAP object configured by IMP. Anytime the IMAP object is accessed, we are syncing the mailbox with the IMP-configured cache. For QRESYNC, we use the SELECT/EXAMINE uidset parameter. The problem is that any particular view may not be sync'd to the same state as the IMP cache. For example, if someone is using the web application and their phone is syncing via ActiveSync, it is quite likely that the activesync cached mod-sequence value will NOT equal the IMP cached mod-sequence value. So this is when explicitly calling FETCH VANISHED CHANGEDSINCE is needed. The good news: once we get the CHANGEDSINCE FETCH information, we don't need to do a separate flags sync since this information has already been cached within the IMAP object (via either the CHANGEDSINCE call or, more likely, a previous FETCH call in another session). Further optimization: in the case where the original QRESYNC/CONDSTORE sync matches the mod-sequence of whatever object/view is accessing the IMAP object, which should be the most common occurrence, there is no need to perform any additional FETCH/SEARCH calls since we cache the results of the initial mailbox sync and return this data. Might be a long-winded explanation, but just wanted to show why FETCH VANISHED CHANGEDSINCE MUST be used by a client even if taking advantage of QRESYNC SELECT/EXAMINE syncing. In other words - I'd like to think that my imap implementation is not broken :) b) Dovecot implemented it slightly better than required by RFC: http://www.ietf.org/mail-archive/web/lemonade/current/msg04771.html I spent a week or so trying to cache message sequence number -> UID mapping. And determined it was more trouble than it was worth. The gains from more compact VANISHED responses in SELECT/EXAMINE are minimal compared to the expense to track them. And the only other reason for tracking - the possibility that EXPUNGEs return EXPUNGED responses instead of VANISHED if the UIDs of the actually expunged messages are needed - can be worked around by doing a UID SEARCH call after the EXPUNGE is over and comparing to the list of UIDs that were given to UID EXPUNGE (with the further optimization that I cache MSN->UID while in a mailbox, which should catch the "STORE (\Deleted)/EXPUNGE" common when using a Trash mailbox or immediate message deletion). Unless I am missing something else that MSNs are necessary? michael
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Timo Sirainen : On 8.11.2012, at 0.08, Michael M Slusarz wrote: I'm not sure changing the defaults is a good idea. But if someone does want to use a particular dovecot server as the backend for activesync clients, for example, it would probably make sense to allow these values to be tweaked via the config files. (I can see an organization having a "normal" IMAP server and a "activesync" IMAP server that differ in these details, and also in things like IDLE timeouts). Well .. I hate adding more settings. :) There are way too many already. Ideally Dovecot would automatically do the right thing anyway. Just like it already caches only those things that are needed. It could also increase these values when QRESYNC is used, or even better to actually have the separate expunge log that I mentioned. Thinking about this more, this can really all be handled by proper MUA design. In short, it is never a good idea to send a '1:*' UID range to a VANISHED CHANGEDSINCE FETCH. It remains a reasonable MUA design decision to not send the actual cached UID list to the FETCH command: if this cached UID list is thousands of messages long, obtaining this list, (optionally) sequence set compressing, and sending via the command may take more time/resources than it saves. But a MUA should, at a minimum, keep track of the minimum UID it is aware of in order to limit the possible response. This is a trivial amount of extra overhead and would prevent a large number of spurious VANISHED UIDs to need to be traversed. michael
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
On 8.11.2012, at 0.34, Michael M Slusarz wrote: > Quoting Michael M Slusarz : > >> I see your point, but the problem is that is not intuitive when reading the >> RFC. One part of the RFC defines the behavior of VANISHED (EARLIER) as only >> returning changes since the mod-sequence given. And you are correct that >> another part of the RFC says that, essentially, a server is allowed to break >> this required response. >> >> I'm thinking that this is more of an issue with the way the RFC is written. >> I'll move this over to the imap protocol list to get further input. > > Sigh. Never mind. For some reason, I completely ignored (missed?) this part > of the RFC: > > Note: A server that receives a mod-sequence smaller than , > where is the value of the smallest expunged mod-sequence > it remembers minus one, MUST behave as if it was requested to report > all expunged messages from the provided UID set parameter. > > So you are right, I was wrong, and the world is good. I wonder how much would it help if you a) Used the uidset/seqset parameters with SELECT command and optionally b) Dovecot implemented it slightly better than required by RFC: http://www.ietf.org/mail-archive/web/lemonade/current/msg04771.html
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Michael M Slusarz : I see your point, but the problem is that is not intuitive when reading the RFC. One part of the RFC defines the behavior of VANISHED (EARLIER) as only returning changes since the mod-sequence given. And you are correct that another part of the RFC says that, essentially, a server is allowed to break this required response. I'm thinking that this is more of an issue with the way the RFC is written. I'll move this over to the imap protocol list to get further input. Sigh. Never mind. For some reason, I completely ignored (missed?) this part of the RFC: Note: A server that receives a mod-sequence smaller than , where is the value of the smallest expunged mod-sequence it remembers minus one, MUST behave as if it was requested to report all expunged messages from the provided UID set parameter. So you are right, I was wrong, and the world is good. michael
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
On 8.11.2012, at 0.08, Michael M Slusarz wrote: >> These defines in mail-transaction-log-private.h anyway can be changed to >> make it much less likely to see your problem: >> >> /* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ >> #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) >> /* If log is larger than MAX_SIZE, rotate regardless of the time */ >> #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) >> #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5) >> >> /* Delete .log.2 files older than this many seconds. Don't be too eager, >> older files are useful for QRESYNC and dsync. */ >> #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2) >> >> Maybe the defaults could be changed.. > > I'm not sure changing the defaults is a good idea. But if someone does want > to use a particular dovecot server as the backend for activesync clients, for > example, it would probably make sense to allow these values to be tweaked via > the config files. (I can see an organization having a "normal" IMAP server > and a "activesync" IMAP server that differ in these details, and also in > things like IDLE timeouts). Well .. I hate adding more settings. :) There are way too many already. Ideally Dovecot would automatically do the right thing anyway. Just like it already caches only those things that are needed. It could also increase these values when QRESYNC is used, or even better to actually have the separate expunge log that I mentioned.
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Timo Sirainen : On 6.11.2012, at 3.49, Michael J Rubinsky wrote: That would require infinitely storing the modseq of when each message was expunged. Not very nice. Also the RFC talks a lot about this situation. The SELECT command has two optional parameters to optimize it. The RFC *does* indicate that a server implementation could, strictly speaking, be considered in compliance without remembering modsequences for all expunged messages, but it does explicitly discourage such implementations. From RFC 5162 [4.1]: Strictly speaking, a server implementation that doesn't remember mod- sequences associated with expunged messages can be considered compliant with this specification. Such implementations return all expunged messages specified in the UID set of the UID FETCH (VANISHED) command every time, without paying attention to the specified CHANGEDSINCE mod-sequence. Such implementations are discouraged, as they can end up returning VANISHED responses that are bigger than the result of a UID SEARCH command for the same UID set. This is talking about a server that doesn't permanently remember ANY modseqs for expunges. Dovecot remembers them, not not infinitely. It also gives advice to avoid infinitely storing the modsequences such as "expiring" sequences associated with older expunged messages, but assigning a single modsequence value to all of the expired expunged messages. Dovecot behaves as the section 4.3 describes. Note especially: Note that indefinitely storing information about expunged messages can cause storage and related problems for an implementation. .. Hence, implementations are encouraged to adopt strategies to protect against such storage problems, such as limiting the size of the queue used to store mod-sequences for expunged messages and "expiring" older records when this limit is reached. When the selected implementation-specific queue limit is reached, the oldest record(s) are deleted from the queue (note that such records are located at the queue head). For all such "expired" records, the server needs to store a single mod-sequence, which is the highest mod-sequence for all "expired" expunged messages. This is exactly what Dovecot does. There is a single modseq associated with all the previously expunged messages. If you try to request expunges for that modseq, it returns all of the expunged messages, which is what you're seeing as a problem. I see your point, but the problem is that is not intuitive when reading the RFC. One part of the RFC defines the behavior of VANISHED (EARLIER) as only returning changes since the mod-sequence given. And you are correct that another part of the RFC says that, essentially, a server is allowed to break this required response. I'm thinking that this is more of an issue with the way the RFC is written. I'll move this over to the imap protocol list to get further input. michael
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Timo Sirainen : On 6.11.2012, at 3.49, Michael J Rubinsky wrote: These defines in mail-transaction-log-private.h anyway can be changed to make it much less likely to see your problem: /* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) /* If log is larger than MAX_SIZE, rotate regardless of the time */ #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5) /* Delete .log.2 files older than this many seconds. Don't be too eager, older files are useful for QRESYNC and dsync. */ #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2) Maybe the defaults could be changed.. I'm not sure changing the defaults is a good idea. But if someone does want to use a particular dovecot server as the backend for activesync clients, for example, it would probably make sense to allow these values to be tweaked via the config files. (I can see an organization having a "normal" IMAP server and a "activesync" IMAP server that differ in these details, and also in things like IDLE timeouts). michael
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
On 6.11.2012, at 14.17, Timo Sirainen wrote: > It would be possible to keep a separate expunge log which could remember the > expunges longer. But that would be yet another different index file for > Dovecot, which annoyingly complicates everything. And currently since it > sounds like the only problem is activesync implementation using it, I'm not > very interested in spending a lot of time on it. Here's an idea how it could maybe be done without too much complexity: Before deleting/replacing dovecot.index.log.2 file, scan all of the expunges from it and append them to dovecot.index.log.expunges file using the normal transaction log file format. Change the reader code to use it only when a special flag is enabled (because it has no other changes, it shouldn't normally be used). Use that flag when looking for old expunges. Once in a while recreate the file and drop some of the oldest expunges. The annoying thing with that is that the file format wastes disk space. An alternative would be to write to the file using yet another format optimized for it, and then have a separate "expunge scanner" API that scans the expunge file and the transaction logs. Anyway, not something I have time to implement myself anytime soon.
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Timo Sirainen : On 6.11.2012, at 3.49, Michael J Rubinsky wrote: That would require infinitely storing the modseq of when each message was expunged. Not very nice. Also the RFC talks a lot about this situation. The SELECT command has two optional parameters to optimize it. The RFC *does* indicate that a server implementation could, strictly speaking, be considered in compliance without remembering modsequences for all expunged messages, but it does explicitly discourage such implementations. From RFC 5162 [4.1]: Strictly speaking, a server implementation that doesn't remember mod- sequences associated with expunged messages can be considered compliant with this specification. Such implementations return all expunged messages specified in the UID set of the UID FETCH (VANISHED) command every time, without paying attention to the specified CHANGEDSINCE mod-sequence. Such implementations are discouraged, as they can end up returning VANISHED responses that are bigger than the result of a UID SEARCH command for the same UID set. This is talking about a server that doesn't permanently remember ANY modseqs for expunges. Dovecot remembers them, not not infinitely. It also gives advice to avoid infinitely storing the modsequences such as "expiring" sequences associated with older expunged messages, but assigning a single modsequence value to all of the expired expunged messages. Dovecot behaves as the section 4.3 describes. Note especially: Note that indefinitely storing information about expunged messages can cause storage and related problems for an implementation. .. Hence, implementations are encouraged to adopt strategies to protect against such storage problems, such as limiting the size of the queue used to store mod-sequences for expunged messages and "expiring" older records when this limit is reached. When the selected implementation-specific queue limit is reached, the oldest record(s) are deleted from the queue (note that such records are located at the queue head). For all such "expired" records, the server needs to store a single mod-sequence, which is the highest mod-sequence for all "expired" expunged messages. This is exactly what Dovecot does. There is a single modseq associated with all the previously expunged messages. If you try to request expunges for that modseq, it returns all of the expunged messages, which is what you're seeing as a problem. It might be helpful in some situations to decrease the precision and remember for example: * UIDs 1-100 were deleted with modseq 10 (in reality multiple times between modseqs 1..10) * UIDs 101-130 were deleted with modseq 15 (in reality with modseqs 11..15) .. and so on But this assumes that the expunged UID ranges compress well. If UIDs are being deleted here and there it's still pretty wasteful to store them. And yes, the current way may be forgetting them a bit too early when a lot of other unrelated changes are happening. It would be possible to keep a separate expunge log which could remember the expunges longer. But that would be yet another different index file for Dovecot, which annoyingly complicates everything. And currently since it sounds like the only problem is activesync implementation using it, I'm not very interested in spending a lot of time on it. These defines in mail-transaction-log-private.h anyway can be changed to make it much less likely to see your problem: /* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) /* If log is larger than MAX_SIZE, rotate regardless of the time */ #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5) /* Delete .log.2 files older than this many seconds. Don't be too eager, older files are useful for QRESYNC and dsync. */ #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2) Maybe the defaults could be changed.. Thanks for the information and clarification, Timo. -- mike The Horde Project (www.horde.org) mrubi...@horde.org smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Robert Schetterer : Am 06.11.2012 13:17, schrieb Timo Sirainen: the only problem is activesync implementation using it Hi, Michael, as i am going to implement horde 5 active sync server in near Future with dovecot , i followed this in high interest I know horde active sync server is different to z-push so sorry my question, i recent use z-push with dovecot and never noticed any problems with it, so how does z-push handle this "problem" comparing horde, any chance to workaround it in horde code ? ZPush's default "imap backend" implementation uses PHP's imap implementation, which has _many_ problems. Horde's uses Horde_Imap_Client a PHP native implementation with tons of improvements over just about any other available IMAP client implementation in PHP. The problem I'm describing stems from our ActiveSync code trying to take advantage of IMAP extensions such as QRESYNC, where the server can track and report things like expunged messages, flag changes etc... since a known point without us having to cache and track every bit of information about the device's state. This is what ZPush does, it uses plaintext files on the server and writes out all the message ids and flag states that were sent to each device. Then during every sync, it retrieves the complete list of message ids and flag state from the IMAP server again and compares them against the data in the text file to determine the deltas. There is already a work around for this in place - if we detect a UID being returned from one of these VANISHED responses that is older than the oldest known UID we know was sent to the device then we reset the device's mailbox state, causing the mailbox to resync. We already track the UIDs of all email sent to the device in our state tables when using a server that does not support QRESYNC, I plan on extending/improving this and getting rid of the current use VANISHED data completely. For further information about our code vs ZPush you can read a basic overview on our wiki page http://wiki.horde.org/ActiveSync. I'm in the process of a more detailed feature comparison grid, but it's far from ready. Please move any further discussion specific to Horde's ActiveSync code to the d...@lists.horde.org mailing list. Hacking dovecot might be complex, as it should fit in so many stuff and setups I would never suggest people should hack server code to get other code to work :) Anyway i agree many people using active sync in their smartphone these days and new outlook will also have a active sync mode, so there shouldnt stay any fixable problems ( where ever they may be ), but in my understanding the imap server ( dovecot ) acts more like a backhand here did you test against recent versions of cyrus etc, how is their behave in this case? I have not personally tested against cyrus, but I know some of our other devs use cyrus with the ActiveSync code base. -- mike The Horde Project (www.horde.org) mrubi...@horde.org smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Am 06.11.2012 13:17, schrieb Timo Sirainen: > the only problem is activesync implementation using it Hi, Michael, as i am going to implement horde 5 active sync server in near Future with dovecot , i followed this in high interest I know horde active sync server is different to z-push so sorry my question, i recent use z-push with dovecot and never noticed any problems with it, so how does z-push handle this "problem" comparing horde, any chance to workaround it in horde code ? Hacking dovecot might be complex, as it should fit in so many stuff and setups Anyway i agree many people using active sync in their smartphone these days and new outlook will also have a active sync mode, so there shouldnt stay any fixable problems ( where ever they may be ), but in my understanding the imap server ( dovecot ) acts more like a backhand here did you test against recent versions of cyrus etc, how is their behave in this case ? Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
On 6.11.2012, at 3.49, Michael J Rubinsky wrote: >> That would require infinitely storing the modseq of when each message >> was expunged. Not very nice. Also the RFC talks a lot about this >> situation. The SELECT command has two optional parameters to optimize >> it. > > The RFC *does* indicate that a server implementation could, strictly > speaking, be considered in compliance without remembering modsequences for > all expunged messages, but it does explicitly discourage such > implementations. From RFC 5162 [4.1]: > > Strictly speaking, a server implementation that doesn't remember mod- > sequences associated with expunged messages can be considered > compliant with this specification. Such implementations return all > expunged messages specified in the UID set of the UID FETCH > (VANISHED) command every time, without paying attention to the > specified CHANGEDSINCE mod-sequence. Such implementations are > discouraged, as they can end up returning VANISHED responses that are > bigger than the result of a UID SEARCH command for the same UID set. This is talking about a server that doesn't permanently remember ANY modseqs for expunges. Dovecot remembers them, not not infinitely. > It also gives advice to avoid infinitely storing the modsequences such as > "expiring" sequences associated with older expunged messages, but assigning a > single modsequence value to all of the expired expunged messages. Dovecot behaves as the section 4.3 describes. Note especially: Note that indefinitely storing information about expunged messages can cause storage and related problems for an implementation. .. Hence, implementations are encouraged to adopt strategies to protect against such storage problems, such as limiting the size of the queue used to store mod-sequences for expunged messages and "expiring" older records when this limit is reached. When the selected implementation-specific queue limit is reached, the oldest record(s) are deleted from the queue (note that such records are located at the queue head). For all such "expired" records, the server needs to store a single mod-sequence, which is the highest mod-sequence for all "expired" expunged messages. This is exactly what Dovecot does. There is a single modseq associated with all the previously expunged messages. If you try to request expunges for that modseq, it returns all of the expunged messages, which is what you're seeing as a problem. It might be helpful in some situations to decrease the precision and remember for example: * UIDs 1-100 were deleted with modseq 10 (in reality multiple times between modseqs 1..10) * UIDs 101-130 were deleted with modseq 15 (in reality with modseqs 11..15) .. and so on But this assumes that the expunged UID ranges compress well. If UIDs are being deleted here and there it's still pretty wasteful to store them. And yes, the current way may be forgetting them a bit too early when a lot of other unrelated changes are happening. It would be possible to keep a separate expunge log which could remember the expunges longer. But that would be yet another different index file for Dovecot, which annoyingly complicates everything. And currently since it sounds like the only problem is activesync implementation using it, I'm not very interested in spending a lot of time on it. These defines in mail-transaction-log-private.h anyway can be changed to make it much less likely to see your problem: /* Rotate when log is older than ROTATE_TIME and larger than MIN_SIZE */ #define MAIL_TRANSACTION_LOG_ROTATE_MIN_SIZE (1024*32) /* If log is larger than MAX_SIZE, rotate regardless of the time */ #define MAIL_TRANSACTION_LOG_ROTATE_MAX_SIZE (1024*1024) #define MAIL_TRANSACTION_LOG_ROTATE_TIME (60*5) /* Delete .log.2 files older than this many seconds. Don't be too eager, older files are useful for QRESYNC and dsync. */ #define MAIL_TRANSACTION_LOG2_STALE_SECS (60*60*24*2) Maybe the defaults could be changed..
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Timo Sirainen : On Mon, 2012-11-05 at 12:43 -0700, Michael M Slusarz wrote: My argument is much simpler: it is blatantly breaking the RFC. From RFC 5162 [3.2]: The VANISHED UID FETCH modifier instructs the server to report those messages from the UID set parameter that have been expunged and whose associated mod-sequence is larger than the specified mod-sequence. **That is, the client requests to be informed of messages from the specified set that were expunged since the specified mod-sequence.** (emphasis added) If you are including UIDs in the FETCH return that have NOT been expunged since the given mod-sequence, that directly contradicts this language. The clear intent of VANISHED UID FETCH is to provide the list of messages that existed in the mailbox at mod-sequence and no longer exist in the mailbox as of the current HIGHESTMODSEQ. That would require infinitely storing the modseq of when each message was expunged. Not very nice. Also the RFC talks a lot about this situation. The SELECT command has two optional parameters to optimize it. The RFC *does* indicate that a server implementation could, strictly speaking, be considered in compliance without remembering modsequences for all expunged messages, but it does explicitly discourage such implementations. From RFC 5162 [4.1]: Strictly speaking, a server implementation that doesn't remember mod- sequences associated with expunged messages can be considered compliant with this specification. Such implementations return all expunged messages specified in the UID set of the UID FETCH (VANISHED) command every time, without paying attention to the specified CHANGEDSINCE mod-sequence. Such implementations are discouraged, as they can end up returning VANISHED responses that are bigger than the result of a UID SEARCH command for the same UID set. Clients that use the message sequence match data can reduce the scope of this VANISHED response substantially in the typical case where expunges have not happened, or happen only toward the end of the mailbox. It also gives advice to avoid infinitely storing the modsequences such as "expiring" sequences associated with older expunged messages, but assigning a single modsequence value to all of the expired expunged messages. -- mike The Horde Project (www.horde.org) mrubi...@horde.org smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
On Mon, 2012-11-05 at 12:43 -0700, Michael M Slusarz wrote: > My argument is much simpler: it is blatantly breaking the RFC. From > RFC 5162 [3.2]: > > The VANISHED UID FETCH modifier instructs the server to report those > messages from the UID set parameter that have been expunged and whose > associated mod-sequence is larger than the specified mod-sequence. > **That is, the client requests to be informed of messages from the > specified set that were expunged since the specified > mod-sequence.** (emphasis added) > > If you are including UIDs in the FETCH return that have NOT been > expunged since the given mod-sequence, that directly contradicts this > language. The clear intent of VANISHED UID FETCH is to provide the > list of messages that existed in the mailbox at mod-sequence and no > longer exist in the mailbox as of the current HIGHESTMODSEQ. That would require infinitely storing the modseq of when each message was expunged. Not very nice. Also the RFC talks a lot about this situation. The SELECT command has two optional parameters to optimize it.
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Michael J Rubinsky : I don't think they become incorrect, just that there are more of them than really necessary? Yes, there's a tipping point. It's when the modseq no longer exists in the dovecot.index.log* files, which get rotated once in a while. This shouldn't happen very often. FYI, I see this about once every two weeks, sometimes more frequently. Is there anything we can do to reduce the chance of this happening? How do you see it? Does it break something? Why? You can change it only by increasing the dovecot.index.log sizes, which requires changing the code. It breaks ActiveSync synchronization of mailboxes. When this happens, the sync state of the mailbox needs to be reset, which causes the entire mailbox to be resynchronized to the mobile device. This can lead to a not-insignificant amount of wasted bandwidth and battery power for the device. There have been times when this has happened multiple times in a single day. Not resetting the state leads to multiple issues on the device due to sending it thousands of deletion commands for messages it knows nothing about. My argument is much simpler: it is blatantly breaking the RFC. From RFC 5162 [3.2]: The VANISHED UID FETCH modifier instructs the server to report those messages from the UID set parameter that have been expunged and whose associated mod-sequence is larger than the specified mod-sequence. **That is, the client requests to be informed of messages from the specified set that were expunged since the specified mod-sequence.** (emphasis added) If you are including UIDs in the FETCH return that have NOT been expunged since the given mod-sequence, that directly contradicts this language. The clear intent of VANISHED UID FETCH is to provide the list of messages that existed in the mailbox at mod-sequence and no longer exist in the mailbox as of the current HIGHESTMODSEQ. As Mike R. has demonstrated, it is plausible that an MUA can only provide the MODSEQ of its cache state and has no knowledge of the UIDs it has actually cached. So having to parse through a (potentially) giant list of UIDs can be a performance killer (imagine the wasted bandwidth of having to upload a million UIDs to a phone every time you sync). michael
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Timo Sirainen : On 5.11.2012, at 20.37, Michael J Rubinsky wrote: On 5.11.2012, at 18.13, Michael J Rubinsky wrote: I've been seeing the following wonky behavior with Dovecot. Currently this is with Dovecot 2.0.19, but I was also seeing it in earlier versions as well, including versions from the 1.x series. Some background, this is from Horde's ActiveSync library, when it is trying to determine what UIDs have recently been expunged. I can guarantee that when this command was sent, there was only one or two emails that were removed since the provided MODSEQ value. .. There's definitely something wonky going on in the code. There's a certain tipping point of modseqs where the values become incorrect. I don't think they become incorrect, just that there are more of them than really necessary? Yes, there's a tipping point. It's when the modseq no longer exists in the dovecot.index.log* files, which get rotated once in a while. This shouldn't happen very often. FYI, I see this about once every two weeks, sometimes more frequently. Is there anything we can do to reduce the chance of this happening? How do you see it? Does it break something? Why? You can change it only by increasing the dovecot.index.log sizes, which requires changing the code. It breaks ActiveSync synchronization of mailboxes. When this happens, the sync state of the mailbox needs to be reset, which causes the entire mailbox to be resynchronized to the mobile device. This can lead to a not-insignificant amount of wasted bandwidth and battery power for the device. There have been times when this has happened multiple times in a single day. Not resetting the state leads to multiple issues on the device due to sending it thousands of deletion commands for messages it knows nothing about. -- mike The Horde Project (www.horde.org) mrubi...@horde.org smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
On 5.11.2012, at 20.37, Michael J Rubinsky wrote: >> On 5.11.2012, at 18.13, Michael J Rubinsky wrote: >> >>> I've been seeing the following wonky behavior with Dovecot. Currently this >>> is with Dovecot 2.0.19, but I was also seeing it in earlier versions as >>> well, including versions from the 1.x series. Some background, this is >>> from Horde's ActiveSync library, when it is trying to determine what UIDs >>> have recently been expunged. I can guarantee that when this command was >>> sent, there was only one or two emails that were removed since the provided >>> MODSEQ value. >> .. >>> >>> There's definitely something wonky going on in the code. There's a certain >>> tipping point of modseqs where the values become incorrect. >> >> I don't think they become incorrect, just that there are more of them than >> really necessary? Yes, there's a tipping point. It's when the modseq no >> longer exists in the dovecot.index.log* files, which get rotated once in a >> while. This shouldn't happen very often. > > FYI, I see this about once every two weeks, sometimes more frequently. Is > there anything we can do to reduce the chance of this happening? How do you see it? Does it break something? Why? You can change it only by increasing the dovecot.index.log sizes, which requires changing the code.
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Timo Sirainen : On 5.11.2012, at 18.13, Michael J Rubinsky wrote: I've been seeing the following wonky behavior with Dovecot. Currently this is with Dovecot 2.0.19, but I was also seeing it in earlier versions as well, including versions from the 1.x series. Some background, this is from Horde's ActiveSync library, when it is trying to determine what UIDs have recently been expunged. I can guarantee that when this command was sent, there was only one or two emails that were removed since the provided MODSEQ value. .. There's definitely something wonky going on in the code. There's a certain tipping point of modseqs where the values become incorrect. I don't think they become incorrect, just that there are more of them than really necessary? Yes, there's a tipping point. It's when the modseq no longer exists in the dovecot.index.log* files, which get rotated once in a while. This shouldn't happen very often. FYI, I see this about once every two weeks, sometimes more frequently. Is there anything we can do to reduce the chance of this happening? -- mike The Horde Project (www.horde.org) mrubi...@horde.org smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
On 5.11.2012, at 18.13, Michael J Rubinsky wrote: > I've been seeing the following wonky behavior with Dovecot. Currently this is > with Dovecot 2.0.19, but I was also seeing it in earlier versions as well, > including versions from the 1.x series. Some background, this is from > Horde's ActiveSync library, when it is trying to determine what UIDs have > recently been expunged. I can guarantee that when this command was sent, > there was only one or two emails that were removed since the provided MODSEQ > value. .. > > There's definitely something wonky going on in the code. There's a certain > tipping point of modseqs where the values become incorrect. I don't think they become incorrect, just that there are more of them than really necessary? Yes, there's a tipping point. It's when the modseq no longer exists in the dovecot.index.log* files, which get rotated once in a while. This shouldn't happen very often. > For a mailbox with HIGHESTMODSEQ of 54004, I see this: > > a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881) > * VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083 > a OK Fetch completed. > b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882) > * VANISHED (EARLIER) 41029:41030,41047:41083 > b OK Fetch completed. > So in b) case Dovecot read the dovecot.index.log and returned the expunges found from there. In a) case it didn't know what was expunged since the modseq, so it returned UIDs of all the expunged mails.
Re: [Dovecot] Issues with VANISHED CHANGEDSINCE
Quoting Michael J Rubinsky : Also, to verify it wasn't something screwy with my server, Michael Slusarz provided me with this from his server: There's definitely something wonky going on in the code. There's a certain tipping point of modseqs where the values become incorrect. For a mailbox with HIGHESTMODSEQ of 54004, I see this: a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881) * VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083 a OK Fetch completed. b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882) * VANISHED (EARLIER) 41029:41030,41047:41083 b OK Fetch completed. FYI, this is output from Dovecot 2.1.10. michael
[Dovecot] Issues with VANISHED CHANGEDSINCE
I've been seeing the following wonky behavior with Dovecot. Currently this is with Dovecot 2.0.19, but I was also seeing it in earlier versions as well, including versions from the 1.x series. Some background, this is from Horde's ActiveSync library, when it is trying to determine what UIDs have recently been expunged. I can guarantee that when this command was sent, there was only one or two emails that were removed since the provided MODSEQ value. C: 8 UID FETCH 1:* UID (VANISHED CHANGEDSINCE 587742) S: * VANISHED (EARLIER) 1:119944,119947:123482,123484:128043,128045:130388,130390:130456,130458:132127,132129:135832,135834:136201,136203:136301,136303:136316,136318:138502,138504:140099,140101:141908,141910:142607,142609:142744,142746:142865,142867:143160,143164:143173,143176:143179,143181:143187,143189:143190,143192:143195,143197:143270,143272:143273,143275:143280,143282:143302,143305:143307,143309:143318,143320:14,143335:143337,143339:143341,143343:143375,143377:143400,143402:143551,143553:143637,143640:143706,143708:144069,144071:144178,144180:144307,144309:144316,144318:144329,144331:144931,144933:144983,144985:145000,145002:145069,145071:145072,145074:145132,145136:145178,145180:145191,145193 Also, to verify it wasn't something screwy with my server, Michael Slusarz provided me with this from his server: There's definitely something wonky going on in the code. There's a certain tipping point of modseqs where the values become incorrect. For a mailbox with HIGHESTMODSEQ of 54004, I see this: a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881) * VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083 a OK Fetch completed. b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882) * VANISHED (EARLIER) 41029:41030,41047:41083 b OK Fetch completed. Thanks for any guidance. -- mike The Horde Project (www.horde.org) mrubi...@horde.org smime.p7s Description: S/MIME Cryptographic Signature
