It looks like the Cyrus people are competent and are aware of the problems.
If anyone has some silver bullets that show how to APPEND to two servers that cannot see each other, I'd _love_ to hear about it. So would the rest of the world. Until then, I'd suggest the "simple" approach of either: a) requiring the databases fail updates when reachability is dubious b) building a separate system to block/defer APPEND when reachability is dubious. The former requires guarantees only the db can make, while the latter wouldn't require anything of the server. If anyone is scared of the token-passing software (implementation) I'm happy to help out here. It's _really_ not as difficult as it looks: it's one of those things. If your implementation isn't simple, you probably messed it up :) But note what's hard is getting dbmail to actually follow it. Right now, APPEND updates the mail store- and it can't do that. Furthermore, the mail store needs to have it's "active" bit for messages so dbmail won't touch messages that are (ahem) inactive. This means changes to indexes, reprofiling this code for various databases, etc. I think it's provable that ANY method will require an active bit (either the database sets it with a trigger or second-agent, or _we_ do using the token passing protocol), so it might as well be time to start adding it. dbmail TRUNK will simply APPEND with that bit set until we add the requisite UID generation part. On Mon, 2005-05-30 at 10:10 -0400, Geo Carncross wrote: > Recapping the thread: > > ``One thing I was thinking of would be a hack to make one server always > use only odd UIDs, and the other always use only even UIDs, and to do > catchups while they are reachable with each other. But this is getting > into hacking code I know nothing about, yet. Maybe a later time.'' > > ``How? IMAP UIDs are defined as strictly increasing integers. See RFC > 2060 section 2.3.1.1. This has nothing to do with Cyrus's > implementation.'' > > ``Using odd and even UIDs and merging would be a novel solution. > However, it may be `too' novel. There is an IMAP requirement that > doesn't permit merging (reordering). Messages can only be appended to > the end of the mailbox because new messages must get a UID >= UIDNEXT. > The situation is never permitted to occur where the client opens a > mailbox which has UIDs 1,3,5 and then at some time later it has UIDs > 1,2,3,4,5 with out a change to UIDVALIDITY. When UIDVALIDITY changes > (which should be larger than before), the client knows that what it > knows about UIDs is now useless.'' > > ``Doing replicated IMAP stores (espeically geographicly distanct ones) > is not an easy problem. '' > > On Fri, 2005-05-27 at 11:57 -0700, Kevin Baker wrote: > > Good thread on a Cyrus forum discussing the same concept > > of UID collistion in server clusters. > > > > > > http://tinyurl.com/9kcgv > > > > > > Might be good to see how they are handling this. Murder > > has been under development for some time now, so this is > > likely resolved. > > > > > > Kevin > > > > > > > > > On Fri, 2005-05-27 at 10:51 -0700, Kevin Baker wrote: > > >> Thanks for the explination I totally understand now. > > >> > > >> That being said, what is the actual harm in one message > > >> id > > >> being even a second off for the UID. They will both be > > >> entered in the mailstore, even if they are slightly out > > >> of > > >> order. > > > > > > It matters because if a client sees a mailbox and the > > > highest UID is 287 > > > and UID 286 comes in later it will NEVER KNOW about UID > > > 286 which means > > > mail is lost and the user is confused. > > > > > > RFC2060 actually _encourages_ IMAP client authors to make > > > this > > > optimization. > > > > > > > > >> Ultimately the Mail client will sort by the header > > >> information and if the clocks are different then they > > >> will > > >> be out of order. > > > > > > No. You're confusing Received-time and UID-time. > > > > > > You also assume clients perform an IMAP SEARCH ALL NOT > > > DELETED operation > > > whenever a new message comes in. There's no such > > > guarantee. If a client > > > saw "LAST UID=287" twice in a raw, then it'll never > > > download UID 286. > > > It'll just never be fetched. > > > > > > Here's the roadmap: > > > > > > UID 287 inserted > > > client sees highest UID number is 287 > > > UID 286 inserted > > > client NEVER DOWNLOADS UID 286 > > > > > > It never does because it has no reason whatsoever to think > > > it exists. > > > Some IMAP clients will detect it later because they do an > > > IMAP SEARCH > > > operation and treat the mailbox as a cache (Netscape, > > > Thunderbird), but > > > others will never notice it until UIDVALIDITY changes. > > > > > > > > >> I should note that I've been running a Cyrus system for > > >> some time now that occasionally has messages arrive one > > >> or > > >> two spots out of order, due to latency with mailfilters. > > >> It really isn't a problem. > > > > > > Cyrus _NEVER_ inserts UIDs before the highest number. > > > Received time has > > > nothing to do with it. > > > > > >> I realize this might bend the RFC slightly... but the > > >> added functionality is great and it will not break any > > >> mail clients. I am all for complete RFC compliance, but > > >> if > > >> we are talking about a matter of a second here and there > > >> due to processor lag I think we are probably going > > >> overboard. > > > > > > No. It's a very serious violation of RFC2060. This > > > "optimization" breaks > > > client optimizations. It breaks _most_ mail clients in > > > very subtle ways. > > > > > > People will complain. > > > > > > -- > > > Internet Connection High Quality Web Hosting > > > http://www.internetconnection.net/ > > > > > > > > > > _______________________________________________ > > Dbmail-dev mailing list > > Dbmail-dev@dbmail.org > > http://twister.fastxs.net/mailman/listinfo/dbmail-dev -- Internet Connection High Quality Web Hosting http://www.internetconnection.net/
