Hi, Actually, Hazelcast gives up consistency upon network partition. This mean any implementation will experience random behavior when using MESSAGE SEQUENCE number during a partition (and this is also true for deletions). In my opinion, it does not really add guaranties compoared to Cassandra, (maybe reduce inconsistency window as operations are expected to be faster)
Moreover, I don't really like the idea of adding yet an other middleware. Note : Hazelcast is a great option if we want to implement a distributed cache mechanism. I guess this is an other issue. Le 10/11/2015 13:47, Eric Charles a écrit : > Would a distributed im-memory backend such as hazelcast be an option to > start? > > http://docs.hazelcast.org/docs/3.5/manual/html/licenses.html > > On 2015-10-26 14:51, Benoit Tellier wrote: >> >> >> Le 26/10/2015 14:08, Matthieu Baechler a écrit : >>> >>> >>> On 26/10/2015 13:53, Benoit Tellier wrote: >>> >>> [...] >>> >>>>> I think we could create a Session row that map MESSAGE SEQUENCE NUMBER >>>>> to the real messages. Then, for a given session, we never remap >>>>> things, >>>>> we only add messages to this row. In case of EXPUNGE, we create a new >>>>> row that won't be used until a new session is open. >>>> >>>> Message Sequence Number are defined as mutable by the RFC. Having them >>>> being immutable look like non RFC compliant to me. >>>> >>>> I mean, the idea is nice, but client using MESSAGE SEQUENCE NUMBERS >>>> won't be designed to work this way, I think. >>>> >>>> https://tools.ietf.org/html/rfc3501#section-2.3.1.2 >>> >>> Let's quote it : >>> >>> Message sequence numbers can be reassigned during the session. For >>> example, when a message is permanently removed (expunged) from the >>> mailbox, the message sequence number for all subsequent messages is >>> decremented. The number of messages in the mailbox is also >>> decremented. Similarly, a new message can be assigned a message >>> sequence number that was once held by some other message prior to an >>> expunge. >>> >>> The only way to reassign Message sequence numbers is EXPUNGED messages. >> >> I agree on your last sentence : "The only way to reassign Message >> sequence numbers is EXPUNGED messages." >> >> The criticism was on this sentence : "In case of EXPUNGE, we create a >> new row that won't be used until a new session is open." >> >> It seems opposed to the RFC, where, from my understanding, the client >> can assume the message sequence number for all subsequent messages to be >> decremented right after his request. >> >>> >>>>> >>>>> That way, a client can't use a MESSAGE SEQUENCE NUMBER wrongly because >>>>> it always maps to a given message. >>>> >>>> With the model you suggest, I understand that a Session Row stops from >>>> being used only when a clients is disconnected. It represents his view >>>> of the correspondence on his mailbox selection moment. He use it until >>>> he is disconnected. >>>> >>>> Is this what you suggest ? >>> >>> Somehow. >>> >>>>> Do you get it ? >>>>> >>>> >>>> Hence, as it is immutable, he can not address added messages after this >>>> selection using MESSAGE SEQUENCE NUMBERS as these changes will be >>>> visible upon re-selection. >>>> >>>> Look like a problem to me... >>> >>> That's not correct : you can add messages to this view. >> >> You are right. I thought, from what you explained that message were only >> added to the future session. >> >>> >>>> Also, you have data races unless you manage APPEND and EXPUNGE >>>> operations in atomic way. This means MESSAGE SEQUENCE NUMBERS should be >>>> included in the process. This means, to have a data race free >>>> implementation, we need the mailbox manager to handle MESSAGE SEQUENCE >>>> NUMBER consistency, and not the protocol part, that did it threw the >>>> event system. Do we agree on this ? >>> >>> What race condition do you think about ? I mean, any mapping should be >>> ok, it can even be generated asynchronously as long as you assign an >>> existing mapping to a given session. >> >> I meant this. >> >> Todays implementation works in two steps : >> >> 1 : Update the database >> 2 : From the event generated from the database, update the mapping. >> >> Consider the following mapping : >> >> UID | 2 | 4 | 6 | 8 | >> --------------------- >> MSN | 1 | 2 | 3 | 4 | >> >> Bobs issue these commands, in this order : >> >> Command 1 : EXPUNGE 1 >> Command 2 : EXPUNGE 2 >> >> The expected result is : >> >> UID | 4 | 8 | >> ------------- >> MSN | 1 | 2 | >> >> Because the client can assume that after executing command 1 mapping >> will be : >> >> UID | 4 | 6 | 8 | >> ----------------- >> MSN | 1 | 2 | 3 | >> >> Now, because the mapping update and the EXPUNGE completion are >> dissociated, mapping update might not have been done (delay in event >> delivery, a long MailboxListener preceding the SelectedMailboxImpl, >> etc... ) >> >> Hence the result might look like : >> >> UID | 6 | 8 | >> ------------- >> MSN | 1 | 2 | >> >> We have deleted the wrong message. >> >> Now, reading this : https://tools.ietf.org/html/rfc3501#section-5.5 >> >> You might argue that these two operations are ambiguous... >> >> However, we have two possible outcomes out of this commands sequence >> depending on the timing. This is what I call a data race. >> >> If you want, I can write an MPT test for this. >> >>> >>> >>>> -------------------- >>>> >>>> A friend (Erwan Guyomarc'h), who is working on James with me, suggested >>>> an other solution. It has its problems, but I put it here as it is >>>> interesting... >>>> >>>> Suppose we have a at least one delivery distributed event system. >>>> >>>> The idea is, instead of maintaining a double index between UID and >>>> Message Sequence Number, we only store on each SelectedMailboxImpl an >>>> uid set. >>>> >>>> ADDED events is an insertion on this set >>>> EXPUNGE events is a deletion on this set >>>> >>>> Of course these operations are not CRDT. >>>> >>>> The problem comes from ADDED operation being reordered after EXPUNGED >>>> operations. >>>> >>>> ADDEDD 7 >>>> EXPUNGED 7 >>>> >>>> is not equivalent to >>>> >>>> EXPUNGED 7 >>>> ADDED 7 >>>> >>>> Now assume we have a causal order. >>>> We also have the certitude to see only one ADDED event per UID. >>>> If we reject EXPUNGE commands if the specified uid is absent from the >>>> SelecteMailboxImpl uid set. >>>> >>>> We might have concurrent problems, but with this we have an eventual >>>> consistent MESSAGE SEQUENCE NUMBERS <=> UID correspondance across >>>> servers. Of course with this solution the difficulty is to have causal >>>> ordering. Which means vector clocks... >>> >>> >>> It means creating the set for every selected mailbox : it looks like a >>> performance killer, don't you think ? >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
