Disclaimer : the following message is long, and might take time to read,
but I think this is a topic we have to exchange on in order to have a
working James in a distributed environment...
=============================================================
Hi every one,
I am working on a distributed event system for the mailbox. The idea is
to make the IDLE functionality work with an arbitrary large number of
servers.
To work on this feature, I had a look to the different types of
MailboxListener. I found this one : SelectedMailboxImpl.
SelectedMailboxImpl is stateful : it maintains (among other things) the
correspondence between UIDs and Message Sequence Number ( called message
index in SelectedMailbox ).
When the mailbox is first selected, all the UIDs and flags for message
in this mailbox are fetch and used to compute the UID for the given
Message Sequence Number. The actual index is event updated using the
mailbox event system.
Well, I have troubles to see how to make this work in a distributed
system. Message systems do not offer perfect guaranties and we might get
a lot of troubles in case of network partitions. Double event delivery,
or no event delivery at all might arise. It is not that bad with IDLE,
but can lead SelectedMailboxImpl to be inconsistent. I guess we have
several options :
# Go stateless on this (note : only for distributed implementation)
## Option 1
- We can recompute the correspondence between UID and Message
Sequence Number each time a message sequence number is used. It might
cost compute and network resources. But no node stores specific data. We
can imagine give a configuration option, that gives the choice between
the two options.
Depending on the implementation, we get different trade-off :
Option 1 a)
On a read request using Message Sequence Number, select all the
data (all the message content) from the database, and select the message
we want. It is consistent but highly ineffective. Note : this works for
Reads, but not deletions...
Option 1 b)
On a read request we first fetch the mailbox to have informations
about the UID to be used, and then gets the message data. But due to
delay between read and write, our information can become inconsistent.
Thus we can do some serious damages (eg : delete the wrong message)
## Option 2
- Store the message sequence number and update it. To do this in a
consistent way, we need a CP data store with the notion of transaction
and attach the Message Sequence Number directly to the Message, stored
in the database. It works, we have ineffective queries like **UPDATE
messages SET seq_number = seq_number - 1 WHERE seq_number > deleted AND
mailboxId=159**. We might have to handle transaction that fails to commit.
Option 2 is still dangerous on databases that lacks the notion of
transaction. For example a process can crash before updating the
sequence number. On Cassandra and other AP data stores, we have
consistencies problems on concurrent updates of sequence number (might
lead to a wrong result, and even messages having the same sequence number).
Other note : adding a message requires to know the last UID used for a
mailbox that stills correspond to an existing message. Here we have two
options : store it or recompute it. In both case we have troubles
without the notion of transactions.
# Keep it stateful
## Option 3 :
- Say that a mailbox should be handled by a single James. This
can't be achieved threw load balancing. Here is a little user story that
shows it :
Bob and Alice uses James.
Bob give the right to Alice to see and add messages to his INBOX
Bob and Alice are handled by different servers. Bob is on James 1 and
Alice on James 2. All there new connections will go to these server.
Bob connects to his INBOX. Alice connects to Bob's INBOX. Problem.
Collocating Bob and Alice ( and more generally user sharing rights) is
hard but also impossible, as this graph might not be disconnected.
The solution with this option might be to proxy Alice commands related
to Bob INBOX selection to Bob's server. To do this, we need to know
which server is in charge of Bob. A solution might be Consistent
Hashing, but it is complicated to implement. Without such a thing,
sharing mailboxes across users might be difficult.
With this option we have other concerns :
- How is our solution better than a Cyrus implementation if we have
the same load balancing troubles ?
- We should document how to deploy such a load balancer
Obviously, with this, we do not need the distributed event system...
## Option 4
Don't care. I don't like this one. Even if inconsistent sequence numbers
have a session lifetime, it can lead to critical scenarios like deleting
the wrong e-mail.
## Option 5
Don't create a distributed implementation. Well, as my involvement on
James project is about to make it distributed, I definitely down vote
this option.
# Avoid the problem
## Option 6
On distributed implementation, partly implement IMAP and refuse requests
based on sequence numbers. Seems stupid, but this is the only
implementation that works safely with Cassandra that comes to my mind...
# Option 7
If you have other ideas...
