Hi Robert, While traveling back from holidays, I took some time to write a quick blog post on the topic :
http://blog.btellier.com/article/72 If you are interrested... And if you have any question, don't hesitate to ask me directly. Benoit Le 02/01/2016 04:06, Robert Munn a écrit : > Benoit, > > This is very interesting work, thank you for contributing. I am interested in > learning more about the event system. > > Robert > > >> On Nov 28, 2015, at 10:09 AM, Tellier Benoit <btell...@apache.org> wrote: >> >> Hi, >> >> I just wanted to present my work on James event system. >> >> ## What is James event system ? >> >> The mailbox event system conveys notifications about modifications of >> the mailboxes and messages states. You can register listener to it so >> that you can be notified. >> >> ## What it is used for ? >> >> It is used for : >> >> - IMAP IDLE : allow one to subscribe to a specific mailbox and gets >> notified about changes without to pull the mailbox. >> >> - Quota system : updates about stored quota are made outside the >> MailboxManager as it may involve large quota calculations >> >> - Indexing of messages for the Search feature (ElasticSearch and Lucene >> implementation ) >> >> - IMAP Sequence Number handling. >> >> - Cache invalidation (caching project, not yet exposed to configuration) >> >> - Many others >> >> ## Why do we need it to be distributed ? >> >> I want to see this feature distributed as I personally really love IDLE >> feature. I want my Thunderbird to be allowed to use this in a >> distributed environment. >> >> I also think one might be interested to make several James work in >> parallel with any kind of architecture (Quotas, messages search indexes). >> >> ## What are different configuration options ? >> >> I reviewed the event system. >> >> First thing is to explicitly specify a listener distributed status. It >> can be either : >> >> - Registered per mailbox >> - The listener needs just to be notified about all local events >> - The listener needs to be notified about all events in your James cluster. >> >> Then, we keep the in memory default implementation (little reworked >> using guava). And I added two other architectures for the event system. >> >> #### Registration based event system >> >> With this implementation, you want to exchange events on the network. >> You want a James system to be only notified about events it explicitly >> registered to. Because of that : >> >> - This approach is thought for architecture with a large number of >> James server >> - It does not support event listener that needs to be notified of all >> events in the cluster. >> >> Each server listens on a message queue and a registration mechanism is >> used to identify to which server we need to send the events. Of course >> you have event serialization / deserialization. >> >> Today : >> - Kafka is used for the messaging >> - Cassandra is used for registration management >> >> This solution was presented at Paris Cassandra Meet-up. >> >> #### Broadcast event system >> >> With this implementation, you want to have several James working >> together but you relies on Mailbox Listeners that needs to be notified >> about every event in your data center. >> >> These listeners could be : >> >> - Lucene document indexing >> - In memory quotas >> - In memory cache >> >> The idea here is to naively broadcast the events to all your James. They >> are notified about every events (so scalability will be limited). >> >> You also have to be aware that events can be duplicated /non emitted >> (james server crash, network partitions) so local data might be >> inconsistent. It seems OK for instance for quota calculation. >> >> ## What do I need to know as an administrator ? >> >> Distributed use of Message Sequence Number (that demands high degree of >> coordination) is risky. The inconsistency window between server may be >> large, and the corresponding between UID and message sequence number is >> not eventually consistent. This topic is in discussion on the dev >> mailing list. >> >> I corrected an issue I spotted month before : a faulty mailbox listener >> might stop the event delivery chain and generate IMAP service >> unavailability. I added a commit to not propagate errors inside mailbox >> Listeners. >> >> I want to finish this section by speaking of event serialization. You >> can either choose : >> >> - JSON >> - MessagePack >> >> The first one is faster to compute but larger. So it let you trade >> compute power versus network. >> >> ## Event delivery modes >> >> As you might have noticed, Mailbox Listener can take a long time to >> execute, and for some of them, they can safely be executed >> asynchronously (IDLE, indexation and even quotas). >> >> I added an Event Delivery abstraction. Thanks to this, you can configure >> your James to : >> >> - Synchronously deliver events (todays behavior) >> - Asynchronously deliver events ( returns before having delivered >> events, Mailbox Listener are notified in parallel in a thread pool) >> - Mixed mode : Every Mailbox Listener indicates if it should be >> synchronously or asynchronously executed. >> >> The asynchronous option can be considered as risky. The mixed one is >> safe, and significantly reduces latencies if you rely on document indexing. >> >> ## Re indexers >> >> I also added the availability to re index documents in a Message Search >> index using the CLI : >> >> - per mailbox : the event system is used to track changes made to the >> given mailbox and significantly reduce the concurrent changes window. >> - your whole James mailboxes : the event system is used to keep track >> of deleted mailboxes. >> >> ## My future works on the event system. >> >> Finish the work on MAILBOX-257 : one should be able to recalculate quotas. >> >> Unfortunately it is not yet planned in my todo list... >> >> Benoit >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org >> For additional commands, e-mail: server-user-h...@james.apache.org >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org > For additional commands, e-mail: server-user-h...@james.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org
