Hello there, The Apache James distributed server has the ambition to scale seamlessly and relies on proven distributed technologies in order to do so (Cassandra, ElasticSearch, S3).
However, we currently rely on RabbitMQ for the messaging part, and despite the addition of Quorum queues in 3.8.0 version (not yet leveraged by James BTW) scalability and distributed set up is still a concern. Here are (some) pitfalls of the current RabbitMQ implementation: - Manual error management (exponential retries and dead lettering) - Custom connection pooling - I had issues with connection recovery and had to reboot James - Complex logic to re-create exclusive queues upon reboot - Scalability of quorum queues - Needs of associated tables to 'view the queue content' - No support for delays. Among the alternatives to RabbitMQ, one stands out: Pulsar... - Handle 1.000.000 topics seamlessly thus can be used as a backbone for PUS/SUB - Scalable - exponential retries and dead-lettering is managed out of the box - As a distributed log, we can iterate the content (but we might still keep track of deleted content) - Handles delays - And it is another Apache TLP project! I know some other community members are some Apache Pulsar enthusiasts. Hence, I wonder if we can not leverage the opportunity of the Summer 2021 of Open Source ( https://summer.iscas.ac.cn/help/en/ ) to conduct a proof of concept regarding Pulsar integration within the James eco-system. (all Apache TLPs projects might be eligible) My views regarding that would be: - Start with a basic MailQueue implementation. Could be contributed on a feature branch / on an unused /server/queue/pulsar mvn project? - From there boostrap a custom MTA Guice assembling of James relying on pulsar. Could it be in /exemple section? - Move forward with an EventBus implementation. Again could be contributed on a feature branch / on an unused /eventbus/pulsar mvn project? - Make the custom MTA proof of concept evolve into a MDA (adding mailbox / jmap bindings). - Implement advanced MailQueue features with Pulsar (delays, browse, purge, flush, ...) - Bind the mailqueue management routes in the proof of concept server - Of course doing some performance tests for this... (This proof of concept can likely still rely on RabbitMQ to back the TaskManager implementation) Thoughts? Would there be people interested into mentoring this? Cheers, Benoit --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
