Hello yukon, Currently FIFO can be achieved with a producer sending to one message queue, and when global ordering is required, multiple producers have to send to a single topic queue.
We want to allow multiple producers to send messages on a topic to multiple message queues and still provide ordering guarantees to the consumer, so that all consumers see the same order of data and also the data is delivered in an ordered fashion. 1) Your idea of using a merge sort with the assumption that the first arriving message is treated as the first message to be delivered, however, I want to propose an approach where when the producer sends a message to a message queue, it must be done in a synchronous fashion and the response will be that the message is accepted, which means that the message follows the convention that all messages delivered prior by that producer have been stored across groups and if not the producer will need to resend the message. We can use a variant of total causal ordering in the layer between the message queue and store. I have been busy with my class project so I couldn't make a lot of progress in detailing my approach. Also, it would be great if you can at a high level, help me understand how the messages in the message queues are stored before the consumer reads them. Does the consumer read directly from the message queue that the producer sends data to? does the broker receive the queued producer messages, store them and then pushes them to another queue for the consumer to read from? For reference on total causal ordering: https://www.cl.cam.ac.uk/teaching/0910/ConcDistS/10b-ProcGp-order.pdf thanks, Sowmya On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote: > Hi Sowmya, > > Sorry for the late reply, do you have any update on this project? > > In RocketMQ, one message queue is a FIFO queue natively, so I proposed a > simple solution that performs merge sort on multiple queues to improve > performance and scalability. > > While the order issue across producers is difficult, we could assume that > the message first arrives the broker should be consumed firstly. > > But it would be wonderful if you have a real design about the order issue > across producers based on the RocketMQ design and the storage structure. > > Looking forward your design ~ > > Regards, > yukon > > On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <sowmya9...@gmail.com> wrote: > >> Hello all, >> >> Adding a few more thoughts on the problem of establishing an order of >> messages across producers. >> >> Consider the Scenario >> >> Producer-1 produces messages a1, b1 and c1 into a queue Queue1 >> Producer-2 produces messages a2, b2 and c2 into queue Queue2. >> >> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2) < >> time(b2) < time(c2) >> >> Are the following orders acceptable to the consumer? >> >> > a1, a2, b1, b2, c1, c2 >> > a1, b1, c1, a2, b2, c2 >> > a1 b1, b2, a2, a3, b3 >> >> and an order displayed at one consumer is consistent across all >> consumers. >> >> This can be achieved using Total Causal Ordering at the Producer or Queue >> level, using Leslie Lamport's clock and synchronization approach. >> >> For reference is the paper attached, http://lamport.azurewebsites.n >> et/pubs/time-clocks.pdf >> >> >> -- >> Regards, >> >> Sowmya >> >> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <sowmya9...@gmail.com> wrote: >> >>> Hello, >>> >>> I'm trying to work on the issue ROCKETMQ-122 >>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as >>> a part of Google Summer of Code 2018. I've been spending some time to >>> understand the system, architecture and the existing Messaging Patterns. >>> I still have a few questions and would like to clarify my assumptions. >>> >>> - Is the current FIFO order example limited to one message queue per >>> producer? Can the producer send the same message to multiple queues? Will >>> the consumers of the queues be able to read messages in Order? >>> - Can I assume that each producer will send messages to one queue? >>> - Global Order is to be identified across all GlobalOrderedProducer >>> (a new producer that is to be used for global order) instances that are >>> running. >>> - I think using a global clock can help establish the order between >>> 2 or more producers, however using some form of vector clock might >>> also help identify the global order of messages between the producers. >>> - A GlobalOrderedConsumer ( consumer that knows how to read globally >>> ordered messages) can then compare messages across all message queues >>> from >>> the corresponding producers and extract the messages. [ this is the >>> approach recommended by yukon on the issue page ] >>> - We can also potentially have another layer in the Message Queue >>> which accumulates all messages sent from producers and provides one >>> ordered >>> message queue for consumers to read from. >>> >>> Thank you for your patience and please let me know if my understanding >>> of the problem and the assumptions are right. >>> >>> Best Regards, >>> Sowmya >>> >>> >> >> >> -- >> Regards, >> >> Sowmya >> >> > -- Regards, Sowmya
