Hey yukon, I submitted my draft on the summer of code homepage a couple of days ago, also attaching the link here for reference,
https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?usp=docslist_api&filetype=msword Thanks, Sowmya On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote: > Hi, > > Of course, we can work together to finetune your design draft. > > Regards, > yukon > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <sowmya9...@gmail.com> wrote: > > > Hello, yukon and Von, > > > > I've shared my GSOC - 18 draft of the project. I'm looking forward to > > working with all of you to finetune the proposal. I will be allocating 20 > > hours per week from now to the proposal acceptance phase to address > > questions and dive deep into any suggestions that you provide. > > I am really looking forward to work on this project. > > > > thanks, > > Sowmya > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <sowmya9...@gmail.com> wrote: > > > > > Hello yukon, > > > > > > Thank you for the inputs. I was able to look at the ~/store and kind of > > > understand the storage structure. I also looked at the > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the > > examples. > > > Now I understand why you proposed a merge sort like approach for > > > performing global ordering. Since the proposals are open, I am > finalizing > > > my draft and will have it up for review very soon. > > > > > > thanks, > > > Sowmya > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote: > > > > > >> Hi Sowmya, > > >> > > >> ``` > > >> Also, it would be great if you can at a high level, help me understand > > >> how the messages in the message queues are stored before the consumer > > reads > > >> them. > > >> ``` > > >> > > >> > > >> > > >> As shown in this figure, messages are sent to brokers by producer and > > >> stored in commit log[1], then messages are dispatched to ConsumeQueue > by > > >> topic, the consumer pulls messages from the queue. > > >> > > >> I recommend you run a broker and send/consume some messages, then > check > > >> out the ~/store directory for details. > > >> > > >> Regards, > > >> yukon > > >> > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <sowmya9...@gmail.com> > wrote: > > >> > > >>> Hello yukon, > > >>> > > >>> Currently FIFO can be achieved with a producer sending to one message > > >>> queue, and when global ordering is required, multiple producers have > to > > >>> send to a single topic queue. > > >>> > > >>> We want to allow multiple producers to send messages on a topic to > > >>> multiple message queues and still provide ordering guarantees to the > > >>> consumer, so that all consumers see the same order of data and also > the > > >>> data is delivered in an ordered fashion. > > >>> > > >>> 1) Your idea of using a merge sort with the assumption that the first > > >>> arriving message is treated as the first message to be delivered, > > however, > > >>> I want to propose an approach where when the producer sends a message > > to a > > >>> message queue, it must be done in a synchronous fashion and the > > response > > >>> will be that the message is accepted, which means that the message > > follows > > >>> the convention that all messages delivered prior by that producer > have > > been > > >>> stored across groups and if not the producer will need to resend the > > >>> message. > > >>> > > >>> We can use a variant of total causal ordering in the layer between > the > > >>> message queue and store. > > >>> > > >>> I have been busy with my class project so I couldn't make a lot of > > >>> progress in detailing my approach. Also, it would be great if you can > > at a > > >>> high level, help me understand how the messages in the message queues > > are > > >>> stored before the consumer reads them. > > >>> > > >>> Does the consumer read directly from the message queue that the > > producer > > >>> sends data to? does the broker receive the queued producer messages, > > store > > >>> them and then pushes them to another queue for the consumer to read > > from? > > >>> > > >>> For reference on total causal ordering: > https://www.cl.cam.ac.uk/teach > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf > > >>> > > >>> thanks, > > >>> Sowmya > > >>> > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote: > > >>> > > >>>> Hi Sowmya, > > >>>> > > >>>> Sorry for the late reply, do you have any update on this project? > > >>>> > > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I > proposed > > >>>> a simple solution that performs merge sort on multiple queues to > > improve > > >>>> performance and scalability. > > >>>> > > >>>> While the order issue across producers is difficult, we could assume > > >>>> that the message first arrives the broker should be consumed > firstly. > > >>>> > > >>>> But it would be wonderful if you have a real design about the order > > >>>> issue across producers based on the RocketMQ design and the storage > > >>>> structure. > > >>>> > > >>>> Looking forward your design ~ > > >>>> > > >>>> Regards, > > >>>> yukon > > >>>> > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <sowmya9...@gmail.com> > > wrote: > > >>>> > > >>>>> Hello all, > > >>>>> > > >>>>> Adding a few more thoughts on the problem of establishing an order > of > > >>>>> messages across producers. > > >>>>> > > >>>>> Consider the Scenario > > >>>>> > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1 > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2. > > >>>>> > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly > > >>>>> time(a2) < time(b2) < time(c2) > > >>>>> > > >>>>> Are the following orders acceptable to the consumer? > > >>>>> > > >>>>> > a1, a2, b1, b2, c1, c2 > > >>>>> > a1, b1, c1, a2, b2, c2 > > >>>>> > a1 b1, b2, a2, a3, b3 > > >>>>> > > >>>>> and an order displayed at one consumer is consistent across all > > >>>>> consumers. > > >>>>> > > >>>>> This can be achieved using Total Causal Ordering at the Producer or > > >>>>> Queue level, using Leslie Lamport's clock and synchronization > > approach. > > >>>>> > > >>>>> For reference is the paper attached, > http://lamport.azurewebsites.n > > >>>>> et/pubs/time-clocks.pdf > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> Regards, > > >>>>> > > >>>>> Sowmya > > >>>>> > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <sowmya9...@gmail.com> > > >>>>> wrote: > > >>>>> > > >>>>>> Hello, > > >>>>>> > > >>>>>> I'm trying to work on the issue ROCKETMQ-122 > > >>>>>> < > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> > > as > > >>>>>> a part of Google Summer of Code 2018. I've been spending some time > > to > > >>>>>> understand the system, architecture and the existing Messaging > > Patterns. > > >>>>>> I still have a few questions and would like to clarify my > > assumptions. > > >>>>>> > > >>>>>> - Is the current FIFO order example limited to one message > queue > > >>>>>> per producer? Can the producer send the same message to > multiple > > queues? > > >>>>>> Will the consumers of the queues be able to read messages in > > Order? > > >>>>>> - Can I assume that each producer will send messages to one > > queue? > > >>>>>> - Global Order is to be identified across all > > >>>>>> GlobalOrderedProducer (a new producer that is to be used for > > global order) > > >>>>>> instances that are running. > > >>>>>> - I think using a global clock can help establish the order > > >>>>>> between 2 or more producers, however using some form of vector > > >>>>>> clock might also help identify the global order of messages > > between the > > >>>>>> producers. > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read > > >>>>>> globally ordered messages) can then compare messages across all > > message > > >>>>>> queues from the corresponding producers and extract the > > messages. [ this is > > >>>>>> the approach recommended by yukon on the issue page ] > > >>>>>> - We can also potentially have another layer in the Message > Queue > > >>>>>> which accumulates all messages sent from producers and provides > > one ordered > > >>>>>> message queue for consumers to read from. > > >>>>>> > > >>>>>> Thank you for your patience and please let me know if my > > >>>>>> understanding of the problem and the assumptions are right. > > >>>>>> > > >>>>>> Best Regards, > > >>>>>> Sowmya > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> Regards, > > >>>>> > > >>>>> Sowmya > > >>>>> > > >>>>> > > >>>> > > >>> > > >>> > > >>> -- > > >>> Regards, > > >>> > > >>> Sowmya > > >>> > > >>> > > >> > > > > > > > > > -- > > > Regards, > > > > > > Sowmya > > > > > > > > > > > > -- > > Regards, > > > > Sowmya > > > -- Regards, Sowmya
