Hi, Sorry for the late reply.
As for: ``` I'm looking at the BrokerController and MessageStore implementation and hooks to understand where the merge logic will best fit. ``` how is it going? Regards On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <sowmya9...@gmail.com> wrote: > Thank you yukon, > > I'm done with my coursework for this semester and have more time now to > improve my proposal. > I'm looking at the BrokerController and MessageStore implementation and > hooks to understand where the merge logic will best fit. So far I've looked > at the codebase from a Producer and Consumer perspective and looked at the > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for understanding the > link between how Producers send and Consumers receive messages. > > thanks, > Sowmya > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote: > > > Cool, let's focus on it and see whether is there anything can be > polished. > > > > Regards > > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <sowmya9...@gmail.com> wrote: > > > > > Hey yukon, > > > I submitted my draft on the summer of code homepage a couple of days > ago, > > > also attaching the link here for reference, > > > > > > https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit? > > > usp=docslist_api&filetype=msword > > > > > > Thanks, > > > Sowmya > > > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote: > > > > > > > Hi, > > > > > > > > Of course, we can work together to finetune your design draft. > > > > > > > > Regards, > > > > yukon > > > > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <sowmya9...@gmail.com> > > wrote: > > > > > > > > > Hello, yukon and Von, > > > > > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking forward > to > > > > > working with all of you to finetune the proposal. I will be > > allocating > > > 20 > > > > > hours per week from now to the proposal acceptance phase to address > > > > > questions and dive deep into any suggestions that you provide. > > > > > I am really looking forward to work on this project. > > > > > > > > > > thanks, > > > > > Sowmya > > > > > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <sowmya9...@gmail.com> > > > wrote: > > > > > > > > > > > Hello yukon, > > > > > > > > > > > > Thank you for the inputs. I was able to look at the ~/store and > > kind > > > of > > > > > > understand the storage structure. I also looked at the > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the > > > > > examples. > > > > > > Now I understand why you proposed a merge sort like approach for > > > > > > performing global ordering. Since the proposals are open, I am > > > > finalizing > > > > > > my draft and will have it up for review very soon. > > > > > > > > > > > > thanks, > > > > > > Sowmya > > > > > > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote: > > > > > > > > > > > >> Hi Sowmya, > > > > > >> > > > > > >> ``` > > > > > >> Also, it would be great if you can at a high level, help me > > > understand > > > > > >> how the messages in the message queues are stored before the > > > consumer > > > > > reads > > > > > >> them. > > > > > >> ``` > > > > > >> > > > > > >> > > > > > >> > > > > > >> As shown in this figure, messages are sent to brokers by > producer > > > and > > > > > >> stored in commit log[1], then messages are dispatched to > > > ConsumeQueue > > > > by > > > > > >> topic, the consumer pulls messages from the queue. > > > > > >> > > > > > >> I recommend you run a broker and send/consume some messages, > then > > > > check > > > > > >> out the ~/store directory for details. > > > > > >> > > > > > >> Regards, > > > > > >> yukon > > > > > >> > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <sowmya9...@gmail.com > > > > > > wrote: > > > > > >> > > > > > >>> Hello yukon, > > > > > >>> > > > > > >>> Currently FIFO can be achieved with a producer sending to one > > > message > > > > > >>> queue, and when global ordering is required, multiple producers > > > have > > > > to > > > > > >>> send to a single topic queue. > > > > > >>> > > > > > >>> We want to allow multiple producers to send messages on a topic > > to > > > > > >>> multiple message queues and still provide ordering guarantees > to > > > the > > > > > >>> consumer, so that all consumers see the same order of data and > > also > > > > the > > > > > >>> data is delivered in an ordered fashion. > > > > > >>> > > > > > >>> 1) Your idea of using a merge sort with the assumption that the > > > first > > > > > >>> arriving message is treated as the first message to be > delivered, > > > > > however, > > > > > >>> I want to propose an approach where when the producer sends a > > > message > > > > > to a > > > > > >>> message queue, it must be done in a synchronous fashion and the > > > > > response > > > > > >>> will be that the message is accepted, which means that the > > message > > > > > follows > > > > > >>> the convention that all messages delivered prior by that > producer > > > > have > > > > > been > > > > > >>> stored across groups and if not the producer will need to > resend > > > the > > > > > >>> message. > > > > > >>> > > > > > >>> We can use a variant of total causal ordering in the layer > > between > > > > the > > > > > >>> message queue and store. > > > > > >>> > > > > > >>> I have been busy with my class project so I couldn't make a lot > > of > > > > > >>> progress in detailing my approach. Also, it would be great if > you > > > can > > > > > at a > > > > > >>> high level, help me understand how the messages in the message > > > queues > > > > > are > > > > > >>> stored before the consumer reads them. > > > > > >>> > > > > > >>> Does the consumer read directly from the message queue that the > > > > > producer > > > > > >>> sends data to? does the broker receive the queued producer > > > messages, > > > > > store > > > > > >>> them and then pushes them to another queue for the consumer to > > read > > > > > from? > > > > > >>> > > > > > >>> For reference on total causal ordering: > > > > https://www.cl.cam.ac.uk/teach > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf > > > > > >>> > > > > > >>> thanks, > > > > > >>> Sowmya > > > > > >>> > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> > wrote: > > > > > >>> > > > > > >>>> Hi Sowmya, > > > > > >>>> > > > > > >>>> Sorry for the late reply, do you have any update on this > > project? > > > > > >>>> > > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I > > > > proposed > > > > > >>>> a simple solution that performs merge sort on multiple queues > to > > > > > improve > > > > > >>>> performance and scalability. > > > > > >>>> > > > > > >>>> While the order issue across producers is difficult, we could > > > assume > > > > > >>>> that the message first arrives the broker should be consumed > > > > firstly. > > > > > >>>> > > > > > >>>> But it would be wonderful if you have a real design about the > > > order > > > > > >>>> issue across producers based on the RocketMQ design and the > > > storage > > > > > >>>> structure. > > > > > >>>> > > > > > >>>> Looking forward your design ~ > > > > > >>>> > > > > > >>>> Regards, > > > > > >>>> yukon > > > > > >>>> > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s < > sowmya9...@gmail.com> > > > > > wrote: > > > > > >>>> > > > > > >>>>> Hello all, > > > > > >>>>> > > > > > >>>>> Adding a few more thoughts on the problem of establishing an > > > order > > > > of > > > > > >>>>> messages across producers. > > > > > >>>>> > > > > > >>>>> Consider the Scenario > > > > > >>>>> > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue > Queue1 > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2. > > > > > >>>>> > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and > similarly > > > > > >>>>> time(a2) < time(b2) < time(c2) > > > > > >>>>> > > > > > >>>>> Are the following orders acceptable to the consumer? > > > > > >>>>> > > > > > >>>>> > a1, a2, b1, b2, c1, c2 > > > > > >>>>> > a1, b1, c1, a2, b2, c2 > > > > > >>>>> > a1 b1, b2, a2, a3, b3 > > > > > >>>>> > > > > > >>>>> and an order displayed at one consumer is consistent across > all > > > > > >>>>> consumers. > > > > > >>>>> > > > > > >>>>> This can be achieved using Total Causal Ordering at the > > Producer > > > or > > > > > >>>>> Queue level, using Leslie Lamport's clock and synchronization > > > > > approach. > > > > > >>>>> > > > > > >>>>> For reference is the paper attached, > > > > http://lamport.azurewebsites.n > > > > > >>>>> et/pubs/time-clocks.pdf > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> -- > > > > > >>>>> Regards, > > > > > >>>>> > > > > > >>>>> Sowmya > > > > > >>>>> > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s < > > sowmya9...@gmail.com> > > > > > >>>>> wrote: > > > > > >>>>> > > > > > >>>>>> Hello, > > > > > >>>>>> > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122 > > > > > >>>>>> < > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> > > > > > as > > > > > >>>>>> a part of Google Summer of Code 2018. I've been spending > some > > > time > > > > > to > > > > > >>>>>> understand the system, architecture and the existing > Messaging > > > > > Patterns. > > > > > >>>>>> I still have a few questions and would like to clarify my > > > > > assumptions. > > > > > >>>>>> > > > > > >>>>>> - Is the current FIFO order example limited to one > message > > > > queue > > > > > >>>>>> per producer? Can the producer send the same message to > > > > multiple > > > > > queues? > > > > > >>>>>> Will the consumers of the queues be able to read messages > > in > > > > > Order? > > > > > >>>>>> - Can I assume that each producer will send messages to > one > > > > > queue? > > > > > >>>>>> - Global Order is to be identified across all > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be used > > for > > > > > global order) > > > > > >>>>>> instances that are running. > > > > > >>>>>> - I think using a global clock can help establish the > order > > > > > >>>>>> between 2 or more producers, however using some form of > > > vector > > > > > >>>>>> clock might also help identify the global order of > messages > > > > > between the > > > > > >>>>>> producers. > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to > read > > > > > >>>>>> globally ordered messages) can then compare messages > across > > > all > > > > > message > > > > > >>>>>> queues from the corresponding producers and extract the > > > > > messages. [ this is > > > > > >>>>>> the approach recommended by yukon on the issue page ] > > > > > >>>>>> - We can also potentially have another layer in the > Message > > > > Queue > > > > > >>>>>> which accumulates all messages sent from producers and > > > provides > > > > > one ordered > > > > > >>>>>> message queue for consumers to read from. > > > > > >>>>>> > > > > > >>>>>> Thank you for your patience and please let me know if my > > > > > >>>>>> understanding of the problem and the assumptions are right. > > > > > >>>>>> > > > > > >>>>>> Best Regards, > > > > > >>>>>> Sowmya > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> -- > > > > > >>>>> Regards, > > > > > >>>>> > > > > > >>>>> Sowmya > > > > > >>>>> > > > > > >>>>> > > > > > >>>> > > > > > >>> > > > > > >>> > > > > > >>> -- > > > > > >>> Regards, > > > > > >>> > > > > > >>> Sowmya > > > > > >>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Sowmya > > > > > > > > > > > > -- > > > Regards, > > > > > > Sowmya > > > > > > > > > -- > Regards, > > Sowmya >
