Hi Yukon, I am sorry for the confusion regarding the google doc, here is the link again, https://drive.google.com/file/d/1nXktUO_TF9- rSHSnGj5z5QZoHzMhosxm/view?usp=sharing
Regarding the hotpot issue, In my document, under the section Optimizing for performance, I had a similar approach to what you mentioned, If the Producer or the implementation of a Global Ordered Producer, attaches to the Message or message properties, an additional attribute which is an atomic counter that gets incremented for every message sent, in combination with the topic and the unique producer group, the messages get to potentially different brokers under different queues. >From the consumer perspective, having a custom RebalanceImplementation and also a custom Implementation for a MQConsumer which can be a Pull or push consumer that orders based on the property we set above in the Global Ordered Producer and sort by producer group, topic all the messages. thanks, Sowmya On Fri, Mar 23, 2018 at 4:48 AM, yukon <yu...@apache.org> wrote: > Hi, > > I added some comments, but it seems your google doc link isn't shared: > > ``` > To resolve this issue, we need a global ordered messaging mechanism to have > a good scalability without hot-pot issue. > > A practical proposal is that sending messages to distributed > queue/partition with an ordered key. Messages are ordered in the same > queue, then implement an OrderConsumer to pull messages from these queues > then perform merge sort, finally deliver these ordered messages to users. > ``` > As mentioned in ROCKETMQ-122, we should address the hot-pot issue. > > That means the messages can be sent to multiple distributed > queues/partitions and still can be consumed in order manner. > > A simple explanation of hot-pot issue: > > Assume that all the generated messages in a hot shop of taobao/amazon > should be consumed in order, in the current state, these massive messages > should be sent to a fixed queue/partition, surely to a fixed server which > may cause high pressure in this server. You can easily infer that we can't > resolve this issue through extending the scale of broker cluster. > > Could your arch resolve this hot-pot issue? > > Regards, > yukon > > > On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <sowmya9...@gmail.com> wrote: > > > It's going great. > > In the DefaultMessageStore, there is a class > > CommitLogDispatcherBuildConsumeQueue, I think that a variant of that > class > > or another implementation which are merged from the commitlog, > > Since the DispachRequest has a store timestamp, the queue can be > populated > > based on that time stamp. > > > > Also, I have some more questions, > > Does the fleet of broker slaves all store the various commit logs that > are > > received by them or is the commit log store, replication managed by the > > broker controller? > > > > The other direction I was thinking about is adding/modifying the > > PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the > > DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding > > this change there will solve the merge issue for the consumers. > > > > Do you think I am on the right track? > > > > thanks, > > Sowmya > > > > > > > > On Mon, Mar 19, 2018 at 8:15 PM, yukon <yu...@apache.org> wrote: > > > > > Hi, > > > > > > Sorry for the late reply. > > > > > > As for: > > > > > > ``` > > > I'm looking at the BrokerController and MessageStore implementation and > > > hooks to understand where the merge logic will best fit. > > > ``` > > > > > > how is it going? > > > > > > Regards > > > > > > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <sowmya9...@gmail.com> > wrote: > > > > > > > Thank you yukon, > > > > > > > > I'm done with my coursework for this semester and have more time now > to > > > > improve my proposal. > > > > I'm looking at the BrokerController and MessageStore implementation > and > > > > hooks to understand where the merge logic will best fit. So far I've > > > looked > > > > at the codebase from a Producer and Consumer perspective and looked > at > > > the > > > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for > understanding > > > the > > > > link between how Producers send and Consumers receive messages. > > > > > > > > thanks, > > > > Sowmya > > > > > > > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <yu...@apache.org> wrote: > > > > > > > > > Cool, let's focus on it and see whether is there anything can be > > > > polished. > > > > > > > > > > Regards > > > > > > > > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <sowmya9...@gmail.com> > > > wrote: > > > > > > > > > > > Hey yukon, > > > > > > I submitted my draft on the summer of code homepage a couple of > > days > > > > ago, > > > > > > also attaching the link here for reference, > > > > > > > > > > > > https://docs.google.com/file/d/1nXktUO_TF9- > > > rSHSnGj5z5QZoHzMhosxm/edit? > > > > > > usp=docslist_api&filetype=msword > > > > > > > > > > > > Thanks, > > > > > > Sowmya > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <yu...@apache.org> wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > Of course, we can work together to finetune your design draft. > > > > > > > > > > > > > > Regards, > > > > > > > yukon > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s < > sowmya9...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hello, yukon and Von, > > > > > > > > > > > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking > > > forward > > > > to > > > > > > > > working with all of you to finetune the proposal. I will be > > > > > allocating > > > > > > 20 > > > > > > > > hours per week from now to the proposal acceptance phase to > > > address > > > > > > > > questions and dive deep into any suggestions that you > provide. > > > > > > > > I am really looking forward to work on this project. > > > > > > > > > > > > > > > > thanks, > > > > > > > > Sowmya > > > > > > > > > > > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s < > > sowmya9...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello yukon, > > > > > > > > > > > > > > > > > > Thank you for the inputs. I was able to look at the ~/store > > and > > > > > kind > > > > > > of > > > > > > > > > understand the storage structure. I also looked at the > > > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used > in > > > the > > > > > > > > examples. > > > > > > > > > Now I understand why you proposed a merge sort like > approach > > > for > > > > > > > > > performing global ordering. Since the proposals are open, I > > am > > > > > > > finalizing > > > > > > > > > my draft and will have it up for review very soon. > > > > > > > > > > > > > > > > > > thanks, > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> > > > wrote: > > > > > > > > > > > > > > > > > >> Hi Sowmya, > > > > > > > > >> > > > > > > > > >> ``` > > > > > > > > >> Also, it would be great if you can at a high level, help > me > > > > > > understand > > > > > > > > >> how the messages in the message queues are stored before > the > > > > > > consumer > > > > > > > > reads > > > > > > > > >> them. > > > > > > > > >> ``` > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> As shown in this figure, messages are sent to brokers by > > > > producer > > > > > > and > > > > > > > > >> stored in commit log[1], then messages are dispatched to > > > > > > ConsumeQueue > > > > > > > by > > > > > > > > >> topic, the consumer pulls messages from the queue. > > > > > > > > >> > > > > > > > > >> I recommend you run a broker and send/consume some > messages, > > > > then > > > > > > > check > > > > > > > > >> out the ~/store directory for details. > > > > > > > > >> > > > > > > > > >> Regards, > > > > > > > > >> yukon > > > > > > > > >> > > > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s < > > > sowmya9...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >>> Hello yukon, > > > > > > > > >>> > > > > > > > > >>> Currently FIFO can be achieved with a producer sending to > > one > > > > > > message > > > > > > > > >>> queue, and when global ordering is required, multiple > > > producers > > > > > > have > > > > > > > to > > > > > > > > >>> send to a single topic queue. > > > > > > > > >>> > > > > > > > > >>> We want to allow multiple producers to send messages on a > > > topic > > > > > to > > > > > > > > >>> multiple message queues and still provide ordering > > guarantees > > > > to > > > > > > the > > > > > > > > >>> consumer, so that all consumers see the same order of > data > > > and > > > > > also > > > > > > > the > > > > > > > > >>> data is delivered in an ordered fashion. > > > > > > > > >>> > > > > > > > > >>> 1) Your idea of using a merge sort with the assumption > that > > > the > > > > > > first > > > > > > > > >>> arriving message is treated as the first message to be > > > > delivered, > > > > > > > > however, > > > > > > > > >>> I want to propose an approach where when the producer > > sends a > > > > > > message > > > > > > > > to a > > > > > > > > >>> message queue, it must be done in a synchronous fashion > and > > > the > > > > > > > > response > > > > > > > > >>> will be that the message is accepted, which means that > the > > > > > message > > > > > > > > follows > > > > > > > > >>> the convention that all messages delivered prior by that > > > > producer > > > > > > > have > > > > > > > > been > > > > > > > > >>> stored across groups and if not the producer will need to > > > > resend > > > > > > the > > > > > > > > >>> message. > > > > > > > > >>> > > > > > > > > >>> We can use a variant of total causal ordering in the > layer > > > > > between > > > > > > > the > > > > > > > > >>> message queue and store. > > > > > > > > >>> > > > > > > > > >>> I have been busy with my class project so I couldn't > make a > > > lot > > > > > of > > > > > > > > >>> progress in detailing my approach. Also, it would be > great > > if > > > > you > > > > > > can > > > > > > > > at a > > > > > > > > >>> high level, help me understand how the messages in the > > > message > > > > > > queues > > > > > > > > are > > > > > > > > >>> stored before the consumer reads them. > > > > > > > > >>> > > > > > > > > >>> Does the consumer read directly from the message queue > that > > > the > > > > > > > > producer > > > > > > > > >>> sends data to? does the broker receive the queued > producer > > > > > > messages, > > > > > > > > store > > > > > > > > >>> them and then pushes them to another queue for the > consumer > > > to > > > > > read > > > > > > > > from? > > > > > > > > >>> > > > > > > > > >>> For reference on total causal ordering: > > > > > > > https://www.cl.cam.ac.uk/teach > > > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf > > > > > > > > >>> > > > > > > > > >>> thanks, > > > > > > > > >>> Sowmya > > > > > > > > >>> > > > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> > > > > wrote: > > > > > > > > >>> > > > > > > > > >>>> Hi Sowmya, > > > > > > > > >>>> > > > > > > > > >>>> Sorry for the late reply, do you have any update on this > > > > > project? > > > > > > > > >>>> > > > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively, > > so > > > I > > > > > > > proposed > > > > > > > > >>>> a simple solution that performs merge sort on multiple > > > queues > > > > to > > > > > > > > improve > > > > > > > > >>>> performance and scalability. > > > > > > > > >>>> > > > > > > > > >>>> While the order issue across producers is difficult, we > > > could > > > > > > assume > > > > > > > > >>>> that the message first arrives the broker should be > > consumed > > > > > > > firstly. > > > > > > > > >>>> > > > > > > > > >>>> But it would be wonderful if you have a real design > about > > > the > > > > > > order > > > > > > > > >>>> issue across producers based on the RocketMQ design and > > the > > > > > > storage > > > > > > > > >>>> structure. > > > > > > > > >>>> > > > > > > > > >>>> Looking forward your design ~ > > > > > > > > >>>> > > > > > > > > >>>> Regards, > > > > > > > > >>>> yukon > > > > > > > > >>>> > > > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s < > > > > sowmya9...@gmail.com> > > > > > > > > wrote: > > > > > > > > >>>> > > > > > > > > >>>>> Hello all, > > > > > > > > >>>>> > > > > > > > > >>>>> Adding a few more thoughts on the problem of > establishing > > > an > > > > > > order > > > > > > > of > > > > > > > > >>>>> messages across producers. > > > > > > > > >>>>> > > > > > > > > >>>>> Consider the Scenario > > > > > > > > >>>>> > > > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue > > > > Queue1 > > > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue > > > Queue2. > > > > > > > > >>>>> > > > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and > > > > similarly > > > > > > > > >>>>> time(a2) < time(b2) < time(c2) > > > > > > > > >>>>> > > > > > > > > >>>>> Are the following orders acceptable to the consumer? > > > > > > > > >>>>> > > > > > > > > >>>>> > a1, a2, b1, b2, c1, c2 > > > > > > > > >>>>> > a1, b1, c1, a2, b2, c2 > > > > > > > > >>>>> > a1 b1, b2, a2, a3, b3 > > > > > > > > >>>>> > > > > > > > > >>>>> and an order displayed at one consumer is consistent > > across > > > > all > > > > > > > > >>>>> consumers. > > > > > > > > >>>>> > > > > > > > > >>>>> This can be achieved using Total Causal Ordering at the > > > > > Producer > > > > > > or > > > > > > > > >>>>> Queue level, using Leslie Lamport's clock and > > > synchronization > > > > > > > > approach. > > > > > > > > >>>>> > > > > > > > > >>>>> For reference is the paper attached, > > > > > > > http://lamport.azurewebsites.n > > > > > > > > >>>>> et/pubs/time-clocks.pdf > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> -- > > > > > > > > >>>>> Regards, > > > > > > > > >>>>> > > > > > > > > >>>>> Sowmya > > > > > > > > >>>>> > > > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s < > > > > > sowmya9...@gmail.com> > > > > > > > > >>>>> wrote: > > > > > > > > >>>>> > > > > > > > > >>>>>> Hello, > > > > > > > > >>>>>> > > > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122 > > > > > > > > >>>>>> < > > > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122? > > filter=12343065 > > > > > > > > > > > > as > > > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been > spending > > > > some > > > > > > time > > > > > > > > to > > > > > > > > >>>>>> understand the system, architecture and the existing > > > > Messaging > > > > > > > > Patterns. > > > > > > > > >>>>>> I still have a few questions and would like to clarify > > my > > > > > > > > assumptions. > > > > > > > > >>>>>> > > > > > > > > >>>>>> - Is the current FIFO order example limited to one > > > > message > > > > > > > queue > > > > > > > > >>>>>> per producer? Can the producer send the same > message > > to > > > > > > > multiple > > > > > > > > queues? > > > > > > > > >>>>>> Will the consumers of the queues be able to read > > > messages > > > > > in > > > > > > > > Order? > > > > > > > > >>>>>> - Can I assume that each producer will send > messages > > to > > > > one > > > > > > > > queue? > > > > > > > > >>>>>> - Global Order is to be identified across all > > > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be > > > used > > > > > for > > > > > > > > global order) > > > > > > > > >>>>>> instances that are running. > > > > > > > > >>>>>> - I think using a global clock can help establish > the > > > > order > > > > > > > > >>>>>> between 2 or more producers, however using some > form > > of > > > > > > vector > > > > > > > > >>>>>> clock might also help identify the global order of > > > > messages > > > > > > > > between the > > > > > > > > >>>>>> producers. > > > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how > > to > > > > read > > > > > > > > >>>>>> globally ordered messages) can then compare > messages > > > > across > > > > > > all > > > > > > > > message > > > > > > > > >>>>>> queues from the corresponding producers and extract > > the > > > > > > > > messages. [ this is > > > > > > > > >>>>>> the approach recommended by yukon on the issue > page ] > > > > > > > > >>>>>> - We can also potentially have another layer in the > > > > Message > > > > > > > Queue > > > > > > > > >>>>>> which accumulates all messages sent from producers > > and > > > > > > provides > > > > > > > > one ordered > > > > > > > > >>>>>> message queue for consumers to read from. > > > > > > > > >>>>>> > > > > > > > > >>>>>> Thank you for your patience and please let me know if > my > > > > > > > > >>>>>> understanding of the problem and the assumptions are > > > right. > > > > > > > > >>>>>> > > > > > > > > >>>>>> Best Regards, > > > > > > > > >>>>>> Sowmya > > > > > > > > >>>>>> > > > > > > > > >>>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> -- > > > > > > > > >>>>> Regards, > > > > > > > > >>>>> > > > > > > > > >>>>> Sowmya > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > >>>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> -- > > > > > > > > >>> Regards, > > > > > > > > >>> > > > > > > > > >>> Sowmya > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Regards, > > > > > > > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Sowmya > > > > > > > > > > > > > > > -- > > Regards, > > > > Sowmya > > > -- Regards, Sowmya
