Hi, Of course, we can work together to finetune your design draft.
Regards, yukon On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <sowmya9...@gmail.com> wrote: > Hello, yukon and Von, > > I've shared my GSOC - 18 draft of the project. I'm looking forward to > working with all of you to finetune the proposal. I will be allocating 20 > hours per week from now to the proposal acceptance phase to address > questions and dive deep into any suggestions that you provide. > I am really looking forward to work on this project. > > thanks, > Sowmya > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <sowmya9...@gmail.com> wrote: > > > Hello yukon, > > > > Thank you for the inputs. I was able to look at the ~/store and kind of > > understand the storage structure. I also looked at the > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the > examples. > > Now I understand why you proposed a merge sort like approach for > > performing global ordering. Since the proposals are open, I am finalizing > > my draft and will have it up for review very soon. > > > > thanks, > > Sowmya > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote: > > > >> Hi Sowmya, > >> > >> ``` > >> Also, it would be great if you can at a high level, help me understand > >> how the messages in the message queues are stored before the consumer > reads > >> them. > >> ``` > >> > >> > >> > >> As shown in this figure, messages are sent to brokers by producer and > >> stored in commit log[1], then messages are dispatched to ConsumeQueue by > >> topic, the consumer pulls messages from the queue. > >> > >> I recommend you run a broker and send/consume some messages, then check > >> out the ~/store directory for details. > >> > >> Regards, > >> yukon > >> > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <sowmya9...@gmail.com> wrote: > >> > >>> Hello yukon, > >>> > >>> Currently FIFO can be achieved with a producer sending to one message > >>> queue, and when global ordering is required, multiple producers have to > >>> send to a single topic queue. > >>> > >>> We want to allow multiple producers to send messages on a topic to > >>> multiple message queues and still provide ordering guarantees to the > >>> consumer, so that all consumers see the same order of data and also the > >>> data is delivered in an ordered fashion. > >>> > >>> 1) Your idea of using a merge sort with the assumption that the first > >>> arriving message is treated as the first message to be delivered, > however, > >>> I want to propose an approach where when the producer sends a message > to a > >>> message queue, it must be done in a synchronous fashion and the > response > >>> will be that the message is accepted, which means that the message > follows > >>> the convention that all messages delivered prior by that producer have > been > >>> stored across groups and if not the producer will need to resend the > >>> message. > >>> > >>> We can use a variant of total causal ordering in the layer between the > >>> message queue and store. > >>> > >>> I have been busy with my class project so I couldn't make a lot of > >>> progress in detailing my approach. Also, it would be great if you can > at a > >>> high level, help me understand how the messages in the message queues > are > >>> stored before the consumer reads them. > >>> > >>> Does the consumer read directly from the message queue that the > producer > >>> sends data to? does the broker receive the queued producer messages, > store > >>> them and then pushes them to another queue for the consumer to read > from? > >>> > >>> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf > >>> > >>> thanks, > >>> Sowmya > >>> > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote: > >>> > >>>> Hi Sowmya, > >>>> > >>>> Sorry for the late reply, do you have any update on this project? > >>>> > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed > >>>> a simple solution that performs merge sort on multiple queues to > improve > >>>> performance and scalability. > >>>> > >>>> While the order issue across producers is difficult, we could assume > >>>> that the message first arrives the broker should be consumed firstly. > >>>> > >>>> But it would be wonderful if you have a real design about the order > >>>> issue across producers based on the RocketMQ design and the storage > >>>> structure. > >>>> > >>>> Looking forward your design ~ > >>>> > >>>> Regards, > >>>> yukon > >>>> > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <sowmya9...@gmail.com> > wrote: > >>>> > >>>>> Hello all, > >>>>> > >>>>> Adding a few more thoughts on the problem of establishing an order of > >>>>> messages across producers. > >>>>> > >>>>> Consider the Scenario > >>>>> > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1 > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2. > >>>>> > >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly > >>>>> time(a2) < time(b2) < time(c2) > >>>>> > >>>>> Are the following orders acceptable to the consumer? > >>>>> > >>>>> > a1, a2, b1, b2, c1, c2 > >>>>> > a1, b1, c1, a2, b2, c2 > >>>>> > a1 b1, b2, a2, a3, b3 > >>>>> > >>>>> and an order displayed at one consumer is consistent across all > >>>>> consumers. > >>>>> > >>>>> This can be achieved using Total Causal Ordering at the Producer or > >>>>> Queue level, using Leslie Lamport's clock and synchronization > approach. > >>>>> > >>>>> For reference is the paper attached, http://lamport.azurewebsites.n > >>>>> et/pubs/time-clocks.pdf > >>>>> > >>>>> > >>>>> -- > >>>>> Regards, > >>>>> > >>>>> Sowmya > >>>>> > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <sowmya9...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Hello, > >>>>>> > >>>>>> I'm trying to work on the issue ROCKETMQ-122 > >>>>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> > as > >>>>>> a part of Google Summer of Code 2018. I've been spending some time > to > >>>>>> understand the system, architecture and the existing Messaging > Patterns. > >>>>>> I still have a few questions and would like to clarify my > assumptions. > >>>>>> > >>>>>> - Is the current FIFO order example limited to one message queue > >>>>>> per producer? Can the producer send the same message to multiple > queues? > >>>>>> Will the consumers of the queues be able to read messages in > Order? > >>>>>> - Can I assume that each producer will send messages to one > queue? > >>>>>> - Global Order is to be identified across all > >>>>>> GlobalOrderedProducer (a new producer that is to be used for > global order) > >>>>>> instances that are running. > >>>>>> - I think using a global clock can help establish the order > >>>>>> between 2 or more producers, however using some form of vector > >>>>>> clock might also help identify the global order of messages > between the > >>>>>> producers. > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read > >>>>>> globally ordered messages) can then compare messages across all > message > >>>>>> queues from the corresponding producers and extract the > messages. [ this is > >>>>>> the approach recommended by yukon on the issue page ] > >>>>>> - We can also potentially have another layer in the Message Queue > >>>>>> which accumulates all messages sent from producers and provides > one ordered > >>>>>> message queue for consumers to read from. > >>>>>> > >>>>>> Thank you for your patience and please let me know if my > >>>>>> understanding of the problem and the assumptions are right. > >>>>>> > >>>>>> Best Regards, > >>>>>> Sowmya > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Regards, > >>>>> > >>>>> Sowmya > >>>>> > >>>>> > >>>> > >>> > >>> > >>> -- > >>> Regards, > >>> > >>> Sowmya > >>> > >>> > >> > > > > > > -- > > Regards, > > > > Sowmya > > > > > > > -- > Regards, > > Sowmya >
