Hi, There were some changes in the proposed architecture after the discussion had with Srinath and Shammi today.
1. There are only queue number of slots. When a message comes A slot will be created to the destination queue of the message if it is not already created and save in the slot. 2. We do not assign the whole map to one particular node, but a range of the queue will be assigned to subscribers. 3. There is a high threshold value for the range we assign. 4. QueueDiliveryWorker checkout the message list from the assigned range and deliver the messages to subscribers. 5. Distributed map should be maintained to keep track of assigned slot ranges. Thank you, Sajini. On Mon, Jul 28, 2014 at 9:27 AM, Srinath Perera <srin...@wso2.com> wrote: > Hazelcast map is OK, but careful on how often you update that. > > For the limit, shall we keep an integer for each slot in Hazelcast, and > once for 1000 messages, shall we update the integer and get the new value. > Then based on that value, we decide slot is full. Slot size limit is a > soft limit. > > Any slot that is assigned, need not used again (unless subscriber died). > When assigning slots, choose slots amongs ones that not yet assigned. > > --Srinath > > > > On Mon, Jul 28, 2014 at 9:20 AM, Sajini De Silva <saj...@wso2.com> wrote: > >> Hi all, >> >> I started implementing the above mentioned features. There are some >> clarifications need to be made. >> >> - In each node I keep a map to keep track of slot assignments. There >> are two possible ways to to that. >> >> >> 1. Use Hazelcast distributed map. >> 2. Keep in-memory map in each node and update the map using >> hazelcast topic message when there is a change in the assignment. >> >> Which one do you think is better? >> >> - How each node knows when the limit of a slot is exceeded? Currently >> what I'm doing is divide the threshold limit by number of nodes and each >> node keep track of its own limit. Is this the best way to do this? >> - When assigning a slot to a node in my current implementation I take >> the all the available slots for a particular queue and get the lowest >> numbered slot. If this to work we have to remove slots which have exceeded >> their limits in all nodes. Any suggestions to do this or is there a better >> way to do this? >> >> Thank you, >> >> Sajini. >> >> >> >> On Sun, Jul 27, 2014 at 6:02 PM, Sajini De Silva <saj...@wso2.com> wrote: >> >>> Hi, >>> >>> According to the discussion we had with Srinath there were some changes >>> of the above proposal. >>> >>> 1. Instead of having queuesCount X nodeCount number of slots, we >>> keep infinite number of slots with a limited number of capacity. The >>> capacity of the slot is configurable. >>> 2. When a message comes to a queue we create a slot and put it in >>> the slot until the slot exceeds its maximum threshold capacity. When a >>> slot >>> exceeds its capacity we create a new slot and put the messages there. >>> 3. When a subscriber subscribes from a node, we assign a slot to >>> that node from free slot pool. >>> 4. If a subscriber disappeared from a node, that subscriber's slot >>> can be assigned to someone else. No need to copy messages to somewhere >>> else. >>> >>> In this manner messages are not needed to be copied around. Message will >>> be write only once and read once. >>> >>> Thank you, >>> >>> Sajini. >>> >>> >>> On Fri, Jul 25, 2014 at 8:56 AM, Srinath Perera <srin...@wso2.com> >>> wrote: >>> >>>> Challenge is we do not know how many subscriptions the topic would >>>> receive. >>>> >>>> If we write message to multiple slots .. and only one subscription >>>> came, we are in trouble. I am inclined to say we write to one slot (slots >>>> are in Cassandra and so slot can be big) and split them only when there are >>>> multiple subscriptions. >>>> >>>> We need to discuss more. Can we chat f2f with Shammi and Hasitha and >>>> others. Key is to minimize the chance of having to do slot merges etc. >>>> >>>> --Srinath >>>> >>>> >>>> On Thu, Jul 24, 2014 at 10:09 PM, Sajini De Silva <saj...@wso2.com> >>>> wrote: >>>> >>>>> Hi Srinath, >>>>> >>>>> While reading this the question came in to my mind was how this >>>>> addresses the problem we were trying to solve using high and low water >>>>> mark levels. So I had an offline discussion with Shammi and came up with >>>>> the following solution. >>>>> >>>>> 1. When we receive messages, if there is no subscription, rather >>>>> than taking the slot and writing all the messages to that slot create >>>>> multiple number of slots which have messages up to HWM level. >>>>> 2. If a queue has subscriptions from multiple nodes, then we will >>>>> give it several slots, assign those slots to corresponding nodes. When >>>>> distributing messages to those slots copy only messages up to HWM >>>>> level. >>>>> >>>>> What do you think? >>>>> >>>>> Thank you, >>>>> >>>>> Sajini. >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Jul 24, 2014 at 3:09 PM, Srinath Perera <srin...@wso2.com> >>>>> wrote: >>>>> >>>>>> Currently we use Global queue to handle the scenario where we receive >>>>>> messages for a queue that does not have any subscriptions. However, due >>>>>> to >>>>>> this large latency is added and message need to copied around lot. >>>>>> >>>>>> Shammi and myself were discussing and look like it is possible to >>>>>> remove it completely. Following is the proposal. >>>>>> >>>>>> >>>>>> 1. We define a concept called slots where we can store messages >>>>>> in Cassandra. Each slot map to a row in Cassandra or a RDBMS table. >>>>>> There >>>>>> are queuesCount X nodeCount slots in the system. >>>>>> 2. Only one node own a slot at a time. When the node own the >>>>>> slot, it reads messages from the slot and send to subscribers. (This >>>>>> is >>>>>> what QueueDeliveryWorker do). This slot assignment we can manage via >>>>>> hazecast. >>>>>> 3. When we receive messages, if there is no subscription, we take >>>>>> a slot and write messages to that slot. When there is subscription to >>>>>> that >>>>>> queue, we assign the slot to associated node for subscription. >>>>>> 4. If a queue has subscriptions from multiple nodes, then we will >>>>>> give it several slots, assign those slots to corresponding nodes.The >>>>>> we >>>>>> distribute messages to those slots. >>>>>> 5. With this model, each message will be only written once and >>>>>> read once mostly. >>>>>> 6. There will be some edge cases (e.g. when a node run out >>>>>> subscriptions etc) where we might need to merge or split messages in >>>>>> slots. >>>>>> These we have to do carefully, but since those are rare, the overhead >>>>>> added >>>>>> is minimal. >>>>>> 7. Each query delivery worker would need to read a slot for each >>>>>> queue. So we need to keep a thread pool and manage reading. >>>>>> >>>>>> This should also reduce number of Cassandra calls by about 50%. >>>>>> >>>>>> WDYT? >>>>>> >>>>>> --Srinath >>>>>> >>>>>> -- >>>>>> ============================ >>>>>> Srinath Perera, Ph.D. >>>>>> http://people.apache.org/~hemapani/ >>>>>> http://srinathsview.blogspot.com/ >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Sajini De SIlva >>>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>>> Email: saj...@wso2.com >>>>> Blog: http://sajinid.blogspot.com/ >>>>> Git hub profile: https://github.com/sajinidesilva >>>>> >>>>> >>>> >>>> >>>> -- >>>> ============================ >>>> Director, Research, WSO2 Inc. >>>> Visiting Faculty, University of Moratuwa >>>> Member, Apache Software Foundation >>>> Research Scientist, Lanka Software Foundation >>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>> Site: http://people.apache.org/~hemapani/ >>>> Photos: http://www.flickr.com/photos/hemapani/ >>>> Phone: 0772360902 >>>> >>> >>> >>> >>> -- >>> Sajini De SIlva >>> Software Engineer; WSO2 Inc.; http://wso2.com , >>> Email: saj...@wso2.com >>> Blog: http://sajinid.blogspot.com/ >>> Git hub profile: https://github.com/sajinidesilva >>> >>> >> >> >> -- >> Sajini De SIlva >> Software Engineer; WSO2 Inc.; http://wso2.com , >> Email: saj...@wso2.com >> Blog: http://sajinid.blogspot.com/ >> Git hub profile: https://github.com/sajinidesilva >> >> > > > -- > ============================ > Director, Research, WSO2 Inc. > Visiting Faculty, University of Moratuwa > Member, Apache Software Foundation > Research Scientist, Lanka Software Foundation > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://people.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > -- Sajini De SIlva Software Engineer; WSO2 Inc.; http://wso2.com , Email: saj...@wso2.com Blog: http://sajinid.blogspot.com/ Git hub profile: https://github.com/sajinidesilva
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
