HI,
How would this approach impact on performance ? this will result in a DB
operation each time the message is published as well the subscriber acks ?
I agree with you on the fact that maintaining the counters in-memory could
result in message content to be persisted in the DB and have no way of
deleting them if the node gets killed.
Also what will be the possibility to check the message content which needs
to be deleted at the start up of the node. Where there should be a
comparison between the meta data and the content column families, all the
ids which are in the content table but not in the meta data CF should be
purged ?
{MessageContentCF} \ {MessageMetaData} = Message Content to be deleted.
this can affect the start up time of the node, but IMO it will not affect
the performance of the main flows.
WDYT ?
Thanks,
Pamod
On Sun, Oct 5, 2014 at 11:09 AM, Asitha Nanayakkara <asi...@wso2.com> wrote:
> Hi Hasitha
>
> In this if a node with a reference count get killed, all the details
> regarding reference counts are lost right? Is there a way to delete the
> content?
>
> Btw what if we have the reference count in database. Something similar to
> what we have for queue message counting now (We create a counter when a
> queue is created and then increment/ decrement count when messages are
> received and sent)
>
> What I suggest is when a topic message is created we add a reference
> counter for the message (Via AndesContextStore a new method
> createReferenceCounter(long
> messageID)) when meta data is duplicated we increment the counter when
> acknowledgment is received we decrement the counter (two methods in context
> store to increment/decrement counts). And we will have a scheduled task to
> periodically check the reference count zero messages and delete the
> content. This way by creating separate insert statement to create a ref
> counter and separate statement to update count we can over come writing
> vendor specific SQL queries for reference counting (For RDBMS). Since the
> idea is to recommend Cassandra for MessageStore and a RDBMS
> AndesContextStore we would be better off that way. Plus this will avoid the
> need to track reference counts in memory avoiding losing the reference
> counts when a node gets killed. WDYT?
>
>
> Thanks
>
> On Sun, Oct 5, 2014 at 6:57 AM, Hasitha Hiranya <hasit...@wso2.com> wrote:
>
>> Hi Team,
>>
>>
>> Following is my vision on intregating topics to MB
>>
>> >> we duplicate metadata per subscriber. It will not create a big
>> overhead.
>> >> we do not duplicate content per subscriber, but we duplicate content
>> per node
>> >> I hereby assume that we do handle acks for topics. We need a reasearch
>> on that.
>>
>> When a topic subscriber is created
>> 1. qpid creates a temp queue
>> 2. qpid creates a binding for that queue to topic exchange using topic
>> name as binding key.
>> 3. qpid creates a subscription for the temp queue.
>>
>> when a topic subscriber is closed qpid does above 3 things in reverse
>> order.
>>
>> Adhering to this model,
>>
>> 1. We store metadata in the same way we use for normal queues.
>> 2. We use the same SlotDelivery worker and the flusher. There is NOTHING
>> called topic delivery worker
>> 3. when show in UI we filter durable ones and show
>> 4. when a subscriber closes, queue is deleted. We do same thing as for
>> normal queues.
>> 5. Whenever we insert metadata, we duplicate metadata for each temp queue
>> (per subscriber). We know the nodes where subscriers lies, do we can
>> duplicate content for those nodes (one copy for node).
>> 6. We need to introduce a new tracking per subscriber in on flight
>> message tracker, which is common for queues as well. when a metadata is
>> inserted for a message id we increase a count.
>> When an ack came for that metadata we decrement it. If it is zero,
>> content is ready to be removed. we do not track this count globally as we
>> have a copy of content per node. Thus reference count do not need to be
>> global. It is a local in-memory tracking.
>> 7. queue change handler - if delete - execute normal delete (remove all
>> metadata), decrement reference counts. Thread that delete content will
>> detect that and will delete offline. This way only if all subscribers are
>> gone, content is removed.
>>
>> 8. Should be careful abt hierarchical topics. We use our maps to identify
>> queues bound to a topic. MQTT, AMQP confusion should be solved there.
>>
>> *Thanks *
>>
>>
>> --
>> *Hasitha Abeykoon*
>> Senior Software Engineer; WSO2, Inc.; http://wso2.com
>> *cell:* *+94 719363063*
>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>
>>
>
>
> --
> *Asitha Nanayakkara*
> Software Engineer
> WSO2, Inc. http://wso2.com/
> Mob: + 94 77 85 30 682
>
>
--
*Pamod Sylvester *
*Senior Software Engineer *
Integration Technologies Team, WSO2 Inc.; http://wso2.com
email: pa...@wso2.com cell: +94 77 7779495
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
