Hi Hasitha

In this if a node with a reference count get killed, all the details
regarding reference counts are lost right? Is there a way to delete the
content?

Btw what if we have the reference count in database. Something similar to
what we have for queue message counting now (We create a counter when a
queue is created and then increment/ decrement count when messages are
received and sent)

What I suggest is when a topic message is created we add a reference
counter for the message (Via AndesContextStore a new method
createReferenceCounter(long
messageID)) when meta data is duplicated we increment the counter when
acknowledgment is received we decrement the counter (two methods in context
store to increment/decrement counts). And we will have a scheduled task to
periodically check the reference count zero messages and delete the
content. This way by creating separate insert statement to create a ref
counter and separate statement to update count we can over come writing
vendor specific SQL queries for reference counting (For RDBMS). Since the
idea is to recommend Cassandra for MessageStore and a RDBMS
AndesContextStore we would be better off that way. Plus this will avoid the
need to track reference counts in memory avoiding losing the reference
counts when a node gets killed. WDYT?


Thanks

On Sun, Oct 5, 2014 at 6:57 AM, Hasitha Hiranya <hasit...@wso2.com> wrote:

> Hi Team,
>
>
> Following is my vision on intregating topics to MB
>
> >> we duplicate metadata per subscriber. It will not create a big overhead.
> >> we do not duplicate content per subscriber, but we duplicate content
> per node
> >> I hereby assume that we do handle acks for topics. We need a reasearch
> on that.
>
> When a topic subscriber is created
> 1. qpid creates a temp queue
> 2. qpid creates a binding for that queue to topic exchange using topic
> name as binding key.
> 3. qpid creates a subscription for the temp queue.
>
> when a topic subscriber is closed qpid does above 3 things in reverse
> order.
>
> Adhering to this model,
>
> 1. We store metadata in the same way we use for normal queues.
> 2. We use the same SlotDelivery worker and the flusher. There is NOTHING
> called topic delivery worker
> 3. when show in UI we filter durable ones and show
> 4. when a subscriber closes, queue is deleted. We do same thing as for
> normal queues.
> 5. Whenever we insert metadata, we duplicate metadata for each temp queue
> (per subscriber). We know the nodes where subscriers lies, do we can
> duplicate content for those nodes (one copy for node).
> 6. We need to introduce a new tracking per subscriber in on flight message
> tracker, which is common for queues as well. when a metadata is inserted
> for a message id we increase a count.
>     When an ack came for that metadata we decrement it. If it is zero,
> content is ready to be removed. we do not track this count globally as we
> have a copy of content per node. Thus reference count do not need to be
> global. It is a local in-memory tracking.
> 7. queue change handler - if delete - execute normal delete (remove all
> metadata), decrement reference counts. Thread that delete content will
> detect that and will delete offline. This way only if all subscribers are
> gone, content is removed.
>
> 8. Should be careful abt hierarchical topics. We use our maps to identify
> queues bound to a topic. MQTT, AMQP confusion should be solved there.
>
> *Thanks *
>
>
> --
> *Hasitha Abeykoon*
> Senior Software Engineer; WSO2, Inc.; http://wso2.com
> *cell:* *+94 719363063*
> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>
>


-- 
*Asitha Nanayakkara*
Software Engineer
WSO2, Inc. http://wso2.com/
Mob: + 94 77 85 30 682
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Reply via email to