Hi Hasitha In this if a node with a reference count get killed, all the details regarding reference counts are lost right? Is there a way to delete the content?
Btw what if we have the reference count in database. Something similar to what we have for queue message counting now (We create a counter when a queue is created and then increment/ decrement count when messages are received and sent) What I suggest is when a topic message is created we add a reference counter for the message (Via AndesContextStore a new method createReferenceCounter(long messageID)) when meta data is duplicated we increment the counter when acknowledgment is received we decrement the counter (two methods in context store to increment/decrement counts). And we will have a scheduled task to periodically check the reference count zero messages and delete the content. This way by creating separate insert statement to create a ref counter and separate statement to update count we can over come writing vendor specific SQL queries for reference counting (For RDBMS). Since the idea is to recommend Cassandra for MessageStore and a RDBMS AndesContextStore we would be better off that way. Plus this will avoid the need to track reference counts in memory avoiding losing the reference counts when a node gets killed. WDYT? Thanks On Sun, Oct 5, 2014 at 6:57 AM, Hasitha Hiranya <hasit...@wso2.com> wrote: > Hi Team, > > > Following is my vision on intregating topics to MB > > >> we duplicate metadata per subscriber. It will not create a big overhead. > >> we do not duplicate content per subscriber, but we duplicate content > per node > >> I hereby assume that we do handle acks for topics. We need a reasearch > on that. > > When a topic subscriber is created > 1. qpid creates a temp queue > 2. qpid creates a binding for that queue to topic exchange using topic > name as binding key. > 3. qpid creates a subscription for the temp queue. > > when a topic subscriber is closed qpid does above 3 things in reverse > order. > > Adhering to this model, > > 1. We store metadata in the same way we use for normal queues. > 2. We use the same SlotDelivery worker and the flusher. There is NOTHING > called topic delivery worker > 3. when show in UI we filter durable ones and show > 4. when a subscriber closes, queue is deleted. We do same thing as for > normal queues. > 5. Whenever we insert metadata, we duplicate metadata for each temp queue > (per subscriber). We know the nodes where subscriers lies, do we can > duplicate content for those nodes (one copy for node). > 6. We need to introduce a new tracking per subscriber in on flight message > tracker, which is common for queues as well. when a metadata is inserted > for a message id we increase a count. > When an ack came for that metadata we decrement it. If it is zero, > content is ready to be removed. we do not track this count globally as we > have a copy of content per node. Thus reference count do not need to be > global. It is a local in-memory tracking. > 7. queue change handler - if delete - execute normal delete (remove all > metadata), decrement reference counts. Thread that delete content will > detect that and will delete offline. This way only if all subscribers are > gone, content is removed. > > 8. Should be careful abt hierarchical topics. We use our maps to identify > queues bound to a topic. MQTT, AMQP confusion should be solved there. > > *Thanks * > > > -- > *Hasitha Abeykoon* > Senior Software Engineer; WSO2, Inc.; http://wso2.com > *cell:* *+94 719363063* > *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> > > -- *Asitha Nanayakkara* Software Engineer WSO2, Inc. http://wso2.com/ Mob: + 94 77 85 30 682
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
