Hi, Idea was when last subscriber goes purge the messages without getting slots involved because it need to happen lightweight and fast.
Thanks On Thu, Mar 12, 2015 at 3:16 AM, Pamod Sylvester <pa...@wso2.com> wrote: > +1 Agreed, also as you and Ramith mentioned deleting the content directly > will not be possible with the current architecture, since matadata is > partitioned based on the queue/topic but the message content is not and > ranges could overlap between multiple topics/queues. > > Therefore, we could invoke the content deletion task for the moment. > > Also another question i have is as to why we cannot partition the message > content the same way we partition the meta information ? > > Also, from the UI when purging we could still refer the unassigned slots i > believe. That's my observation. Do correct me if i am wrong. > > Thanks, > Pamod > > On Thu, Mar 12, 2015 at 1:29 PM, Hasitha Amal De Silva <hasit...@wso2.com> > wrote: > >> Hi Pamod, >> >> +1 for the solution. as discussed, we can use a single service call to >> fetch all slots from the coordinator for the given queue, and trigger >> single message deletions (referring to OnflightMessageTracker and >> SlotDeliveryWorker) . This will ensure that tombstones are not read when >> purging in a subscription closed scenario. >> >> However, we also need to think about triggering a queue purge from the >> UI, at which point we may not have active subscriptions or slot references. >> >> Thanks >> >> On Thu, Mar 12, 2015 at 11:56 AM, Pamod Sylvester <pa...@wso2.com> wrote: >> >>> Hi All, >>> >>> During the subscription disconnection/deletion purge operation is being >>> called on the the relevant non durable topic the subscription/s was bound >>> to. During purge operation all existing data (if any) relevant to that >>> topic will be removed. >>> >>> When using hector the removal is done through a range query following >>> LOC, >>> >>> List<Long> currentPage = >>> HectorDataAccessHelper.getNumericColumnKeysOfRow >>> (keyspace, >>> CassandraConstants.META_DATA_COLUMN_FAMILY, storageQueueName, pageSize, >>> lastProcessedID); >>> >>> Initially the relevant message ids are retrieved from MetaData CF and >>> then removed from the MessageContent CF. Once more than 200,000 messages >>> are sent since the range query queries the entire row of data belonging to >>> a relevant topic, it results in the following, >>> >>> >>> >>> *Caused by: >>> org.apache.cassandra.db.filter.TombstoneOverwhelmingException: nullat >>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) >>> ~[apache-cassandra-2.1.2.jar:2.1.2]at >>> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) >>> ~[apache-cassandra-2.1.2.jar:2.1.2]* >>> >>> And causes *me.prettyprint.hector.api.exceptions.HTimedOutException: >>> TimedOutException() *in the broker end, >>> >>> As solution, >>> >>> Since the idea of purging during disconnection is to clear out remaining >>> data which have not being given out to its subscribers, we could query the >>> slots which are still remaining and directly delete them from the >>> MessageContentCF instead of querying them from meta data CF. Would there be >>> any implication to this ? WDYT ? >>> >>> Thanks, >>> Pamod >>> >>> -- >>> *Pamod Sylvester * >>> *Senior Software Engineer * >>> Integration Technologies Team, WSO2 Inc.; http://wso2.com >>> email: pa...@wso2.com cell: +94 77 7779495 >>> >> >> >> >> -- >> Cheers, >> >> Hasitha Amal De Silva >> Software Engineer >> Mobile : 0772037426 >> Blog : http://devnutshell.tumblr.com/ >> WSO2 Inc.: http://wso2.com ( lean.enterprise.middleware. ) >> > > > > -- > *Pamod Sylvester * > *Senior Software Engineer * > Integration Technologies Team, WSO2 Inc.; http://wso2.com > email: pa...@wso2.com cell: +94 77 7779495 > -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
