Hi Pamod Querying the remaining slots and removing content directly is OK. As an improvement you can add a separate method in slot coordinator to get that message range of all the slots, to reduce cluster communication, and then do the necessary processing in the local node it self.
Thanks, Asitha On Thu, Mar 12, 2015 at 11:56 AM, Pamod Sylvester <pa...@wso2.com> wrote: > Hi All, > > During the subscription disconnection/deletion purge operation is being > called on the the relevant non durable topic the subscription/s was bound > to. During purge operation all existing data (if any) relevant to that > topic will be removed. > > When using hector the removal is done through a range query following LOC, > > List<Long> currentPage = > HectorDataAccessHelper.getNumericColumnKeysOfRow > (keyspace, > CassandraConstants.META_DATA_COLUMN_FAMILY, storageQueueName, pageSize, > lastProcessedID); > > Initially the relevant message ids are retrieved from MetaData CF and then > removed from the MessageContent CF. Once more than 200,000 messages are > sent since the range query queries the entire row of data belonging to a > relevant topic, it results in the following, > > > > *Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException: > nullat > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) > ~[apache-cassandra-2.1.2.jar:2.1.2]at > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) > ~[apache-cassandra-2.1.2.jar:2.1.2]* > > And causes *me.prettyprint.hector.api.exceptions.HTimedOutException: > TimedOutException() *in the broker end, > > As solution, > > Since the idea of purging during disconnection is to clear out remaining > data which have not being given out to its subscribers, we could query the > slots which are still remaining and directly delete them from the > MessageContentCF instead of querying them from meta data CF. Would there be > any implication to this ? WDYT ? > > Thanks, > Pamod > > -- > *Pamod Sylvester * > *Senior Software Engineer * > Integration Technologies Team, WSO2 Inc.; http://wso2.com > email: pa...@wso2.com cell: +94 77 7779495 > -- *Asitha Nanayakkara* Software Engineer WSO2, Inc. http://wso2.com/ Mob: + 94 77 85 30 682
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
