I do not think the feature is supposed to crash the server. It could be that the message is the logs and the crash is not related to this message. WARN might be a better logging level for any message, even though the first threshold is WARN and the second is FAIL. ERROR is usually something more dramatic.
On Wed, Dec 25, 2013 at 1:02 PM, Laing, Michael <michael.la...@nytimes.com>wrote: > It's a feature: > > In the stock cassandra.yaml file for 2.03 see: > > # When executing a scan, within or across a partition, we need to keep the >> # tombstones seen in memory so we can return them to the coordinator, >> which >> # will use them to make sure other replicas also know about the deleted >> rows. >> # With workloads that generate a lot of tombstones, this can cause >> performance >> # problems and even exaust the server heap. >> # ( >> http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets >> ) >> # Adjust the thresholds here if you understand the dangers and want to >> # scan more tombstones anyway. These thresholds may also be adjusted at >> runtime >> # using the StorageService mbean. >> tombstone_warn_threshold: 1000 >> tombstone_failure_threshold: 100000 > > > You are hitting the failure threshold. > > ml > > > On Wed, Dec 25, 2013 at 12:17 PM, Rahul Menon <ra...@apigee.com> wrote: > >> Sanjeeth, >> >> Looks like the error is being populated from the hintedhandoff, what is >> the size of your hints cf? >> >> Thanks >> Rahul >> >> >> On Wed, Dec 25, 2013 at 8:54 PM, Sanjeeth Kumar <sanje...@exotel.in>wrote: >> >>> Hi all, >>> One of my cassandra nodes crashes with the following exception >>> periodically - >>> ERROR [HintedHandoff:33] 2013-12-25 20:29:22,276 SliceQueryFilter.java >>> (line 200) Scanned over 100000 tombstones; query aborted (see >>> tombstone_fail_thr >>> eshold) >>> ERROR [HintedHandoff:33] 2013-12-25 20:29:22,278 CassandraDaemon.java >>> (line 187) Exception in thread Thread[HintedHandoff:33,1,main] >>> org.apache.cassandra.db.filter.TombstoneOverwhelmingException >>> at >>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201) >>> at >>> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122) >>> at >>> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) >>> at >>> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72) >>> at >>> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297) >>> at >>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306) >>> at >>> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351) >>> at >>> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309) >>> at >>> org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92) >>> at >>> org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> >>> Why does this happen? Does this relate to any incorrect config value? >>> >>> The Cassandra Version I'm running is >>> ReleaseVersion: 2.0.3 >>> >>> - Sanjeeth >>> >>> >> >
