Rahul,
Please see this DataStax article which suggests you might be using Cassandra as a queue-like dataset – and that’s an anti-pattern for Cassandra. It could be you need to use a different database. It could be your data model is wrong: https://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets Kenneth Brotman From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Saturday, February 23, 2019 4:47 PM To: user@cassandra.apache.org Subject: Re: Tombstones in memtable Would also be good to see your schema (anonymized if needed) and the select queries you’re running -- Jeff Jirsa On Feb 23, 2019, at 4:37 PM, Rahul Reddy <rahulreddy1...@gmail.com> wrote: Thanks Jeff, I'm having gcgs set to 10 mins and changed the table ttl also to 5 hours compared to insert ttl to 4 hours . Tracing on doesn't show any tombstone scans for the reads. And also log doesn't show tombstone scan alerts. Has the reads are happening 5-8k reads per node during the peak hours it shows 1M tombstone scans count per read. On Fri, Feb 22, 2019, 11:46 AM Jeff Jirsa <jji...@gmail.com> wrote: If all of your data is TTL’d and you never explicitly delete a cell without using s TTL, you can probably drop your GCGS to 1 hour (or less). Which compaction strategy are you using? You need a way to clear out those tombstones. There exist tombstone compaction sub properties that can help encourage compaction to grab sstables just because they’re full of tombstones which will probably help you. -- Jeff Jirsa On Feb 22, 2019, at 8:37 AM, Kenneth Brotman <kenbrot...@yahoo.com.invalid> wrote: Can we see the histogram? Why wouldn’t you at times have that many tombstones? Makes sense. Kenneth Brotman From: Rahul Reddy [mailto:rahulreddy1...@gmail.com] Sent: Thursday, February 21, 2019 7:06 AM To: user@cassandra.apache.org Subject: Tombstones in memtable We have small table records are about 5k . All the inserts comes as 4hr ttl and we have table level ttl 1 day and gc grace seconds has 3 hours. We do 5k reads a second during peak load During the peak load seeing Alerts for tomstone scanned histogram reaching million. Cassandra version 3.11.1. Please let me know how can this tombstone scan can be avoided in memtable
