Changing gcgs didn't help
CREATE KEYSPACE ksname WITH replication = {'class':
'NetworkTopologyStrategy', 'dc1': '3', 'dc2': '3'} AND durable_writes =
true;
```CREATE TABLE keyspace."table" (
"column1" text PRIMARY KEY,
"column2" text
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 18000
AND gc_grace_seconds = 60
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
flushed table and took tsstabledump
grep -i '"expired" : true' SSTables.txt|wc -l
16439
grep -i '"expired" : false' SSTables.txt |wc -l
2657
ttl is 4 hours.
INSERT INTO keyspace."TABLE_NAME" ("column1", "column2") VALUES (?, ?)
USING TTL(4hours) ?';
SELECT * FROM keyspace."TABLE_NAME" WHERE "column1" = ?';
metric to scan tombstones
increase(cassandra_Table_TombstoneScannedHistogram{keyspace="mykeyspace",Table="tablename",function="Count"}[5m])
during peak hours. we only have couple of hundred inserts and 5-8k reads/s
per node.
```
```tablestats
Read Count: 605231874
Read Latency: 0.021268529760215503 ms.
Write Count: 2763352
Write Latency: 0.027924007871599422 ms.
Pending Flushes: 0
Table: name
SSTable count: 1
Space used (live): 1413203
Space used (total): 1413203
Space used by snapshots (total): 0
Off heap memory used (total): 28813
SSTable Compression Ratio: 0.5015090954531143
Number of partitions (estimate): 19568
Memtable cell count: 573
Memtable data size: 22971
Memtable off heap memory used: 0
Memtable switch count: 6
Local read count: 529868919
Local read latency: 0.020 ms
Local write count: 2707371
Local write latency: 0.024 ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 1
Bloom filter false ratio: 0.00000
Bloom filter space used: 23888
Bloom filter off heap memory used: 23880
Index summary off heap memory used: 4717
Compression metadata off heap memory used: 216
Compacted partition minimum bytes: 73
Compacted partition maximum bytes: 124
Compacted partition mean bytes: 99
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
Dropped Mutations: 0
histograms
Percentile SSTables Write Latency Read Latency Partition Size
Cell Count
(micros) (micros) (bytes)
50% 0.00 20.50 17.08 86
1
75% 0.00 24.60 20.50 124
1
95% 0.00 35.43 29.52 124
1
98% 0.00 35.43 42.51 124
1
99% 0.00 42.51 51.01 124
1
Min 0.00 8.24 5.72 73
0
Max 1.00 42.51 152.32 124
1
```
3 node in dc1 and 3 node in dc2 cluster. With instanc type aws ec2
m4.xlarge
On Sat, Feb 23, 2019, 7:47 PM Jeff Jirsa <jji...@gmail.com> wrote:
> Would also be good to see your schema (anonymized if needed) and the
> select queries you’re running
>
>
> --
> Jeff Jirsa
>
>
> On Feb 23, 2019, at 4:37 PM, Rahul Reddy <rahulreddy1...@gmail.com> wrote:
>
> Thanks Jeff,
>
> I'm having gcgs set to 10 mins and changed the table ttl also to 5 hours
> compared to insert ttl to 4 hours . Tracing on doesn't show any tombstone
> scans for the reads. And also log doesn't show tombstone scan alerts. Has
> the reads are happening 5-8k reads per node during the peak hours it shows
> 1M tombstone scans count per read.
>
> On Fri, Feb 22, 2019, 11:46 AM Jeff Jirsa <jji...@gmail.com> wrote:
>
>> If all of your data is TTL’d and you never explicitly delete a cell
>> without using s TTL, you can probably drop your GCGS to 1 hour (or less).
>>
>> Which compaction strategy are you using? You need a way to clear out
>> those tombstones. There exist tombstone compaction sub properties that can
>> help encourage compaction to grab sstables just because they’re full of
>> tombstones which will probably help you.
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Feb 22, 2019, at 8:37 AM, Kenneth Brotman <
>> kenbrot...@yahoo.com.invalid> wrote:
>>
>> Can we see the histogram? Why wouldn’t you at times have that many
>> tombstones? Makes sense.
>>
>>
>>
>> Kenneth Brotman
>>
>>
>>
>> *From:* Rahul Reddy [mailto:rahulreddy1...@gmail.com
>> <rahulreddy1...@gmail.com>]
>> *Sent:* Thursday, February 21, 2019 7:06 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Tombstones in memtable
>>
>>
>>
>> We have small table records are about 5k .
>>
>> All the inserts comes as 4hr ttl and we have table level ttl 1 day and gc
>> grace seconds has 3 hours. We do 5k reads a second during peak load During
>> the peak load seeing Alerts for tomstone scanned histogram reaching million.
>>
>> Cassandra version 3.11.1. Please let me know how can this tombstone scan
>> can be avoided in memtable
>>
>>
