Hi Ayub,
Is everything flushing to SSTables? It has to be somewhere right? So is it in the memtables? Or is it that there are tombstones that are sometimes detected and sometimes not detected as described in the very detailed article on The Last Pickle by Alex Dejanovski called Undetectable tombstones in Apache Cassandra: http://thelastpickle.com/blog/2018/07/05/undetectable-tombstones-in-apache-cassandra.html . I thought that was resolved in 3.11.2 by CASSANDRA-8527; and you are running 3.11.3! Is there still an outstanding issue? Kenneth Brotman From: Ayub M [mailto:hia...@gmail.com] Sent: Saturday, February 16, 2019 9:58 PM To: user@cassandra.apache.org Subject: tombstones threshold warning In the logs I see tombstone warning threshold. Read 411 live rows and 1644 tombstone cells for query SELECT * FROM ks.tbl WHERE key = XYZ LIMIT 5000 (see tombstone_warn_threshold) This is Cassandra 3.11.3, I see there are 2 sstables for this table and the partition XYZ exists in only one file. Now I dumped this sstable into json using sstabledump. I extracted the data of only this partition and I see there are only 411 rows in it. And all of them are active/live records, so I do not understand from where these tombstone are coming from? This table has collection columns and there are cell tombstones for the collection columns when they were inserted. Does collection cell tombstones get counted as tombstones cells in the warning displayed? Did a small test to see if collection tombstones are counted as tombstones and it does not seem so. So wondering where are those tombstones coming from in my above query. CREATE TABLE tbl ( col1 text, col2 text, c1 int, col3 map<text, text>, PRIMARY KEY (col1, col2) ) WITH CLUSTERING ORDER BY (col2 ASC) cassandra@cqlsh:dev_test> insert into tbl (col1 , col2 , c1, col3 ) values('3','3',3,{'key':'value'}); cassandra@cqlsh:dev_test> select * from tbl where col1 = '3'; col1 | col2 | c1 | col3 ----------------+----------+----+------------------ 3 | 3 | 3 | {'key': 'value'} (1 rows) Tracing session: 4c2a1894-3151-11e9-838d-29ed5fcf59ee activity | timestamp | source | source_elapsed | client ------------------------------------------------------------------------------------------+----------------------------+---------------+----------------+----------- Execute CQL3 query | 2019-02-15 18:41:25.145000 | 10.216.1.1 | 0 | 127.0.0.1 Parsing select * from tbl where col1 = '3'; [CoreThread-3] | 2019-02-15 18:41:25.145000 | 10.216.1.1 | 177 | 127.0.0.1 Preparing statement [CoreThread-3] | 2019-02-15 18:41:25.145001 | 10.216.1.1 | 295 | 127.0.0.1 Reading data from [/10.216.1.1] [CoreThread-3] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 491 | 127.0.0.1 Executing single-partition query on tbl [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 770 | 127.0.0.1 Acquiring sstable references [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 897 | 127.0.0.1 Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 1096 | 127.0.0.1 Merged data from memtables and 1 sstables [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 1235 | 127.0.0.1 Read 1 live rows and 0 tombstone cells [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 1317 | 127.0.0.1 Request complete | 2019-02-15 18:41:25.146529 | 10.216.1.1 | 1529 | 127.0.0.1 [root@localhost tbl-8aaa6bc1315011e991e523330936276b]# sstabledump aa-1-bti-Data.db [ { "partition" : { "key" : [ "3" ], "position" : 0 }, "rows" : [ { "type" : "row", "position" : 41, "clustering" : [ "3" ], "liveness_info" : { "tstamp" : "2019-02-15T18:36:16.838103Z" }, "cells" : [ { "name" : "c1", "value" : 3 }, { "name" : "col3", "deletion_info" : { "marked_deleted" : "2019-02-15T18:36:16.838102Z", "local_delete_time" : "2019-02-15T18:36:17Z" } }, { "name" : "col3", "path" : [ "key" ], "value" : "value" } ] } ] }
