Thanks Ken, further investigating what I found is the tombstones which I am
seeing are from null values in the collection objects. Tombstones are also
inserted into when initial collection values are inserted but seems like
they are not counted towards threshold warning and do not show up in
tracing. These tests I did on 3.11.3 version of cassandra.
Record inserted with values in collection objects - I see tombstones but
they do not show up in tracing.
"partition" : {
"key" : [ "e7cd5752-bc0d-4157-a80f-7523add8dbcd" ],
"position" : 121
},
"rows" : [
{
"type" : "row",
"position" : 528,
"clustering" : [ "ELVIS" ],
"liveness_info" : { "tstamp" : "2019-02-21T21:34:06.574848Z" },
"cells" : [
{ "name" : "frozen_race", "value" : {"race_title": "Ronde van
Gelderland", "race_date": "2015-04-19 00:00:00.000Z", "race_time":
"03:22:23"} },
{ "name" : "basics_udt", "deletion_info" : { "marked_deleted" :
"2019-02-21T21:34:06.574847Z", "local_delete_time" : "2019-02-21T21:34:06Z"
} },
{ "name" : "basics_udt", "path" : [ "birthday" ], "value" :
"1993-06-18 00:00:00.000Z" },
{ "name" : "basics_udt", "path" : [ "nationality" ], "value" :
"New Zealand" },
{ "name" : "events_list", "deletion_info" : { "marked_deleted" :
"2019-02-21T21:34:06.574847Z", "local_delete_time" : "2019-02-21T21:34:06Z"
} },
{ "name" : "events_list", "path" : [
"6ab41176-3620-11e9-81ac-87caa6eca935" ], "value" : "list-element1" },
{ "name" : "events_list", "path" : [
"6ab41177-3620-11e9-81ac-87caa6eca935" ], "value" : "list-element2" },
{ "name" : "frozen_race_list", "deletion_info" : {
"marked_deleted" : "2019-02-21T21:34:06.574847Z", "local_delete_time" :
"2019-02-21T21:34:06Z" } },
{ "name" : "frozen_race_list", "path" : [
"6ab41178-3620-11e9-81ac-87caa6eca935" ], "value" : {"race_title":
"Rabobank 7-Dorpenomloop Aalburg", "race_date": "2015-05-09 00:00:00.000Z",
"race_time": "02:58:33"} },
{ "name" : "frozen_race_list", "path" : [
"6ab41179-3620-11e9-81ac-87caa6eca935" ], "value" : {"race_title": "Ronde
van Gelderland", "race_date": "2015-04-19 00:00:00.000Z", "race_time":
"03:22:23"} },
{ "name" : "teams_map", "deletion_info" : { "marked_deleted" :
"2019-02-21T21:34:06.574847Z", "local_delete_time" : "2019-02-21T21:34:06Z"
} },
{ "name" : "teams_map", "path" : [ "1" ], "value" : "map-value1"
},
{ "name" : "teams_map", "path" : [ "2" ], "value" : "map-value2"
},
{ "name" : "teams_set", "deletion_info" : { "marked_deleted" :
"2019-02-21T21:34:06.574847Z", "local_delete_time" : "2019-02-21T21:34:06Z"
} },
{ "name" : "teams_set", "path" : [ "set-element1" ], "value" : ""
},
{ "name" : "teams_set", "path" : [ "set-element2" ], "value" : ""
}
cassandra@cqlsh:dev_ticket> select * from collsndudt where id =
e7cd5752-bc0d-4157-a80f-7523add8dbcd and lastname = 'ELVIS';
id | lastname |
basics_udt
| events_list |
frozen_race
|
frozen_race_list
| teams_map | teams_set
--------------------------------------+----------+-------------------------------------------------------------------------------------------------------+------------------------------------+-----------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------+----------------------------------
e7cd5752-bc0d-4157-a80f-7523add8dbcd | ELVIS | {birthday: '1993-06-18
00:00:00.000000+0000', nationality: 'New Zealand', height: null, weight:
null} | ['list-element1', 'list-element2'] | {race_title: 'Ronde van
Gelderland', race_date: '2015-04-19 00:00:00.000000+0000', race_time:
'03:22:23'} | [{race_title: 'Rabobank 7-Dorpenomloop Aalburg', race_date:
'2015-05-09 00:00:00.000000+0000', race_time: '02:58:33'}, {race_title:
'Ronde van Gelderland', race_date: '2015-04-19 00:00:00.000000+0000',
race_time: '03:22:23'}] | {1: 'map-value1', 2: 'map-value2'} |
{'set-element1', 'set-element2'}
(1 rows)
Tracing session: aa82f2c0-3621-11e9-81ac-87caa6eca935
activity
|
timestamp | source | source_elapsed | client
-------------------------------------------------------------------------------------------------------------------------+----------------------------+---------------+----------------+-----------
Execute CQL3 query | 2019-02-21 21:43:03.404000 | 10.216.87.180
| 0 | 127.0.0.1
Parsing select * from collsndudt where id =
e7cd5752-bc0d-4157-a80f-7523add8dbcd and lastname = 'ELVIS'; [CoreThread-8]
| 2019-02-21 21:43:03.404000 | 10.216.87.180 | 60 | 127.0.0.1
Preparing statement [CoreThread-8] | 2019-02-21 21:43:03.404000 |
10.216.87.180 | 182 | 127.0.0.1
Reading data from [/10.216.87.180] [CoreThread-8] | 2019-02-21
21:43:03.404000 | 10.216.87.180 | 328 | 127.0.0.1
Executing
single-partition query on collsndudt [CoreThread-3] | 2019-02-21
21:43:03.404001 | 10.216.87.180 | 401 | 127.0.0.1
Acquiring sstable references [CoreThread-3] | 2019-02-21 21:43:03.404001 |
10.216.87.180 | 401 | 127.0.0.1
Skipped 0/1 non-slice-intersecting
sstables, included 0 due to tombstones [CoreThread-3] | 2019-02-21
21:43:03.404001 | 10.216.87.180 | 481 | 127.0.0.1
Merged data
from memtables and 1 sstables [CoreThread-3] | 2019-02-21 21:43:03.404001 |
10.216.87.180 | 617 | 127.0.0.1
Read 1
live rows and 0 tombstone cells[CoreThread-3] | 2019-02-21 21:43:03.404001
| 10.216.87.180 | 617 | 127.0.0.1
Request complete | 2019-02-21 21:43:03.404741 | 10.216.87.180 |
741 | 127.0.0.1
Here null values were inserted into the collection and they do show up in
tracing -
"partition" : {
"key" : [ "cb07baad-eac8-4f65-b28a-bddc06a0de23" ],
"position" : 0
},
"rows" : [
{
"type" : "row",
"position" : 76,
"clustering" : [ "ADAMS" ],
"liveness_info" : { "tstamp" : "2019-02-21T21:34:20.024606Z" },
"cells" : [
{ "name" : "frozen_race", "deletion_info" : { "local_delete_time"
: "2019-02-21T21:34:20Z" }
},
{ "name" : "basics_udt", "deletion_info" : { "marked_deleted" :
"2019-02-21T21:34:20.024605Z", "local_delete_time" : "2019-02-21T21:34:20Z"
} },
{ "name" : "events_list", "deletion_info" : { "marked_deleted" :
"2019-02-21T21:34:20.024605Z", "local_delete_time" : "2019-02-21T21:34:20Z"
} },
{ "name" : "frozen_race_list", "deletion_info" : {
"marked_deleted" : "2019-02-21T21:34:20.024605Z", "local_delete_time" :
"2019-02-21T21:34:20Z" } },
{ "name" : "teams_map", "deletion_info" : { "marked_deleted" :
"2019-02-21T21:34:20.024605Z", "local_delete_time" : "2019-02-21T21:34:20Z"
} },
{ "name" : "teams_set", "deletion_info" : { "marked_deleted" :
"2019-02-21T21:34:20.024605Z", "local_delete_time" : "2019-02-21T21:34:20Z"
} }
cassandra@cqlsh:dev_ticket> select * from collsndudt where id =
cb07baad-eac8-4f65-b28a-bddc06a0de23;
id | lastname | basics_udt | events_list
| frozen_race | frozen_race_list | teams_map | teams_set
--------------------------------------+----------+------------+-------------+-------------+------------------+-----------+-----------
cb07baad-eac8-4f65-b28a-bddc06a0de23 | ADAMS | null | null
| null | null | null | null
Tracing session: 63b75250-3621-11e9-81ac-87caa6eca935
activity
| timestamp | source | source_elapsed
| client
--------------------------------------------------------------------------------------------------+----------------------------+---------------+----------------+-----------
Execute CQL3 query | 2019-02-21 21:41:04.629000 | 10.216.87.180
| 0 | 127.0.0.1
Parsing select * from collsndudt where id =
cb07baad-eac8-4f65-b28a-bddc06a0de23; [CoreThread-8] | 2019-02-21
21:41:04.629000 | 10.216.87.180 | 118 | 127.0.0.1
Preparing
statement [CoreThread-8] | 2019-02-21 21:41:04.629000 | 10.216.87.180
| 177 | 127.0.0.1
Reading data from [/
10.216.87.180] [CoreThread-8] | 2019-02-21 21:41:04.629000 | 10.216.87.180
| 318 | 127.0.0.1
Executing single-partition query on
collsndudt [CoreThread-6] | 2019-02-21 21:41:04.629001 | 10.216.87.180
| 460 | 127.0.0.1
Acquiring sstable
references [CoreThread-6] | 2019-02-21 21:41:04.629001 | 10.216.87.180
| 460 | 127.0.0.1
Skipped 0/1 non-slice-intersecting sstables, included 0 due to
tombstones [CoreThread-6] | 2019-02-21 21:41:04.630000 | 10.216.87.180
| 543 | 127.0.0.1
Merged data from memtables and 1
sstables [CoreThread-6] | 2019-02-21 21:41:04.630000 | 10.216.87.180
| 611 | 127.0.0.1
Read 1 live rows and 1
tombstone cells [CoreThread-6] | 2019-02-21 21:41:04.630000 | 10.216.87.180
| 611 | 127.0.0.1
Request complete | 2019-02-21
When it reports 1 tombstone cells, does it mean 1 records? Otherwise it
read more than one tombstone cell.
On Wed, Feb 20, 2019 at 1:30 AM Kenneth Brotman
<kenbrot...@yahoo.com.invalid> wrote:
> There is another good article called Common Problems with Cassandra
> Tombstones by Alla Babkina at
> https://opencredo.com/blogs/cassandra-tombstones-common-issues/ . It
> says interesting stuff like:
>
>
>
> 1. You can get tombstones without deleting anything
>
> 2. Inserting null values causes tombstones
>
> 3. Inserting values into collection columns results in tombstones
> even if you never delete a value
>
> 4. Expiring Data with TTL results in tombstones (of course)
>
> 5. The Invisible Column Ranges Tombstones – Resolved in
> CASSANDRA-11166 though; I should have said that in the last email too not
> CASSANDRA-8527. But it shouldn’t be this since you are on 3.11.3.
>
>
>
> I think number three above answers your questions based on your original
> post. See the article for the details. It’s really good.
>
>
>
> Kenneth Brotman
>
>
>
> *From:* Kenneth Brotman [mailto:kenbrot...@yahoo.com]
> *Sent:* Tuesday, February 19, 2019 10:12 PM
> *To:* 'user@cassandra.apache.org'
> *Subject:* RE: tombstones threshold warning
>
>
>
> Hi Ayub,
>
>
>
> Is everything flushing to SSTables? It has to be somewhere right? So is
> it in the memtables?
>
>
>
> Or is it that there are tombstones that are sometimes detected and
> sometimes not detected as described in the very detailed article on The
> Last Pickle by Alex Dejanovski called Undetectable tombstones in Apache
> Cassandra:
> http://thelastpickle.com/blog/2018/07/05/undetectable-tombstones-in-apache-cassandra.html
> .
>
>
>
> I thought that was resolved in 3.11.2 by CASSANDRA-8527; and you are
> running 3.11.3! Is there still an outstanding issue?
>
>
>
> Kenneth Brotman
>
>
>
>
>
> *From:* Ayub M [mailto:hia...@gmail.com]
> *Sent:* Saturday, February 16, 2019 9:58 PM
> *To:* user@cassandra.apache.org
> *Subject:* tombstones threshold warning
>
>
>
> In the logs I see tombstone warning threshold.
>
> Read 411 live rows and 1644 tombstone cells for query SELECT * FROM ks.tbl
> WHERE key = XYZ LIMIT 5000 (see tombstone_warn_threshold)
>
> This is Cassandra 3.11.3, I see there are 2 sstables for this table and
> the partition XYZ exists in only one file. Now I dumped this sstable into
> json using sstabledump. I extracted the data of only this partition and I
> see there are only 411 rows in it. And all of them are active/live records,
> so I do not understand from where these tombstone are coming from?
>
> This table has collection columns and there are cell tombstones for the
> collection columns when they were inserted. Does collection cell tombstones
> get counted as tombstones cells in the warning displayed?
>
> Did a small test to see if collection tombstones are counted as tombstones
> and it does not seem so. So wondering where are those tombstones coming
> from in my above query.
>
> CREATE TABLE tbl (
>
> col1 text,
>
> col2 text,
>
> c1 int,
>
> col3 map<text, text>,
>
> PRIMARY KEY (col1, col2)
>
> ) WITH CLUSTERING ORDER BY (col2 ASC)
>
>
>
> cassandra@cqlsh:dev_test> insert into tbl (col1 , col2 , c1, col3 )
> values('3','3',3,{'key':'value'});
>
> cassandra@cqlsh:dev_test> select * from tbl where col1 = '3';
>
> col1 | col2 | c1 | col3
>
> ----------------+----------+----+------------------
>
> 3 | 3 | 3 | {'key': 'value'}
>
> (1 rows)
>
>
>
> Tracing session: 4c2a1894-3151-11e9-838d-29ed5fcf59ee
>
> activity
> | timestamp | source | source_elapsed |
> client
>
> ------------------------------------------------------------------------------------------+----------------------------+---------------+----------------+-----------
>
>
> Execute CQL3 query | 2019-02-15 18:41:25.145000 | 10.216.1.1 | 0
> | 127.0.0.1
>
> Parsing select * from tbl where col1 = '3'; [CoreThread-3]
> | 2019-02-15 18:41:25.145000 | 10.216.1.1 | 177 |
> 127.0.0.1
>
> Preparing statement
> [CoreThread-3] | 2019-02-15 18:41:25.145001 | 10.216.1.1 | 295 |
> 127.0.0.1
>
> Reading data from [/10.216.1.1]
> [CoreThread-3] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 491
> | 127.0.0.1
>
> Executing single-partition query on tbl
> [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 770
> | 127.0.0.1
>
> Acquiring sstable references
> [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 897 |
> 127.0.0.1
>
> Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones
> [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 1096 |
> 127.0.0.1
>
> Merged data from memtables and 1 sstables
> [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 1235 |
> 127.0.0.1
>
> Read 1 live rows and 0 tombstone cells
> [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 | 1317 |
> 127.0.0.1
>
>
> Request complete | 2019-02-15 18:41:25.146529 | 10.216.1.1 | 1529 |
> 127.0.0.1
>
> [root@localhost tbl-8aaa6bc1315011e991e523330936276b]# sstabledump
> aa-1-bti-Data.db
>
> [
>
> {
>
> "partition" : {
>
> "key" : [ "3" ],
>
> "position" : 0
>
> },
>
> "rows" : [
>
> {
>
> "type" : "row",
>
> "position" : 41,
>
> "clustering" : [ "3" ],
>
> "liveness_info" : { "tstamp" : "2019-02-15T18:36:16.838103Z" },
>
> "cells" : [
>
> { "name" : "c1", "value" : 3 },
>
> { "name" : "col3", "deletion_info" : { "marked_deleted" :
> "2019-02-15T18:36:16.838102Z", "local_delete_time" : "2019-02-15T18:36:17Z" }
> },
>
> { "name" : "col3", "path" : [ "key" ], "value" : "value" }
>
> ]
>
> }
>
> ]
>
> }
>
>
--
Regards,
Ayub
