Right. In this case, cleanup should have done the necessary work here. Thomas
From: Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com] Sent: Freitag, 02. Februar 2018 06:59 To: user@cassandra.apache.org Subject: Re: Old tombstones not being cleaned up We did start with a 3 node cluster and a RF of 3, then added another 3 nodes and again another 3 nodes. So it is a good guess :) But I have run both repair and cleanup against the table on all nodes, would that not have removed any stray partitions? tor. 1. feb. 2018 kl. 22.31 skrev Steinmaurer, Thomas <thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>>: Did you started with a 9 node cluster from the beginning or did you extend / scale out your cluster (with vnodes) beyond the replication factor? If later applies and if you are deleting by explicit deletes and not via TTL, then nodes might not see the deletes anymore, as a node might not own the partition anymore after a topology change (e.g. scale out beyond the keyspace RF). Just a very wild guess. Thomas From: Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com<mailto:bo.gunder...@gmail.com>] Sent: Donnerstag, 01. Februar 2018 22:14 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Old tombstones not being cleaned up We do not use TTL anywhere...records are inserted and deleted "manually" by our software. tor. 1. feb. 2018 kl. 18.29 skrev Jonathan Haddad <j...@jonhaddad.com<mailto:j...@jonhaddad.com>>: Changing the defaul TTL doesn’t change the TTL on the existing data, only new data. It’s only set if you don’t supply one yourself. On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen <bo.gunder...@gmail.com<mailto:bo.gunder...@gmail.com>> wrote: Hi, We are running a small 9 node Cassandra v2.1.17 cluster. The cluster generally runs fine, but we have one table that are causing OOMs because an enormous amount of tombstones. Looking at the data in the table (sstable2json), the first of the tombstones are almost a year old. The table was initially created with a gc_grace_period of 10 days, but I have now lowered it to 1 hour. I have run a full repair of the table across all nodes. I have forced several major compactions of the table by using "nodetool compact", and also tried to switch from LeveledCompaction to SizeTierCompaction and back. What could cause cassandra to keep these tombstones? sstable2json: {"key": "foo", "cells": [["0000082f-25ef-4324-bb8a-8cf013c823c1:_","0000082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135], ["000010f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","000010f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731], ["00001d7a-ce95-4c74-b67e-f8cdffec4f85:_","00001d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102], ["00001dd3-ae22-4f6e-944a-8cfa147cde68:_","00001dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006], ["000022cc-d69c-4596-89e5-3e976c0cb9a8:_","000022cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448], ["00002777-4b1a-4267-8efc-c43054e63170:_","00002777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691], ["000061e8-f48b-4484-96f1-f8b6a3ed8f9f:_","000061e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300], ["000063da-f165-449b-b65d-2b7869368734:_","000063da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634], ["0000656f-f8b5-472b-93ed-1a893002f027:_","0000656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716], ... {"key": "bar", "metadata": {"deletionInfo": {"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}}, "cells": [["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965], ["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186], ["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840], .... sstablemetadata: stablemetadata /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db SSTable: /data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741 Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Bloom Filter FP chance: 0.100000 Minimum timestamp: 1488976211688000 Maximum timestamp: 1517468644066000 SSTable max local deletion time: 2147483647 Compression ratio: 0.5121956624389545 Estimated droppable tombstones: 18.00161766553587 SSTable Level: 0 Repaired at: 0 ReplayPosition(segmentId=1517168739626, position=22690189<tel:22%2069%2001%2089>) Estimated tombstone drop times:%n 1488976211: 1 1489906506: 4706 1490174752: 6111 1490449759: 6554 1490735410: 6559 1491016789: 6369 1491347982: 10216 1491680214: 13502 ... desc: CREATE TABLE xxx.yyy ( ti text, uuid text, json_data text, PRIMARY KEY (ti, uuid) ) WITH CLUSTERING ORDER BY (uuid ASC) AND bloom_filter_fp_chance = 0.1 AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 3600 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; jmx props(picture): [image001.png] The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g> The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313