Amazing thank you!
I have an overlapping timestamp data issue in multiple SSTables. This is a
production cluster of 27 nodes using Kairosdb front-end on cassandra 3.11. Our
kairosdb data_points table, looks like this:
CREATE TABLE kairosdb.data_points (
key blob,
column1 blob,
value blob,
PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE
AND CLUSTERING ORDER BY (column1 ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'com.jeffjirsa.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '1', 'compaction_window_unit': 'DAYS',
'max_threshold': '32', 'min_threshold': '4', 'timestamp_resolution':
'MILLISECONDS', 'tombstone_compaction_interval': '432000',
'tombstone_threshold': '0.2', 'unchecked_tombstone_compaction': 'true'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 7884009
AND gc_grace_seconds = 432000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.1
AND speculative_retry = '99PERCENTILE’;
I run repairs once every 3 days on this 27 node cluster. I ran a manual
compaction on individual sstables that had very old timestamps. It seems that
all that did was merge the old stale data, into a less sstables, rather than
deleting the tombstoned data completely. Here’s a quick look at a few entries
from the sstable metadata:
Max: 01/01/2019 Min: 12/31/2018 Estimated droppable tombstones:
0.6407875588103883 6.8G Mar 4 22:40 mc-230833-big-Data.db
Max: 01/02/2019 Min: 01/01/2019 Estimated droppable tombstones:
0.6717275763312848 8.7G Mar 5 12:38 mc-231449-big-Data.db
Max: 01/03/2019 Min: 01/02/2019 Estimated droppable tombstones:
0.7037095860184683 14G Mar 5 14:01 mc-231502-big-Data.db
Max: 01/04/2019 Min: 01/03/2019 Estimated droppable tombstones:
0.7081188342023961 17G Mar 5 02:03 mc-230946-big-Data.db
Max: 01/05/2019 Min: 01/04/2019 Estimated droppable tombstones:
0.7148984115492688 21G Mar 5 05:06 mc-231068-big-Data.db
Max: 01/06/2019 Min: 01/05/2019 Estimated droppable tombstones:
0.7190351559091802 21G Mar 5 10:40 mc-231315-big-Data.db
Max: 01/07/2019 Min: 01/06/2019 Estimated droppable tombstones:
0.7174924163467192 22G Mar 4 20:05 mc-230680-big-Data.db
Max: 01/08/2019 Min: 01/07/2019 Estimated droppable tombstones:
0.7209061510375004 22G Mar 5 09:02 mc-231243-big-Data.db
Max: 01/09/2019 Min: 01/08/2019 Estimated droppable tombstones:
0.7152589769956947 19G Mar 4 20:02 mc-230685-big-Data.db
Max: 01/10/2019 Min: 01/09/2019 Estimated droppable tombstones:
0.6791591497664088 8.0G Mar 5 04:43 mc-231091-big-Data.db
Max: 01/11/2019 Min: 01/10/2019 Estimated droppable tombstones:
0.6903846423101958 12G Mar 5 15:58 mc-231600-big-Data.db
Max: 01/12/2019 Min: 01/11/2019 Estimated droppable tombstones:
0.7960901678466651 14G Mar 5 05:48 mc-231118-big-Data.db
Max: 01/13/2019 Min: 01/12/2019 Estimated droppable tombstones:
0.7925577980175544 13G Mar 5 18:45 mc-231725-big-Data.db
Max: 01/14/2019 Min: 01/13/2019 Estimated droppable tombstones:
0.7977456322563183 14G Mar 5 15:38 mc-231577-big-Data.db
Max: 01/15/2019 Min: 01/14/2019 Estimated droppable tombstones:
0.7914742161174189 14G Mar 5 17:39 mc-231674-big-Data.db
Max: 01/16/2019 Min: 01/15/2019 Estimated droppable tombstones:
0.7844429396813951 13G Mar 5 11:07 mc-231363-big-Data.db
Max: 01/17/2019 Min: 01/16/2019 Estimated droppable tombstones:
0.5902276606951279 6.6G Mar 5 02:13 mc-230984-big-Data.db
Max: 01/18/2019 Min: 01/17/2019 Estimated droppable tombstones:
0.6547576878709388 7.4G Mar 5 00:57 mc-230924-big-Data.db
Max: 01/19/2019 Min: 01/18/2019 Estimated droppable tombstones:
0.6892596936899507 8.0G Mar 4 19:59 mc-230714-big-Data.db
Max: 01/20/2019 Min: 01/19/2019 Estimated droppable tombstones:
0.7076946624407203 8.2G Mar 5 19:14 mc-231761-big-Data.db
Max: 01/21/2019 Min: 01/20/2019 Estimated droppable tombstones:
0.7112744492848729 8.8G Mar 5 11:53 mc-231418-big-Data.db
Max: 01/22/2019 Min: 01/21/2019 Estimated droppable tombstones:
0.7098184906956251 9.0G Mar 5 12:59 mc-231466-big-Data.db
Max: 01/23/2019 Min: 01/22/2019 Estimated droppable tombstones:
0.7280378416128286 9.4G Mar 4 20:04 mc-230712-big-Data.db
Max: 01/24/2019 Min: 01/23/2019 Estimated droppable tombstones:
0.7375989189486377 9.9G Mar 5 08:03 mc-231233-big-Data.db
Max: 01/25/2019 Min: 01/24/2019 Estimated droppable tombstones:
0.7550665105646288 11G Mar 5 05:54 mc-231130-big-Data.db
Max: 01/26/2019 Min: 01/25/2019 Estimated droppable tombstones:
0.755651643899716 11G Mar 5 17:19 mc-231662-big-Data.db
Max: 01/27/2019 Min: 01/26/2019 Estimated droppable tombstones:
0.7737130829029405 12G Mar 5 08:23 mc-231245-big-Data.db
Max: 01/28/2019 Min: 01/27/2019 Estimated droppable tombstones:
0.7698542540694441 12G Mar 5 19:47 mc-231774-big-Data.db
Max: 01/29/2019 Min: 01/28/2019 Estimated droppable tombstones:
0.7751910087975257 13G Mar 5 04:25 mc-231064-big-Data.db
Max: 01/30/2019 Min: 01/29/2019 Estimated droppable tombstones:
0.7808428227545806 13G Mar 5 18:38 mc-231719-big-Data.db
Max: 01/31/2019 Min: 01/30/2019 Estimated droppable tombstones:
0.7833226924138593 14G Mar 5 02:26 mc-230969-big-Data.db
Max: 02/01/2019 Min: 01/31/2019 Estimated droppable tombstones:
0.7857791249217135 14G Mar 5 19:10 mc-231733-big-Data.db
Max: 02/02/2019 Min: 02/01/2019 Estimated droppable tombstones:
0.7866910479817485 17G Mar 5 06:54 mc-231166-big-Data.db
Max: 02/03/2019 Min: 02/02/2019 Estimated droppable tombstones:
0.5542345959736666 18G Mar 5 11:45 mc-231396-big-Data.db
Max: 02/04/2019 Min: 02/03/2019 Estimated droppable tombstones:
0.5119360095151949 19G Mar 5 03:28 mc-231022-big-Data.db
Max: 02/05/2019 Min: 02/04/2019 Estimated droppable tombstones:
0.2738841238834289 19G Mar 5 10:11 mc-231322-big-Data.db
Max: 02/06/2019 Min: 02/05/2019 Estimated droppable tombstones:
0.2613869586284185 19G Mar 5 11:35 mc-231388-big-Data.db
Max: 02/07/2019 Min: 02/06/2019 Estimated droppable tombstones:
0.16169525425765702 19G Mar 6 18:39 mc-232699-big-Data.db
Max: 02/08/2019 Min: 02/07/2019 Estimated droppable tombstones:
0.1705846230960788 18G Mar 6 19:59 mc-232756-big-Data.db
Max: 02/09/2019 Min: 02/08/2019 Estimated droppable tombstones:
0.20397309825287832 18G Mar 5 06:14 mc-231141-big-Data.db
Max: 02/10/2019 Min: 02/09/2019 Estimated droppable tombstones:
0.21416796761523366 18G Mar 5 05:46 mc-231119-big-Data.db
Max: 02/11/2019 Min: 02/10/2019 Estimated droppable tombstones:
0.2167303814761698 19G Mar 5 13:20 mc-231468-big-Data.db
Max: 02/12/2019 Min: 02/11/2019 Estimated droppable tombstones:
0.2125492567433847 19G Mar 5 14:25 mc-231520-big-Data.db
Max: 02/13/2019 Min: 02/12/2019 Estimated droppable tombstones:
0.21271803514892473 19G Mar 5 15:17 mc-231565-big-Data.db
Max: 02/14/2019 Min: 02/13/2019 Estimated droppable tombstones:
0.223317411505499 19G Mar 5 12:52 mc-231443-big-Data.db
Max: 02/15/2019 Min: 02/14/2019 Estimated droppable tombstones:
0.2380465637337778 19G Mar 5 07:14 mc-231185-big-Data.db
Max: 02/16/2019 Min: 02/15/2019 Estimated droppable tombstones:
0.24416048715449007 19G Mar 4 23:06 mc-230822-big-Data.db
Max: 02/17/2019 Min: 02/16/2019 Estimated droppable tombstones:
0.24840416482860103 19G Mar 6 15:23 mc-232556-big-Data.db
Max: 02/18/2019 Min: 02/17/2019 Estimated droppable tombstones:
0.2388316490362356 19G Mar 6 15:51 mc-232578-big-Data.db
Max: 02/19/2019 Min: 02/18/2019 Estimated droppable tombstones:
0.24455869859040452 20G Mar 6 16:20 mc-232596-big-Data.db
Max: 02/20/2019 Min: 02/19/2019 Estimated droppable tombstones:
0.22053576261936442 21G Mar 6 16:49 mc-232619-big-Data.db
Max: 02/21/2019 Min: 02/20/2019 Estimated droppable tombstones:
0.12749878111824647 21G Mar 6 17:17 mc-232637-big-Data.db
Max: 02/22/2019 Min: 02/21/2019 Estimated droppable tombstones:
0.01626749369647037 21G Mar 6 17:45 mc-232660-big-Data.db
Max: 02/23/2019 Min: 02/22/2019 Estimated droppable tombstones:
0.09138094337300502 21G Mar 6 18:13 mc-232676-big-Data.db
Max: 02/24/2019 Min: 02/23/2019 Estimated droppable tombstones: 0.0 21G
Mar 6 19:06 mc-232719-big-Data.db
Max: 02/25/2019 Min: 02/24/2019 Estimated droppable tombstones:
3.539084062959639E-8 22G Mar 6 19:35 mc-232736-big-Data.db
Max: 02/26/2019 Min: 02/25/2019 Estimated droppable tombstones:
1.5737638608259986E-8 22G Mar 6 20:28 mc-232772-big-Data.db
Max: 02/27/2019 Min: 02/26/2019 Estimated droppable tombstones:
1.2888956685492647E-8 22G Mar 6 20:57 mc-232796-big-Data.db
Max: 02/28/2019 Min: 02/27/2019 Estimated droppable tombstones:
6.44921720903649E-9 22G Mar 1 01:12 mc-226629-big-Data.db
Max: 03/01/2019 Min: 02/28/2019 Estimated droppable tombstones:
6.461017817804411E-9 22G Mar 2 01:43 mc-227693-big-Data.db
Max: 03/02/2019 Min: 03/01/2019 Estimated droppable tombstones:
9.270087478134665E-10 21G Mar 3 01:19 mc-228763-big-Data.db
Max: 03/03/2019 Min: 03/02/2019 Estimated droppable tombstones: 0.0 21G
Mar 4 02:35 mc-229907-big-Data.db
Max: 03/04/2019 Min: 03/03/2019 Estimated droppable tombstones:
2.6800683107535888E-9 22G Mar 5 01:50 mc-230915-big-Data.db
Max: 03/05/2019 Min: 03/04/2019 Estimated droppable tombstones:
2.703574931824122E-9 22G Mar 6 00:35 mc-231983-big-Data.db
Max: 03/06/2019 Min: 03/05/2019 Estimated droppable tombstones: 0.0 22G
Mar 7 00:38 mc-232951-big-Data.db
Max: 03/07/2019 Min: 03/06/2019 Estimated droppable tombstones: 0.0 11G
Mar 7 12:17 mc-233426-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0 3.9M
Mar 7 01:32 mc-233018-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0 427M
Mar 7 12:42 mc-233459-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0 437M
Mar 7 13:01 mc-233469-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0 455M
Mar 7 12:52 mc-233464-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0 62M
Mar 7 01:32 mc-233017-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0 681M
Mar 7 12:17 mc-233443-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0 694M
Mar 7 12:11 mc-233438-big-Data.db
Max: 03/07/2019 Min: 03/07/2019 Estimated droppable tombstones: 0.0 795M
Mar 7 12:34 mc-233454-big-Data.db
Max: 08/17/2018 Min: 08/17/2018 Estimated droppable tombstones:
4.202054543912095E-4 141M Mar 6 21:17 mc-232845-big-Data.db
Max: 08/18/2018 Min: 08/17/2018 Estimated droppable tombstones:
0.052106761370398144 1.1G Mar 6 21:08 mc-232827-big-Data.db
Max: 08/19/2018 Min: 08/18/2018 Estimated droppable tombstones:
0.0656319007897974 1.1G Mar 6 21:15 mc-232838-big-Data.db
Max: 08/20/2018 Min: 08/19/2018 Estimated droppable tombstones:
0.12785613299472212 1.1G Mar 6 21:05 mc-232823-big-Data.db
Max: 08/21/2018 Min: 08/20/2018 Estimated droppable tombstones:
0.11578795798812021 1.1G Mar 6 21:09 mc-232829-big-Data.db
Max: 08/22/2018 Min: 08/21/2018 Estimated droppable tombstones:
0.11781698286628624 1.1G Mar 6 21:06 mc-232824-big-Data.db
Max: 08/23/2018 Min: 08/22/2018 Estimated droppable tombstones:
0.015310752384131633 1.1G Mar 6 21:11 mc-232832-big-Data.db
Max: 08/24/2018 Min: 08/23/2018 Estimated droppable tombstones:
6.239976748893364E-4 1.1G Mar 6 21:14 mc-232837-big-Data.db
Max: 08/25/2018 Min: 08/24/2018 Estimated droppable tombstones:
0.007267002827070718 1.1G Mar 6 21:16 mc-232842-big-Data.db
Max: 08/26/2018 Min: 08/25/2018 Estimated droppable tombstones:
0.015487525347912895 1.1G Mar 6 21:12 mc-232834-big-Data.db
Max: 08/27/2018 Min: 08/26/2018 Estimated droppable tombstones:
0.22586925686932946 1.2G Mar 5 03:53 mc-231079-big-Data.db
Max: 08/28/2018 Min: 08/27/2018 Estimated droppable tombstones:
0.20038331556336905 1.2G Mar 5 08:11 mc-231268-big-Data.db
Max: 08/29/2018 Min: 08/28/2018 Estimated droppable tombstones:
0.2530757099874583 1.3G Mar 4 19:01 mc-230693-big-Data.db
Max: 08/30/2018 Min: 08/29/2018 Estimated droppable tombstones:
0.2558579833614155 1.3G Mar 5 17:23 mc-231701-big-Data.db
Max: 08/31/2018 Min: 08/30/2018 Estimated droppable tombstones:
0.252704679692973 1.2G Mar 5 17:17 mc-231695-big-Data.db
Max: 09/01/2018 Min: 08/31/2018 Estimated droppable tombstones:
0.251179630123946 1.2G Mar 5 15:59 mc-231633-big-Data.db
Max: 09/02/2018 Min: 09/01/2018 Estimated droppable tombstones:
0.2554852155936904 1.2G Mar 5 16:29 mc-231658-big-Data.db
Max: 09/03/2018 Min: 09/02/2018 Estimated droppable tombstones:
0.25881950027198586 1.2G Mar 5 16:00 mc-231635-big-Data.db
Max: 09/04/2018 Min: 09/03/2018 Estimated droppable tombstones:
0.7886645476349738 4.7G Mar 4 20:41 mc-230737-big-Data.db
Max: 09/05/2018 Min: 09/04/2018 Estimated droppable tombstones:
0.8831512575108397 13G Mar 5 03:13 mc-230996-big-Data.db
Max: 09/06/2018 Min: 09/05/2018 Estimated droppable tombstones:
0.881454703411404 12G Mar 5 06:33 mc-231151-big-Data.db
Max: 09/07/2018 Min: 09/06/2018 Estimated droppable tombstones:
0.8771459061085191 12G Mar 5 11:24 mc-231372-big-Data.db
Max: 09/08/2018 Min: 09/07/2018 Estimated droppable tombstones:
0.8753203300842707 12G Mar 4 23:23 mc-230835-big-Data.db
Max: 09/09/2018 Min: 09/08/2018 Estimated droppable tombstones:
0.8808274303834315 11G Mar 5 07:58 mc-231212-big-Data.db
Max: 09/10/2018 Min: 09/09/2018 Estimated droppable tombstones:
0.8809198437763183 12G Mar 5 08:03 mc-231217-big-Data.db
Max: 09/11/2018 Min: 09/10/2018 Estimated droppable tombstones:
0.8723055307614438 11G Mar 5 07:31 mc-231196-big-Data.db
Max: 09/12/2018 Min: 09/11/2018 Estimated droppable tombstones:
0.8747882208752266 11G Mar 5 05:19 mc-231097-big-Data.db
Max: 09/13/2018 Min: 09/12/2018 Estimated droppable tombstones:
0.05224083315449286 1.3G Mar 6 21:00 mc-232816-big-Data.db
Max: 09/14/2018 Min: 09/13/2018 Estimated droppable tombstones:
0.07163238619554278 1.2G Mar 6 21:02 mc-232818-big-Data.db
Max: 09/15/2018 Min: 09/14/2018 Estimated droppable tombstones:
0.10299448148502635 1.2G Mar 6 21:03 mc-232820-big-Data.db
Max: 09/16/2018 Min: 09/15/2018 Estimated droppable tombstones:
0.12947413624859622 1.2G Mar 1 14:51 mc-227283-big-Data.db
Max: 09/17/2018 Min: 09/16/2018 Estimated droppable tombstones:
0.14577734759443223 1.2G Mar 2 14:55 mc-228346-big-Data.db
Max: 09/18/2018 Min: 09/17/2018 Estimated droppable tombstones:
0.15627725735914008 1.2G Mar 3 12:35 mc-229331-big-Data.db
Max: 09/19/2018 Min: 09/18/2018 Estimated droppable tombstones:
0.19208129168867066 1.3G Mar 4 10:23 mc-230295-big-Data.db
Max: 09/20/2018 Min: 09/19/2018 Estimated droppable tombstones:
0.249042970270552 1.4G Mar 5 10:21 mc-231370-big-Data.db
Max: 09/21/2018 Min: 09/20/2018 Estimated droppable tombstones:
0.2923327861443509 1.4G Mar 5 10:39 mc-231385-big-Data.db
Max: 09/22/2018 Min: 09/21/2018 Estimated droppable tombstones:
0.4694543993137898 1.7G Mar 5 07:25 mc-231230-big-Data.db
Max: 09/23/2018 Min: 09/22/2018 Estimated droppable tombstones:
0.8812610467316245 7.0G Mar 5 14:31 mc-231532-big-Data.db
Max: 09/24/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8963227271743491 9.6G Mar 4 18:56 mc-230653-big-Data.db
Max: 09/25/2018 Min: 09/24/2018 Estimated droppable tombstones:
0.8828679015527205 13G Mar 4 22:07 mc-230775-big-Data.db
Max: 09/26/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8706920830020076 14G Mar 5 07:01 mc-231162-big-Data.db
Max: 09/27/2018 Min: 09/26/2018 Estimated droppable tombstones:
0.8774074013483522 14G Mar 5 13:03 mc-231437-big-Data.db
Max: 09/28/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8446102849522781 14G Mar 5 00:43 mc-230864-big-Data.db
Max: 09/29/2018 Min: 09/28/2018 Estimated droppable tombstones:
0.8808040881298336 12G Mar 5 05:54 mc-231117-big-Data.db
Max: 09/30/2018 Min: 09/29/2018 Estimated droppable tombstones:
0.88253551363474 12G Mar 5 03:46 mc-231028-big-Data.db
Max: 10/01/2018 Min: 09/24/2018 Estimated droppable tombstones:
0.8788732337462403 13G Mar 5 12:12 mc-231403-big-Data.db
Max: 10/02/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.883335202175054 12G Mar 5 00:29 mc-230874-big-Data.db
Max: 10/03/2018 Min: 10/02/2018 Estimated droppable tombstones:
0.8771751169697322 12G Mar 5 17:47 mc-231677-big-Data.db
Max: 10/04/2018 Min: 10/03/2018 Estimated droppable tombstones:
0.8822618238791344 13G Mar 5 19:33 mc-231737-big-Data.db
Max: 10/05/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8865947931678021 13G Mar 4 20:45 mc-230717-big-Data.db
Max: 10/06/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8800199984853445 13G Mar 5 03:32 mc-231010-big-Data.db
Max: 10/07/2018 Min: 10/06/2018 Estimated droppable tombstones:
0.8912633931978029 13G Mar 5 14:36 mc-231519-big-Data.db
Max: 10/08/2018 Min: 10/07/2018 Estimated droppable tombstones:
0.8776893254692907 13G Mar 5 01:58 mc-230939-big-Data.db
Max: 10/09/2018 Min: 10/08/2018 Estimated droppable tombstones:
0.8856079371268669 13G Mar 5 14:08 mc-231496-big-Data.db
Max: 10/10/2018 Min: 10/09/2018 Estimated droppable tombstones:
0.8779972870791256 13G Mar 5 16:48 mc-231621-big-Data.db
Max: 10/11/2018 Min: 10/10/2018 Estimated droppable tombstones:
0.8798199877431634 13G Mar 5 00:22 mc-230863-big-Data.db
Max: 10/12/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8818200837221603 13G Mar 5 11:07 mc-231347-big-Data.db
Max: 10/13/2018 Min: 10/12/2018 Estimated droppable tombstones:
0.8792880344026048 13G Mar 4 20:01 mc-230689-big-Data.db
Max: 10/14/2018 Min: 10/13/2018 Estimated droppable tombstones:
0.8893721720823108 13G Mar 4 21:58 mc-230770-big-Data.db
Max: 10/15/2018 Min: 10/14/2018 Estimated droppable tombstones:
0.8866647009668137 13G Mar 5 01:58 mc-230933-big-Data.db
Max: 10/16/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8800707952691067 14G Mar 5 11:00 mc-231343-big-Data.db
Max: 10/17/2018 Min: 09/19/2018 Estimated droppable tombstones:
0.8810867844724584 15G Mar 5 05:00 mc-231072-big-Data.db
Max: 10/18/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8785127729091343 15G Mar 5 09:07 mc-231256-big-Data.db
Max: 10/19/2018 Min: 09/20/2018 Estimated droppable tombstones:
0.8876021364430524 15G Mar 5 04:42 mc-231060-big-Data.db
Max: 10/20/2018 Min: 10/19/2018 Estimated droppable tombstones:
0.881872857492371 13G Mar 5 17:58 mc-231682-big-Data.db
Max: 10/21/2018 Min: 10/20/2018 Estimated droppable tombstones:
0.8848086228545486 13G Mar 5 13:51 mc-231483-big-Data.db
Max: 10/22/2018 Min: 08/27/2018 Estimated droppable tombstones:
0.8792001876878928 15G Mar 5 15:24 mc-231552-big-Data.db
Max: 10/23/2018 Min: 08/27/2018 Estimated droppable tombstones:
0.886772503866838 14G Mar 5 00:37 mc-230872-big-Data.db
Max: 10/24/2018 Min: 10/23/2018 Estimated droppable tombstones:
0.883097444361784 13G Mar 4 21:55 mc-230766-big-Data.db
Max: 10/25/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8725515297254708 14G Mar 5 03:38 mc-231015-big-Data.db
Max: 10/26/2018 Min: 10/25/2018 Estimated droppable tombstones:
0.8818153097020685 12G Mar 5 02:17 mc-230953-big-Data.db
Max: 10/27/2018 Min: 10/26/2018 Estimated droppable tombstones:
0.8848182273248961 12G Mar 5 01:12 mc-230905-big-Data.db
Max: 10/28/2018 Min: 10/27/2018 Estimated droppable tombstones:
0.8863910139910516 12G Mar 5 15:16 mc-231554-big-Data.db
Max: 10/29/2018 Min: 10/28/2018 Estimated droppable tombstones:
0.8804707550662745 13G Mar 5 19:53 mc-231768-big-Data.db
Max: 10/30/2018 Min: 10/29/2018 Estimated droppable tombstones:
0.8848917993137158 13G Mar 5 17:06 mc-231631-big-Data.db
Max: 10/31/2018 Min: 09/04/2018 Estimated droppable tombstones:
0.8744904945244535 15G Mar 5 16:01 mc-231581-big-Data.db
Max: 11/01/2018 Min: 10/31/2018 Estimated droppable tombstones:
0.8844023760442303 13G Mar 5 16:56 mc-231623-big-Data.db
Max: 11/02/2018 Min: 09/06/2018 Estimated droppable tombstones:
0.876832422513591 15G Mar 5 17:42 mc-231663-big-Data.db
Max: 11/03/2018 Min: 11/02/2018 Estimated droppable tombstones:
0.8884698595625823 13G Mar 5 12:00 mc-231393-big-Data.db
Max: 11/04/2018 Min: 11/03/2018 Estimated droppable tombstones:
0.8856117904594102 13G Mar 5 05:03 mc-231078-big-Data.db
Max: 11/05/2018 Min: 11/04/2018 Estimated droppable tombstones:
0.8850104959605515 13G Mar 5 06:28 mc-231131-big-Data.db
Max: 11/06/2018 Min: 11/05/2018 Estimated droppable tombstones:
0.8876672835148812 13G Mar 4 23:03 mc-230811-big-Data.db
Max: 11/07/2018 Min: 11/06/2018 Estimated droppable tombstones:
0.8819559232651852 13G Mar 5 19:52 mc-231763-big-Data.db
Max: 11/08/2018 Min: 11/07/2018 Estimated droppable tombstones:
0.8890240896109401 14G Mar 5 09:34 mc-231277-big-Data.db
Max: 11/09/2018 Min: 09/13/2018 Estimated droppable tombstones:
0.8889195336887382 16G Mar 5 14:22 mc-231498-big-Data.db
Max: 11/10/2018 Min: 11/09/2018 Estimated droppable tombstones:
0.8901968237826773 14G Mar 5 13:12 mc-231440-big-Data.db
Max: 11/11/2018 Min: 11/10/2018 Estimated droppable tombstones:
0.8823935870159266 14G Mar 5 12:29 mc-231407-big-Data.db
Max: 11/12/2018 Min: 11/11/2018 Estimated droppable tombstones:
0.8821515266485579 14G Mar 5 07:09 mc-231164-big-Data.db
Max: 11/13/2018 Min: 09/17/2018 Estimated droppable tombstones:
0.8821863572044562 16G Mar 5 17:27 mc-231638-big-Data.db
Max: 11/14/2018 Min: 11/13/2018 Estimated droppable tombstones:
0.8837352268527888 14G Mar 5 19:24 mc-231730-big-Data.db
Max: 11/15/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8790743796626552 15G Mar 5 09:27 mc-231264-big-Data.db
Max: 11/16/2018 Min: 09/23/2018 Estimated droppable tombstones:
0.8844277571061905 15G Mar 5 16:35 mc-231607-big-Data.db
Max: 11/17/2018 Min: 11/16/2018 Estimated droppable tombstones:
0.8862673108309931 14G Mar 5 08:03 mc-231204-big-Data.db
Max: 11/18/2018 Min: 11/17/2018 Estimated droppable tombstones:
0.8842595514003214 14G Mar 5 18:52 mc-231717-big-Data.db
Max: 11/19/2018 Min: 09/24/2018 Estimated droppable tombstones:
0.881639236418788 15G Mar 4 23:34 mc-230826-big-Data.db
Max: 11/20/2018 Min: 11/19/2018 Estimated droppable tombstones:
0.886075536820344 14G Mar 4 23:00 mc-230805-big-Data.db
Max: 11/21/2018 Min: 11/20/2018 Estimated droppable tombstones:
0.889928801493522 14G Mar 5 09:35 mc-231279-big-Data.db
Max: 11/22/2018 Min: 11/21/2018 Estimated droppable tombstones:
0.8861353317060062 14G Mar 5 20:00 mc-231776-big-Data.db
Max: 11/23/2018 Min: 11/22/2018 Estimated droppable tombstones:
0.8829576920997242 14G Mar 5 07:49 mc-231198-big-Data.db
Max: 11/24/2018 Min: 11/23/2018 Estimated droppable tombstones:
0.8821505816554567 13G Mar 5 03:25 mc-230998-big-Data.db
Max: 11/25/2018 Min: 11/24/2018 Estimated droppable tombstones:
0.880930424220212 13G Mar 4 20:17 mc-230696-big-Data.db
Max: 11/26/2018 Min: 11/25/2018 Estimated droppable tombstones:
0.8820117087435542 13G Mar 4 21:32 mc-230743-big-Data.db
Max: 11/27/2018 Min: 11/26/2018 Estimated droppable tombstones:
0.8791004516800335 13G Mar 5 03:23 mc-230997-big-Data.db
Max: 11/28/2018 Min: 11/27/2018 Estimated droppable tombstones:
0.8788965149724617 13G Mar 5 01:52 mc-230929-big-Data.db
Max: 11/29/2018 Min: 11/28/2018 Estimated droppable tombstones:
0.8877500310616938 13G Mar 4 22:50 mc-230803-big-Data.db
Max: 11/30/2018 Min: 11/29/2018 Estimated droppable tombstones:
0.8754630223343176 13G Mar 4 21:26 mc-230742-big-Data.db
Max: 12/01/2018 Min: 11/30/2018 Estimated droppable tombstones:
0.8762374681745563 12G Mar 4 21:34 mc-230755-big-Data.db
Max: 12/02/2018 Min: 12/01/2018 Estimated droppable tombstones:
0.8787902962561704 12G Mar 5 04:31 mc-231059-big-Data.db
Max: 12/03/2018 Min: 12/02/2018 Estimated droppable tombstones:
0.8853087223736261 12G Mar 5 15:16 mc-231551-big-Data.db
Max: 12/04/2018 Min: 12/03/2018 Estimated droppable tombstones:
0.8803649205154546 12G Mar 5 10:06 mc-231309-big-Data.db
Max: 12/05/2018 Min: 12/04/2018 Estimated droppable tombstones:
0.8824805012470633 12G Mar 5 10:33 mc-231334-big-Data.db
Max: 12/06/2018 Min: 12/05/2018 Estimated droppable tombstones:
0.7605552563033167 4.1G Mar 5 08:12 mc-231253-big-Data.db
Max: 12/07/2018 Min: 12/06/2018 Estimated droppable tombstones:
0.7748787995519647 3.9G Mar 5 10:55 mc-231386-big-Data.db
Max: 12/08/2018 Min: 12/07/2018 Estimated droppable tombstones:
0.7998981602050579 4.1G Mar 5 08:37 mc-231275-big-Data.db
Max: 12/09/2018 Min: 12/08/2018 Estimated droppable tombstones:
0.8047662079574316 4.5G Mar 5 03:35 mc-231043-big-Data.db
Max: 12/10/2018 Min: 12/09/2018 Estimated droppable tombstones:
0.7987046261073453 4.8G Mar 4 23:36 mc-230870-big-Data.db
Max: 12/11/2018 Min: 12/10/2018 Estimated droppable tombstones:
0.8346316850246404 5.6G Mar 5 13:10 mc-231478-big-Data.db
Max: 12/12/2018 Min: 12/11/2018 Estimated droppable tombstones:
0.8336216107728608 6.1G Mar 5 00:06 mc-230888-big-Data.db
Max: 12/13/2018 Min: 12/12/2018 Estimated droppable tombstones: 0.8566337089121
7.2G Mar 5 02:46 mc-230993-big-Data.db
Max: 12/14/2018 Min: 12/13/2018 Estimated droppable tombstones:
0.8137644691768783 4.7G Mar 5 10:32 mc-231358-big-Data.db
Max: 12/15/2018 Min: 12/14/2018 Estimated droppable tombstones:
0.8166609937509232 4.6G Mar 5 13:59 mc-231525-big-Data.db
Max: 12/16/2018 Min: 12/15/2018 Estimated droppable tombstones:
0.8085604043211527 4.8G Mar 5 05:00 mc-231110-big-Data.db
Max: 12/17/2018 Min: 12/16/2018 Estimated droppable tombstones:
0.8124008277006111 5.0G Mar 4 20:34 mc-230739-big-Data.db
Max: 12/18/2018 Min: 12/17/2018 Estimated droppable tombstones:
0.8197544452946743 5.0G Mar 5 12:03 mc-231430-big-Data.db
Max: 12/19/2018 Min: 12/18/2018 Estimated droppable tombstones:
0.7604684134873694 5.7G Mar 4 21:08 mc-230768-big-Data.db
Max: 12/20/2018 Min: 12/19/2018 Estimated droppable tombstones:
0.6276716162431576 6.8G Mar 4 22:39 mc-230832-big-Data.db
Max: 12/21/2018 Min: 12/20/2018 Estimated droppable tombstones:
0.6262830796548643 6.9G Mar 4 21:23 mc-230778-big-Data.db
Max: 12/22/2018 Min: 12/21/2018 Estimated droppable tombstones:
0.6245678218315354 6.7G Mar 5 09:22 mc-231304-big-Data.db
Max: 12/23/2018 Min: 12/22/2018 Estimated droppable tombstones:
0.6339901894339154 6.7G Mar 5 00:06 mc-230897-big-Data.db
Max: 12/24/2018 Min: 12/23/2018 Estimated droppable tombstones:
0.6401085489180292 6.8G Mar 5 00:17 mc-230901-big-Data.db
Max: 12/25/2018 Min: 12/24/2018 Estimated droppable tombstones:
0.648027924752315 6.9G Mar 4 22:04 mc-230809-big-Data.db
Max: 12/26/2018 Min: 12/25/2018 Estimated droppable tombstones:
0.6465660696516876 7.0G Mar 4 23:16 mc-230856-big-Data.db
Max: 12/27/2018 Min: 12/26/2018 Estimated droppable tombstones:
0.5464676457788102 5.9G Mar 5 08:46 mc-231285-big-Data.db
Max: 12/28/2018 Min: 12/27/2018 Estimated droppable tombstones:
0.5556336150105652 5.8G Mar 5 09:03 mc-231298-big-Data.db
Max: 12/29/2018 Min: 12/28/2018 Estimated droppable tombstones:
0.5884672237873865 6.1G Mar 4 20:32 mc-230741-big-Data.db
Max: 12/30/2018 Min: 12/29/2018 Estimated droppable tombstones:
0.6116207911770754 6.3G Mar 4 21:52 mc-230801-big-Data.db
Max: 12/31/2018 Min: 12/30/2018 Estimated droppable tombstones:
0.6156449592384619 6.6G Mar 5 09:48 mc-231332-big-Data.db
Currently our data on disk is filling up quickly because we are unable to
successfully evict this data. Is there a way to
1.
1. Cleanup what is currently taking up so much disk space
2.
2. Mitigate this entirely in the future
Any help would be greatly appreciated!!
Thanks,
Nick Hatfield
From: Surbhi Gupta <surbhi.gupt...@gmail.com<mailto:surbhi.gupt...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Thursday, March 7, 2019 at 11:50 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: can i...
Send the details
On Thu, Mar 7, 2019 at 8:45 AM Nick Hatfield
<nick.hatfi...@metricly.com<mailto:nick.hatfi...@metricly.com>> wrote:
Use this email to get some insight on how to fix database issues in our cluster?
