sounds like the foreground read repair can cause issues to twcs (mix old
and new data in same sstable), is there a way to disable the foreground
read repair? is that indeed the case that it's problematic?
On Mon, Dec 17, 2018 at 9:21 AM Gil Ganz <gilg...@gmail.com> wrote:
> hey jeff, attaching more information.
> so this the situation before - 3 nodes in the cluster (3.11.3 in this case
> but i saw same thing in 2.1 and 3.0), there is a script writing one row
> every minute and another script doing nodetool flush every 10 minute.
> window is defined as two hours, so after a few days this is how the
> directory listing looks :
>
> drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups
> -rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
> mc-171-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25 mc-171-big-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
> mc-172-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25 mc-172-big-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
> mc-173-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25 mc-173-big-Statistics.db
> .
> .
> -rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10
> mc-873-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10 mc-873-big-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt
> .
> .
> .
> -rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50 mc-1150-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50
> mc-1150-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50
> mc-1150-big-Statistics.db
>
>
>
> check out sstable 873 is from dec 15. at this point i remove node no 3.
> and the log shows streamed data is creating sstables 1150-1200. it also
> shows one of these sstables being compacted with sstable 873, resulting in
> sstable number 1228, an operation which from what i understand should not
> happen.
>
> DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199
> StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136]
> Received 58 sstables from /10.106.12.34
> ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0
>
>
>
> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
> TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in
> current bucket, compacting what's here:
> [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]
>
> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
> CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136)
> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0,
> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0,
> ]
>
> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578
> CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136)
> 2 sstables to
> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,]
> to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read
> Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput
> = ~184/s. 71 total partitions merged to 71. Partition merge counts were
> {1:71, }
>
> see sstablemetadata output for sstable 873 and 1228. as you can see the
> 1228 min timstamp is old.
>
> 873:
> WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider
> adding more capacity to your cluster or removing obsolete snapshots
> SSTable: /root/gil_sstables_before/mc-873-big
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.010000
> Minimum timestamp: 1544896864561707
> Maximum timestamp: 1544903882074190
> SSTable min local deletion time: 2147483647
> SSTable max local deletion time: 2147483647
> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
> Compression ratio: 0.6954296160877513
> TTL min: 0
> TTL max: 0
> First token: -7160136740246525330 (key=42)
> Last token: 8209703677368372043 (key=34)
> minClustringValues: [2018-12-15 18:01Z]
> maxClustringValues: [2018-12-15 19:58Z]
> Estimated droppable tombstones: 0.0
> SSTable Level: 0
> Repaired at: 0
> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
> position=19800215)=CommitLogPosition(segmentId=1544706489001,
> position=20586499)}
> totalColumnsSet: 62
> totalRows: 62
> Estimated tombstone drop times:
> Count Row Size Cell Count
> 1 0 24
> 2 0 7
> 3 0 5
> 4 0 1
> 5 0 1
> 6 0 0
> 7 0 0
> 8 0 0
> 10 0 0
> 12 0 0
> 14 0 0
> 17 0 0
> 20 0 0
> 24 0 0
> 29 0 0
> 35 0 0
> 42 0 0
> 50 7 0
> 60 17 0
> 72 0 0
> 86 5 0
> 103 2 0
> 124 5 0
> 149 1 0
> 179 0 0
> 215 1 0
> 258 0 0
> 310 0 0
> 372 0 0
> 446 0 0
> 535 0 0
> 642 0 0
> 770 0 0
> 924 0 0
> 1109 0 0
> 1331 0 0
> 1597 0 0
> 1916 0 0
> 2299 0 0
> 2759 0 0
> 3311 0 0
> 3973 0 0
> 4768 0 0
> 5722 0 0
> 6866 0 0
> 8239 0 0
> 9887 0 0
> 11864 0 0
> 14237 0 0
> 17084 0 0
> 20501 0 0
> 24601 0 0
> 29521 0 0
> 35425 0 0
> 42510 0 0
> 51012 0 0
> 61214 0 0
> 73457 0 0
> 88148 0 0
> 105778 0 0
> 126934 0 0
> 152321 0 0
> 182785 0 0
> 219342 0 0
> 263210 0 0
> 315852 0 0
> 379022 0 0
> 454826 0 0
> 545791 0 0
> 654949 0 0
> 785939 0 0
> 943127 0 0
> 1131752 0 0
> 1358102 0 0
> 1629722 0 0
> 1955666 0 0
> 2346799 0 0
> 2816159 0 0
> 3379391 0 0
> 4055269 0 0
> 4866323 0 0
> 5839588 0 0
> 7007506 0 0
> 8409007 0 0
> 10090808 0 0
> 12108970 0 0
> 14530764 0 0
> 17436917 0 0
> 20924300 0 0
> 25109160 0 0
> 30130992 0 0
> 36157190 0 0
> 43388628 0 0
> 52066354 0 0
> 62479625 0 0
> 74975550 0 0
> 89970660 0 0
> 107964792 0 0
> 129557750 0 0
> 155469300 0 0
> 186563160 0 0
> 223875792 0 0
> 268650950 0 0
> 322381140 0 0
> 386857368 0 0
> 464228842 0 0
> 557074610 0 0
> 668489532 0 0
> 802187438 0 0
> 962624926 0 0
> 1155149911 0 0
> 1386179893 0 0
> 1663415872 0 0
> 1996099046 0 0
> 2395318855 0 0
> 2874382626 0
> 3449259151 0
> 4139110981 0
> 4966933177 0
> 5960319812 0
> 7152383774 0
> 8582860529 0
> 10299432635 0
> 12359319162 0
> 14831182994 0
> 17797419593 0
> 21356903512 0
> 25628284214 0
> 30753941057 0
> 36904729268 0
> 44285675122 0
> 53142810146 0
> 63771372175 0
> 76525646610 0
> 91830775932 0
> 110196931118 0
> 132236317342 0
> 158683580810 0
> 190420296972 0
> 228504356366 0
> 274205227639 0
> 329046273167 0
> 394855527800 0
> 473826633360 0
> 568591960032 0
> 682310352038 0
> 818772422446 0
> 982526906935 0
> 1179032288322 0
> 1414838745986 0
> Estimated cardinality: 38
> EncodingStats minTTL: 0
> EncodingStats minLocalDeletionTime: 1442880000
> EncodingStats minTimestamp: 1544896864561707
> KeyType: org.apache.cassandra.db.marshal.Int32Type
> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
> StaticColumns: {}
> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
> You have new mail in /var/spool/mail/root
>
>
>
> 1228 :
>
> WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider
> adding more capacity to your cluster or removing obsolete snapshots
> SSTable:
> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.010000
> Minimum timestamp: 1544896805142916
> Maximum timestamp: 1544903882074190
> SSTable min local deletion time: 2147483647
> SSTable max local deletion time: 2147483647
> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
> Compression ratio: 0.6942828485456369
> TTL min: 0
> TTL max: 0
> First token: -9108684050423740263 (key=53)
> Last token: 8669466869824630438 (key=77)
> minClustringValues: [2018-12-15 18:00Z]
> maxClustringValues: [2018-12-15 19:58Z]
> Estimated droppable tombstones: 0.0
> SSTable Level: 0
> Repaired at: 0
> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
> position=19800215)=CommitLogPosition(segmentId=1544706489001,
> position=20586499)}
> totalColumnsSet: 113
> totalRows: 113
> Estimated tombstone drop times:
> Count Row Size Cell Count
> 1 0 41
> 2 0 21
> 3 0 7
> 4 0 1
> 5 0 1
> 6 0 0
> 7 0 0
> 8 0 0
> 10 0 0
> 12 0 0
> 14 0 0
> 17 0 0
> 20 0 0
> 24 0 0
> 29 0 0
> 35 0 0
> 42 0 0
> 50 16 0
> 60 25 0
> 72 0 0
> 86 17 0
> 103 4 0
> 124 7 0
> 149 1 0
> 179 0 0
> 215 1 0
> 258 0 0
> 310 0 0
> 372 0 0
> 446 0 0
> 535 0 0
> 642 0 0
> 770 0 0
> 924 0 0
> 1109 0 0
> 1331 0 0
> 1597 0 0
> 1916 0 0
> 2299 0 0
> 2759 0 0
> 3311 0 0
> 3973 0 0
> 4768 0 0
> 5722 0 0
> 6866 0 0
> 8239 0 0
> 9887 0 0
> 11864 0 0
> 14237 0 0
> 17084 0 0
> 20501 0 0
> 24601 0 0
> 29521 0 0
> 35425 0 0
> 42510 0 0
> 51012 0 0
> 61214 0 0
> 73457 0 0
> 88148 0 0
> 105778 0 0
> 126934 0 0
> 152321 0 0
> 182785 0 0
> 219342 0 0
> 263210 0 0
> 315852 0 0
> 379022 0 0
> 454826 0 0
> 545791 0 0
> 654949 0 0
> 785939 0 0
> 943127 0 0
> 1131752 0 0
> 1358102 0 0
> 1629722 0 0
> 1955666 0 0
> 2346799 0 0
> 2816159 0 0
> 3379391 0 0
> 4055269 0 0
> 4866323 0 0
> 5839588 0 0
> 7007506 0 0
> 8409007 0 0
> 10090808 0 0
> 12108970 0 0
> 14530764 0 0
> 17436917 0 0
> 20924300 0 0
> 25109160 0 0
> 30130992 0 0
> 36157190 0 0
> 43388628 0 0
> 52066354 0 0
> 62479625 0 0
> 74975550 0 0
> 89970660 0 0
> 107964792 0 0
> 129557750 0 0
> 155469300 0 0
> 186563160 0 0
> 223875792 0 0
> 268650950 0 0
> 322381140 0 0
> 386857368 0 0
> 464228842 0 0
> 557074610 0 0
> 668489532 0 0
> 802187438 0 0
> 962624926 0 0
> 1155149911 0 0
> 1386179893 0 0
> 1663415872 0 0
> 1996099046 0 0
> 2395318855 0 0
> 2874382626 0
> 3449259151 0
> 4139110981 0
> 4966933177 0
> 5960319812 0
> 7152383774 0
> 8582860529 0
> 10299432635 0
> 12359319162 0
> 14831182994 0
> 17797419593 0
> 21356903512 0
> 25628284214 0
> 30753941057 0
> 36904729268 0
> 44285675122 0
> 53142810146 0
> 63771372175 0
> 76525646610 0
> 91830775932 0
> 110196931118 0
> 132236317342 0
> 158683580810 0
> 190420296972 0
> 228504356366 0
> 274205227639 0
> 329046273167 0
> 394855527800 0
> 473826633360 0
> 568591960032 0
> 682310352038 0
> 818772422446 0
> 982526906935 0
> 1179032288322 0
> 1414838745986 0
> Estimated cardinality: 71
> EncodingStats minTTL: 0
> EncodingStats minLocalDeletionTime: 1442880000
> EncodingStats minTimestamp: 1544896805142916
> KeyType: org.apache.cassandra.db.marshal.Int32Type
> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
> StaticColumns: {}
> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>
> thanks
> gil
>
>
>>
>> ---------- Forwarded message ---------
>> From: Jeff Jirsa <jji...@gmail.com>
>> Date: Fri, Dec 14, 2018 at 8:56 AM
>> Subject: Re: TWCS sstables gets merged following node removal
>> To: <user@cassandra.apache.org>
>>
>>
>> Remove node will stream data from all windows to remote nodes , so some
>> compaction is expected
>>
>> Would need to see the sstablemetadata to understand what’s happening
>> there.
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Dec 13, 2018, at 10:26 PM, Roy Burstein <burstein....@gmail.com>
>> wrote:
>>
>> Hi all ,
>>
>> My colleague opened Jira ticket for the issue but we are struggling with
>> this issue for a while and we have space issues :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-14929
>>
>>
>> After removing a node from the cluster, a table that is defined as TWCS,
>> has sstables from different time windows merged together, making old and
>> new data sit in the same sstable(the full example is in the ticket).
>>
>> Any idea about this?workaround?Known issues?
>>
>> Thanks!
>>
>> Roy
>>
>>
