[ https://issues.apache.org/jira/browse/CASSANDRA-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616612#comment-14616612 ]
Jeff Williams commented on CASSANDRA-9577: ------------------------------------------ We are also running on EC2. The data disk is a raid0 of the instance's ephemeral storage (2 x 40GB SSD) mounted as: /dev/md0 on /var/lib/cassandra type xfs (rw,_netdev). Yes, this is just happening on a single node. There are 8 other nodes in the cluster and only this one is showing this behaviour. > Cassandra not performing GC on stale SStables after compaction > -------------------------------------------------------------- > > Key: CASSANDRA-9577 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9577 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.0.12.200 / DSE 4.6.1. > Reporter: Jeff Ferland > Assignee: Marcus Eriksson > > Space used (live), bytes: 878681716067 > Space used (total), bytes: 2227857083852 > jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ sudo lsof *-Data.db > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > java 4473 cassandra 446r REG 0,26 17582559172 39241 > trends-trends-jb-144864-Data.db > java 4473 cassandra 448r REG 0,26 62040962 37431 > trends-trends-jb-144731-Data.db > java 4473 cassandra 449r REG 0,26 829935047545 21150 > trends-trends-jb-143581-Data.db > java 4473 cassandra 452r REG 0,26 8980406 39503 > trends-trends-jb-144882-Data.db > java 4473 cassandra 454r REG 0,26 8980406 39503 > trends-trends-jb-144882-Data.db > java 4473 cassandra 462r REG 0,26 9487703 39542 > trends-trends-jb-144883-Data.db > java 4473 cassandra 463r REG 0,26 36158226 39629 > trends-trends-jb-144889-Data.db > java 4473 cassandra 468r REG 0,26 105693505 39447 > trends-trends-jb-144881-Data.db > java 4473 cassandra 530r REG 0,26 17582559172 39241 > trends-trends-jb-144864-Data.db > java 4473 cassandra 535r REG 0,26 105693505 39447 > trends-trends-jb-144881-Data.db > java 4473 cassandra 542r REG 0,26 9487703 39542 > trends-trends-jb-144883-Data.db > java 4473 cassandra 553u REG 0,26 6431729821 39556 > trends-trends-tmp-jb-144884-Data.db > jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ ls *-Data.db > trends-trends-jb-142631-Data.db trends-trends-jb-143562-Data.db > trends-trends-jb-143581-Data.db trends-trends-jb-144731-Data.db > trends-trends-jb-144883-Data.db > trends-trends-jb-142633-Data.db trends-trends-jb-143563-Data.db > trends-trends-jb-144530-Data.db trends-trends-jb-144864-Data.db > trends-trends-jb-144889-Data.db > trends-trends-jb-143026-Data.db trends-trends-jb-143564-Data.db > trends-trends-jb-144551-Data.db trends-trends-jb-144881-Data.db > trends-trends-tmp-jb-144884-Data.db > trends-trends-jb-143533-Data.db trends-trends-jb-143578-Data.db > trends-trends-jb-144552-Data.db trends-trends-jb-144882-Data.db > jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ cd - > /mnt/cassandra/data/trends/trends > jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ sudo lsof * > jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ ls *-Data.db > trends-trends-jb-124502-Data.db trends-trends-jb-141113-Data.db > trends-trends-jb-141377-Data.db trends-trends-jb-141846-Data.db > trends-trends-jb-144890-Data.db > trends-trends-jb-125457-Data.db trends-trends-jb-141123-Data.db > trends-trends-jb-141391-Data.db trends-trends-jb-141871-Data.db > trends-trends-jb-41121-Data.db > trends-trends-jb-130016-Data.db trends-trends-jb-141137-Data.db > trends-trends-jb-141538-Data.db trends-trends-jb-141883-Data.db > trends-trends.trends_date_idx-jb-2100-Data.db > trends-trends-jb-139563-Data.db trends-trends-jb-141358-Data.db > trends-trends-jb-141806-Data.db trends-trends-jb-142033-Data.db > trends-trends-jb-141102-Data.db trends-trends-jb-141363-Data.db > trends-trends-jb-141829-Data.db trends-trends-jb-144553-Data.db > Compaction started INFO [CompactionExecutor:6661] 2015-06-05 14:02:36,515 > CompactionTask.java (line 120) Compacting > [SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-124502-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141358-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141883-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141846-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141871-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141391-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-139563-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-125457-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141806-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141102-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141123-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-41121-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141137-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141113-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-142033-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141829-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141377-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141538-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141363-Data.db'), > > SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-130016-Data.db')] > Compaction ended: INFO [CompactionExecutor:6661] 2015-06-09 06:54:27,527 > CompactionTask.java (line 296) Compacted 20 sstables to > [/ebs/cassandra/data/trends/trends/trends-trends-jb-143581,]. > 1,260,539,306,144 bytes to 829,935,047,545 (~65% of original) in > 319,910,989ms = 2.474087MB/s. 1,913,263,636 total partitions merged to > 873,858,309. Partition merge counts were {1:391213541, 2:241603436, > 3:102139024, 4:52438965, 5:47749892, 6:13962210, 7:9923980, 8:8363815, > 9:3626260, 10:1413014, 11:615314, 12:336503, 13:416749, 14:55606, } > date: Wed Jun 10 23:43:31 UTC 2015 -- This message was sent by Atlassian JIRA (v6.3.4#6332)