On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams <je...@wherethebitsroam.com> wrote:
> 1) Cassandra version is 2.0.12. > > 2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies > -> trackcontent -> track_content -> Attributes, I get: > > LiveDiskSpaceUsed: 17788740448, i.e. ~17GB > LiveSSTableCount: 3 > TotalDiskSpaceUsed: 55714084629, i.e. ~55GB > > So it obviously knows about the extra disk space even though the "live" > space looks correct. I couldn't find anything to identify the actual files > though. > That's what I would expect. > 3) So that was even more interesting. After restarting the cassandra > daemon, the sstables were not deleted and now the same JMX attributes are: > > LiveDiskSpaceUsed: 55681040579, i.e. ~55GB > LiveSSTableCount: 8 > TotalDiskSpaceUsed: 55681040579, i.e. ~55GB > > So some of my non-live tables are back live again, and obviously some of > the big ones!! > This is permanently fatal to consistency; sorry if I was not clear enough that if they were not live, there was some risk of Cassandra considering them live again upon restart. If I were you, I would either stop the node and remove the files you know shouldn't be live or do a major compaction ASAP. The behavior you just encountered sounds like a bug, and it is a rather serious one. SSTables which should be dead being marked live is very bad for consistency. Do you see any exceptions in your logs or anything? If you can repro, you should file a JIRA ticket with the apache project... =Rob
