Thanks Rob, Jeff. I have updated the Jira issue with my information. On 6 July 2015 at 23:46, Jeff Ferland <j...@tubularlabs.com> wrote:
> I’ve seen the same thing: > https://issues.apache.org/jira/browse/CASSANDRA-9577 > > I’ve had cases where a restart clears the old tables, and I’ve had cases > where a restart considers the old tables to be live. > > On Jul 6, 2015, at 1:51 PM, Robert Coli <rc...@eventbrite.com> wrote: > > On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams <je...@wherethebitsroam.com> > wrote: > >> 1) Cassandra version is 2.0.12. >> > > >> 2) Interesting. Looking at JMX org.apache.cassandra.db -> ColumnFamilies >> -> trackcontent -> track_content -> Attributes, I get: >> >> LiveDiskSpaceUsed: 17788740448, i.e. ~17GB >> LiveSSTableCount: 3 >> TotalDiskSpaceUsed: 55714084629, i.e. ~55GB >> >> So it obviously knows about the extra disk space even though the "live" >> space looks correct. I couldn't find anything to identify the actual files >> though. >> > > That's what I would expect. > > >> 3) So that was even more interesting. After restarting the cassandra >> daemon, the sstables were not deleted and now the same JMX attributes are: >> >> LiveDiskSpaceUsed: 55681040579, i.e. ~55GB >> LiveSSTableCount: 8 >> TotalDiskSpaceUsed: 55681040579, i.e. ~55GB >> >> So some of my non-live tables are back live again, and obviously some of >> the big ones!! >> > > This is permanently fatal to consistency; sorry if I was not clear enough > that if they were not live, there was some risk of Cassandra considering > them live again upon restart. > > If I were you, I would either stop the node and remove the files you know > shouldn't be live or do a major compaction ASAP. > > The behavior you just encountered sounds like a bug, and it is a rather > serious one. SSTables which should be dead being marked live is very bad > for consistency. > > Do you see any exceptions in your logs or anything? If you can repro, you > should file a JIRA ticket with the apache project... > > =Rob > > >
