[ https://issues.apache.org/jira/browse/CASSANDRA-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007060#comment-14007060 ]
Marcus Eriksson commented on CASSANDRA-6696: -------------------------------------------- Just pushed a version to https://github.com/krummas/cassandra/commits/marcuse/6696-4 - I'll spend some more time writing tests, but I figure it is ready for feedback now atleast. * Flush to one sstable per disk: ** Split the total range in #disks parts ** Flush whole vnodes, if a vnode starts on a disk, it stays there. Note though that if a vnode wraps around the tokenspace, it will be split in 2 parts and be on different disks. * SSTables flushed during startup will not get placed correctly since we don't yet know the local ranges. * LeveledCompaction needs to know what ranges we have, calling startup() on the CompactionStrategy has been moved out of the CFS constructor * LCS: ** One manifest per vnode, with a global L0. ** L1 is now aims to contain one sstable ** Same prios as before, first STCS in L0, then compactions in L1+, and last L0 -> L1. ** STCS in L0 will create big per-disk files, not per-vnode ones. * STCS: ** We now have L0 and L1, L1 contains per-vnode sstables, but within the vnode-sstables, we give no overlappiness-guarantees ** Compactions in L0 only include L0 sstables, and L1 compactions only include L1 compactions, all compactions end up as per-vnode sstables in L1 ** When we get 4 sstables of similar size in L0, we will compact those, and create num_tokens L1 sstables. ** When one L1 vnode gets 4 sstables of similar size, it will compact those together ** L0 -> L1 compactions are prioritized over L1 -> L1 ones (though, these will run in parallel) * Introduces originalFirst to keep track of the original first key of the sstable, we need this when figuring out which manifest the sstable belongs to during replace(..). * If we get new ring version (i.e. we get a new token or lose one), we only reinitialize the LeveledManifestWrapper, this means that we might have sstables that start in one vnode, but does not end in it. * "nodetool rebalancedata" will iterate over all sstables and make sure they are in the correct places. * If a disk breaks/runs out of space we will flush/compact to the remaining disks > Drive replacement in JBOD can cause data to reappear. > ------------------------------------------------------ > > Key: CASSANDRA-6696 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6696 > Project: Cassandra > Issue Type: Improvement > Components: Core > Reporter: sankalp kohli > Assignee: Marcus Eriksson > Fix For: 3.0 > > > In JBOD, when someone gets a bad drive, the bad drive is replaced with a new > empty one and repair is run. > This can cause deleted data to come back in some cases. Also this is true for > corrupt stables in which we delete the corrupt stable and run repair. > Here is an example: > Say we have 3 nodes A,B and C and RF=3 and GC grace=10days. > row=sankalp col=sankalp is written 20 days back and successfully went to all > three nodes. > Then a delete/tombstone was written successfully for the same row column 15 > days back. > Since this tombstone is more than gc grace, it got compacted in Nodes A and B > since it got compacted with the actual data. So there is no trace of this row > column in node A and B. > Now in node C, say the original data is in drive1 and tombstone is in drive2. > Compaction has not yet reclaimed the data and tombstone. > Drive2 becomes corrupt and was replaced with new empty drive. > Due to the replacement, the tombstone in now gone and row=sankalp col=sankalp > has come back to life. > Now after replacing the drive we run repair. This data will be propagated to > all nodes. > Note: This is still a problem even if we run repair every gc grace. > -- This message was sent by Atlassian JIRA (v6.2#6252)