[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091987#comment-13091987 ]
Jonathan Ellis commented on CASSANDRA-1608: ------------------------------------------- ... didn't quite get the DT registration right the first time, but it's committed again now. Still getting seeing a few failures when I make Leveled the default for the test suite. Most are bogus (e.g. test assuming it can request compaction of arbitrary sstables) but some are not: {noformat} [junit] Test org.apache.cassandra.db.ColumnFamilyStoreTest FAILED (crashed) [junit] Testsuite: org.apache.cassandra.db.ColumnFamilyTest [junit] Test org.apache.cassandra.db.ColumnFamilyStoreTest FAILED (crashed) [junit] Testsuite: org.apache.cassandra.db.ColumnFamilyTest {noformat} Actually, it's quite likely that these are manifestations of the CASSANDRA-3085 problem, which is more likely to bite us in practice now that there are many more sstables generated. So I'll address that first before worrying about these. Additional note: test suite runs about 20% slower for me w/ Leveled compactions. Unsure if that should be expected. > Redesigned Compaction > --------------------- > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core > Reporter: Chris Goffinet > Assignee: Benjamin Coverston > Attachments: 1608-22082011.txt, 1608-v2.txt, 1608-v4.txt, 1608-v5.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira