[ https://issues.apache.org/jira/browse/CASSANDRA-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963159#comment-13963159 ]
Benedict commented on CASSANDRA-6916: ------------------------------------- Probably nothing - it looks like an eminently sensible idea. I'll give it a go for my next version and let you know if it goes pear shaped :) FTR, if possible, I'll probably extend SSTableWriter to an SSTableRewriter, to keep the functionality separate. > Preemptive opening of compaction result > --------------------------------------- > > Key: CASSANDRA-6916 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6916 > Project: Cassandra > Issue Type: Bug > Components: Core > Reporter: Benedict > Assignee: Benedict > Priority: Minor > Fix For: 2.1 > > > Related to CASSANDRA-6812, but a little simpler: when compacting, we mess > quite badly with the page cache. One thing we can do to mitigate this problem > is to use the sstable we're writing before we've finished writing it, and to > drop the regions from the old sstables from the page cache as soon as the new > sstables have them (even if they're only written to the page cache). This > should minimise any page cache churn, as the old sstables must be larger than > the new sstable, and since both will be in memory, dropping the old sstables > is at least as good as dropping the new. > The approach is quite straight-forward. Every X MB written: > # grab flushed length of index file; > # grab second to last index summary record, after excluding those that point > to positions after the flushed length; > # open index file, and check that our last record doesn't occur outside of > the flushed length of the data file (pretty unlikely) > # Open the sstable with the calculated upper bound > Some complications: > # must keep running copy of compression metadata for reopening with > # we need to be able to replace an sstable with itself but a different lower > bound > # we need to drop the old page cache only when readers have finished -- This message was sent by Atlassian JIRA (v6.2#6252)