[ https://issues.apache.org/jira/browse/CASSANDRA-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McGuire updated CASSANDRA-6916: ------------------------------------ Attachment: 6916v3-preempive-open-compact.mixed.2.logs.tar.gz [~benedict] Yea, this is killer. Here's a mixed read/write workload: http://riptano.github.io/cassandra_performance/graph_v3/graph.html?stats=stats.6916v3-preempive-open-compact.mixed.2.json&metric=op_rate&operation=mixed Logs are attached as: 6916v3-preempive-open-compact.mixed.2.logs.tar.gz > Preemptive opening of compaction result > --------------------------------------- > > Key: CASSANDRA-6916 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6916 > Project: Cassandra > Issue Type: Bug > Components: Core > Reporter: Benedict > Assignee: Benedict > Priority: Minor > Labels: performance > Fix For: 2.1 > > Attachments: 6916v3-preempive-open-compact.logs.gz, > 6916v3-preempive-open-compact.mixed.2.logs.tar.gz > > > Related to CASSANDRA-6812, but a little simpler: when compacting, we mess > quite badly with the page cache. One thing we can do to mitigate this problem > is to use the sstable we're writing before we've finished writing it, and to > drop the regions from the old sstables from the page cache as soon as the new > sstables have them (even if they're only written to the page cache). This > should minimise any page cache churn, as the old sstables must be larger than > the new sstable, and since both will be in memory, dropping the old sstables > is at least as good as dropping the new. > The approach is quite straight-forward. Every X MB written: > # grab flushed length of index file; > # grab second to last index summary record, after excluding those that point > to positions after the flushed length; > # open index file, and check that our last record doesn't occur outside of > the flushed length of the data file (pretty unlikely) > # Open the sstable with the calculated upper bound > Some complications: > # must keep running copy of compression metadata for reopening with > # we need to be able to replace an sstable with itself but a different lower > bound > # we need to drop the old page cache only when readers have finished -- This message was sent by Atlassian JIRA (v6.2#6252)