[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Attachment: 6357-revert-v4.txt > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 rc1 > > Attachments: 6357-revert-v2.txt, 6357-revert-v3.txt, > 6357-revert-v4.txt, 6357-revert.txt, 6357-v2.txt, 6357.txt, > c6357-2.1-stress-write-adj-ops-sec.png, > c6357-2.1-stress-write-latency-99th.png, > c6357-2.1-stress-write-latency-median.png, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Attachment: 6357-v4.txt v4. > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 rc1 > > Attachments: 6357-revert-v2.txt, 6357-revert-v3.txt, > 6357-revert-v4.txt, 6357-revert.txt, 6357-v2.txt, 6357.txt, > c6357-2.1-stress-write-adj-ops-sec.png, > c6357-2.1-stress-write-latency-99th.png, > c6357-2.1-stress-write-latency-median.png, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Attachment: (was: 6357-v4.txt) > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 rc1 > > Attachments: 6357-revert-v2.txt, 6357-revert-v3.txt, > 6357-revert-v4.txt, 6357-revert.txt, 6357-v2.txt, 6357.txt, > c6357-2.1-stress-write-adj-ops-sec.png, > c6357-2.1-stress-write-latency-99th.png, > c6357-2.1-stress-write-latency-median.png, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Fix Version/s: 2.1 beta2 > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 beta1, 2.1 beta2 > > Attachments: 6357-revert-v2.txt, 6357-revert.txt, 6357-v2.txt, > 6357.txt, c6357-2.1-stress-write-adj-ops-sec.png, > c6357-2.1-stress-write-latency-99th.png, > c6357-2.1-stress-write-latency-median.png, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Attachment: 6357-revert-v2.txt All right. I'll put the mutable set back here in the meantime to keep the revert as clean as possible. > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 beta1 > > Attachments: 6357-revert-v2.txt, 6357-revert.txt, 6357-v2.txt, > 6357.txt, c6357-2.1-stress-write-adj-ops-sec.png, > c6357-2.1-stress-write-latency-99th.png, > c6357-2.1-stress-write-latency-median.png, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Attachment: 6357-revert.txt > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 beta1 > > Attachments: 6357-revert.txt, 6357-v2.txt, 6357.txt, > c6357-2.1-stress-write-adj-ops-sec.png, > c6357-2.1-stress-write-latency-99th.png, > c6357-2.1-stress-write-latency-median.png, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dan jatnieks updated CASSANDRA-6357: Attachment: c6357-2.1-stress-write-adj-ops-sec.png c6357-2.1-stress-write-latency-median.png c6357-2.1-stress-write-latency-99th.png > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 beta1 > > Attachments: 6357-v2.txt, 6357.txt, > c6357-2.1-stress-write-adj-ops-sec.png, > c6357-2.1-stress-write-latency-99th.png, > c6357-2.1-stress-write-latency-median.png, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Tester: dan jatnieks (was: Quentin Conner) Labels: performance (was: ) > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 beta1 > > Attachments: 6357-v2.txt, 6357.txt, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Fix Version/s: (was: 2.0.6) 2.1 v3 pushed to https://github.com/jbellis/cassandra/commits/6357-3, with some renaming pulled into a separate commit. bit messier than I was hoping so retargetting for 2.1. > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Fix For: 2.1 > > Attachments: 6357-v2.txt, 6357.txt, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Attachment: 6357-v2.txt v2 attached that handles blacklisted-flush-location > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis > Fix For: 2.0.6 > > Attachments: 6357-v2.txt, 6357.txt, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Reviewer: Aleksey Yeschenko > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis > Fix For: 2.0.6 > > Attachments: 6357-v2.txt, 6357.txt, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Priority: Minor (was: Major) Fix Version/s: 2.0.6 Issue Type: New Feature (was: Improvement) > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis >Priority: Minor > Fix For: 2.0.6 > > Attachments: 6357-v2.txt, 6357.txt, > c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dan jatnieks updated CASSANDRA-6357: Attachment: c6357-stress-write-latency-99th-1.png > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis > Attachments: 6357.txt, c6357-stress-write-latency-99th-1.png > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6357) Flush memtables to separate directory
[ https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-6357: -- Tester: Quentin Conner Summary: Flush memtables to separate directory (was: Flush memtables to seperate directory) > Flush memtables to separate directory > - > > Key: CASSANDRA-6357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6357 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Patrick McFadin >Assignee: Jonathan Ellis > Attachments: 6357.txt > > > Flush writers are a critical element for keeping a node healthy. When several > compactions run on systems with low performing data directories, IO becomes a > premium. Once the disk subsystem is saturated, write IO is blocked which will > cause flush writer threads to backup. Since memtables are large blocks of > memory in the JVM, too much blocking can cause excessive GC over time > degrading performance. In the worst case causing an OOM. > Since compaction is running on the data directories. My proposal is to create > a separate directory for flushing memtables. Potentially we can use the same > methodology of keeping the commit log separate and minimize disk contention > against the critical function of the flushwriter. -- This message was sent by Atlassian JIRA (v6.1#6144)
