[
https://issues.apache.org/jira/browse/CASSANDRA-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13945219#comment-13945219
]
dan jatnieks commented on CASSANDRA-6357:
-----------------------------------------
Retested this using trunk (as of Mar 13). I switched to different hardware with
7200rpm disks because the slow 5400rpm disks on the other system just couldn't
keep up without throttling stress op/s.
The machine this time was a Dell with two quad core hyper-threaded Intel Xeon
E5620 CPU, 32Gb memory, and 8 disks (500Gb, 7200 rpm).
The two scenarios were the same as last time and used the same stress
parameters.
The results were much less dramatic than the 2.0 test.
The base test results (data and flush on the same device) :
{noformat}
real op rate : 9433
adjusted op rate : 9435
adjusted op rate stderr : 0
key rate : 9433
latency mean : 5.3
latency median : 1.4
latency 95th percentile : 5.4
latency 99th percentile : 12.9
latency 99.9th percentile : 259.4
latency max : 20873.9
Total operation time : 00:17:40
{noformat}
The flush test results (data and flush on separate devices):
{noformat}
real op rate : 10391
adjusted op rate : 10391
adjusted op rate stderr : 0
key rate : 10391
latency mean : 4.8
latency median : 1.4
latency 95th percentile : 5.4
latency 99th percentile : 14.2
latency 99.9th percentile : 245.0
latency max : 17035.2
Total operation time : 00:16:02
{noformat}
See attached graphs:
[2.1 Stress Write Latency 99.9th
Percentile|^c6357-2.1-stress-write-latency-99th.png]
[2.1 Stress Write Median Latency|^c6357-2.1-stress-write-latency-median.png]
[2.1 Stress Write Adjusted Ops/sec|^c6357-2.1-stress-write-adj-ops-sec.png]
> Flush memtables to separate directory
> -------------------------------------
>
> Key: CASSANDRA-6357
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6357
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Patrick McFadin
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: performance
> Fix For: 2.1 beta1
>
> Attachments: 6357-v2.txt, 6357.txt,
> c6357-2.1-stress-write-adj-ops-sec.png,
> c6357-2.1-stress-write-latency-99th.png,
> c6357-2.1-stress-write-latency-median.png,
> c6357-stress-write-latency-99th-1.png
>
>
> Flush writers are a critical element for keeping a node healthy. When several
> compactions run on systems with low performing data directories, IO becomes a
> premium. Once the disk subsystem is saturated, write IO is blocked which will
> cause flush writer threads to backup. Since memtables are large blocks of
> memory in the JVM, too much blocking can cause excessive GC over time
> degrading performance. In the worst case causing an OOM.
> Since compaction is running on the data directories. My proposal is to create
> a separate directory for flushing memtables. Potentially we can use the same
> methodology of keeping the commit log separate and minimize disk contention
> against the critical function of the flushwriter.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
