[
https://issues.apache.org/jira/browse/CASSANDRA-14466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504784#comment-16504784
]
Brian O'Neill edited comment on CASSANDRA-14466 at 6/7/18 3:57 PM:
-------------------------------------------------------------------
Although I don't know how Cassandra behaves in this case, my own experiments
with O_DIRECT have shown that it only improves performance in very few
applications. In particular, applications which do very little computation (low
CPU load) and are accessing only the fastest available SSDs. Cassandra tends to
be a bit heavy on the CPU load, and so I'm skeptical that switching to O_DIRECT
by itself is the reason performance improved. In addition, I've only seen
significant improvement with O_DIRECT when bypassing the file system and
accessing the block device directly.
As was already suggested, switching off read-ahead might be the reason why
you're seeing improved performance. Although the file system is generally able
to adapt to usage patterns, I've found that explicitly setting
POSIX_FADV_RANDOM really helps for random access workloads. This behavior is
implicit with O_DIRECT. Although I see that Cassandra has utility code to call
fadvise, only the POSIX_FADV_DONTNEED option ever appears to be used.
Considering that the test machine had 128GB of RAM, you really want the OS to
manage the cache instead of Cassandra. How large was the JVM heap when running
the test? Far larger data sizes, caching more data in the Java heap will lead
to GC problems.
was (Author: bronee):
Although I don't know how Cassandra behaves in this case, my own experiments
with O_DIRECT have show that it only improves performance in very few
applications. In particular, applications which do very little computation (low
CPU load) and are accessing only the fastest available SSDs. Cassandra tends to
be a bit heavy on the CPU load, and so I'm skeptical that switching to O_DIRECT
by itself is the reason performance improved. In addition, I've only seen
significant improvement with O_DIRECT when bypassing the file system and
accessing the block device directly.
As was already suggested, switching off read-ahead might be the reason why
you're seeing improved performance. Although the file system is generally able
to adapt to usage patterns, I've found that explicitly setting
POSIX_FADV_RANDOM really helps for random access workloads. This behavior is
implicit with O_DIRECT. Although I see that Cassandra has utility code to call
fadvise, only the POSIX_FADV_DONTNEED option ever appears to be used.
Considering that the test machine had 128GB of RAM, you really want the OS to
manage the cache instead of Cassandra. How large was the JVM heap when running
the test? Far larger data sizes, caching more data in the Java heap will lead
to GC problems.
> Enable Direct I/O
> ------------------
>
> Key: CASSANDRA-14466
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14466
> Project: Cassandra
> Issue Type: New Feature
> Components: Local Write-Read Paths
> Reporter: Mulugeta Mammo
> Priority: Major
> Attachments: direct_io.patch
>
>
> Hi,
> JDK 10 introduced a new API for Direct IO that enables applications to bypass
> the file system cache and potentially improve performance. Details of this
> feature can be found at [https://bugs.openjdk.java.net/browse/JDK-8164900].
> This patch uses the JDK 10 API to enable Direct IO for the Cassandra read
> path. By default, we have disabled this feature; but it can be enabled using
> a new configuration parameter, enable_direct_io_for_read_path. We have
> conducted a Cassandra read-only stress test and measured a throughput gain of
> up to 60% on flash drives.
> The patch requires JDK 10 Cassandra Support -
> https://issues.apache.org/jira/browse/CASSANDRA-9608
> Please review the patch and let us know your feedback.
> Thanks,
> [^direct_io.patch]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
