[
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661948#comment-16661948
]
Benjamin Roth commented on CASSANDRA-13241:
-------------------------------------------
4-8kb will not "destroy" the OS page cache. Linux Pages are 4kb by default, so
4kb chunks perfectly fit into cache pages. Actually read-ahead will kill your
performance if you have a lot of disk-reads going on. This can kill your page
cache if your dataset is a lot larger than available memory and you are doing
many random reads with small resultsets.
We use 4kb chunks and we observed a TREMENDOUS difference in IO reads when
disabling read ahead completely. With default read ahead kernel settings, the
physical read IO is roughly 20-30x in our use case, specifically it was like
~20MB/s vs 600MB/s.
Sum-up: Not 4KB chunk size alone is the problem but all components have to be
tuned and aligned to remove bottlenecks and make the whole system performant.
The specific params always depend on the particular case.
> Lower default chunk_length_in_kb from 64kb to 16kb
> --------------------------------------------------
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
> Issue Type: Wish
> Components: Core
> Reporter: Benjamin Roth
> Assignee: Ariel Weisberg
> Priority: Major
> Attachments: CompactIntegerSequence.java,
> CompactIntegerSequenceBench.java, CompactSummingIntegerSequence.java
>
>
> Having a too low chunk size may result in some wasted disk space. A too high
> chunk size may lead to massive overreads and may have a critical impact on
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) /
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but
> if the model consists rather of small rows or small resultsets, the read
> overhead with 64kb chunk size is insanely high. This applies for example for
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic
> snitch magic": https://cl.ly/3E0t1T1z2c0J
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
