[
https://issues.apache.org/jira/browse/CASSANDRA-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441223#comment-16441223
]
Ariel Weisberg commented on CASSANDRA-13896:
--------------------------------------------
I'm not sure what [~PrinceNana] was proposing to eliminate the contention
entirely. Unless we have each thread allocate from its own memory region they
will be shared and it will have to be an atomic operation.
If we could {{incrementAndGet}} instead of {{compareAndSet}} it might also be
faster under contention.
I think splitting the contended resource is a classic solution to scaling
contended access to shared state. The way Cassandra currently works with large
numbers of threads it's the right kind of optimization. It's not totally crazy
to optimize this based on microbenchmarks of the allocator rather then end to
end benchmarks. Having headroom in the allocator is worth the risk of this
particular change IMO, but we should know what we are getting. If we attempted
to prove the correctness of the allocator we could be more confident about
changing it. Like have multiple threads do a bunch of allocations and then
compare the allocations across threads for correctness. They don't overlap,
they are one after another, maybe other checks.
The contended annotation is not used correctly (I think) as we don't
technically know where the contended atomic integer is stored and it would
probably be stored away from the object just due to the annotation. The
annotation only applies to the memory around the class or field being annotated
(I think). We also have two contended fields, the allocation and the count. I
think maybe count can just go away or alternatively be CASed into a single
atomic long along with the offset. Assuming we can't manage to make that
incrementAndGet.
Once we have consensus on this approach vs some other I can look closer at is
this correct. And the answer for that is usually going to be well is there a
test that convinces me it's correct?
> Improving Cassandra write performance
> ---------------------------------------
>
> Key: CASSANDRA-13896
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13896
> Project: Cassandra
> Issue Type: Improvement
> Components: Local Write-Read Paths
> Environment: Skylake server with 2 sockets, 192GB RAM, 3xPCIe SSDs
> OS: Centos 7.3
> Java: Oracle JDK1.8.0_121
> Reporter: Prince Nana Owusu Boateng
> Priority: Major
> Labels: Performance
> Fix For: 4.x
>
> Attachments: Screen Shot 2017-09-22 at 11.22.43 AM.png, Screen Shot
> 2017-09-22 at 3.31.09 PM.png
>
>
> During our Cassandra performance testing, we see high percentage of the CPU
> spent in *org.apache.cassandra.utils.memory.SlabAllocator.allocate(int,
> OpOrder Group) * method. Appears to be high contention of the
> *<nextFreeOffset>* atomic Integer in write workloads. This structure is
> used by the threads for keeping track of the region bytebuffer allocation.
> When the contention appears, adding more clients, modifications of write
> specific parameters does not change write throughput performance. Attached
> are the details of Java Flight Recorder (JFR), showing hot functions and also
> performance results. When we see this contention, we still have plenty of
> CPU and throughput left ( *<20%* Total average CPU utilization and *<11%*
> of the storage write total throughput). This occurs on Cassandra 3.10.0-src
> version using the Cassandra-Stress.
> Proposal:
> We will like to introduce a solution which eliminates the atomic operations
> on the *<nextFreeOffset>* atomic Integer. This implementation will allow
> concurrent allocation of bytebuffers without an atomic compareAndSet and
> incrementAndGet operations. The solution is expected to increase overall
> write performance while improving CPU utilization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
