[
https://issues.apache.org/jira/browse/CASSANDRA-19785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868730#comment-17868730
]
Paul Chandler commented on CASSANDRA-19785:
-------------------------------------------
[~brandon.williams] thanks for your advice, we have been spending the week,
creating a test harness for this 4.0.13. This is now a 3 node cluster running a
vanilla apache cassandra 4.0.13.
We found the largest memory increase when the test harness was sending batch
writes. The following describes the test:
Traffic was started around 10:10am where we had been sending batch write
requests. The total batch requests that we were sending was around 5K where the
batch size was 20. The batch creates different keys and there are chances that
it might have put the same key as well as its just a simulator that takes the
keys within a range. The traffic was reduced around 12:30 pm. At around
13:30pm, the traffic was stopped. Along with the batch requests, read requests
were run where the simulator was fetching 1 key and the no of read requests
that was pumped was around 15-20K.
During the run we were taking heap dumps at different times, examining the heap
dumps and concentrating on the MutationStage threads we could see the number of
threads increasing, along with the total memory footprint and memory per
thread. After the traffic stopped the memory footprint dropped back slightly,
but no where near what it was at the start of the run. The details can be seen
here:
|Time|Threads| Memory Usage | |Memory per thread|
|10:10| | |Process starts| |
|10:16|92| 88,923,216| | 966,557|
|11:11|80| 138,417,912| | 1,730,224|
|12:36|106| 228,661,192| | 2,157,181|
|13:21|168| 375,284,728| | 2,233,838|
|13:30| | |Processed stopped |
|14:14|168| 341,755,952| | 2,034,262|
|14:56|167| 340,695,720| | 2,040,094|
We are not really sure what this all means, but it does look a bit like a
memory leak somewhere
> Possible memory leak in BTree.FastBuilder
> ------------------------------------------
>
> Key: CASSANDRA-19785
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19785
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Reporter: Paul Chandler
> Priority: Normal
> Fix For: 4.0.x
>
> Attachments: image-2024-07-19-08-44-56-714.png,
> image-2024-07-19-08-45-17-289.png, image-2024-07-19-08-45-33-933.png,
> image-2024-07-19-08-45-50-383.png, image-2024-07-19-08-46-06-919.png,
> image-2024-07-19-08-46-42-979.png, image-2024-07-19-08-46-56-594.png,
> image-2024-07-19-08-47-19-517.png, image-2024-07-19-08-47-34-582.png
>
>
> We are having a problem with the heap growing in size, This is a large
> cluster > 1,000 nodes across a large number of dc’s. This is running version
> 4.0.11.
>
> Each node has a 32GB heap, and the amount used continues to grow until it
> reaches 30GB, it then struggles with multiple Full GC pauses, as can be seen
> here:
> !image-2024-07-19-08-44-56-714.png!
> We took 2 heap dumps on one node a few days after it was restarted, and the
> heap had grown by 2.7GB
>
> 9{^}th{^} July
> !image-2024-07-19-08-45-17-289.png!
> 11{^}th{^} July
> !image-2024-07-19-08-45-33-933.png!
> This can be seen as mainly an increase of memory used by
> FastThreadLocalThread, increasing from 5.92GB to 8.53GB
> !image-2024-07-19-08-45-50-383.png!
> !image-2024-07-19-08-46-06-919.png!
> Looking deeper into this it can be seen that the growing heap is contained
> within the threads for the MutationStage, Native-transport-Requests,
> ReadStage etc. We would expect the memory used within these threads to be
> short lived, and not grow as time goes on. We recently increased the size of
> theses threadpools, and that has increased the size of the problem.
>
> Top memory usage for FastThreadLocalThread
> 9{^}th{^} July
> !image-2024-07-19-08-46-42-979.png!
> 11{^}th{^} July
> !image-2024-07-19-08-46-56-594.png!
> This has led us to investigate whether there could be a memory leak, and we
> have found the following issues within the retained references in
> BTree.FastBuilder objects. The issue appears to stem from the reset() method,
> which does not properly clear all buffers. We are not really sure how the
> BTree.FastBuilder works, but this this is our analysis of where a leak might
> occur.
>
> Specifically:
> Leaf Buffer Not Being Cleared:
> When leaf().count is 0, the statement Arrays.fill(leaf().buffer, 0,
> leaf().count, null); does not clear the buffer because the end index is 0.
> This leaves the buffer with references to potentially large objects,
> preventing garbage collection and increasing heap usage.
> Branch inUse Property:
> If the inUse property of the branch is set to false elsewhere in the code,
> the while loop while (branch != null && branch.inUse) does not execute,
> resulting in uncleared branch buffers and retained references.
>
> This is based on the following observations:
> Heap Dumps: Analysis of heap dumps shows that leaf().count is often 0,
> and as a result, the buffer is not being cleared, leading to high heap
> utilization.
> !image-2024-07-19-08-47-19-517.png!
> Remote Debugging: Debugging sessions indicate that the drain() method
> sets count to 0, and the inUse flag for the parent branch is set to false,
> preventing the while loop in reset() from clearing the branch buffers.
> !image-2024-07-19-08-47-34-582.png!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
