[ https://issues.apache.org/jira/browse/CASSANDRA-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16838232#comment-16838232 ]
Sumanth Pasupuleti commented on CASSANDRA-15013: ------------------------------------------------ Thanks for the feedback [~benedict]. I have incorporated it in the branch https://github.com/sumanth-pasupuleti/cassandra/commits/15013_trunk_2 However, with this change, we will lose the ability to change the endpoint/global limits on a node on the fly, without restarting C* process. Added forceAllocate method to ResourceLimits, to accommodate backpressure scenario. A few DTest failures, seemingly from race conditions on allocate/release. Working on figuring out where the race is coming from. Dtest results: https://circleci.com/gh/sumanth-pasupuleti/cassandra/369#tests https://circleci.com/gh/sumanth-pasupuleti/cassandra/368#tests > Message Flusher queue can grow unbounded, potentially running JVM out of > memory > ------------------------------------------------------------------------------- > > Key: CASSANDRA-15013 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15013 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client > Reporter: Sumanth Pasupuleti > Assignee: Sumanth Pasupuleti > Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 3.0.x, 3.11.x > > Attachments: BlockedEpollEventLoopFromHeapDump.png, > BlockedEpollEventLoopFromThreadDump.png, RequestExecutorQueueFull.png, heap > dump showing each ImmediateFlusher taking upto 600MB.png > > > This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue > bounded, since, in the current state, items get added to the queue without > any checks on queue size, nor with any checks on netty outbound buffer to > check the isWritable state. > We are seeing this issue hit our production 3.0 clusters quite often. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org