[ https://issues.apache.org/jira/browse/CASSANDRA-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819331#comment-16819331 ]
Benedict commented on CASSANDRA-15013: -------------------------------------- [~sumanth.pasupuleti] thanks for the update. As to addressing the slow leak, there are a number of similar approaches, but I would propose the following: * Instead of a {{Map<InetAddress, AtomicLong>}} in {{Dispatcher}} have a {{Map<InetAddress, Dispatcher>}} in {{Server.Initializer}} * Add a reference count to the {{Dispatcher}} as well as an atomic {{bytesInFlight}} * In {{Server.Initializer.initChannel}}, lookup the socket InetAddress: *# If there is no {{Dispatcher}} create one *# If the {{Dispatcher}} cannot increment its reference count, remove it from the map and goto (1) *# Otherwise we've taken ownership of the {{Dispatcher}} and can use it * Then, in the {{Dispatcher}}, override {{channelInactive}} to decrement our reference count and remove ourselves from the map if we've been freed This also marginally reduces the per-message cost of enforcing these constraints. WDYT? > Message Flusher queue can grow unbounded, potentially running JVM out of > memory > ------------------------------------------------------------------------------- > > Key: CASSANDRA-15013 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15013 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client > Reporter: Sumanth Pasupuleti > Assignee: Sumanth Pasupuleti > Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 3.0.x, 3.11.x > > Attachments: BlockedEpollEventLoopFromHeapDump.png, > BlockedEpollEventLoopFromThreadDump.png, RequestExecutorQueueFull.png, heap > dump showing each ImmediateFlusher taking upto 600MB.png > > > This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue > bounded, since, in the current state, items get added to the queue without > any checks on queue size, nor with any checks on netty outbound buffer to > check the isWritable state. > We are seeing this issue hit our production 3.0 clusters quite often. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org