[ https://issues.apache.org/jira/browse/CASSANDRA-16616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Caleb Rackliffe updated CASSANDRA-16616: ---------------------------------------- Reviewers: Benjamin Lerer, David Capwell (was: Benjamin Lerer, Caleb Rackliffe, David Capwell) > Harden internode message resource limit accounting against serialization > failures > --------------------------------------------------------------------------------- > > Key: CASSANDRA-16616 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16616 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode > Reporter: Jon Meredith > Assignee: Jon Meredith > Priority: Normal > Fix For: 4.0-rc > > Time Spent: 10m > Remaining Estimate: 0h > > If the internode messaging exception recovery code fails and is unable to > correctly adjust the resource limits for an OutboundConnection, it affects > the other connection types sharing the same OutboundConnections so that any > of the connections could hit {{assert using >= 0;}} in > {{org.apache.cassandra.net.ResourceLimits.Concurrent#release}}. > While it is possible to modify all of the outbound connection code to > re-initialize all of the connections with a correct limit, the effort to test > and maintain the recovery code seems too high for something that should > "never happen" (except it did once, which is why it needs hardening). The > safer option is to kill the JVM and have whatever external monitoring is in > place restart the instance in a known good state. > Additionally, the logging for dropping outbound messages that have expired or > are unserializable messages takes place after the recovery handling logic. If > there are problems with the recovery logic that throw an exception, the > message is never logged for future diagnosis. Logging should take place > first, and then releasing capacity/handling the expiration/serialization. > Discovered on a branch modified for testing that threw an exception in the > Verb.serializeSize method. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org