[ https://issues.apache.org/jira/browse/CASSANDRA-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032696#comment-13032696 ]
Jonathan Ellis commented on CASSANDRA-2034: ------------------------------------------- bq. there is the potential to take us back to the Bad Old Days when HH could cause cascading failure To elaborate, the scenario here is, we did a write that succeeded on some nodes, but not others. So we need to write a local hint to replay to the down-or-slow nodes later. But, those nodes being down-or-slow mean load has increased on the rest of the cluster, and writing the extra hint will increase that further, possibly enough that other nodes will see this coordinator as down-or-slow, too, and so on. So I think what we want to do, with this option on, is to attempt the hint write but if we can't do it in a reasonable time, throw back a TimedOutException which is already our signal that "your cluster may be overloaded, you need to back off." Specifically, we could add a separate executor here, with a blocking, capped queue. When we go to do a hint-after-failure we enqueue the write but if it is rejected because queue is full we throw the TOE. Otherwise, we wait for the write and then return success to the client. The tricky part is the queue needs to be large enough to handle load spikes but small enough that wait-for-success-post-enqueue is negligible compared to RpcTimeout. If we had different timeouts for writes than reads (which we don't -- CASSANDRA-959) then it might be nice to use say 80% of the timeout for the normal write, and reserve 20% for the hint phase. > Make Read Repair unnecessary when Hinted Handoff is enabled > ----------------------------------------------------------- > > Key: CASSANDRA-2034 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2034 > Project: Cassandra > Issue Type: Improvement > Components: Core > Reporter: Jonathan Ellis > Assignee: Jonathan Ellis > Fix For: 1.0 > > Original Estimate: 8h > Remaining Estimate: 8h > > Currently, HH is purely an optimization -- if a machine goes down, enabling > HH means RR/AES will have less work to do, but you can't disable RR entirely > in most situations since HH doesn't kick in until the FailureDetector does. > Let's add a scheduled task to the mutate path, such that we return to the > client normally after ConsistencyLevel is achieved, but after RpcTimeout we > check the responseHandler write acks and write local hints for any missing > targets. > This would making disabling RR when HH is enabled a much more reasonable > option, which has a huge impact on read throughput. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira