[ https://issues.apache.org/jira/browse/CASSANDRA-15442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994025#comment-16994025 ]
Yifan Cai commented on CASSANDRA-15442: --------------------------------------- Putting some formula here to analyze the impact of allowing longer mutation time for read repair prudently. For simplicity, using a simple and stable queue to model the request handling. We have, _M / Lm = Rm_ where _M_ is the average amount of mutations at a given time window, _Lm_ is the average latency and _Rm_ is the rate. If some of the mutation can have a longer timeout, the new average latency is _Lm’ = (Lm * M1 + L * M2) / M_ where _M1_ is the regular mutation, and _M2_ is the amount of the super slow mutation (without increasing the timeout, they will timeout) from read repair, and _L_ is the average latency those slow mutations take. _M1_ and _M2_ satisfies, * _M1 + M2 = M_ * _M2 = M * Prr * Pmto_, _Prr_ is the observed percentage of read repair mutation vs. total, and _Pmto_ is the observed percentage of the timeouted mutation vs. total. The range of _L_ is _Lm < L <= R_, _R_ is the configured read timeout Therefore, we have _Lm' <= Lm + (R - Lm) * Prr * Pmto_ _Rm' >= M / (Lm + (R - Lm) * Prr * Pmto)_ Let _K = (R - Lm) / Lm_, since they are constants. The ratio between the prior rate and the new rate with the change is, _1 < Rm / Rm' <= 1 + K * Prr * Pmto_ Based on the equation, the percentage of read repair and mutation timeouts are the factors to the throughput. In a health cluster, i.e. low mutation timeout rate, the ratio should be close to 1, meaning the impact is small. > Read repair implicitly increases read timeout value > --------------------------------------------------- > > Key: CASSANDRA-15442 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15442 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core > Reporter: Yifan Cai > Assignee: Yifan Cai > Priority: Normal > > When read repair occurs during a read, internally, it starts several > _blocking_ operations in sequence. See > {{org.apache.cassandra.service.StorageProxy#fetchRows}}. > The timeline of the blocking operations > # Regular read, wait for full data/digest read response to complete. > {{reads[*].awaitResponses();}} > # Read repair read, wait for full data read response to complete. > {{reads[*].awaitReadRepair();}} > # Read repair write, wait for write response to complete. > {{concatAndBlockOnRepair(results, repairs);}} > Step 1 and 2 share the same timeout, and wait for the duration of read > timeout, say 5 s. > Step 3 waits for the duration of write timeout, say 2 s. > In the worse case, the actual time taken for a read could accumulate to ~7 s, > if each individual step does not exceed the timeout value. > From the client perspective, it may not expect a request taken higher than > the database configured timeout value. > Such scenario is especially bad for the clients that have set up client-side > timeout monitoring close to the configured one. The clients think the > operations timed out and abort, but they are in fact still running on server. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org