[ https://issues.apache.org/jira/browse/CASSANDRA-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693173#comment-17693173 ]
Stefan Miklosovic commented on CASSANDRA-14715: ----------------------------------------------- [~jaid] is this happening in 4.0 too? I am trying to figure that out. The logic around this stuff was rewritten there. https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/reads/repair/BlockingReadRepair.java#L82-L106 > Read repairs can result in bogus timeout errors to the client > ------------------------------------------------------------- > > Key: CASSANDRA-14715 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14715 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths > Reporter: Cameron Zemek > Priority: Low > Time Spent: 50m > Remaining Estimate: 0h > > In RepairMergeListener:close() it does the following: > > {code:java} > try > { > FBUtilities.waitOnFutures(repairResults, > DatabaseDescriptor.getWriteRpcTimeout()); > } > catch (TimeoutException ex) > { > // We got all responses, but timed out while repairing > int blockFor = consistency.blockFor(keyspace); > if (Tracing.isTracing()) > Tracing.trace("Timed out while read-repairing after receiving all {} > data and digest responses", blockFor); > else > logger.debug("Timeout while read-repairing after receiving all {} > data and digest responses", blockFor); > throw new ReadTimeoutException(consistency, blockFor-1, blockFor, true); > } > {code} > This propagates up and gets sent to the client and we have customers get > confused cause they see timeouts for CL ALL requiring ALL replicas even > though they have read_repair_chance = 0 and using a LOCAL_* CL. > At minimum I suggest instead of using the consistency level of DataResolver > (which is always ALL with read repairs) for the timeout it instead use > repairResults.size(). That is blockFor = repairResults.size() . But saying it > received _blockFor - 1_ is bogus still. Fixing that would require more > changes. I was thinking maybe like so: > > {code:java} > public static void waitOnFutures(List<AsyncOneResponse> results, long ms, > MutableInt counter) throws TimeoutException > { > for (AsyncOneResponse result : results) > { > result.get(ms, TimeUnit.MILLISECONDS); > counter.increment(); > } > } > {code} > > > > Likewise in SinglePartitionReadLifecycle:maybeAwaitFullDataRead() it says > _blockFor - 1_ for how many were received, which is also bogus. > > Steps used to reproduce was modify RepairMergeListener:close() to always > throw timeout exception. With schema: > {noformat} > CREATE KEYSPACE weather WITH replication = {'class': > 'NetworkTopologyStrategy', 'dc1': '3', 'dc2': '3'} AND durable_writes = true; > CREATE TABLE weather.city ( > cityid int PRIMARY KEY, > name text > ) WITH bloom_filter_fp_chance = 0.01 > AND dclocal_read_repair_chance = 0.0 > AND read_repair_chance = 0.0 > AND speculative_retry = 'NONE'; > {noformat} > Then using the following steps: > # ccm node1 cqlsh > # INSERT INTO weather.city(cityid, name) VALUES (1, 'Canberra'); > # exit; > # ccm node1 flush > # ccm node1 stop > # rm -rf > ~/.ccm/test_repair/node1/data0/weather/city-ff2fade0b18d11e8b1cd097acbab1e3d/mc-1-big-* > # remove the sstable with the insert > # ccm node1 start > # ccm node1 cqlsh > # CONSISTENCY LOCAL_QUORUM; > # select * from weather.city where cityid = 1; > You get result of: > {noformat} > ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting > for replica nodes' responses] message="Operation timed out - received only 5 > responses." info={'received_responses': 5, 'required_responses': 6, > 'consistency': 'ALL'}{noformat} > But was expecting: > {noformat} > ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting > for replica nodes' responses] message="Operation timed out - received only 1 > responses." info={'received_responses': 1, 'required_responses': 2, > 'consistency': 'LOCAL_QUORUM'}{noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org