[ https://issues.apache.org/jira/browse/CASSANDRA-14983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820091#comment-16820091 ]
Marcus Olsson commented on CASSANDRA-14983: ------------------------------------------- I don't necessarily think we should remove the fast path, I might have phrased my previous comment a bit ambiguous on that point. For me it depends on what expectations and guarantees we want to have for the requests (wrt. timeouts/speculative retries). In any case I think it would be really hard to have guarantees for both speculative retries and timeouts as there are multiple things affecting us here (gc pauses being a major one). I think your suggestion makes sense, but there is still a problematic situation with cluster imbalance. If one node has a slow disk, more sstables or similar issues, should it act mostly as a proxy to other nodes or do we accept slow requests? If we accept that requests could be slow in those situations then I don't think there is any reason to remove the fast path. I'm not sure which behavior would be best here as there are arguments for both. I also want to mention that if we do want to have a feature that waits for local requests before doing speculative retries I think we should keep it separate from the fast path. The requests can and will go through the normal path still and I guess we want to have a similar behavior when that happens. > Local reads potentially blocking remote reads > --------------------------------------------- > > Key: CASSANDRA-14983 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14983 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Coordination > Reporter: Marcus Olsson > Assignee: Marcus Olsson > Priority: Low > Attachments: graph_local_read.html, graph_local_read_trunk.html, > local_read_trace.log > > > Since CASSANDRA-4718 there is a fast path allowing local requests to continue > to [work in the same > thread|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java#L157] > rather than being sent over to the read stage. > Based on the comment > {code:java} > // We delay the local (potentially blocking) read till the end to avoid > stalling remote requests. > {code} > it seems like this should be performed last in the chain to avoid blocking > remote requests but that does not seem to be the case when the local request > is a data request. The digest request(s) are sent after the data requests are > sent (and now the transient replica requests as well). When the fast path is > used for local data/transient data requests this will block the next type of > request from being sent away until the local read is finished and add > additional latency to the request. > In addition to this it seems like local requests are *always* data requests > (might not be a problem), but the log message can say either ["digest" or > "data"|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java#L156] > as the type of request. > I have tried to run performance measurements to see the impact of this in 3.0 > (by moving local requests to the end of ARE#executeAsync()) but I haven't > seen any big difference yet. I'll continue to run some more tests to see if I > can find a use case affected by this. > Attaching a trace (3.0) where this happens. Reproduction: > # Create a three node CCM cluster > # Provision data with stress (rf=3) > # In parallel: > ## Start stress read run > ## Run multiple manual read queries in cqlsh with tracing on and > local_quorum (as this does not always happen) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org