[
https://issues.apache.org/jira/browse/KUDU-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17967383#comment-17967383
]
Alexey Serbin commented on KUDU-3668:
-------------------------------------
Thanks a lot for the report!
The outcome of the scenario described is quite unexpected from the
application's standpoint, indeed. And it's certainly a bug if Kudu Java client
library retries RPCs that it has already reported as timed out to the client
application via its client API.
Just to provide more context: if not the higher-level retry of write operations
by the client application itself, the operation retried by the Java Kudu client
library under the described conditions wouldn't be a problem because Write RPCs
in Kudu have exactly-once semantics (see
[KUDU-568|https://issues.apache.org/jira/browse/KUDU-568] for details). In
other words, even if Write RPC has succeeded at the server side but the client
side didn't receive acknowledgement due to a network issue and retried the
operation, Kudu server side would recognize that it has already processed the
operation and wouldn't try to apply it again. In addition, there is one extra
validation of the RPC timeout before admitting the operation into the prepare
queue in Kudu 1.17.0 and later versions: see
[KUDU-3500|https://issues.apache.org/jira/browse/KUDU-3500] for details.
Maybe, you could put together a reproduction scenario for the issue using
Kudu's test scaffolding? It's possible to inject delays and various sorts of
artificial failures at the server side: search for gflags named like
{{\-\-inject_xxx}} and {{\-\-xxx_inject_xxx}}.
It would be great to have an automated test scenario to:
* reproduce the issue and confirm that the RCA is correct
* use the test scenario to validate the fix if/when the issue is addressed
* spot future regressions (if any) once the issue is addressed
Meanwhile, you might consider picking up the fix for
[KUDU-3500|https://issues.apache.org/jira/browse/KUDU-3500] as a temporary
workaround.
Thank you!
> Kudu Client Write Retry Behavior After Timeout
> ----------------------------------------------
>
> Key: KUDU-3668
> URL: https://issues.apache.org/jira/browse/KUDU-3668
> Project: Kudu
> Issue Type: Bug
> Components: client
> Reporter: shen yushi
> Priority: Major
>
> *Issue Description*
> The Kudu client continues internal retries of write requests _after_
> returning a timeout status to the caller. In our implementation, callers may
> initiate new write operations upon receiving this timeout status. This can
> cause overlapping writes if the client's internal retry completes
> concurrently, potentially resulting in data duplication or conflicts.
> *Technical Analysis*
> Based on our investigation (referencing
> [gerrit.cloudera.org/c/12338|https://gerrit.cloudera.org/c/12338]):
> # Each KuduRpc object contains an RpcTimeoutTask.
> # When a tablet server stalls (simulated via gdb process suspension), this
> task triggers:
> * Invokes the RPC's error callback
> * Returns timeout status to the caller
> # Meanwhile, the Connection object retains the original KuduRpc.
> # Upon tablet server recovery:
> * The connection may receive exceptions/errors
> * AsyncKuduClient.handleRetryableError initiates internal retries
> *Environment*
> * Kudu Version: 1.15.0
> *Key Question*
> Is this dual-retry behavior (caller + client) an intentional design? We found
> no existing issues addressing this scenario in the community. We'd appreciate
> your insights on:
> * Whether this constitutes a bug
> * Recommended solutions or configuration adjustments
> We're committed to providing additional details and contributing fixes if
> needed. Thank you for your time and expertise!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
