cshuo commented on issue #19902: URL: https://github.com/apache/hudi/issues/19902#issuecomment-5629028749
This is a valid issue: the RPC can time out while instant creation waits for the lock, causing the writer task to fail and potentially triggering a job restart. Flink’s RPC ask timeout defaults to 10 seconds (pekko.ask.timeout; see the [configuration definition](https://github.com/apache/flink/blob/release-1.20/flink-core/src/main/java/org/apache/flink/configuration/RpcOptions.java#L65-L73)), so lock contention lasting longer than that can trigger this failure. Before introducing a more complex state machine, could we first try bounded client-side retries with backoff and jitter? Since requests are processed serially and reuse the checkpoint-to-instant mapping, this could address transient contention with a smaller change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
