nsivabalan opened a new pull request, #20444: URL: https://github.com/apache/hudi/pull/20444
### Describe the issue this Pull Request addresses #20415 made `AsyncCompactService#updateWriteClient` and `AsyncClusteringService#updateWriteClient` wait for the service's running compaction or clustering run: both the run and the client switch now hold the same monitor (`synchronized (compactor)` / `synchronized (clusteringClient)`) for the whole run. The thread calling `updateWriteClient` is the writer's driver thread: - **HoodieStreamer**: `StreamSync#reInitWriteClient` → `HoodieStreamer#onInitializingWriteClient`, on every write client re-creation (e.g. a source schema change). - **Spark streaming sink**: `HoodieStreamingSink#retireWriteClient`, inside `addBatch` when the writer schema changes. A compaction or clustering run is not bounded in time. On a busy cluster it routinely waits for executors, and large compactions/clusterings run for an hour or more. With the run-long lock, ingestion stops for that whole time: HoodieStreamer does not sync, and the streaming query's micro-batch does not complete (so the query's trigger, progress and checkpoint all stall). That is a regression of ingestion latency driven by an unrelated table service, and we cannot block the driver until a compaction completes. Related: #20400, #20429. ### Summary and Changelog - Restore the non-blocking `updateWriteClient` in `AsyncCompactService` and `AsyncClusteringService`: the switch only replaces the client used by later runs, and the run loop no longer holds a run-long lock. - Make `BaseCompactor#compactionClient` and `BaseClusterer#clusteringClient` `volatile`, since the writer switches them while the service thread reads them. Both Spark implementations read the client once at the start of a run, so a run never mixes two clients. - Replace `TestAsyncTableServiceClientHandoff#testHandoffWaitsForActiveRun` with `testHandoffDoesNotWaitForActiveRun`: while a compaction/clustering run is held, the switch returns immediately, the running run keeps the client it started with, and the next run uses the new client. - Restore the original `HoodieStreamingSink#retireWriteClient` doc. ### Impact Switching write clients never waits for a running async compaction or clustering, as before #20415. As before #20415, the writer closes the previous client once it has switched the services, so a run that is still in progress on that client can fail. Handling that failure (rolling back and retrying the run with the new client, and reporting non-retryable failures to the writer with their cause) is tracked in #20429, which is reworked on top of this change. ### Risk Level low. Restores the pre-#20415 behaviour of `updateWriteClient`; adds `volatile` to two fields. ### Documentation Update None. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
