nsivabalan opened a new pull request, #20412: URL: https://github.com/apache/hudi/pull/20412
### Describe the issue this Pull Request addresses Closes #20400 (same root cause as #12652). `HoodieStreamingSink` could not keep an embedded timeline server alive across micro-batches. The writer closed the write client after every batch (unless async table services were on), which stopped the server it owned, and the sink then reused that closed client. To paper over this, the sink disabled the timeline server by default (HUDI-3636). Users who turned it on with a `REMOTE_ONLY` view failed on the second micro-batch with `Connection refused`. This aligns the sink with `HoodieStreamer`: one embedded timeline server per streaming query, reused across every commit. ### Summary and Changelog - `HoodieStreamingSink` owns an `EmbeddedTimelineService`. It is started on the first micro-batch and passed into the write client, which does not own it (as `StreamSync` does), so closing or rebuilding a client never stops the server. - The write client is kept open across micro-batches; the sink owns its lifecycle. - A `StreamingQueryListener` keyed on the query id closes the client and stops the server when the query terminates. A V1 `Sink` has no close hook, and long-lived drivers (e.g. a Spark Connect server running Spark Declarative Pipelines flows, #19997) would otherwise hold them until JVM exit. - Removed the HUDI-3636 default that set `hoodie.embed.timeline.server=false` for streaming writes. Setting it explicitly still opts out. - `HoodieSparkSqlWriter`: new `StreamingWriteParams.writeClientFactory`. When set, the writer builds the client through it and does not close it after the write. - `DataSourceUtils.createHoodieClient`: new overload taking a client factory; the existing method delegates to it. - Markers stay forced to `DIRECT` in this PR. Switching to timeline-server-based markers is a separate stacked PR. ### Impact - The timeline server is now on by default for Spark structured streaming writes into Hudi, giving them the cached file-system view. - On query termination, in-flight async compaction/clustering is force-stopped rather than left running until JVM exit (a graceful stop would block Spark's listener bus thread). - Known follow-up (pre-existing, also true for `HoodieStreamer`): the metadata table writer starts and stops its own short-lived timeline server per commit. ### Risk Level medium. Changes the write client and timeline server lifecycle for every streaming write. Verified with `TestStructuredStreaming` (all tests pass), including a new COW/MOR test that runs 3 micro-batches with `REMOTE_ONLY` view and no backup view (the #12652 scenario). ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
