danny0405 opened a new pull request, #20453: URL: https://github.com/apache/hudi/pull/20453
### Describe the issue this Pull Request addresses Closes #20402. RFC-112 needs repeatable coverage of the real `spark-pipelines` CLI against a separately configured Spark Connect server, using the candidate Hudi bundle. Scala dataflow-graph tests do not exercise that remote boundary. The general Spark 4.1 entries in the current `bot.yml` matrices are disabled. ### Summary and Changelog - Add a reusable Python harness that launches a foreground Spark 4.1.1 Connect server with the built Hudi bundle, extension, catalog, Kryo configuration, and isolated local storage. Pin the client environment to `pyspark[pipelines,connect]==4.1.1`. - Run a deterministic JSON-to-Hudi streaming pipeline through the actual CLI, then verify exact rows through a separate Connect client using both catalog and Hudi path reads. Check Hudi metadata and an SDP-managed streaming checkpoint. - Bound startup and client execution, detect server exits, retain client/server logs, and clean up temporary resources and owned process groups on success, failure, timeout, or interruption. - Add a focused Spark 4.1 / Scala 2.13 / JDK 17 CI job in `bot.yml` that builds the candidate bundle, runs lifecycle regressions and the remote harness, and uploads diagnostics. Validation: - Spark 4.1 candidate bundle reactor build passed with JDK 17 (`mvn package -T 2 -Pjava17 -Dspark4.1 -Dscala-2.13 -DskipTests -Dgpg.skip -Djacoco.skip -am -pl packaging/hudi-spark-bundle`). - Real CLI/Connect smoke test passed locally with that bundle and PySpark 4.1.1: the Hudi flow completed, both reads returned all three expected rows, and checkpoint/resource checks passed. - All five standard-library lifecycle tests passed, covering client failure diagnostics, timeout isolation from an unrelated process, server exit, forced termination, and cleanup after startup failure. - Python syntax, workflow YAML and shell-step syntax, `git diff --check`, and Apache RAT passed. The PR description was validated with `scripts/pr_compliance.py` before publication. ### Impact Adds test infrastructure and one CI lane; no production APIs, storage formats, defaults, or writer semantics change. The smoke test establishes the remote execution baseline. Repeat materialized-view refresh, streaming option forwarding, restart/full-refresh, chaining, and keyed update/delete acceptance remain with the sibling tasks under #20401. ### Risk Level Low. The new CI lane adds a Spark bundle build, a Python dependency installation, and a local Connect server. Process-group cleanup is scoped to harness-owned processes, clients have explicit time limits, and diagnostics are retained. Local end-to-end and lifecycle checks passed; the new Ubuntu CI lane has not yet completed on GitHub. ### Documentation Update Add `hudi-integ-test/spark-pipelines/README.md` with local build/install/run commands, server versus session/flow configuration ownership, timeout and cleanup behavior, log locations, CI integration, and instructions for adding sibling scenarios. No website update is needed for this developer test harness; this PR does not declare a new supported runtime feature. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
