nsivabalan commented on PR #20453:
URL: https://github.com/apache/hudi/pull/20453#issuecomment-6485349284
Following up on the CI-placement discussion, I looked at how Apache Spark
tests Connect and declarative pipelines, to see what we could borrow. Spark
uses three layers:
1. **Python unit tests** (`python/pyspark/pipelines/tests`). These use
`ReusedConnectTestCase`, which calls `.remote("local[4]")` so PySpark boots a
local Connect server in-process. They call `create_dataflow_graph` /
`start_run` directly, not the CLI.
2. **End-to-end CLI tests driven from ScalaTest**
([`EndToEndAPISuite`](https://github.com/apache/spark/blob/32dd961c36decf1d5f9799d0e87592d1dd94b939/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/EndToEndAPISuite.scala)).
`SparkConnectServerTest` starts the Connect server inside the test JVM on a
random port. Each test then:
- writes the spec and sources to a temp dir;
- runs the real `pyspark/pipelines/cli.py` as a subprocess against
`sc://localhost:<port>`;
- waits with a timeout and fails with the CLI's stdout/stderr on a
non-zero exit;
- skips via `assume(...)` when the Python deps are missing.
The scenarios live in a shared `APITest` trait so they can be reused with
other runners.
3. **A separately launched server process.** On the JVM side this is
`RemoteSparkSession`, which uses `spark-submit` via `ProcessBuilder`. On the
Python side it is
[`build_python_connect.yml`](https://github.com/apache/spark/blob/32dd961c36decf1d5f9799d0e87592d1dd94b939/.github/workflows/build_python_connect.yml),
which runs `start-connect-server.sh` plus the client tests on a cron schedule,
not per PR.
This harness is layer 3, and that is the layer Spark runs on a schedule. So
my proposal is to split the coverage:
**Per PR: a small layer-2 ScalaTest in `hudi-spark` (Spark 4.1 profile),
modelled on `EndToEndAPISuite`**
- Start a Connect server in the test JVM with Hudi already on the classpath
(`SparkConnectService.start(spark.sparkContext)` on the shared session).
Spark's `SparkConnectServerTest` is test-only, so we'd copy the ~30 lines of
setup.
- Run the pipelines CLI from the pip-installed PySpark 4.1.1 (same
`requirements.txt`) as a subprocess against `sc://localhost:<port>`, with a
timeout, and fail with its output.
- `assume(...)` when PySpark is not installed, so local `mvn test` runs
without Python skip instead of fail.
- Keep it to 2–3 scenarios that exercise the Hudi wiring through the real
CLI:
- an MV plus a streaming table writing to Hudi;
- a per-flow `SET` reaching the sink (#20403 / #20455);
- a failing flow surfacing a clear error.
- It runs inside the existing Maven test jobs, so it needs no bundle build,
no `spark-submit` and no new CI lane. The only extra CI step is a cached `pip
install` in the Spark 4.1 job.
**Nightly: this harness (layer 3), as already proposed above**
- It keeps the cases only a separately launched server can show:
- static settings applied at server launch;
- the shared server surviving a failed flow (#20370 / #20452);
- resources released across runs (#20400);
- the real bundle jar on a real `spark-submit` classpath.
- Same blocking policy as in my earlier comment: a red nightly blocks merges
until fixed, with the escape hatch for the fix PR.
With this split, regressions in the Hudi sink/SDP wiring are caught on the
PR that introduces them at almost no extra CI cost. The heavier process-level
checks still run every night. @danny0405 would you be open to adding the
layer-2 suite, either in this PR or as a follow-up? @yihua thoughts welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
