HyukjinKwon opened a new pull request, #54:
URL: https://github.com/apache/spark-connect-rust/pull/54
> **Draft / WIP** — opened to validate the new parity and coverage CI jobs
and to
> read CI's authoritative coverage number. The coverage-gap-filling is in
progress.
### What changes were proposed in this pull request?
Two improvements to the Connect test setup, now that the client is at full
parity.
**1. Parity gate runs the official suite through our client only (drops the
double
reference run).** Previously the gate ran every official `sql/tests/connect`
file *twice*
(reference pyspark **and** our client via transport injection) to derive a
per-file
environmental-failure baseline — ~1h20m–2h. Since the client is at parity,
this replaces
that with:
- `scripts/run_official_tests.py` — runs the suite **only** through the Rust
transport
plugin and requires every test to pass except a checked-in manifest of
*known
environmental failures*, with a per-file retry for the timing-sensitive
streaming-listener/observation tests.
- `scripts/gen_parity_skiplist.py` — regenerates that manifest from a
reference-client run
(on a Spark version bump), recording each env-failure test id + reason.
- `scripts/parity_known_failures.txt` — the generated manifest (65 entries:
foreachBatch /
streaming that need a real streaming setup, python-datasource worker
cases, a few
arrow/udf/functions edge cases the reference also fails here).
- The CI job is rewritten accordingly (roughly halving it).
**2. A Rust test-coverage gate across all crates, including the `pyspark-rs`
PyO3 layer.**
`scripts/rust_coverage.sh` measures merged coverage with `cargo-llvm-cov` —
the pure-Rust
crates plus the `pyspark-rs` extension driven by the Python suite — and
fails under
`COVERAGE_MIN` (90%). A new `rust-coverage` CI job runs it. Coverage gaps
are being filled
with golden/e2e/unit tests, prioritizing full coverage of the user-facing
API.
Also fixes a stale e2e expectation: `SELECT ROUND(3.14159, 2)` yields
`DECIMAL(4,2)` in
Spark (the SQL literal is decimal, not double).
### Why are the changes needed?
The reference run in the parity gate no longer earns its ~1h cost once the
client is at
parity; running our client alone against a checked-in env-failure manifest
keeps the same
guarantee ("pass everything the reference can pass here") at roughly half
the time. The
coverage gate guards against untested code, especially on the user-facing
API surface.
### Does this PR introduce any user-facing change?
No — CI/infra and tests only.
### How was this patch tested?
CI on this PR. Locally: the ours-only gate passes the official suite against
a live Spark
4.2.0 server with the manifest applied (`test_client.py` included); coverage
is measured
with `cargo-llvm-cov`.
JIRA: https://issues.apache.org/jira/browse/SPARK-58987
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
