HyukjinKwon opened a new pull request, #74:
URL: https://github.com/apache/spark-connect-rust/pull/74
### What changes were proposed in this pull request?
Mirror Apache Spark's `build_python_connect.yml`, which runs the
resource-profile and artifact tests against a **local-cluster** master (a real
multi-executor cluster) rather than `local[*]`. The single-JVM local mode
cannot exercise resource profiles or artifact distribution across executors, so
those tests were previously either not meaningful (run against `local[*]`) or
not run at all (`test_connect_resources` lives outside the discovered
`sql/tests/connect` scope).
- `run_official_tests.py` gains:
- `--only`: run an explicit list of tests, given either as dotted module
names (like Apache's `run-tests --testnames`, e.g.
`pyspark.resource.tests.test_connect_resources`) or `spark_py`-relative paths.
This can reach files outside the `sql/tests/connect` scope `discover()` walks.
- `--exclude`: drop file basenames from discovery.
- The parity job bounces the Connect server into `local-cluster[2, 4, 1024]`
after the main `local[*]` run and runs the same four tests Apache does, through
our Rust client:
- `pyspark.resource.tests.test_connect_resources`
- `pyspark.sql.tests.connect.client.test_artifact`
- `pyspark.sql.tests.connect.client.test_artifact_localcluster`
- `pyspark.sql.tests.connect.test_parity_resources`
- The three that live under `sql/tests/connect` are excluded from the
`local[*]` run so they execute only where they are meaningful (matching Apache,
which runs them only under local-cluster).
### Why are the changes needed?
Resource profiles and cross-executor artifact distribution are untested
against a single-JVM server. This adds the local-cluster coverage Apache
already has.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
CI (this PR). `run_official_tests.py --only` resolution verified locally;
the local-cluster phase runs in CI.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
