HyukjinKwon opened a new pull request, #82: URL: https://github.com/apache/spark-connect-rust/pull/82
### What changes were proposed in this pull request? Make the entire `pyspark.pandas` (pandas-on-Spark) package a fully-vendored, drift-checked mirror of Apache Spark, and keep tests out of the packaged wheel. - **Vendored-drift CI** (`scripts/check_vendored_upstream.sh`): replace the per-subdirectory / per-file `pyspark/pandas` entries with the whole `pandas` tree, compared byte-for-byte against the `v4.2.0` tag. `check_one` now takes an optional extra `--exclude`, and `pandas` is checked with `--exclude=tests` because we intentionally do not ship `pandas/tests`. Any hand-edit to a file under `pyspark/pandas` now fails CI. (Our `pyspark/pandas` is already byte-identical to upstream, so no source changes were needed.) - **Packaging** (`pyproject.toml`): add a maturin `exclude` for `pyspark/**/tests/**` and `pyspark/**/test_*.py`, so the vendored upstream test modules (`pyspark/pipelines/tests`, `pyspark/testing/tests`) stay out of the built wheel/sdist. ### Why are the changes needed? `pyspark.pandas` is vendored verbatim from Apache Spark; it must never be locally edited (edits would silently diverge from upstream and break the "drop-in" guarantee). Enforcing the whole tree in the drift gate makes that a hard CI failure instead of a convention. Excluding tests keeps the shipped package lean and free of upstream test suites end users don't need. ### How was this patch tested? - `diff -r --exclude=tests --exclude=__pycache__ python/pyspark/pandas <v4.2.0>/python/pyspark/pandas` → no differences (the tree matches upstream). - `bash -n scripts/check_vendored_upstream.sh` passes; the new whole-tree `pandas` check with the `tests` exclude reports OK against a local v4.2.0 checkout. - `maturin sdist` → 0 Python test files packaged (`pyspark/**/tests`, `test_*.py`), pandas modules intact. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
