Hyukjin Kwon created SPARK-61359:
------------------------------------
Summary: Read the crate directory when the proto build script
runs, and pin the hash seed and the UDTF file timeout of the parity scripts
Key: SPARK-61359
URL: https://issues.apache.org/jira/browse/SPARK-61359
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Three small build and test-infrastructure fixes from the s16 review:
- crates/spark-connect-proto/build.rs takes the directory of its .proto files
from env!("CARGO_MANIFEST_DIR"), which is baked in when the script is compiled.
A build directory shared by several checkouts (worktrees) can run a script
compiled in another checkout; once that checkout is deleted, the build fails
with "protoc failed: Could not make proto path relative: <deleted
checkout>/crates/spark-connect-proto/proto/spark/connect/base.proto". The
script now reads CARGO_MANIFEST_DIR when it runs, as Cargo sets it for every
build-script run.
- scripts/gen_parity_skiplist.py, which runs the reference client to regenerate
the manifest of environmental failures, does not pin PYTHONHASHSEED, while
scripts/run_official_tests.py pins it to 0 (as Apache's bin/pyspark does). With
a random seed, about 7% of the runs of the official
test_crossvalidator_with_random_forest_classifier fail, so a regenerated
manifest could pick up that flake. It now pins it too.
- scripts/run_official_tests.py runs test_parity_udtf.py (387 tests, most with
a Python worker on the server) under the default 360 s per-file cap. It takes
145-204 s on CI runners but 466 s on a loaded machine, so it gets the 1200 s
cap of test_parity_arrow_python_udf.py.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]