Hyukjin Kwon created SPARK-61385:
------------------------------------
Summary: Run the tests of the execution hook without a server on a
client that does not retry, so that they do not wait 10 minutes for the plan
compression lookup
Key: SPARK-61385
URL: https://issues.apache.org/jira/browse/SPARK-61385
Project: Spark
Issue Type: Test
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Since SPARK-61263 (plan compression), the first plan of a session reads the
plan compression confs of the server with a Config call before the execution
hook runs, as PySpark's _get_plan_compression_threshold_and_algorithm does. The
two offline tests of crates/spark-connect/tests/execute_plan_hook.rs use a
session of the dead port sc://127.0.0.1:1 with the default retry policy, so
that Config call is retried for its whole backoff before the hook runs: the
test binary takes 639 s, where it took 0.00 s before.
The CI jobs run it several times: "Rust build and test (stable)" twice, and
"Rust checks", "WASM UDF feature" and "Measure and publish coverage badges"
once each, 10.7 minutes each time. "Rust behavioral e2e" runs it one test at a
time, 1278 s. The "Rust build and test (stable)" job went from 7.1 minutes at
aad559aa to 34.7 minutes at 2c768214, the first commit after SPARK-61263 with a
full run.
Fix: run those two tests in the crate on a session of the dead port whose
client does not retry (RetryPolicy::no_retries, as the tests of the streaming
module do), and keep the test of a server in the integration test file.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]