Hyukjin Kwon created SPARK-61385:
------------------------------------

             Summary: Run the tests of the execution hook without a server on a 
client that does not retry, so that they do not wait 10 minutes for the plan 
compression lookup
                 Key: SPARK-61385
                 URL: https://issues.apache.org/jira/browse/SPARK-61385
             Project: Spark
          Issue Type: Test
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Since SPARK-61263 (plan compression), the first plan of a session reads the 
plan compression confs of the server with a Config call before the execution 
hook runs, as PySpark's _get_plan_compression_threshold_and_algorithm does. The 
two offline tests of crates/spark-connect/tests/execute_plan_hook.rs use a 
session of the dead port sc://127.0.0.1:1 with the default retry policy, so 
that Config call is retried for its whole backoff before the hook runs: the 
test binary takes 639 s, where it took 0.00 s before.

The CI jobs run it several times: "Rust build and test (stable)" twice, and 
"Rust checks", "WASM UDF feature" and "Measure and publish coverage badges" 
once each, 10.7 minutes each time. "Rust behavioral e2e" runs it one test at a 
time, 1278 s. The "Rust build and test (stable)" job went from 7.1 minutes at 
aad559aa to 34.7 minutes at 2c768214, the first commit after SPARK-61263 with a 
full run.

Fix: run those two tests in the crate on a session of the dead port whose 
client does not retry (RetryPolicy::no_retries, as the tests of the streaming 
module do), and keep the test of a server in the integration test file.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to