HyukjinKwon opened a new pull request, #201: URL: https://github.com/apache/spark-connect-rust/pull/201
### What changes were proposed in this pull request? [SPARK-60964](https://issues.apache.org/jira/browse/SPARK-60964): two changes to the reattachable execution of the Rust transport, as PySpark's `ExecutePlanResponseReattachableIterator` does. - **Each response read has the retries of the policy afresh.** `ReattachableResponseStream::message` makes a retry state of its own, for the reattaches that it needs, as `_has_next` makes a `Retrying` of its own. Before, one retry state served the whole stream. - **An execution whose first `ExecutePlan` call fails is released.** An analysis or parse error is an example. `execute_plan_reattachable` sends `ReleaseExecute` (`ReleaseAll`) before it returns the error, as the iterator releases an execution on any error. The client now sends releases with the metadata of the execution, for a stream and for such a failure alike. ### Why are the changes needed? - The attempts of the shared retry state were never reset, and its elapsed-time limit counted from the start of the stream. So a query, a statement or the streaming listener that ran past the limit, or that reattached more often than the maximum number of retries, could not retry a transient `UNAVAILABLE` of a later reattach. The call failed and the execution was released. - The error of a failing first call returned before the reattachable stream existed, so no `ReleaseExecute` was sent, and the server kept the execution until its detached timeout. ### Does this PR introduce _any_ user-facing change? Yes. Long executions survive transient failures of their reattaches, and executions that fail at once are released, as with PySpark 4.3.0. ### How was this patch tested? - New tests against the mock Connect server of the transport's reattach tests. The server can now fail the `ExecutePlan` call itself, and script the results of `ReattachExecute`. - An `ExecutePlan` call that fails at once is released once, with the identity of the request. - Three reattaches that each fail once with `UNAVAILABLE` read the whole result, under a policy of two retries. Both fail without this change. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (660 passed, and 125 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (886 passed, 9 skipped), and the live drop-in and Arrow suites (81 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
