[
https://issues.apache.org/jira/browse/SPARK-61362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61362:
------------------------------------
Assignee: Hyukjin Kwon
> Report and retry transport failures as grpc-python does, and send the request
> fields, keepalive errors and user agent of PySpark
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61362
> URL: https://issues.apache.org/jira/browse/SPARK-61362
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Low-priority transport findings of the s16 review of the Rust core
> (spark-connect-core), each checked against PySpark 4.3.0 / grpc-python 1.76:
> - Retries: is_transport_error retried any HTTP/2 error, also a stream that
> the server resets with a reason that tonic leaves UNKNOWN (STREAM_CLOSED,
> FRAME_SIZE_ERROR, HTTP_1_1_REQUIRED): 15 retries over about 10 minutes, where
> grpc-python reports INTERNAL and PySpark does not retry. A GOAWAY that
> processed no stream was retried only when tonic happened to report it as
> UNKNOWN; grpc-python always reports UNAVAILABLE.
> - Codes: a dropped connection is UNKNOWN in tonic but UNAVAILABLE in
> grpc-python, and the Rust transport that the pandas-parity gate runs official
> PySpark over (RustConnectStub) passed UNKNOWN on, so official PySpark did not
> reattach a reset. SparkError::grpc_code and the drop-in's getGrpcStatusCode
> reported the tonic codes too.
> - Request fields: a user_id gRPC header that PySpark never sends;
> client_observed_server_side_session_id on ReleaseExecute, CloneSession,
> AddArtifacts and ArtifactStatus, where PySpark sends none; no client_type on
> ExecutePlan, AnalyzePlan, Config, Interrupt, FetchErrorDetails,
> ReattachExecute, ReleaseExecute and ReleaseSession; CloneSession with the
> user agent instead of "python"; and a first ExecutePlan without the observed
> server-side session ID.
> - A keepalive parameter that is not an integer (grpc_keepalive_time_ms=abc)
> was ignored, where PySpark raises ValueError "invalid literal for int() with
> base 10: 'abc'" when it opens its channel; and the user agent named macOS
> "macos" where PySpark's platform.uname().system.lower() gives "darwin".
> - Plan compression encoded every relation and command only to compare its
> size with the threshold; the plan of a DataFrame that an on_execute_plan hook
> of the drop-in put in went uncompressed, where PySpark's hook would put in
> the plan of to_proto, which compresses it; and no live test checked that a
> server runs compressed plans.
> The fix maps transport failures to grpc-python's code (new
> retries::grpc_python_code) for retries, errors and the transport seam, sends
> the request fields as PySpark per request type, raises PySpark's ValueError
> for a bad keepalive parameter, names the OS as Python does, sizes plans with
> encoded_len, compresses the plan again after a hook ran, and adds a live test
> of compressed plans.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]