[ 
https://issues.apache.org/jira/browse/SPARK-61362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61362:
-----------------------------------
    Labels: pull-request-available  (was: )

> Report and retry transport failures as grpc-python does, and send the request 
> fields, keepalive errors and user agent of PySpark
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61362
>                 URL: https://issues.apache.org/jira/browse/SPARK-61362
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Low-priority transport findings of the s16 review of the Rust core 
> (spark-connect-core), each checked against PySpark 4.3.0 / grpc-python 1.76:
> - Retries: is_transport_error retried any HTTP/2 error, also a stream that 
> the server resets with a reason that tonic leaves UNKNOWN (STREAM_CLOSED, 
> FRAME_SIZE_ERROR, HTTP_1_1_REQUIRED): 15 retries over about 10 minutes, where 
> grpc-python reports INTERNAL and PySpark does not retry. A GOAWAY that 
> processed no stream was retried only when tonic happened to report it as 
> UNKNOWN; grpc-python always reports UNAVAILABLE.
> - Codes: a dropped connection is UNKNOWN in tonic but UNAVAILABLE in 
> grpc-python, and the Rust transport that the pandas-parity gate runs official 
> PySpark over (RustConnectStub) passed UNKNOWN on, so official PySpark did not 
> reattach a reset. SparkError::grpc_code and the drop-in's getGrpcStatusCode 
> reported the tonic codes too.
> - Request fields: a user_id gRPC header that PySpark never sends; 
> client_observed_server_side_session_id on ReleaseExecute, CloneSession, 
> AddArtifacts and ArtifactStatus, where PySpark sends none; no client_type on 
> ExecutePlan, AnalyzePlan, Config, Interrupt, FetchErrorDetails, 
> ReattachExecute, ReleaseExecute and ReleaseSession; CloneSession with the 
> user agent instead of "python"; and a first ExecutePlan without the observed 
> server-side session ID.
> - A keepalive parameter that is not an integer (grpc_keepalive_time_ms=abc) 
> was ignored, where PySpark raises ValueError "invalid literal for int() with 
> base 10: 'abc'" when it opens its channel; and the user agent named macOS 
> "macos" where PySpark's platform.uname().system.lower() gives "darwin".
> - Plan compression encoded every relation and command only to compare its 
> size with the threshold; the plan of a DataFrame that an on_execute_plan hook 
> of the drop-in put in went uncompressed, where PySpark's hook would put in 
> the plan of to_proto, which compresses it; and no live test checked that a 
> server runs compressed plans.
> The fix maps transport failures to grpc-python's code (new 
> retries::grpc_python_code) for retries, errors and the transport seam, sends 
> the request fields as PySpark per request type, raises PySpark's ValueError 
> for a bad keepalive parameter, names the OS as Python does, sizes plans with 
> encoded_len, compresses the plan again after a hook ran, and adds a live test 
> of compressed plans.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to