Hyukjin Kwon created SPARK-61362:
------------------------------------

             Summary: Report and retry transport failures as grpc-python does, 
and send the request fields, keepalive errors and user agent of PySpark
                 Key: SPARK-61362
                 URL: https://issues.apache.org/jira/browse/SPARK-61362
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Low-priority transport findings of the s16 review of the Rust core 
(spark-connect-core), each checked against PySpark 4.3.0 / grpc-python 1.76:
- Retries: is_transport_error retried any HTTP/2 error, also a stream that the 
server resets with a reason that tonic leaves UNKNOWN (STREAM_CLOSED, 
FRAME_SIZE_ERROR, HTTP_1_1_REQUIRED): 15 retries over about 10 minutes, where 
grpc-python reports INTERNAL and PySpark does not retry. A GOAWAY that 
processed no stream was retried only when tonic happened to report it as 
UNKNOWN; grpc-python always reports UNAVAILABLE.
- Codes: a dropped connection is UNKNOWN in tonic but UNAVAILABLE in 
grpc-python, and the Rust transport that the pandas-parity gate runs official 
PySpark over (RustConnectStub) passed UNKNOWN on, so official PySpark did not 
reattach a reset. SparkError::grpc_code and the drop-in's getGrpcStatusCode 
reported the tonic codes too.
- Request fields: a user_id gRPC header that PySpark never sends; 
client_observed_server_side_session_id on ReleaseExecute, CloneSession, 
AddArtifacts and ArtifactStatus, where PySpark sends none; no client_type on 
ExecutePlan, AnalyzePlan, Config, Interrupt, FetchErrorDetails, 
ReattachExecute, ReleaseExecute and ReleaseSession; CloneSession with the user 
agent instead of "python"; and a first ExecutePlan without the observed 
server-side session ID.
- A keepalive parameter that is not an integer (grpc_keepalive_time_ms=abc) was 
ignored, where PySpark raises ValueError "invalid literal for int() with base 
10: 'abc'" when it opens its channel; and the user agent named macOS "macos" 
where PySpark's platform.uname().system.lower() gives "darwin".
- Plan compression encoded every relation and command only to compare its size 
with the threshold; the plan of a DataFrame that an on_execute_plan hook of the 
drop-in put in went uncompressed, where PySpark's hook would put in the plan of 
to_proto, which compresses it; and no live test checked that a server runs 
compressed plans.
The fix maps transport failures to grpc-python's code (new 
retries::grpc_python_code) for retries, errors and the transport seam, sends 
the request fields as PySpark per request type, raises PySpark's ValueError for 
a bad keepalive parameter, names the OS as Python does, sizes plans with 
encoded_len, compresses the plan again after a hook ran, and adds a live test 
of compressed plans.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to