Hyukjin Kwon created SPARK-61520:
------------------------------------
Summary: Unpickle deep Columns and DataFrames, and call session
hooks on deep plans, in the Python client as PySpark does
Key: SPARK-61520
URL: https://issues.apache.org/jira/browse/SPARK-61520
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A Column of more than about 50 nested expressions, or a DataFrame of more than
about 50 plan levels, pickles but does not unpickle: the client decodes its
spark.connect bytes with prost, and a Python worker of the server with
protobuf's FromString, and both stop at 100 levels of nesting ("failed to
decode Protobuf message" / DecodeError). With hook_factories, a plan of about
50 levels fails the same way, as the request is decoded for the hooks and
decoded again after them. PySpark pickles its Python expression and plan
objects, and gives the hooks its Python request, so the same Columns and
DataFrames round-trip (e.g. 100 additions, 90 withColumn levels) and the hooks
run on deep plans, up to Python's own recursion limit.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]