Hyukjin Kwon created SPARK-61328:
------------------------------------

             Summary: Raise RecursionError instead of crashing the process on a 
deep expression, in the Python client as PySpark does
                 Key: SPARK-61328
                 URL: https://issues.apache.org/jira/browse/SPARK-61328
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


SPARK-61260 bounded the depth of a plan, but not the depth of an expression. 
Serializing an expression recursed once per level with no bound, so in the 
Python client (and the Rust API) a deep expression crashed the process. Against 
an OSS Spark 4.3.0 Connect server, select((col("id") + 1 + ... + 
1).alias("x")).collect():
- PySpark 4.3.0 raises RecursionError from 744 additions, before any request, 
and the process lives.
- master: 744 to a few thousand additions were sent and rejected by the server 
with "[Unknown error] Application error processing RPC", and from about 30,000 
the process died with SIGSEGV, on the main thread of Python. The serialized 
proto of such an expression is also dropped and encoded by recursing.

Serialize and render expressions on a stack that grows on demand (stacker, 
already in Cargo.lock), and raise RecursionError for a request whose plan has 
an expression deeper than 745 levels, as PySpark does, without serializing the 
expression past the bound.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to