Hyukjin Kwon created SPARK-61400:
------------------------------------

             Summary: Clone and drop the request of a deep plan on a grown 
stack, so that a deep plan or expression does not crash a thread with a small 
stack, in Spark Connect Rust
                 Key: SPARK-61400
                 URL: https://issues.apache.org/jira/browse/SPARK-61400
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


A follow-up of SPARK-61260, SPARK-61328 and SPARK-61332, found in a post-merge 
review. The stack guard of those changes covers the serialization, 
Expression::to_proto, render and drop, but not the derived Clone of the request 
protos: analyze_plan clones the request on every attempt of with_retry and 
execute_plan_reattachable clones it too, on the calling thread. A 256-level 
withColumn chain analyzed from a thread with a 512 KiB stack, or a 746-level 
one from a 1 MiB thread, crashes the process with SIGBUS in Relation::clone 
instead of being sent or raising RecursionError. The merge condition and 
actions of MergeIntoWriter are also serialized outside the expression depth 
bound, and the comments that give PySpark's bound as 744 or 746 levels do not 
hold for either protobuf backend of PySpark.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to