Hyukjin Kwon created SPARK-61695:
------------------------------------

             Summary: Clone the expression of an unpickled Column on a stack 
that holds its levels, so that a deep one is sent from a small thread
                 Key: SPARK-61695
                 URL: https://issues.apache.org/jira/browse/SPARK-61695
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


An unpickled Column of the Python client keeps the `spark.connect.Expression` 
that it was pickled with (`ExpressionNode::Sent`), and serializing it clones 
that proto with prost's derived `Clone`, which recurses once per level of the 
expression, on the caller's stack. The clone of a `LogicalPlan::Raw` relation 
runs on a stack grown for its levels; the clone of a sent expression did not. A 
Column of 300 levels, unpickled and then used on a Python thread of 512 KiB 
(for example `spark.range(1).select(column).schema`), overflows the thread's 
stack in that clone and crashes the process with SIGBUS (seen on macOS x86_64, 
where `test_a_deep_column_unpickles_and_is_used_on_a_thread_of_512_kib` fails). 
The clone should run on a stack that holds its levels, as the clone of a raw 
relation does.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to