Hyukjin Kwon created SPARK-61695:
------------------------------------
Summary: Clone the expression of an unpickled Column on a stack
that holds its levels, so that a deep one is sent from a small thread
Key: SPARK-61695
URL: https://issues.apache.org/jira/browse/SPARK-61695
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
An unpickled Column of the Python client keeps the `spark.connect.Expression`
that it was pickled with (`ExpressionNode::Sent`), and serializing it clones
that proto with prost's derived `Clone`, which recurses once per level of the
expression, on the caller's stack. The clone of a `LogicalPlan::Raw` relation
runs on a stack grown for its levels; the clone of a sent expression did not. A
Column of 300 levels, unpickled and then used on a Python thread of 512 KiB
(for example `spark.range(1).select(column).schema`), overflows the thread's
stack in that clone and crashes the process with SIGBUS (seen on macOS x86_64,
where `test_a_deep_column_unpickles_and_is_used_on_a_thread_of_512_kib` fails).
The clone should run on a stack that holds its levels, as the clone of a raw
relation does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]