[ 
https://issues.apache.org/jira/browse/SPARK-61695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61695:
-----------------------------------
    Labels: pull-request-available  (was: )

> Clone the expression of an unpickled Column on a stack that holds its levels, 
> so that a deep one is sent from a small thread
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61695
>                 URL: https://issues.apache.org/jira/browse/SPARK-61695
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> An unpickled Column of the Python client keeps the `spark.connect.Expression` 
> that it was pickled with (`ExpressionNode::Sent`), and serializing it clones 
> that proto with prost's derived `Clone`, which recurses once per level of the 
> expression, on the caller's stack. The clone of a `LogicalPlan::Raw` relation 
> runs on a stack grown for its levels; the clone of a sent expression did not. 
> A Column of 300 levels, unpickled and then used on a Python thread of 512 KiB 
> (for example `spark.range(1).select(column).schema`), overflows the thread's 
> stack in that clone and crashes the process with SIGBUS (seen on macOS 
> x86_64, where 
> `test_a_deep_column_unpickles_and_is_used_on_a_thread_of_512_kib` fails). The 
> clone should run on a stack that holds its levels, as the clone of a raw 
> relation does.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to