Hyukjin Kwon created SPARK-61416:
------------------------------------
Summary: Raise RecursionError for the expressions of merges,
writeTo, function registrations, AutoCDC flows and pickled Columns too deep to
send, as PySpark does
Key: SPARK-61416
URL: https://issues.apache.org/jira/browse/SPARK-61416
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
SPARK-61328 and SPARK-61332 bound the depth of the expressions and plans that a
relation serializes (MAX_EXPRESSION_DEPTH, 745 levels, a RecursionError past
it). Expressions that requests serialize outside the plan of a relation have no
bound: the condition and the clause conditions and assignments of a merge, the
partitioning and the overwrite condition of writeTo, the arguments of a
function registration, the expressions of an AutoCDC flow, and a pickled Column
(Column.__reduce__, to_plan). A merge or an overwrite of 20,000 levels kills
the process with SIGSEGV (a 1,000-level one is sent), and pickling a Column of
20,000 levels takes 21 s and crashes at 150,000. PySpark raises RecursionError
for them before any request.
Fix: serialize those expressions with the same bound (Expression::try_to_proto,
Column::try_to_proto), so that they are a RecursionError past 745 levels and
are not built past the bound; the drop-in's merge clause methods raise it as
PySpark's _build_merge_action does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]