Hyukjin Kwon created SPARK-61332:
------------------------------------
Summary: Send plans of up to 746 levels as PySpark does, and count
the plans of subqueries for the depth bound without overflowing the stack
Key: SPARK-61332
URL: https://issues.apache.org/jira/browse/SPARK-61332
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
SPARK-61260 bounded the depth of a plan at 256 levels, on the premise that
PySpark raises RecursionError from about 245 levels. It does not: official
PySpark 4.3.0 serializes plans up to about 746 levels (bisected with
_plan.to_proto: 745 filters, 746 withColumns or selects, 747 limits) and raises
RecursionError from there, and the server runs plans up to about 510 levels. So
the Rust client and the Python drop-in reject plans of 257 to about 510 levels
that PySpark, the previous master and the server all run, such as Apache's own
test_recursion_handling_for_plan_logging (400 withColumns, then the schema).
Also, the thread that SPARK-61260 serializes a deep plan on is sized for the
levels of the outer plan only, but the plans of its subqueries (EXISTS, scalar,
IN, table arguments) are serialized on it too: an outer plan of about 34 levels
with an EXISTS over a plan of 256 levels (Rust) or 300 (drop-in) overflows that
thread and aborts the process, as do much shallower ones in debug builds. The
previous master ran these plans.
Raise the bound to 746 levels, as PySpark's, serialize plans on a stack that
grows on demand (as SPARK-61328 does for expressions) instead of a sized
thread, and count the levels of the plans of subqueries under the plan that has
them, as they are sent, for the bound.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]