Hyukjin Kwon created SPARK-61332:
------------------------------------

             Summary: Send plans of up to 746 levels as PySpark does, and count 
the plans of subqueries for the depth bound without overflowing the stack
                 Key: SPARK-61332
                 URL: https://issues.apache.org/jira/browse/SPARK-61332
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


SPARK-61260 bounded the depth of a plan at 256 levels, on the premise that 
PySpark raises RecursionError from about 245 levels. It does not: official 
PySpark 4.3.0 serializes plans up to about 746 levels (bisected with 
_plan.to_proto: 745 filters, 746 withColumns or selects, 747 limits) and raises 
RecursionError from there, and the server runs plans up to about 510 levels. So 
the Rust client and the Python drop-in reject plans of 257 to about 510 levels 
that PySpark, the previous master and the server all run, such as Apache's own 
test_recursion_handling_for_plan_logging (400 withColumns, then the schema).

Also, the thread that SPARK-61260 serializes a deep plan on is sized for the 
levels of the outer plan only, but the plans of its subqueries (EXISTS, scalar, 
IN, table arguments) are serialized on it too: an outer plan of about 34 levels 
with an EXISTS over a plan of 256 levels (Rust) or 300 (drop-in) overflows that 
thread and aborts the process, as do much shallower ones in debug builds. The 
previous master ran these plans.

Raise the bound to 746 levels, as PySpark's, serialize plans on a stack that 
grows on demand (as SPARK-61328 does for expressions) instead of a sized 
thread, and count the levels of the plans of subqueries under the plan that has 
them, as they are sent, for the bound.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to