txwyy123 commented on PR #59529:
URL: https://github.com/apache/spark/pull/59529#issuecomment-6377284231
Thanks for the thorough review @cloud-fan!
**Design choice (Q1):** I'm keeping this as a targeted guard on this
producer, with a narrower scope. The broader rewrite-boundary fix can be
tracked as a follow-up — the remaining query shapes (DataFrame API
`.orderBy(...).select(..., <correlated scalar>).limit(3)`) take a different
path that doesn't go through this rule. Updated the PR title and description to
reflect the narrower claim.
**Guard breadth (Q2):** Narrowed the predicate to only block subqueries that
become joins:
```scala
!SubqueryExpression.hasCorrelatedSubquery(e) &&
!SubqueryExpression.hasInOrCorrelatedExistsSubquery(e)
```
Uncorrelated subqueries are now pushed through, preserving the
`CollectLimitExec`/`TakeOrderedAndProjectExec` plan shape from SPARK-40501.
Updated the Scaladoc to explain why only correlated/IN/EXISTS subqueries are
blocked.
**Test fixes (P3s):**
- Plan-shape tests now use **correlated** scalar subqueries (with
`OuterReference` + `outerAttrs`), plus correlated EXISTS and IN (`ListQuery`).
Added a bare `localLimit()` case to cover the first match arm.
- Added an **uncorrelated scalar subquery regression test** that verifies it
IS still pushed through.
- All end-to-end SQL tests now run with `autoBroadcastJoinThreshold=-1` to
force SortMergeJoin. The OFFSET test data is inserted in score-descending order
(reverse of the requested sort) so the test actually fails without the fix.
Added an IN-subquery end-to-end test.
- Removed the limit-over-limit plan scenario that expected a plan no rule in
the batch produces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
