txwyy123 commented on PR #59529: URL: https://github.com/apache/spark/pull/59529#issuecomment-6492141276
Pushed another revision -- fixed the IN subquery end-to-end test. **What was wrong:** The IN test had `WHERE id IN (SELECT a FROM k5)` -- the IN was in a `Filter` node, not the `Project`. `PushProjectionThroughLimitAndOffset` only operates on `Project` nodes, so the test wasn't exercising our guard at all. The IN in WHERE goes through `RewritePredicateSubquery` (a different code path). **The fix:** Moved the IN to the SELECT list: `SELECT name, (id IN (SELECT a FROM k5)) AS is_matched`. When IN appears in a `Project`, `handleUnaryNode` rewrites it to an `ExistenceJoin` via `rewriteExistentialExprWithAttrs` -- which is still a join that repartitions and destroys sort order. This is the same bug mechanism, now correctly exercised. **Verified the other 3 tests are correct:** - Scalar subquery tests: `(SELECT count/sum(...)) AS col` -- in the SELECT list ✅ - EXISTS test: `EXISTS(SELECT 1 ...) AS has_match` -- in the SELECT list ✅ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
