mhatipog opened a new pull request, #59445: URL: https://github.com/apache/spark/pull/59445
What changes were proposed? Prevent subquery-containing projections from being pushed below LIMIT or OFFSET. Ordinary deterministic projections retain the existing optimization. Added four regression cases covering correlated scalar and EXISTS subqueries over sorted LIMIT and OFFSET. Why are the changes needed? Subqueries may be rewritten into joins that change row ordering after projection pushdown, silently returning incorrect results. Fixes #59438. How was this patch tested? Regression tests added to SubquerySuite. Not run locally; CI validation pending. Was generative AI used? Yes. The patch was developed with AI assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
