rich7420 opened a new pull request, #7195: URL: https://github.com/apache/datafusion-comet/pull/7195
## Which issue does this PR close? Part of #7147. ## Rationale for this change A native projection immediately below LocalLimit or GlobalLimit can evaluate an ANSI cast on rows beyond the limit and fail where Spark succeeds, for example when LIMIT is inside a subquery. ## What changes are included in this PR? Extend the existing CollectLimit projection input cap to adjacent LocalLimit and GlobalLimit. With native local limits enabled, a limit below the projection bounds its input before expression evaluation. Keep the outer limit and include the OFFSET prefix in the cap. Preserve configuration gating, column-only projection pruning, AQE links and repeated planning. Exclude CollectTail and offset-only limits, and never move the cap across filters or sorts. Update compatibility documentation. Throwing filters, GlobalLimit codegen OFFSET-prefix differences and the investigation of already-ordered TakeOrderedAndProject inputs remain outside this PR. ## How are these changes tested? - Spark 4.0.4: 16 focused tests passed; seven regressions failed before the fix. - Spark 4.1.3: 131 tests passed, including the full planner suite and focused LIMIT suites. After strengthening the final assertions, all four affected tests passed again. - Coverage includes direct LocalLimit/GlobalLimit, offsets, zero limits, batch boundaries, multiple partitions, AQE and consumed invalid values. - Native debug build, Spotless, Scalastyle, Prettier and `git diff --check` passed. - Spark 3.4/3.5/4.2 were not run locally. Draft pending Spark SQL CI. Please apply `run-spark-4.1-tests`, covering `InsertSortForLimitAndOffsetSuite` (sql_core-1) and `StreamSuite` (sql_core-3), before review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
