ErikBPF commented on PR #5903: URL: https://github.com/apache/datafusion-comet/pull/5903#issuecomment-6376832061
Rebased onto current `main` (`5a50bd6f2`), collapsed to one commit (`8c5700c5a`), and addressed the open review threads: - **Dedup** (comphead): `CollectSet`/`CollectList` now share one branch over `aggFunc.children.head.dataType.asNullable`. - **Spill assertion** (comphead): removed rather than weakened. On this fixture no stage spills: `Final` and `Partial` both report `spill_count = 0` with and without `COMET_RESPECT_DATAFUSION_CONFIGS` / `sort_spill_reservation_bytes`, and a 400 KiB pool fails the Final allocation (`fair limit is 429496 bytes`) instead of spilling. The grouped test keeps both nested shapes plus native-execution and Spark-answer assertions. - **Ungrouped placement** (comphead): the `Accumulator` cases moved to `collect_list.sql` / `collect_set.sql`, both shapes and both functions; the SQL harness asserts native execution by default. - **andygrove**: the ungrouped query from the review is covered by those SQL fixtures; removing `asNullable` still makes the `collect_list` cases fail with the nullability mismatch. Local validation on this head (Spark 4.1.3): compile + spotless clean; grouped regression 1/1; `CometSqlFileTestSuite collect_list` 2/2 and `collect_set` 8/8; full `CometAggregateSuite` 163 passed, 0 failed, 2 ignored. The PR body is updated with the new scope and evidence. Comet CI and CodeQL are `action_required` on this head, as they were on every previous head, so no hosted verdict exists yet. Could a maintainer approve the gated CI and CodeQL runs, and enable `run-spark-4.1-tests` for the Spark SQL verdict the contributor guide asks for? I will follow up with the hosted results and fix anything they surface. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
