akshaytayal opened a new issue, #13438: URL: https://github.com/apache/gluten/issues/13438
Part of #13181 (sub-theme #13176). Found while re-enabling the Spark 4.2 UTs disabled in #13163. ### Problem `array_agg`/`collect_list(x) RESPECT NULLS` (SPARK-55256) drops nulls on Gluten: `[2, 4]` instead of Spark's `[2, null, 4]`. This is a silent wrong result. Disabled test: `GlutenDataFrameAggregateSuite` `SPARK-55256: array_agg with RESPECT NULLS preserves nulls`. ### Root cause Gluten rewrites to `VeloxCollectList(child, ignoreNulls = false)`, but Velox Spark `collect_list` has no per-call flag: it only reads the query config `spark.collect_list.ignore_nulls` (`velox/functions/sparksql/aggregates/CollectListAggregate.cpp`) and registers only `collect_list(E)`. A query-level config can't express `collect_list(a)` and `collect_list(b) RESPECT NULLS` in the same query. ### Proposed fix - Velox: add `collect_list(E, constant boolean ignoreNulls)`, mirroring `collect_set`, and read the constant input; keep the config only as the default for the one-argument form. - Gluten (after the Velox bump): in `VeloxIntermediateData.getRawInputs` add `case c: VeloxCollectList if !c.ignoreNulls => Seq(c.child, Literal.FalseLiteral)` (same as `collect_set` in the companion PR), and remove the exclude. - Interim option: fall back `collect_list`/`array_agg` with RESPECT NULLS to vanilla Spark to avoid the wrong result. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
