tlfua opened a new issue, #7527: URL: https://github.com/apache/datafusion-comet/issues/7527
### Describe the bug With `spark.sql.ansi.enabled=true`, a TopK (`ORDER BY ... LIMIT n`) whose whole input is a single row fails in Comet with `ARITHMETIC_OVERFLOW` when its first sort key overflows on that row, while Spark returns the row. Spark's `TakeOrderedAndProjectExec` goes through `takeOrdered`, which uses Guava's `Ordering.leastOf` (`TopKSelector`) per partition and a `BoundedPriorityQueue` to merge. Both store the first element without a comparison, and sorting a single element compares nothing, so Spark's lazily generated ordering never evaluates any sort key when the input has one row. Comet's native TopK evaluates every sort key over the whole batch. With two or more rows Spark compares and evaluates the first key for every row, so both engines fail. This is pre-existing and independent of https://github.com/apache/datafusion-comet/pull/7457 , which falls back when a sort key after the first can fail; it was raised in review there. ### Steps to reproduce ```sql SET spark.sql.ansi.enabled=true; SET spark.sql.adaptive.enabled=false; CREATE TABLE one_row(a int, b int) USING parquet; INSERT INTO one_row VALUES (1, 2147483647); SELECT a, b FROM one_row ORDER BY b + 1 LIMIT 10; ``` Spark returns `1, 2147483647`. Comet fails with `[ARITHMETIC_OVERFLOW] integer overflow` (checked on Spark 4.1 with a `CometSqlFileTestSuite` fixture, current main plus #7457). ### Expected behavior Same result as Spark. Since it only affects a single-row input, it may be enough to document it rather than fall back for every TopK with a fallible first key. ### Additional context Suggested labels: priority:low, correctness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
