dwsmith1983 opened a new pull request, #6302: URL: https://github.com/apache/datafusion-comet/pull/6302
## Which issue does this PR close? Part of #5712 (item 3, the test for the per-batch element limit). ## Rationale for this change Comet's native `sequence` fails a batch whose total element count would pass `Int.MaxValue`, the limit of Arrow's i32 list offsets, and the error tells the user to lower `spark.comet.batchSize`. Spark builds one array per row and completes the same query. That makes it the one known case where Comet fails a query Spark finishes, and nothing tested it. ## What changes are included in this PR? One test in `CometArrayExpressionSuite`. A single-file table holds 8192 rows of `sequence(0, 262143)`, which is 8192 * 262144 = 2^31 elements in one batch at the default batch size. The test checks that: - Spark returns 2147483648 for `sum(size(sequence(a, b)))`; - Comet fails, and the error names both the 2147483648 total and `spark.comet.batchSize`; - with `spark.comet.batchSize=256`, Comet matches Spark and runs natively. The issue sketched this as a `sequence.sql` case with `expect_error`, but `expect_error` in the SQL file tests requires Spark to throw as well, and here Spark succeeds, so it is a Scala test. The lower batch size is 256 rather than 4096 because each successful batch materializes its elements in native memory: 256 rows are 256 MB, while 4096 rows would be about 4 GB. ## How are these changes tested? The test passes on Spark 3.5 and 4.1. Changing it to expect Comet to succeed at the default batch size makes it fail with the per-batch limit error, so it does reach that path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
