SEPURI-SAI-KRISHNA opened a new pull request, #58707: URL: https://github.com/apache/spark/pull/58707
### What changes were proposed in this pull request? `Flatten.genCodeForNumberOfElements` accumulates the result element count into a `long` and hands it to `CodeGenerator.createArrayData` without checking it against `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`. Interpreted evaluation does check. This PR adds the same check to the generated code so both paths raise the same error. This is the `flatten` half of the pattern that [SPARK-58631](https://issues.apache.org/jira/browse/SPARK-58631) (#57832) fixed for `array_repeat`. `concat` has the identical gap and is fixed in #58703 ([SPARK-59374](https://issues.apache.org/jira/browse/SPARK-59374)). ### Why are the changes needed? The two evaluation paths report different error conditions for the same input: - Interpreted: `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION`, from `QueryExecutionErrors.arrayFunctionWithElementsExceedLimitError`. - Codegen: no check, so execution reaches `ArrayData.allocateArrayData`, whose own bound check raises the internal `_LEGACY_ERROR_TEMP_2176` through `QueryExecutionErrors.cannotCreateArrayWithElementsExceedLimitError`. An internal error condition is not meant to be user facing, and which one a query gets should not depend on whether codegen was used. Worth noting for reviewers that this is latent rather than something a user hits today. Unlike `array_repeat`, where the element count comes straight from an argument, `flatten` needs its inputs to actually hold that many elements, so an executor will normally run out of memory before the limit is reached. The existing test reaches it only by using a `ConstantColumnVector`, which reports its length without allocating. ### Does this PR introduce _any_ user-facing change? Yes, in the sense that the error condition changes for an input that already fails. `flatten` with more than `MAX_ROUNDED_ARRAY_LENGTH` total elements now raises `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` under codegen, where it previously raised the internal `_LEGACY_ERROR_TEMP_2176`. That matches what interpreted evaluation has always raised. The set of inputs that fail is unchanged. Added a migration guide entry alongside the one #57832 added for `array_repeat` and `array_insert`, and the one #58703 adds for `concat`, since this is the same kind of change. Happy to drop it if you would rather not document a case that is this hard to reach. ### How was this patch tested? Extended the existing `Elements exceed limit for flatten()` test in `QueryExecutionErrorsSuite` to run the expression through `GenerateUnsafeProjection` as well as `eval`, asserting the same error from both. That test already builds a large array cheaply with `ConstantColumnVector`, which reports its length without allocating, so nothing large is allocated. It sits in `QueryExecutionErrorsSuite` rather than in `CollectionExpressionsSuite`, where #57832 put its tests, because `ConstantColumnVector` lives in `sql/core` and is not reachable from a catalyst suite. Extending the existing test also seemed better than adding a second one: the test was already there and simply never exercised codegen, which is how the gap survived. Reverting the change makes the new codegen case fail with `_LEGACY_ERROR_TEMP_2176`, which is the behaviour being fixed. ``` build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryExecutionErrorsSuite -- -z \"Elements exceed limit\"" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
