[
https://issues.apache.org/jira/browse/SPARK-58631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58631:
-----------------------------------
Labels: pull-request-available (was: )
> array_repeat and array_insert raise inconsistent array size limit errors in
> codegen and interpreted execution
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58631
> URL: https://issues.apache.org/jira/browse/SPARK-58631
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: sepuri sai krishna
> Priority: Minor
> Labels: pull-request-available
>
> {{array_repeat}} and {{array_insert}} raise a different error for an array
> that exceeds the maximum array length depending on whether the expression
> runs through the interpreted path or through generated code.
> h3. array_repeat
> {code}
> eval -> COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER (SQLSTATE 54000)
> codegen -> _LEGACY_ERROR_TEMP_2176
> {code}
> {{ArrayRepeat.eval}} checks {{count}} against
> {{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}} and raises
> {{COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER}}.
> {{ArrayRepeat.genCodeForNumberOfElements}} performs no such check, so the
> generated code proceeds to the array allocation, which fails with the
> internal {{_LEGACY_ERROR_TEMP_2176}}. Codegen is the default execution path,
> so the internal error class is what users normally see. The interpreted path
> was migrated to the named condition by SPARK-45710 (4.0.0); the codegen path
> was not.
> h3. array_insert
> {code}
> eval -> COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION
> codegen -> COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER
> {code}
> The codegen branch taken when {{pos}} is not foldable calls
> {{createArrayWithElementsExceedLimitError}}, which hardcodes {{"parameter" ->
> toSQLId("count")}}. The resulting message reads "the value of parameter(s)
> `count` in the function `array_insert` is invalid", but {{array_insert(array,
> pos, value)}} has no {{count}} parameter. Every other site in the expression
> -- all three interpreted ones and the two sibling codegen ones -- uses
> {{arrayFunctionWithElementsExceedLimitError}}
> ({{COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION}}). Introduced by SPARK-56567
> (4.2.0).
> h3. Why this was not caught
> The existing test "Elements exceed limit for array_repeat()" in
> {{QueryExecutionErrorsSuite}} runs {{select array_repeat(1, 2147483647)}},
> which is constant-folded and therefore only exercises the interpreted path.
> The {{array_insert}} branch is reachable only with a non-foldable {{pos}},
> because a foldable positive {{pos}} takes a different codegen branch.
> h3. Scope
> Error reporting only -- neither function produces wrong results, and the
> conditions under which each raises an error are unchanged.
> {{Sequence}} also reports {{COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER}} naming
> a {{count}} parameter it does not have, but it does so from
> {{sequenceLength}}, which both execution paths share. There is no
> interpreted-vs-codegen inconsistency there, so it is left alone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
