[
https://issues.apache.org/jira/browse/SPARK-55897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-55897:
-----------------------------------
Labels: pull-request-available (was: )
> ColumnarRow.get() and ColumnarBatchRow.get() throw on UserDefinedType
> ---------------------------------------------------------------------
>
> Key: SPARK-55897
> URL: https://issues.apache.org/jira/browse/SPARK-55897
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.1
> Environment: I don't think this is hardware-dependent but I
> discovered this on an M3 Macbook pro.
> Reporter: James Willis
> Priority: Major
> Labels: pull-request-available
>
> {{ColumnarRow.get()}} and {{ColumnarBatchRow.get()}} do not handle
> {{{}UserDefinedType{}}}, throwing
> {{SparkUnsupportedOperationException("_LEGACY_ERROR_TEMP_3155")}} when a UDT
> field is accessed via the interpreted eval path (e.g.,
> {{GetStructField.nullSafeEval}} on a nested struct from the vectorized
> Parquet reader).
> {code:java}
> org.apache.spark.SparkException: [INTERNAL_ERROR] Undefined error message
> parameter for error class:
> '_LEGACY_ERROR_TEMP_3155', MessageTemplate: Datatype not supported
> <dataType>, Parameters: Map()
> at org.apache.spark.sql.vectorized.ColumnarRow.get(ColumnarRow.java:221)
> at
> org.apache.spark.sql.catalyst.expressions.GetStructField.nullSafeEval(complexTypeExtractors.scala:207){code}
> *This happens when:*
> # The vectorized Parquet reader produces a {{ColumnarBatch}}
> # {{ColumnarToRowExec}} (in WSCG codegen mode) reads top-level columns via
> typed accessors ({{{}getArray{}}}, {{{}getBinary{}}}, etc.), which works fine
> # But for *nested* structures (e.g., {{{}array[0].field{}}}), the top-level
> {{getArray()}} returns a {{{}ColumnarArray{}}}, and indexing into it returns
> a {{ColumnarRow}} — these remain as columnar objects, not copied to
> {{UnsafeRow}}
> # A downstream expression that *can't be codegenned* (like
> {{InferredExpression}} in PR #611) falls back to {{{}eval(){}}}, which calls
> {{GetStructField.nullSafeEval()}} on the {{ColumnarRow}}
> # {{GetStructField}} passes the UDT type (from the expression's schema) to
> {{{}ColumnarRow.get(){}}}, which doesn't handle UDT → crash
> *The codegen path never hits this* because {{CodeGenerator.getValue()}} (line
> 1683) resolves {{UserDefinedType }}to {{sqlType}} before generating code, so
> it generates {{getBinary()}} instead of {{{}get(ordinal, UDT){}}}.
> h3. Root Cause
> {{ColumnarRow.get()}} and {{ColumnarBatchRow.get()}} dispatch on {{dataType}}
> via {{instanceof}} checks for all concrete Spark types but have no branch for
> {{{}UserDefinedType{}}}. When {{{}GetStructField.nullSafeEval(){}}}passes a
> UDT type to {{{}get(){}}}, it falls through to the default error branch.
> The codegen path is unaffected because {{CodeGenerator.getValue()}} unwraps
> {{udt.sqlType()}} before generating type-specific accessor calls
> ({{{}getInt{}}}, {{{}getStruct{}}}, etc.), bypassing {{get()}} entirely. This
> is why the existing SPARK-39086 tests pass — they run through whole-stage
> codegen.
> The bug surfaces when the interpreted path is used (codegen disabled, codegen
> fallback, or exceeding {{{}spark.sql.codegen.maxFields{}}}).
> h3. Affected Code
> * {{ColumnarRow.java:184-223}} — {{get(int ordinal, DataType dataType)}}
> * {{ColumnarBatchRow.java:179-222}} — {{get(int ordinal, DataType dataType)}}
> * {{ColumnarArray.java:215-217}} — {{get(int ordinal, DataType dataType)}}
> *
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
