[
https://issues.apache.org/jira/browse/SPARK-61500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar Yu updated SPARK-61500:
------------------------------
Summary: Handle null values in ColumnarArray.get() and
ColumnarBatchRow.get() (was: Handle null elements in ColumnarArray.get())
> Handle null values in ColumnarArray.get() and ColumnarBatchRow.get()
> --------------------------------------------------------------------
>
> Key: SPARK-61500
> URL: https://issues.apache.org/jira/browse/SPARK-61500
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.4
> Reporter: Wechar Yu
> Priority: Major
>
> ColumnarArray.get(ordinal, dataType) does not check whether the element is
> null before reading its value. For primitive-backed types, it can return
> the underlying stored value even when isNullAt(ordinal) returns true.
> For example:
> {code:scala}
> val vector = new OnHeapColumnVector(1, IntegerType)
> try {
> vector.putInt(0, 42)
> vector.putNull(0)
> val array = new ColumnarArray(vector, 0, 1)
> assert(array.isNullAt(0))
> array.get(0, IntegerType) // Returns 42; expected null.
> } finally {
> vector.close()
> }
> {code}
> The same issue affects OffHeapColumnVector.
> Additionally, get(ordinal, NullType) throws an unsupported-data-type
> exception instead of returning null.
> Enable null handling in the SpecializedGettersReader.read() call used by
> ColumnarArray.get(), consistent with UnsafeArrayData.get() and
> UnsafeRow.get().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]