Wechar Yu created SPARK-61500:
---------------------------------
Summary: Handle null elements in ColumnarArray.get()
Key: SPARK-61500
URL: https://issues.apache.org/jira/browse/SPARK-61500
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.4
Reporter: Wechar Yu
ColumnarArray.get(ordinal, dataType) does not check whether the element is
null before reading its value. For primitive-backed types, it can return
the underlying stored value even when isNullAt(ordinal) returns true.
For example:
{code:scala}
val vector = new OnHeapColumnVector(1, IntegerType)
try {
vector.putInt(0, 42)
vector.putNull(0)
val array = new ColumnarArray(vector, 0, 1)
assert(array.isNullAt(0))
array.get(0, IntegerType) // Returns 42; expected null.
} finally {
vector.close()
}
{code}
The same issue affects OffHeapColumnVector.
Additionally, get(ordinal, NullType) throws an unsupported-data-type
exception instead of returning null.
Enable null handling in the SpecializedGettersReader.read() call used by
ColumnarArray.get(), consistent with UnsafeArrayData.get() and
UnsafeRow.get().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]