Wechar Yu created SPARK-61500:
---------------------------------

             Summary: Handle null elements in ColumnarArray.get()
                 Key: SPARK-61500
                 URL: https://issues.apache.org/jira/browse/SPARK-61500
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.1.4
            Reporter: Wechar Yu


ColumnarArray.get(ordinal, dataType) does not check whether the element is
null before reading its value. For primitive-backed types, it can return
the underlying stored value even when isNullAt(ordinal) returns true.

For example:

{code:scala}
val vector = new OnHeapColumnVector(1, IntegerType)
try {
  vector.putInt(0, 42)
  vector.putNull(0)

  val array = new ColumnarArray(vector, 0, 1)
  assert(array.isNullAt(0))
  array.get(0, IntegerType) // Returns 42; expected null.
} finally {
  vector.close()
}
{code}

The same issue affects OffHeapColumnVector.

Additionally, get(ordinal, NullType) throws an unsupported-data-type
exception instead of returning null.

Enable null handling in the SpecializedGettersReader.read() call used by
ColumnarArray.get(), consistent with UnsafeArrayData.get() and
UnsafeRow.get().



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to