[ 
https://issues.apache.org/jira/browse/SPARK-61500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wechar Yu updated SPARK-61500:
------------------------------
    Summary: Handle null values in ColumnarArray.get() and 
ColumnarBatchRow.get()  (was: Handle null elements in ColumnarArray.get())

> Handle null values in ColumnarArray.get() and ColumnarBatchRow.get()
> --------------------------------------------------------------------
>
>                 Key: SPARK-61500
>                 URL: https://issues.apache.org/jira/browse/SPARK-61500
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.1.4
>            Reporter: Wechar Yu
>            Priority: Major
>
> ColumnarArray.get(ordinal, dataType) does not check whether the element is
> null before reading its value. For primitive-backed types, it can return
> the underlying stored value even when isNullAt(ordinal) returns true.
> For example:
> {code:scala}
> val vector = new OnHeapColumnVector(1, IntegerType)
> try {
>   vector.putInt(0, 42)
>   vector.putNull(0)
>   val array = new ColumnarArray(vector, 0, 1)
>   assert(array.isNullAt(0))
>   array.get(0, IntegerType) // Returns 42; expected null.
> } finally {
>   vector.close()
> }
> {code}
> The same issue affects OffHeapColumnVector.
> Additionally, get(ordinal, NullType) throws an unsupported-data-type
> exception instead of returning null.
> Enable null handling in the SpecializedGettersReader.read() call used by
> ColumnarArray.get(), consistent with UnsafeArrayData.get() and
> UnsafeRow.get().



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to