[
https://issues.apache.org/jira/browse/SPARK-39220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539876#comment-17539876
]
XiDuo You commented on SPARK-39220:
-----------------------------------
is it possible to also provide a stack log ?
> codegen cause NullPointException
> --------------------------------
>
> Key: SPARK-39220
> URL: https://issues.apache.org/jira/browse/SPARK-39220
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0, 2.4.6, 3.2.1
> Reporter: chenxusheng
> Priority: Major
>
> The following code raises NullPointException
> {code:sql}
> SELECT
> fk4c7a8cfc,
> fka54f2a73,
> fk37e266f7
> FROM
> be2a04fad4a24848bee641825e5b3466
> WHERE
> (
> fk4c7a8cfc is not null
> and fk4c7a8cfc<> ''
> )
> LIMIT
> 1000
> {code}
> However, if so, it is normal
> {code:sql}
> SELECT
> fk4c7a8cfc,
> fka54f2a73,
> fk37e266f7
> FROM
> be2a04fad4a24848bee641825e5b3466
> WHERE
> (
> fk4c7a8cfc is not null
> and '' <> fk4c7a8cfc
> )
> LIMIT
> 1000
> {code}
> I just put the '' in where in front.
> The reason for this problem is that the data contains null values.
> *_org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext#genEqual_*
> {code:scala}
> def genEqual(dataType: DataType, c1: String, c2: String): String = dataType
> match {
> case BinaryType => s"java.util.Arrays.equals($c1, $c2)"
> case FloatType =>
> s"((java.lang.Float.isNaN($c1) && java.lang.Float.isNaN($c2)) || $c1 ==
> $c2)"
> case DoubleType =>
> s"((java.lang.Double.isNaN($c1) && java.lang.Double.isNaN($c2)) || $c1
> == $c2)"
> case dt: DataType if isPrimitiveType(dt) => s"$c1 == $c2"
> case dt: DataType if dt.isInstanceOf[AtomicType] => s"$c1.equals($c2)"
> case array: ArrayType => genComp(array, c1, c2) + " == 0"
> case struct: StructType => genComp(struct, c1, c2) + " == 0"
> case udt: UserDefinedType[_] => genEqual(udt.sqlType, c1, c2)
> case NullType => "false"
> case _ =>
> throw new IllegalArgumentException(
> "cannot generate equality code for un-comparable type: " +
> dataType.catalogString)
> }
> {code}
> {code:scala}
> case dt: DataType if dt.isInstanceOf[AtomicType] => s"$c1.equals($c2)"
> {code}
> Missing null value judgment?
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
