[
https://issues.apache.org/jira/browse/SPARK-59818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119968#comment-18119968
]
Jubin Soni commented on SPARK-59818:
------------------------------------
I'm working on this
> approx_top_k_estimate and approx_top_k_combine throw NullPointerException on
> a NULL sketch state
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-59818
> URL: https://issues.apache.org/jira/browse/SPARK-59818
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Jubin Soni
> Priority: Major
>
> h2. Summary
> The sketch-state argument of \{{approx_top_k_estimate}} and
> \{{approx_top_k_combine}} is an ordinary nullable input column, but neither
> function guards against a \{{NULL}} value. Both dereference the evaluated
> `state` unconditionally and fail with a \{{NullPointerException}}.
> Analysis does not prevent this: \{{checkStateFieldAndType}} validates only
> the _schema_ of the state struct, not its nullness. Therefore, a
> correctly-typed nullable struct column passes analysis and then fails at
> optimization or execution time.
> A \{{NULL}} sketch can arise naturally from an outer join, a UNION with a
> missing side, or a partially populated column.
> h2. 1) approx_top_k_estimate
> {code:sql}
> SELECT approx_top_k_estimate(s, 5) FROM (
> SELECT CAST(NULL AS STRUCT<sketch: BINARY, maxItemsTracked: INT,
> itemDataType: INT, itemDataTypeDDL: STRING>) AS s
> );
> {code}
> The NPE is raised inside \{{ConstantFolding}} and is reported to the user as
> an \{{INTERNAL_ERROR}}:
> {code}
> [INTERNAL_ERROR] The Spark SQL phase optimization failed with an internal
> error.
> You hit a bug in Spark or the Spark plugins you use. SQLSTATE: XX000
> Caused by: java.lang.NullPointerException: Cannot invoke
> "org.apache.spark.sql.catalyst.InternalRow.getBinary(int)" because
> "stateEval" is null
> at ApproxTopKEstimate.eval(ApproxTopKExpressions.scala:113)
> at
> org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.tryFold(expressions.scala:65)
> {code}
> There are two distinct problems here:
> * {\{ApproxTopKEstimate}} overrides \{{BinaryExpression.eval}} directly
> instead of using \{{nullSafeEval}}, so the inherited null handling is
> bypassed.
> * It declares \{{override def nullable: Boolean = false}} even though the
> result is \{{NULL}} whenever the state is \{{NULL}}. The reported output
> schema is therefore incorrect: it declares \{{nullable=false}} for an
> expression that can return \{{NULL}}. This can also allow the optimizer to
> reason incorrectly about the expression.
> h2. 2) approx_top_k_combine
> {code:sql}
> SELECT approx_top_k_estimate(approx_top_k_combine(s), 5) FROM (
> SELECT CAST(NULL AS STRUCT<sketch: BINARY, maxItemsTracked: INT,
> itemDataType: INT, itemDataTypeDDL: STRING>) AS s
> );
> {code}
> {\{ApproxTopKCombine.update}} has the same unguarded dereference. This one
> fails at execution time on the executors and aborts the job:
> {code}
> org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 0.0 failed 1 times ...
> Caused by: java.lang.NullPointerException
> at ApproxTopKCombine.update(ApproxTopKAggregates.scala:908)
> at TypedImperativeAggregate.update(interfaces.scala:583)
> {code}
> h2. Expected behaviour
> Consistent with SQL semantics for a nullable input column:
> * {\{approx_top_k_estimate(NULL, k)}} should return \{{NULL}}, and the
> expression should report itself as nullable when its state argument is
> nullable.
> * {\{approx_top_k_combine}} should skip \{{NULL}} sketches, as other
> aggregates ignore \{{NULL}} inputs.
> This is different from the existing \{{APPROX_TOP_K_NULL_ARG}} error, which
> is raised for \{{NULL}} _foldable configuration_ arguments (\{{k}},
> \{{maxItemsTracked}}).
> That check, \{{ApproxTopK.checkExpressionNotNull}}, calls \{{expr.eval()}}
> with no input row, so it applies only to literals and is not applicable to a
> per-row data column.
> h2. Relationship to earlier tickets
> * SPARK-53960 ("Let approx_top_k_accumulate/combine/estimate handle NULLs")
> added handling for NULLs _within_ the accumulated data (counting them as
> items). It did not address the NULL-state path or the nullable declaration.
> * SPARK-58095 concerns _empty_ input (zero rows) in combine's
> eval/serialize, which is a different code path and a different trigger from a
> NULL state value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
