[
https://issues.apache.org/jira/browse/SPARK-61589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-61589:
----------------------------
Priority: Critical (was: Blocker)
> Define NULL semantics for KLL quantile and rank getter arrays
> -------------------------------------------------------------
>
> Key: SPARK-61589
> URL: https://issues.apache.org/jira/browse/SPARK-61589
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Yan Yan
> Priority: Critical
>
> The KLL quantile and rank getter functions accept a constant scalar or an
> array of requests. NULL elements inside a non-NULL request array are
> currently read as numeric zero and produce non-NULL results. This behavior is
> not explained in the public API contract, so users cannot distinguish it from
> scalar or whole-array NULL propagation.
> This affects both getter families for bigint, float and double:
> * kll_sketch_get_quantile_bigint / float / double
> * kll_sketch_get_rank_bigint / float / double
> For a sketch containing -10, 0 and 10, the current bigint results are:
> {code:sql}
> SELECT kll_sketch_get_quantile_bigint(kll_sketch_agg_bigint(value),
> array(0.5D, NULL))
> FROM VALUES (-10L), (0L), (10L) AS t(value);
> -- [0, -10]
> SELECT kll_sketch_get_rank_bigint(kll_sketch_agg_bigint(value), array(10L,
> NULL))
> FROM VALUES (-10L), (0L), (10L) AS t(value);
> -- [1.0, 0.6666666666666666]
> {code}
> The second example is significant: interpreting a NULL element as a
> zero-valued quantile returns the rank of zero, which need not be zero. An
> all-NULL request array also produces an array of zero-request results rather
> than a NULL array.
> Define and regression-test the following contract consistently across SQL,
> Scala and Python:
> * A NULL sketch, scalar NULL request, or whole-array NULL request returns
> NULL.
> * A NULL rank element passed to a quantile getter requests rank zero,
> yielding the minimum quantile.
> * A NULL quantile element passed to a rank getter requests the rank of zero.
> * This applies to mixed and all-NULL request arrays for all three numeric
> sketch types.
> The proposed resolution preserves existing accepted-query behavior. The APIs
> do not establish a clear non-NULL-element precondition today; introducing
> rejection would change those queries. Documenting the existing semantics and
> adding focused regression coverage makes the contract explicit without
> changing runtime results.
> Implementation and regression coverage:
> https://github.com/apache/spark/pull/60358
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]