Yibo Dong created SPARK-61525:
---------------------------------
Summary: histogram_numeric produces duplicate NaN bins and loses a
finite input value
Key: SPARK-61525
URL: https://issues.apache.org/jira/browse/SPARK-61525
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.3.0
Environment:
{code}
Spark: 4.3.0
Commit: 98bc0212c94705434471627068f41e374ccab143
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
Master: local[2]
{code}
Reporter: Yibo Dong
For the input values \{{NaN}}, \{{NaN}}, and \{{1.0}}, \{{histogram_numeric(v,
2)}} returns two NaN-centered bins, with heights 2 and 1. The finite value
\{{1.0}} is not represented by either bin.
The input contains only two distinct values under Spark SQL equality semantics,
and two bins are requested. However, both bins are centered at NaN, with a
combined height of 3, although only two input rows contain NaN.
h2. Steps to Reproduce
Save the following SQL as {{tmp.sql}}:
{code}
WITH input AS (
SELECT /*+ REPARTITION(1) */ v
FROM VALUES
(CAST('NaN' AS DOUBLE)),
(CAST('NaN' AS DOUBLE)),
(1D) AS t(v)
)
SELECT histogram_numeric(v, 2) AS h FROM input;
{code}
Execute using stock Spark 4.3.0:
{code}
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
-f tmp.sql
{code}
h2. Expected Result
The two bins should represent the two distinct input values, with heights
matching their frequencies:
{code}
[{"x":1.0,"y":1.0},{"x":"NaN","y":2.0}]
{code}
Spark SQL treats NaN values as equal and groups them together in aggregations
([NaN
semantics|https://spark.apache.org/docs/4.3.0/sql-ref-datatypes.html#nan-semantics]).
Although {{histogram_numeric}} is approximate, the input contains only two
distinct values and requests two bins. It should not produce duplicate
NaN-centered bins while losing the finite value.
h2. Actual Result
{code}
[{"x":"NaN","y":2.0},{"x":"NaN","y":1.0}]
{code}
Both bins are centered at NaN, and the finite value {{1.0}} is missing from the
bin centers. The same result occurs when the input is repartitioned into two
partitions.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]