Yibo Dong created SPARK-61525:
---------------------------------

             Summary: histogram_numeric produces duplicate NaN bins and loses a 
finite input value
                 Key: SPARK-61525
                 URL: https://issues.apache.org/jira/browse/SPARK-61525
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.3.0
         Environment: 
{code}
Spark: 4.3.0 
Commit: 98bc0212c94705434471627068f41e374ccab143
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
Master: local[2]
{code}


            Reporter: Yibo Dong


For the input values \{{NaN}}, \{{NaN}}, and \{{1.0}}, \{{histogram_numeric(v, 
2)}} returns two NaN-centered bins, with heights 2 and 1. The finite value 
\{{1.0}} is not represented by either bin.
The input contains only two distinct values under Spark SQL equality semantics, 
and two bins are requested. However, both bins are centered at NaN, with a 
combined height of 3, although only two input rows contain NaN.

h2. Steps to Reproduce

Save the following SQL as {{tmp.sql}}:

{code}
WITH input AS (
SELECT /*+ REPARTITION(1) */ v
FROM VALUES
(CAST('NaN' AS DOUBLE)),
(CAST('NaN' AS DOUBLE)),
(1D) AS t(v)
)
SELECT histogram_numeric(v, 2) AS h FROM input;
{code}

Execute using stock Spark 4.3.0:

{code}
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
-f tmp.sql
{code}

h2. Expected Result

The two bins should represent the two distinct input values, with heights 
matching their frequencies:

{code}
[{"x":1.0,"y":1.0},{"x":"NaN","y":2.0}]
{code}

Spark SQL treats NaN values as equal and groups them together in aggregations 
([NaN 
semantics|https://spark.apache.org/docs/4.3.0/sql-ref-datatypes.html#nan-semantics]).

Although {{histogram_numeric}} is approximate, the input contains only two 
distinct values and requests two bins. It should not produce duplicate 
NaN-centered bins while losing the finite value.

h2. Actual Result

{code}
[{"x":"NaN","y":2.0},{"x":"NaN","y":1.0}]
{code}

Both bins are centered at NaN, and the finite value {{1.0}} is missing from the 
bin centers. The same result occurs when the input is repartitioned into two 
partitions.
 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to