[
https://issues.apache.org/jira/browse/SPARK-60865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60865:
-----------------------------------
Labels: pull-request-available (was: )
> assertDataFrameEqual fails for unordered rows with floating-point tolerance
> ---------------------------------------------------------------------------
>
> Key: SPARK-60865
> URL: https://issues.apache.org/jira/browse/SPARK-60865
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Aditya Ranjan
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> {{pyspark.testing.assertDataFrameEqual}} can incorrectly report unequal rows
> when {{checkRowOrder=False}} and floating-point values compare equal within
> the configured {{rtol}} and {{atol}} tolerances.
> The current implementation sorts rows using their string representations and
> compares the sorted rows positionally. Lexicographic string ordering does not
> preserve tolerance-aware numerical equality.
> Consequently, two collections of rows can have a valid one-to-one matching
> while the positional comparison fails.
> h3. Steps to Reproduce
> {{from pyspark.sql import Row
> from pyspark.testing import assertDataFrameEqual
> actual = [
> Row(value=0.0),
> Row(value=0.1),
> ]
> expected = [
> Row(value=0.1),
> Row(value=0.2),
> ]
> assertDataFrameEqual(
> actual,
> expected,
> rtol=0,
> atol=0.11,
> checkRowOrder=False,
> )}}
> h3. Actual Behavior
> The assertion can fail even though every actual row can be matched uniquely
> to an expected row within the configured absolute tolerance.
> h3. Expected Behavior
> When {{{}checkRowOrder=False{}}}, the assertion should succeed if a complete
> one-to-one matching exists between the actual and expected rows under the
> configured comparison tolerances.
> When {{{}checkRowOrder=True{}}}, positional comparison should remain enforced.
> Genuinely unequal rows must continue to fail.
> h3. Root Cause
> Sorting rows by {{str(row)}} followed by positional comparison is not
> sufficient for unordered, tolerance-aware comparisons.
> A greedy matching strategy can also fail when tolerance ranges overlap and an
> earlier match needs reassignment.
> h3. Proposed Fix
> Retain the existing deterministic sorting and fast positional comparison.
> If positional comparison fails for equally sized unordered collections,
> attempt one-to-one row matching using the existing tolerance-aware comparison
> logic.
> Allow previously matched rows to be reassigned when necessary to obtain a
> complete matching.
> h3. Testing
> The proposed implementation includes regression tests for:
> * Approximate floating-point equality across unordered rows.
> * Overlapping tolerance ranges requiring reassignment.
> * Ordered comparisons that must detect differences.
> * Unequal rows that must continue to fail.
> Local validation included seven focused tests, randomized and exhaustive
> matching checks, Python compilation, and Git whitespace validation.
> h3. References
> Original GitHub issue: [https://github.com/apache/spark/issues/60027]
> Implementation PR: [https://github.com/apache/spark/pull/60047]
> Note: GitHub issue #60027 is distinct from Jira SPARK-60027, which tracks an
> unrelated Spark SQL issue.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
