adiranjan25 opened a new pull request, #60047: URL: https://github.com/apache/spark/pull/60047
## What changes were proposed in this pull request? Fix `assertDataFrameEqual` when `checkRowOrder=False` and floating-point values are equal within the configured tolerances but have different string representations. Previously, rows were sorted by their string representations and compared positionally. This could produce false assertion failures because string ordering does not preserve tolerance-aware equality. This change retains deterministic sorting and introduces a fallback matching procedure when positional comparison fails. It finds a complete one-to-one matching of approximately equal rows, including cases where overlapping tolerances require reassignment of earlier matches. ## How was this patch tested? Added regression tests covering: - Approximately equal floating-point values with different representations. - Overlapping tolerances requiring reassignment of matches. - Ordered comparisons that must still detect differences. - Genuinely unequal rows that must still fail. Local validation included seven focused tests, exhaustive and randomized matching checks, Python syntax compilation, and `git diff --check`. The fallback matching procedure has additional computational cost, with approximately quadratic scaling in measured cases and potentially higher worst-case complexity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
