KazMiddelhoek opened a new issue, #60027:
URL: https://github.com/apache/spark/issues/60027
the assertDataFrameEqual function sorts rows based on the row values as
strings when checkRowOrder is False. This can lead to unexpected test failures:
When small differences occur due to floating point arithmetic, the sorting can
make it so that the wrong rows are compared (even though when the proper rows
would be compared, the comparison would work using the atol and rtol). An
example:
```
from pyspark.sql import SparkSession
from pyspark.testing import assertDataFrameEqual
def test_test(spark_session: SparkSession):
df1 = spark_session.createDataFrame(
[(0.1 + 0.1 + 0.1, "a"), (0.3, "b")],
["id", "value"]
)
df2 = spark_session.createDataFrame(
[(0.3, "a"), (0.3, "b")],
["id", "value"]
)
> assertDataFrameEqual(df1, df2)
E pyspark.errors.exceptions.base.PySparkAssertionError:
[DIFFERENT_ROWS] Results do not match: ( 100.00000 % )
E *** actual ***
E Row(id=0.3, value='b')
E - Row(id=0.30000000000000004, value='a')
E
E
E *** expected ***
E + Row(id=0.3, value='a')
E Row(id=0.3, value='b')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
