laserninja opened a new pull request, #58789: URL: https://github.com/apache/spark/pull/58789
### What changes were proposed in this pull request? Check Row lengths before recursively comparing field values. Add tests for either side having extra fields, empty Rows, nested Rows, arrays, maps, and both row-order settings. ### Why are the changes needed? The Row comparator uses zip without a length check, so trailing fields are ignored. List-of-Row inputs are supported and do not undergo a DataFrame schema comparison, allowing unequal results to pass assertions. ```python from pyspark.sql import Row from pyspark.testing import assertDataFrameEqual assertDataFrameEqual([Row(x=1)], [Row(x=1, y=2)]) # Returns without raising ``` ### Does this PR introduce _any_ user-facing change? Yes. assertDataFrameEqual raises DIFFERENT_ROWS when corresponding Rows contain different numbers of fields, including nested Rows. ### How was this patch tested? The new regression tests failed before the fix and passed afterward on Python 3.10.11 with NumPy 2.2.6. The focused run passed 2 tests, including parameterized subtests: ```bash PYTHONPATH=python TERM=dumb python -m unittest pyspark.sql.tests.test_utils.RowComparisonTests -q PYTHON_EXECUTABLE=python ./dev/lint-python --compile --custom-pyspark-error --ruff ``` Run the commands with the Python virtual environment activated; the local run shared an environment from a sibling checkout. The tests call public Python APIs directly and require no SparkSession. Python compilation, custom-error checks, Ruff lint, and Ruff formatting passed. JVM-backed suites were not run because these fixes are confined to Python-side logic. Draft pending creation and linking of the corresponding ASF Jira issue. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
