ChuckLin2025 opened a new pull request, #60112:
URL: https://github.com/apache/spark/pull/60112
### What changes were proposed in this pull request?
`RowComparisonUtils.compare`, which backs `QueryTest.compare` and
`checkDataset`, has no case
for `Set`. Sets fall into the generic `Iterable` case, which zips the two
sides and compares
them position by position in iteration order.
This PR adds a `Set` case that matches each element of one set with a
distinct element of the
other, using `compare` for the elements. Element comparison is unchanged:
arrays are still
compared by content, `NaN` equals `NaN`, and `0.0` differs from `-0.0`.
Sequences are still
compared in order.
### Why are the changes needed?
Scala keeps the insertion order of immutable sets with up to four elements,
and a `Set`
decoded from an array keeps the array's element order. So a correct result
can fail
`checkDataset`:
```scala
checkDataset(Seq(Seq(3, 1, 2)).toDF("c").as[Set[Int]], Set(1, 2, 3))
// Decoded objects do not match expected objects:
// expected: ArraySeq(Set(1, 2, 3))
// actual: ArraySeq(Set(3, 1, 2))
```
Tests that compare sets this way only pass when the result happens to come
back in the
expected order, for example when checking functions with unspecified element
order such as
`collect_set` or `collect_union`.
### Does this PR introduce _any_ user-facing change?
No. Test utilities only.
### How was this patch tested?
- New `RowComparisonUtilsSuite`: sets in different orders (with up to four
and with more
elements, nested in `Seq` and `Row`, mixed set types, null elements), sets
with different
elements, element comparison through `compare` (arrays, `NaN`, `-0.0`,
nested sets), and
sequences still compared in order.
- A new case in `DatasetPrimitiveSuite."arbitrary sets"` decoding `array(3,
1, 2)` as a `Set`.
- Without the fix, both new `RowComparisonUtilsSuite` order tests and the new
`DatasetPrimitiveSuite` case fail. With it, `RowComparisonUtilsSuite`,
`DatasetPrimitiveSuite`, `LateralColumnAliasSuite`, `MathFunctionsSuite`,
and the
`DataFrameAggregateSuite` collect tests pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
