kz930 opened a new pull request, #8975: URL: https://github.com/apache/texera/pull/8975
### What changes were proposed in this PR? Distinct's exported Python called `drop_duplicates`, which counts two NaN values as equal and 0.0 as equal to -0.0. The engine's `DistinctOpExec` keeps rows in a `LinkedHashSet` that compares them with Scala's `==`, under which NaN equals nothing, and hashes each double by its bits, so 0.0 and -0.0 land apart. The export therefore emitted fewer rows than the run it came from. The export now selects rows with `_texera_set_select`, the helper the set operators use: one key per row, equal exactly where the engine's rows are, keeping the first row of each key in input order. ### Any related issues, documentation, discussions? Closes #8974. Depends on #8506, which adds `StandaloneHelpers.SetRowKeys`; until it merges this branch does not compile. ### How was this PR tested? `DistinctOpDescSpec` gains a test that runs `DistinctOpExec` and the exported Python over 1.0, NaN, 0.0, -0.0, null, 1.0, NaN, null, -0.0 and compares the rows in order. The engine keeps 6. The old export kept 4 and fails the test. It replaces the test that pinned the old `drop_duplicates` line. With #8506 applied, all 11 Distinct tests pass, and Distinct's verification run still matches. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
