kz930 opened a new issue, #8974: URL: https://github.com/apache/texera/issues/8974
### What happened? Distinct's exported Python calls `drop_duplicates`, which treats two NaN values as equal and 0.0 as equal to -0.0. The engine's `DistinctOpExec` keeps tuples in a `LinkedHashSet`, where NaN never equals NaN and 0.0 and -0.0 hash apart, so it keeps all of them. The export therefore emits fewer rows than the workflow it was exported from. ### How to reproduce? Run Distinct on one DOUBLE column holding NaN, NaN, 0.0, -0.0, 1.0, 1.0. - Engine (`DistinctOpExec`): NaN, NaN, 0.0, -0.0, 1.0 (5 rows) - Exported Python (`drop_duplicates`): NaN, 0.0, 1.0 (3 rows) ### Version/Branch 1.4.0-incubating-SNAPSHOT (main) ### Commit Hash (Optional) 6a29c59c6 ### What browsers are you seeing the problem on? _No response_ ### Relevant log output _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
