kz930 opened a new pull request, #8980: URL: https://github.com/apache/texera/pull/8980
### What changes were proposed in this PR? Filter's exported Python compared the column to the value with pandas operators. The engine's `FilterPredicate.evaluate` compares by the column's type instead, so the two disagreed on numbers stored as text, booleans, NaN and -0.0. Each comparison in the export now goes through `_texera_filter_keep`, a transcription of `FilterPredicate.evaluate`: | Column type | Comparison | |---|---| | STRING | as Java doubles when both sides parse as one, otherwise by UTF-16 text | | BOOLEAN | the lower-case text "true"/"false" | | INTEGER, DOUBLE | by `Double.compareTo`: NaN equals NaN and sorts above every number, -0.0 sorts below 0.0 | | LONG | as integers, rejecting what `Long.valueOf` rejects | | TIMESTAMP | by whole milliseconds | The column type comes from the input schema. Without one, it is inferred from the pandas dtype. `is null` and `is not null` are unchanged. One approximation remains. The engine reads a TIMESTAMP value with DateParserUtils, which accepts more formats than pandas. A value pandas cannot read raises an error, so the export never gives a silently wrong answer. ### Any related issues, documentation, discussions? Closes #8977 ### How was this PR tested? A new test in `SpecializedFilterOpDescSpec` runs 26 predicates across the six column types through `SpecializedFilterOpExec` and through the export, and compares the kept rows by input position. It fails on main and passes with this change. All 58 Filter tests pass, as do Filter's verify run and `TranslatedPlanRunSpec` on the integration branch. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
