andygrove commented on issue #5149: URL: https://github.com/apache/datafusion-comet/issues/5149#issuecomment-5908802425
Status as of 2026-09-30: **Done** - #5150: boolean, byte/short/int/long, float/double and decimal now trim exactly Spark's byte sets, through the two helpers in `native/spark-expr/src/conversion_funcs/trim.rs`. It also added a Rust parity test over the full codepoint matrix (boolean, integral, date, float/double, decimal), Spark-oracle parity tests in `CometNativeCastSuite` (boolean, integral, float/double, decimal) and the `cast_string_trim.sql` fixture. - #5364: `to_time` / `try_to_time`. - Date already matched Spark and is covered by the tests above. **Remaining** 1. `CAST(string AS timestamp)` and `CAST(string AS timestamp_ntz)` still trim Unicode whitespace. Both parsers use `str::trim`, the Spark 4 check for padding before a leading `T` uses `trim_start()`, and the `cast_utf8_to_timestamp!` macro calls `trim_end()` on every value before parsing, so trailing non-ASCII whitespace would still be stripped even once the parsers are fixed. `test_cast_string_to_timestamp_unicode_whitespace_divergence` pins the current behavior, and the compatibility guide documents it. 2. Found while checking the above: an empty or whitespace-only string cast to timestamp or timestamp_ntz returns NULL under ANSI in Comet, where Spark raises `CAST_INVALID_INPUT` (`parseTimestampString` returns no segments and `stringToTimestampAnsi` throws on `None`). Confirmed against the Spark 4.1.1 jars; 3.4 and 3.5 take the same path. The Spark-oracle ANSI tests don't catch it because they only check that both sides throw somewhere in the batch. 3. The separate issue for `to_csv`'s `ignoreLeadingWhiteSpace` / `ignoreTrailingWhiteSpace` trimming has not been filed yet. **Next** - I'm bringing #5130 up to date with main, since #5172 is stacked on it. - I'll then open a PR for items 1 and 2: both timestamp parsers move onto `trim_all`, and the parity tests extend to timestamp and timestamp_ntz. That should close out this epic. It overlaps the trimming half of #5165 / #5172 (cc @peterxcli); the leading `+` ANSI fix there is unaffected. cc @coderfender since this is assigned to you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
