sunchao commented on code in PR #5682:
URL: https://github.com/apache/datafusion-comet/pull/5682#discussion_r3942101881
##########
native/spark-expr/src/conversion_funcs/string.rs:
##########
@@ -1470,7 +1474,13 @@ fn timestamp_parser<T: TimeZone>(
if !has_direct_match {
if let Some((stripped, suffix_tz)) = extract_offset_suffix(value) {
- return timestamp_parser_with_tz(stripped, eval_mode, &suffix_tz);
+ let stripped = stripped.trim_end();
Review Comment:
### Correctness
[P2] Preserve the Spark 4 leading-whitespace check on the new suffix path
For a STRING column containing `\tT1:2:3 +08:00` (a leading TAB), Spark 4
returns NULL in Legacy/TRY and CAST_INVALID_INPUT in ANSI. The guard above
checks the whole trimmed value, so its anchored time-only patterns do not match
while ` +08:00` is still present. This new trim then turns the extracted
`T1:2:3 ` into `T1:2:3` and sends it directly to `timestamp_parser_with_tz`,
which produces a value without another Spark-4 check. At `0a1fc76a`, the same
prefix failed the seconds-segment gate and the input was rejected. Please
preserve/reapply the raw-leading-whitespace restriction after suffix extraction
and add this column case in all three modes. This is specific to timezone-aware
TIMESTAMP on Spark 4; maintained Spark 3.5 accepts the leading whitespace,
while NTZ rejects time-only strings. The Spark result was confirmed with 4.0.4
Cast evaluation and generated projections; the Comet result follows from the
changed source path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
