peterxcli opened a new issue, #5674:
URL: https://github.com/apache/datafusion-comet/issues/5674
### Describe the bug
Comet's string→timestamp parser is a fixed list of anchored regexes
(`native/spark-expr/src/conversion_funcs/string.rs:1646-1660`) plus an
offset-suffix fallback (`:1471-1475`, `:1567-1635`). Spark's
`SparkDateTimeUtils.parseTimestampString` is a segment scanner with per-segment
digit rules (`isValidDigits`). They disagree in four ways, all on the
default-on path, for both TIMESTAMP and TIMESTAMP_NTZ (LEGACY returns NULL,
ANSI raises `CAST_INVALID_INPUT`):
1. Month/day/hour/minute/second must be exactly 2 digits in Comet; Spark
accepts 1–2 digits: `'2020-1-1'`, `'2020-01-01 12:34:5'` → Spark valid, Comet
NULL / error.
2. Empty fraction: `'2020-01-01 12:34:56.'` → Spark valid, Comet NULL
(`RE_MICROSECOND` requires `\.\d+`).
3. Zone suffix on a date-only string: `'2020-10-01Z'`, `'2020-01-01+05:30'`,
`'2020-10-01 UTC'` → Spark NULL (a zone is only allowed after a time segment),
Comet accepts them (midnight in that zone).
4. 7-digit years: Spark's timestamp `maxDigitsYear` is 6 (only
`stringToDate` allows 7); `'0002020-01-01 00:00:00'` → Spark NULL, Comet
`2020-01-01 00:00:00`.
### Steps to reproduce
```sql
CREATE TABLE t USING parquet AS SELECT * FROM VALUES
('2020-1-1'), ('2020-01-01 12:34:5'), ('2020-01-01 12:34:56.'),
('2020-10-01Z'), ('0002020-01-01 00:00:00') AS t(s);
SELECT s, CAST(s AS TIMESTAMP) FROM t;
```
| input | Spark (UTC) | Comet |
|---|---|---|
| `2020-1-1` | 2020-01-01 00:00:00 | NULL |
| `2020-01-01 12:34:5` | 2020-01-01 12:34:05 | NULL |
| `2020-01-01 12:34:56.` | 2020-01-01 12:34:56 | NULL |
| `2020-10-01Z` | NULL | 2020-10-01 00:00:00 |
| `0002020-01-01 00:00:00` | NULL | 2020-01-01 00:00:00 |
With `spark.sql.ansi.enabled=true` the first three fail the whole query in
Comet.
### Expected behavior
The same accept/reject set as Spark.
### Proposed solution
Port Spark's byte-scanning segment machine (segments 0–8 with per-segment
digit-count validation; a zone suffix only after a time segment) for both the
TZ and NTZ parsers. At minimum: relax month/day/hour/minute/second to
`\d{1,2}`, allow an empty fraction, cap timestamp years at 6 digits, and only
attempt suffix extraction when the remainder ends after a time segment. Add
these strings to the cast suite — the fuzz alphabet `0123456789/:T` contains
neither `-` nor `.`, so it cannot generate them.
### Additional context
Open PR #5130 re-implements the same 14 shapes as a byte classifier with
bit-identical output, so it preserves these divergences. #5165 tracks sibling
trimming divergences in the same parser; #3776 asks to port
`DateTimeUtilsSuite`, which would surface some of these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
