TianJialong opened a new issue, #13471:
URL: https://github.com/apache/gluten/issues/13471
### Description
**Motivation**
TO_DATE, TO_TIMESTAMP and TO_TIMESTAMP_LTZ are the temporal string/number
conversion expressions in Flink SQL. Today gluten-flink does not map them, so
any query using them fails function offload (Function not supported) and stays
on the native path.
**Proposal**
Offload them to the velox flinksql function set:
- **velox side (companion PR)**: implementations under
`velox/functions/flinksql/` in `DateTimeFunctions.{h,cpp}` (impl library
`velox_functions_flink_impl`), registrations in
`registration/TemporalFunctionsRegistration.cpp`:
- `to_date(string)`: line-by-line port of Flink's
`DateTimeUtils.parseDate` — parse failure yields NULL, a year number out of int
range fails the job. Full fidelity.
- `to_date(string, format)`: thin wrapper over the shared `functions/lib`
DateTimeFormatter lenient SimpleDateFormat-pattern engine
(`buildSimpleDateTimeFormatter(..., lenient=true)`). Known deviations from
Flink's JDK engine (parse failure NULL instead of a garbage date, no lenient
rollover, proleptic calendar, defaulted year 2000, fixed two-digit-year pivot)
are marked `TODO` and pinned by unit tests; an illegal format fails the job,
mirroring Flink.
- `to_timestamp(string)`: hand-written mirror of Flink's java.time SMART
formatter — optional-section ladders with back-off, strict-width years
(EXCEEDS_PAD), day-of-month clamping to the month end, `24:00:00` rolling to
the next day. Every parse failure is a NULL row; the function has no
job-failure path, matching Flink's fallible codegen.
- `to_timestamp(string, format)`: same engine wrapper, keeping millisecond
precision. A formatter build failure is a NULL row, not a job failure (Flink
try-catches `ofPattern` per row) — deliberately the opposite disposition from
`to_date`.
- `to_timestamp_ltz(numeric, precision)`: epoch seconds (`p=0`) /
milliseconds (`p=3`) against Flink's epoch domain, out-of-range values NULL;
any other precision fails the job with Flink's `TableException` text. Java
`(long)` cast semantics for doubles (NaN → 0, saturation), toward-zero
truncation for decimals.
- **planner side (this repo)**: plain name mappings for TO_DATE /
TO_TIMESTAMP / TO_TIMESTAMP_LTZ in `RexCallConverterFactory`;
`RexNodeConverter` gains the DATE literal channel (epoch-day `IntegerValue`)
and the TIMESTAMP_WITH_LOCAL_TIME_ZONE literal channel (shared with TIMESTAMP,
epoch millis), so planner-folded literals reach velox. Identity casts
(`isSameTypeIgnoringNullability`, type equality ignoring nullability only) pass
the operand through unchanged — velox rejects same-type casts, so without this
every folded literal dies at conversion.
- **Semantics to preserve**: NULL propagation on all forms; one-argument
`to_date` fidelity; the documented two-argument deviation surface;
`to_timestamp`'s all-failures-NULL channel vs `to_date`'s fail-the-job channel;
`to_timestamp_ltz` precision 0/3 with job failure otherwise; session-timezone
independence of the LTZ epoch semantics.
**Gluten version**: main branch (gluten-flink)
### Gluten version
main branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
