AnissaLiu opened a new issue, #13485:
URL: https://github.com/apache/gluten/issues/13485
### Description
## Description
### Motivation
FROM_UNIXTIME, UNIX_TIMESTAMP and TIMESTAMPADD are the three most commonly
used temporal scalar expressions in Flink SQL — epoch-second ↔ datetime-string
conversion and calendar-aware timestamp addition. Today gluten-flink does not
map any of them:
- `FROM_UNIXTIME(...)` / `UNIX_TIMESTAMP(...)` fail function offload with
`Function not supported: FROM_UNIXTIME` / `Function not supported:
UNIX_TIMESTAMP`, forcing the whole Calc to stay on the native path.
- `TIMESTAMPADD(...)` is rewritten by Calcite into interval arithmetic (`ts
+ INTERVAL` / `ts + (interval × n)`), which then fails at the converter layer
with `No suitable converter found for: +` — the `+`/`-`/`*` converters have no
(datetime, interval) branch. Additionally, `LogicalTypeConverter` maps
`DayTimeIntervalType` to `BigIntType` (wrong target type for interval operands)
and has **no entry** for `YearMonthIntervalType`, causing `Unsupported logical
type: INTERVAL MONTH` at the type-mapping layer before the converter is even
reached.
### Proposal
Offload all three expressions to the velox flinksql function set:
**velox side** (`velox/functions/flinksql/`):
- `DateTimeFunctions.h`: FROM_UNIXTIME (1-arg / 2-arg) and UNIX_TIMESTAMP
(0/1/2-arg) with Flink semantics:
- `unix_timestamp` returns `Long.MIN_VALUE` (not NULL) on parse failure —
Flink's sentinel from `DateTimeUtils.internalParseTimestampMillis`
- `from_unixtime` with an invalid format fails the job (deferred to the
first evaluated row so all-null inputs still return NULL)
- Zero-arg `unix_timestamp` is non-deterministic (evaluated per row, not
constant-folded)
- Missing-year patterns with month/day tokens default to 1970 (Java
`SimpleDateFormat` default), not the Joda 2000 fallback
- Milliseconds truncate toward zero via total millis
- `IntervalArithmetic.h`: `plus`/`minus`/`multiply` for the TIMESTAMPADD
rewrite shapes (10 registrations):
- `Timestamp ± IntervalDayTime`: millis arithmetic preserving
sub-millisecond precision
- `Timestamp ± IntervalYearMonth`: **timezone-naive** wall-calendar add
with end-of-month clamping (Flink semantics — prestosql's existing tz-aware
version diverges by 1 hour across DST boundaries)
- `Date ± IntervalDayTime / IntervalYearMonth`: keeps the DATE type with
day-truncation and clamping
- `IntervalDayTime × BIGINT` / `IntervalYearMonth × BIGINT`: for the
column-amount rewrite `ts + (interval × n)`
**planner side** (gluten-flink):
- `FromUnixtimeRexCallConverter`: INTEGER-family first operand cast to
BIGINT (mirroring Flink codegen), optional STRING format operand
- `UnixTimestampRexCallConverter`: 0/1/2 string operands
- `IntervalArithmeticRexCallConverter`: plus/minus for (timestamp/date,
interval) shapes; interval-on-left plus is flipped to the canonical (datetime,
interval) order
- `IntervalMultiplyRexCallConverter`: interval × numeric, normalized to
(interval, BIGINT)
- `LogicalTypeConverter` fixes: `DayTimeIntervalType → IntervalDayTimeType`
(was BigIntType), add `YearMonthIntervalType → IntervalYearMonthType`
- `RexNodeConverter.toVariant`: interval literal cases (day-time family →
millis BigIntValue, year-month family → months IntegerValue)
### Semantics to preserve
| Expression | Flink behavior | Divergence from existing velox registrations
|
|---|---|---|
| `UNIX_TIMESTAMP('garbage')` | `Long.MIN_VALUE` | sparksql returns NULL |
| `FROM_UNIXTIME(0, 'qq')` | job fails | sparksql returns NULL |
| `UNIX_TIMESTAMP()` | per-row evaluation (non-deterministic) | sparksql is
constant-folded once per query |
| `UNIX_TIMESTAMP('12-31', 'MM-dd')` | year 1970 | velox Joda-style defaults
to 2000 |
| `TIMESTAMPADD(MONTH, 1, ts)` | timezone-naive wall calendar + end-of-month
clamping (Jan-31 → Feb-28) | prestosql uses session-timezone round-trip
(tz-aware) |
| `TIMESTAMPADD(DAY, 1, date_col)` | returns DATE (day-truncation) |
prestosql has no (Date, Interval) signature |
| `TIMESTAMPADD(HOUR, n_col, ts)` | rewritten to `ts + (interval × n)` —
requires interval multiplication | no existing interval × numeric registration |
### Gluten version
main branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
