andygrove opened a new issue, #235:
URL: https://github.com/apache/datafusion-spark/issues/235
Part of #175 and apache/datafusion-comet#7300.
### What the function does
`timestamp_millis(milliseconds)` returns the `TIMESTAMP` that is
`milliseconds` ms after `1970-01-01 00:00:00 UTC`.
The argument must be an integral type (`TINYINT`, `SMALLINT`, `INT`,
`BIGINT`); the expression uses `ExpectsInputTypes`
with `IntegralType`, so `DECIMAL`, `FLOAT`, `DOUBLE` and `STRING` are
analysis errors rather than being cast. The value
is multiplied by 1000 with `Math.multiplyExact` into microseconds.
The result is always `TIMESTAMP` (LTZ), independent of
`spark.sql.timestampType`; only its display depends on the
session time zone. NULL input gives NULL; nullability follows the input.
Since 3.1.0; the inverse is `unix_millis`
(already in this crate).
### Examples
Session time zone `America/Los_Angeles` (as in Spark's docs and golden
files):
```sql
SELECT timestamp_millis(1230219000123);
-- 2008-12-25 07:30:00.123
SELECT timestamp_millis(-1230219000123);
-- 1931-01-07 00:29:59.877
SELECT timestamp_millis(null);
-- NULL
SELECT timestamp_millis(92233720368547758);
-- java.lang.ArithmeticException: long overflow
SELECT timestamp_millis(-92233720368547758);
-- java.lang.ArithmeticException: long overflow
```
### Edge cases to test
- Overflow: `|millis| > i64::MAX / 1000` (9223372036854775) raises
`ArithmeticException` ("long overflow") with ANSI
on and off (identical in Spark's ANSI and non-ANSI golden files). Test the
boundary values on both sides.
- Negative values: `-1230219000123` is `...:59.877`, i.e. no rounding toward
zero; `-1` is one millisecond before the
epoch.
- `TINYINT` / `SMALLINT` / `INT` / `BIGINT` inputs; `DECIMAL`, `DOUBLE`,
`STRING` fail with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`.
- Values far outside years 0001..9999 are valid (any `i64` microsecond value
is a valid Spark timestamp).
- The stored value must not depend on `spark.sql.session.timeZone`; display
across a DST change does.
- Literal vs column input; dictionary-encoded input.
### Differences across Spark versions
None found (3.4 through 4.2 have the same evaluation). ANSI mode does not
affect this function.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala`,
`MillisToTimestamp` (extends `IntegralToTimestampBase`, `upScaleFactor =
MICROS_PER_MILLIS`).
- DataFusion core `to_timestamp_millis` differs: it returns
`Timestamp(Millisecond, tz)`, accepts floats, decimals and
strings, and never fails on overflow (the value is stored as
milliseconds). Spark's microsecond range is 1000x
smaller, so values beyond `i64::MAX / 1000` must error. Write a small
kernel (checked multiply) returning
`Timestamp(Microsecond, Some(tz))`, with the zone from
`datafusion.execution.time_zone`.
- Share code with `timestamp_seconds` / `timestamp_micros` (separate issues)
in `crates/spark/src/function/datetime/`,
next to `unix.rs`.
- Tests: `crates/sqllogictest/test_files/datetime/timestamp_millis.slt`
exists but every query is commented out
(ported placeholders). Uncomment and expand it. Document with
`#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]