mbutrovich commented on issue #26673:
URL: https://github.com/apache/datafusion/issues/26673#issuecomment-6371214584
Thanks @alamb for filing this. The cutoff is 2.147483647 seconds, which is
`i32::MAX` nanoseconds. That matches your cases (2.10 keeps its fraction, 2.50
loses it):
```sql
SELECT d, EXTRACT(EPOCH FROM d) AS epoch
FROM (VALUES
(arrow_cast(2147483647, 'Duration(ns)')),
(arrow_cast(2147483648, 'Duration(ns)'))
) AS v(d);
```
```
| 0 days 0 hours 0 mins 2.147483647 secs | 2.147483647 |
| 0 days 0 hours 0 mins 2.147483648 secs | 2.0 |
```
As I read it,
[`epoch`](https://github.com/apache/datafusion/blob/c76ed5396ad3f4a4dc9d9ceeddc419a146b871f9/datafusion/functions/src/datetime/date_part.rs#L592)
sends `Duration` to
[`seconds`](https://github.com/apache/datafusion/blob/c76ed5396ad3f4a4dc9d9ceeddc419a146b871f9/datafusion/functions/src/datetime/date_part.rs#L512-L524),
which reads `date_part(array, Nanosecond)` as an `Int32` array. For a
`Duration(ns)`, arrow-rs returns the total nanoseconds with
[`try_into().ok()`](https://github.com/apache/arrow-rs/blob/df4dc3ede59bd43b5f71847231212381d068bb72/arrow-arith/src/temporal.rs#L774),
so the value is NULL above `i32::MAX`. `seconds` treats a NULL sub-second part
as zero. I have not stepped through it in a debugger. This looks like the same
`Int32` overflow on `Duration` as #14738.
`EXTRACT(EPOCH FROM INTERVAL '4.322 seconds')` returns `4.322`, so only the
`Duration` path is affected. The same happens for `Duration(µs)` and
`Duration(ms)` inputs.
This is the cause of the remaining SF1 wrong results in the benchmark: q124,
q127, q128, q197 and q405 all average or take a percentile of
`CAST(EXTRACT(EPOCH FROM (ts1 - ts2)) AS NUMERIC(18,6))`. With the expression
replaced by `(CAST(ts1 AS BIGINT) - CAST(ts2 AS BIGINT)) / 1000000.0`, all five
match the reference results. Together with the `year_month` dialect change and
#26671, DataFusion 55.0.0 at SF1 has no wrong results left (331 equal, 0
different, 78 queries that return errors).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
