Subodh-17 opened a new pull request, #119:
URL: https://github.com/apache/datafusion-spark/pull/119
## Summary
Fixes premature integer overflow in Spark's `make_dt_interval` expression
(#109).
### Context & Defect
`make_dt_interval` returns an Arrow `Duration(Microsecond)` backed by a
64-bit signed integer (`i64`), supporting durations up to $\approx \pm 106.75$
million days. However, intermediate conversions previously accumulated hours
and minutes using 32-bit signed integers (`i32`). Consequently, any interval
with `days >= 1,491,309` (or `days <= -1,491,309`) overflowed `i32` during
minute calculation ($1,491,309 \times 24 \times 60 = 2,147,484,960 >
\text{i32::MAX}$), prematurely returning `NULL` for valid day-time intervals
that Apache Spark evaluates successfully.
### Key Changes
1. **Widening Intermediate Accumulation to `i64`:** Promoted intermediate
conversion constants `HOURS_PER_DAY` and `MINS_PER_HOUR` to `i64` and widened
`day`, `hour`, and `min` to `i64` before multiplication and addition.
2. **Defensive Arithmetic:** All accumulation steps use checked arithmetic
(`checked_mul`, `checked_add`, `checked_sub`), safely returning `None` (`NULL`)
on genuine overflow.
3. **Float Seconds Range Guard:** Added float guard `const MAX_SECS_ABS: f64
= (1i64 << 53) as f64;` rejecting non-finite floats and $|sec| \ge 2^{53}$.
This prevents float-to-int cast saturation in `sec.trunc() as i64` without
rejecting any representable interval (guard margin is $>44\times$ beyond the
theoretical maximum seconds that could ever produce a representable duration
under extreme opposite-sign cancellation).
4. **Elimination of `.abs()`:** Replaced `frac_us.abs()` with safe signed
comparisons (`frac_us >= MICROS_PER_SEC` and `frac_us <= -MICROS_PER_SEC`) and
checked carry operations (`checked_add(1)?` / `checked_sub(1)?`), eliminating
debug panic risks on `i64::MIN`.
5. **Preserved True Overflow Contract (#20):** Genuine overflows beyond
64-bit duration limits continue to return `NULL`, preserving the distinct
architectural boundary tracked under #20 (error-raising on true overflow).
### Spark Parity & Compatibility
- Verified against live Apache Spark 4.3.0 execution (with Py4J 0.10.9.9 and
ANSI enabled). Spark returns valid day-time intervals up to 106,751,991 days,
matching this implementation.
- Note: The original issue description references PySpark 4.2.0; behavior is
identical across Spark 4.x releases.
### Verification & Tests
- **Focused Unit Tests:** 23/23 passed in
`crates/spark/src/function/datetime/make_dt_interval.rs` (testing
positive/negative boundaries, exact `i64::MAX`/`i64::MIN`, $\pm 1\ \mu\text{s}$
overflows, fractional carry rounding, and extreme floats).
- **Focused SQLLogicTests:** 1/1 passed in
`crates/sqllogictest/test_files/datetime/make_dt_interval.slt` (updating
1,500,000 days from NULL to valid interval and adding positive/negative
boundary assertions).
- **Full Crate Tests:** 345/345 passed (`cargo test -p datafusion-spark`).
- **Full SQLLogicTest Suite:** 245/245 test files passed (`cargo test -p
datafusion-spark-sqllogictest`).
- **Linters:** `cargo fmt --all -- --check` clean; `cargo clippy -p
datafusion-spark --all-targets -- -D warnings` clean (0 warnings); `git diff
--check` clean.
Fixes #109.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]