dongjoon-hyun opened a new pull request, #555:
URL: https://github.com/apache/spark-connect-swift/pull/555
### What changes were proposed in this pull request?
This PR aims to fix the vendored Arrow `Date32` and `Date64` code to handle
Arrow date values as signed integers.
1. `Date32Array` loads `Int32` instead of `UInt32`, and multiplies by
`86400` in `TimeInterval`. Loading `Int32` alone is not enough because `Int32 *
86400` overflows for dates after `2038-01-19`.
2. `Date64Array` loads `Int64` instead of `UInt64`.
3. `Date32BufferBuilder` uses `.rounded(.down)` instead of truncating toward
zero, so pre-epoch values are floored to their day just like post-epoch values.
The same bugs exist in upstream `apache/arrow-swift`; they will be reported
there separately.
### Why are the changes needed?
Arrow `Date32` and `Date64` are signed day and millisecond counts since the
UNIX epoch, and Spark sends `DATE` columns as `Date32`. Previously:
- `Date32Array` read a day such as `-1` (`1969-12-31`) as `4294967295`, and
`UInt32 * 86400` trapped with `Swift runtime failure: arithmetic overflow`.
Only `1970-01-01` through `2106-02-07` could be read; any other `DATE` crashed
the client process in both debug and release builds, without any error message.
This affects `DataFrame.collect()`, `DataFrame.collect(as:)`, and
`ARRAY`/`STRUCT` columns containing `DATE`.
- `Date64Array` read negative milliseconds as huge positive values.
- `createDataFrame` stored a pre-epoch `Date` that is not at UTC midnight
one day later. For example, `1969-12-31T12:00:00Z` was stored as `1970-01-01`.
Reproducer (crashes before this fix, passes after):
```swift
let spark = try await SparkSession.builder.getOrCreate()
let rows = try await spark.sql("SELECT DATE'1969-12-31'").collect()
```
### Does this PR introduce _any_ user-facing change?
No behavior change for dates from `1970-01-01` to `2106-02-07`. This is a
bug fix: `DATE` values outside that range can now be collected instead of
crashing the client, and `createDataFrame` stores pre-epoch dates on the
correct day.
### How was this patch tested?
Pass the CIs with newly added test cases.
- `DataFrameTests.collectDate`
- `CreateDataFrameTests.dateType`
- `CreateDataFrameTests.collectAsWithDateBeforeEpoch`
I also verified manually against an Apache Spark 4.2.0 Connect server that
all three new tests crash with signal 5 without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
