andygrove opened a new pull request, #327:
URL: https://github.com/apache/datafusion-spark/pull/327
Closes #151.
## Summary
Adds a Spark-compatible `dayofweek(date)`, which returns an INT from 1
(Sunday) to 7 (Saturday). It is ported from Comet's
`datetime_funcs/day_of_week.rs`.
- New `crates/spark/src/function/datetime/dayofweek.rs`. The result comes
straight from the `Date32` epoch day: `((days + 4) mod 7) + 1`, computed in
`i64`. This matches Spark's
`LocalDate.ofEpochDay(days).getDayOfWeek.plus(1).getValue`, which is defined
for every `Int`.
- `invoke_with_args` handles scalar and array inputs, NULL and untyped NULL,
and dictionary-encoded `Date32`. For a dictionary, it maps the values, rewraps
the keys, and unpacks the result to plain `Int32`.
- The signature is the same `date_signature()` that `dayname` and
`monthname` use. Strings and timestamps are implicitly cast to DATE, as in
Spark.
- Documented with `#[user_doc]` and registered in `datetime/mod.rs`.
- `weekday` now uses the same epoch-day helper instead of arrow's
`date_part(DayOfWeekMonday0)`. arrow builds a chrono date for each row, so
dates outside chrono's range (such as `date_from_unix_date(±2^31)`) came back
NULL. Spark returns 1 and 4 for those dates, and so does `weekday` now. The
`weekday` signature is unchanged.
## Differences from Comet's version
- Comet uses `Signature::any(1)` and expects the planner to hand it a
`Date32`. Here the function has a coercible date signature, so strings and
timestamps work from SQL. Comet's name `spark_dayofweek` becomes `dayofweek`.
- A scalar input returns a scalar here. Comet always expands it to an array.
- `return_field_from_args` passes the input's nullability through to the
result, the same as `weekday`.
- The kernel, the null-density switch between `unary` and `unary_opt`, and
the dictionary handling are the same as in Comet. I found no behavioral
difference between Comet and Spark 4.2.0 for `Date32` input.
Known differences from Spark, which also apply to the existing `dayname` and
`weekday` and are left as commented-out cases in the `.slt`:
- With ANSI mode off, Spark returns NULL for a malformed string.
DataFusion's cast errors regardless of ANSI mode (#88).
- Spark casts a timestamp with a time zone to a date in the session time
zone. Here it is cast in the timestamp's own time zone (#88).
## Testing
- Every expected value was checked against PySpark 4.2.0. This covers all
seven weekdays, the epoch, a leap day, 0001-01-01, 9999-12-31, the 1582
Gregorian cutover, `date_from_unix_date(-2147483648)` = 3 and `(2147483647)` =
6, NULL and untyped NULL (`int`), strings, TIMESTAMP_NTZ, TIMESTAMP in UTC and
in America/New_York, malformed strings with ANSI on and off, and an INT
argument (DATATYPE_MISMATCH).
- `crates/sqllogictest/scripts/pyspark_slt_runner.py` on `dayofweek.slt` and
`weekday.slt`: every `dayofweek` record passes on Spark 4.2.0. Skipped records
are the `arrow_typeof` checks and the dictionary case, marked `skipif spark`.
The one `weekday` mismatch, `weekday('not-a-date')`, was already there (strings
are not cast for `weekday`, #41).
- Unit tests ported from Comet: the week around the epoch, nulls at both
densities, the full `i32` range, agreement with `weekday` over a 400-year
cycle, dictionary input, non-date input rejected, and scalar input. A
nullability test was added.
- Passing: `cargo fmt --all`, `cargo clippy --all-targets --all-features --
-D warnings`, `cargo test -p datafusion-spark` and `cargo test -p
datafusion-spark-sqllogictest` (249 of 249 files).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]