andygrove opened a new issue, #270:
URL: https://github.com/apache/datafusion-spark/issues/270

   Part of #178. Split out of #155, which was closed as a duplicate of #211 and 
#212.
   
   ### What the functions do
   
   Spark 4.1 added the TIME type, and the hour, minute and second functions 
accept it. datafusion-spark's `hour`, `minute`,
   `second` and `date_part` don't accept TIME yet.
   
   | Spark SQL | Spark expression (4.1) | Result |
   |---|---|---|
   | `hour(t)`, `extract(HOUR FROM t)` | `HoursOfTime` | INT |
   | `minute(t)`, `extract(MINUTE FROM t)` | `MinutesOfTime` | INT |
   | `second(t)` | `SecondsOfTime` | INT (whole seconds) |
   | `extract(SECOND FROM t)`, `date_part('SECOND', t)` | 
`SecondsOfTimeWithFraction` | DECIMAL(8,6) |
   
   - `hour`, `minute` and `second` are `ExpressionBuilder`s that pick the TIME 
expression when the argument is a `TimeType`.
     `EXTRACT` and `date_part` go through `TimePart.parseExtractField`, which 
accepts `HOUR`/`H`/`HOURS`/`HR`/`HRS`,
     `MINUTE`/`M`/`MIN`/`MINS`/`MINUTES` and 
`SECOND`/`S`/`SEC`/`SECONDS`/`SECS`. Any other field fails analysis.
   - `SecondsOfTimeWithFraction` truncates the fraction to the precision `p` of 
`TIME(p)` (`DateTimeUtils.getSecondsOfTimeWithFraction`).
   - All of these are `RuntimeReplaceable` and become `StaticInvoke` calls into 
`DateTimeUtils`.
     The source is 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala`.
   
   ### Examples
   
   From `sql/core/src/test/resources/sql-tests/results/time.sql.out` (Spark 
4.1.1):
   
   ```sql
   SELECT extract(HOUR FROM time'23:59:00.987654');       -- 23   (int)
   SELECT extract(MINUTE FROM time'00:59:00.987654');     -- 59   (int)
   SELECT extract(SECOND FROM time'00:59:00.987654');     -- 0.987654   
(decimal(8,6))
   SELECT extract(SECOND FROM cast('09:08:01.987654' AS time(0)));  -- 1.000000
   SELECT extract(SECOND FROM cast('09:08:01.987654' AS time(3)));  -- 1.987000
   SELECT extract(SECOND FROM cast('09:08:01.987654' AS time(6)));  -- 1.987654
   SELECT second(TIME'13:25:59.999999');                  -- 59   (from the 
ExpressionDescription)
   ```
   
   ### Comet implementation
   
   Comet implements `extract(SECOND FROM time)` (`spark_seconds_of_time`). It 
takes the TIME value as
   `Time64(Nanosecond)` and the precision as a literal, and returns 
DECIMAL(8,6):
   - 
[`datetime_funcs/extract_date_part.rs`](https://github.com/apache/datafusion-comet/blob/3878424d90c23de3c6355a53b293c22822876944/native/spark-expr/src/datetime_funcs/extract_date_part.rs)
   
   ### Edge cases to test
   
   - NULL input, midnight, `23:59:59.999999`.
   - Every TIME precision from 0 to 6 for `extract(SECOND ...)`.
   - All the field aliases above, and an unsupported field such as `DAY` on a 
TIME.
   
   ### Differences across Spark versions
   
   - TIME exists only in Spark 4.1 and later. In 3.4, 3.5 and 4.0 these 
functions take only timestamps (and strings cast to timestamps).
   
   ### Implementation notes
   
   - The existing `hour`, `minute` and `second` are in 
`crates/spark/src/function/datetime/extract.rs`, and `date_part` is in
     `date_part.rs`. Add TIME handling there rather than new functions.
   - #38 (`date_part('SECOND', ts)` drops the fractional seconds) is the 
timestamp counterpart of the DECIMAL(8,6) result here.
   
   ---
   _This issue was generated by an LLM (Claude Code) from the Apache Spark 
source code, Spark's `.sql.out` golden files and
   Comet's source. Please verify the details against Spark before relying on 
them._
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to