andygrove opened a new pull request, #317:
URL: https://github.com/apache/datafusion-spark/pull/317

   ## Which issue does this PR close?
   
   Part of #88: the "Rework the `spark_cast` UDF" item. Builds on #304, which 
builds on #89. Until those merge, this PR also shows their commits. Only the 
last commit, `feat(spark): parse Spark type names in spark_cast and call the 
cast kernel`, is new here.
   
   ## Rationale for this change
   
   `spark_cast(expr, 'timestamp')` only supported a `timestamp` target and 
integer or float input, with its own conversion code. To test the 
Spark-compatible cast kernel from SQL and SLT as each family is ported, the 
function needs to accept any Spark type name and call the kernel.
   
   ## What changes are included in this PR?
   
   - **Target type parser** (`cast/data_type.rs`): parses the second argument 
with Spark's `dataType` grammar from `SqlBaseParser.g4`, case-insensitively.
     - **Primitive types:** `boolean`, `tinyint`/`byte`, `smallint`/`short`, 
`int`/`integer`, `bigint`/`long`, `float`/`real`, `double`, `date`, `binary` 
and `string`.
     - **Decimals:** `decimal`/`dec`/`numeric`, with an optional precision and 
scale. A bare `decimal` is `decimal(10, 0)`, and precision and scale are 
validated.
     - **Character types:** `char(n)` and `varchar(n)` become `STRING`, which 
is what Spark casts them to by default.
     - **Timestamps:**
       - `timestamp`, `timestamp_ltz` and `timestamp with local time zone` give 
a `TIMESTAMP` tagged with the session time zone. That matches Spark's default 
`spark.sql.timestampType`.
       - `timestamp_ntz` and `timestamp without time zone` give a 
`TIMESTAMP_NTZ`.
     - **Complex types:** `array<…>`, `map<…, …>` (with Spark's `ArrowUtils` 
field names `entries`, `key` and `value`), and `struct<…>`. Struct fields can 
be backquoted, `NOT NULL` or have a `COMMENT`, and duplicate field names are 
allowed, as in Spark.
     - **Unsupported types:** intervals, `TIME`, `VARIANT`, `VOID`, geospatial 
types and collated strings get an "unsupported" error. Malformed type names get 
a parse error that names the problem.
   - **`spark_cast` UDF:**
     - accepts any input type
     - requires the target type to be a string literal
     - computes its return field from the parsed type
     - calls the kernel with `EvalMode::from_config` and the session time zone 
as a `SparkTimeZone`
   
     The old conversion code in the UDF is removed.
   - **Integer and float to `TIMESTAMP`:** ported into the kernel 
(`cast/numeric.rs`). Spark has no cast from a number to `TIMESTAMP_NTZ`, so 
that cast is rejected.
   
     Float input follows Spark 4.2.0's `doubleToTimestamp` and 
`doubleToTimestampAnsi`:
   
     | Mode | NaN or infinite | Out of range |
     |---|---|---|
     | Legacy | NULL | saturates, like Scala's `toLong` |
     | ANSI | `CAST_INVALID_INPUT` | `CAST_OVERFLOW`, such as `1.0E25D` from 
`DOUBLE` to `BIGINT` |
     | TRY | NULL | NULL |
   
     The previous UDF already saturated in Legacy mode. Comet returns NULL 
there, which differs from Spark.
   
   ## Are these changes tested?
   
   Yes.
   
   - **Unit tests:**
     - the type parser: every primitive and complex type, invalid names and 
unsupported types
     - integer and float to timestamp in each eval mode, including saturation, 
a NaN under a null, `FLOAT` widening and the error messages
     - the UDF: return fields, a non-literal target type, array and scalar 
input, and reading ANSI mode from the config
   - **New SLT file** `conversion/cast_target_type.slt`: return types from 
`arrow_typeof`, the session time zone on `timestamp`, casts that already work 
through Arrow, invalid and unsupported targets, and casts that are not 
implemented yet.
   - **Existing SLT files:** `cast_int_to_timestamp.slt` and 
`cast_float_to_timestamp.slt` still pass, with these changes:
     - The old "unsupported target" test used `spark_cast(100, 'string')`, 
which now works, so it now uses `interval` instead.
     - It also now checks that a number can't be cast to `timestamp_ntz`.
     - The ANSI errors are checked against Spark's messages.
     - It has a Legacy saturation test that casts the result back to `bigint`. 
The old file said this couldn't be shown in SLT.
   
   ## Are there any user-facing changes?
   
   Yes. `spark_cast` accepts any Spark type name as its target and any input 
type.
   
   - **Newly working:** casts that are already Spark-compatible through Arrow 
now succeed. One example is `spark_cast(42, 'string')`.
   - **Errors for casts not ported yet:** those return an explicit "not 
supported" or "not implemented yet" error rather than a result that differs 
from Spark's. Integer narrowing is one example. The unsuffixed literal `1` is a 
`BIGINT`, so even `spark_cast(1, 'int')` needs that port.
   - **Error message format:** in ANSI mode, errors for NaN and out-of-range 
input now use Spark's `[CAST_INVALID_INPUT]` and `[CAST_OVERFLOW]` messages.
   
   _This PR was written by an LLM (Claude) from the Comet and Spark sources, 
and reviewed before opening._
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to