andygrove opened a new pull request, #304:
URL: https://github.com/apache/datafusion-spark/pull/304

   ## Which issue does this PR close?
   
   Part of #88. Builds on #89, which adds `EvalMode`. Until #89 merges, this PR 
also shows its commit. Only the last commit, `feat(spark): add the 
Spark-compatible cast framework`, is new here.
   
   ## Rationale for this change
   
   #88 ports Comet's Spark-compatible `CAST` 
(`native/spark-expr/src/conversion_funcs`) one source type family at a time. 
Each family needs the same kernel, dispatcher and error type to plug into, and 
these can be reviewed separately from the roughly 11k lines of conversion code. 
This PR adds that framework, following the design proposals in #88.
   
   ## What changes are included in this PR?
   
   - **`SparkError`** (`datafusion_spark::SparkError`): a typed error for 
`CAST_INVALID_INPUT` (as `CastInvalidValue` and `InvalidInputInCastToDatetime`, 
the split Comet uses to pick the JVM exception type), `CAST_OVERFLOW`, 
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` and 
`NUMERIC_OUT_OF_SUPPORTED_RANGE`. It renders the message from Spark 4.2.0's 
`error-conditions.json` and reaches callers as `DataFusionError::External`. 
`SparkError::find` gets the variant back through the `Context`, `Shared` and 
`ArrowError(ExternalError)` wrappers, so Comet can map it to a Spark exception 
without parsing the message. Spark 3.x ended the `CAST_INVALID_INPUT` and 
`CAST_OVERFLOW` messages with `If necessary set "spark.sql.ansi.enabled" to 
"false" ...`, which Spark 4.0 dropped. Comet's messages still have it.
   - **Kernel** (`datafusion_spark::function::conversion::cast`): 
`SparkCastOptions { eval_mode, timezone }` and `spark_cast(ColumnarValue, 
&DataType, &SparkCastOptions)`. `SparkCastOptions` is `#[non_exhaustive]`, so 
the Spark-version and formatting options can be added as the casts that use 
them are ported. The `cast_array` dispatcher handles:
     - dictionary targets, which are rejected because Spark cannot name them
     - NULL input
     - dictionary-encoded string and binary input
     - `TIMESTAMP` to `TIMESTAMP_NTZ` and back, in the session time zone
   - **Arrow fallthrough**: `is_datafusion_spark_compatible` lists the casts 
where Arrow's result matches Spark's. Every other cast returns a 
`NotImplemented` error.
   - **Pending ports**: Comet's allowlist accepts some casts that it actually 
implements itself before reaching Arrow, such as `BIGINT` to `INT` in Legacy 
and ANSI mode, float and decimal to integral, decimal to decimal in ANSI mode, 
decimal to string in Legacy mode, and binary to string. Falling through to 
Arrow would silently give NULL instead of Spark's wrapped value or error. 
`is_pending_port` rejects them as "not implemented yet" until their family is 
ported.
   - **Helpers ported from Comet**: post-processing of Arrow's timestamp casts 
(timestamp to long rounds down to seconds, and timestamp to string drops 
trailing zeros) in `cast/utils.rs`, and the session time zone helpers 
(`array_with_timezone`, `resolve_local_datetime`, `timestamp_ntz_to_timestamp`, 
`cast_timestamp_to_ntz`) in `function/timezone_utils.rs`. The pre-cast for 
timestamp to string or date now handles millisecond timestamps, where Comet's 
version assumed microseconds and would panic.
   - The existing `spark_cast` UDF moves unchanged from `conversion/cast.rs` to 
`conversion/cast/udf.rs`. Reworking it to parse the target type and call this 
kernel is a separate item in #88.
   
   `trim.rs`, `cast_overflow` and `timestamp_micros_with_tz` are not ported 
yet, because nothing uses them yet and they would be dead code. They come with 
the first cast that uses them.
   
   ## Are these changes tested?
   
   Yes, with Rust unit tests:
   
   - `SparkError`'s messages and `find`
   - the dispatcher: dictionary targets, identity casts, an unsupported cast, 
pending casts not falling through to Arrow, Arrow fallthrough, scalars, NULL 
arrays, dictionary input, and the timestamp casts, including missing and 
invalid time zones
   - the post-processing
   - the time zone helpers, including DST gaps and overlaps, ported from Comet
   
   No SQL function calls the kernel yet, so there are no SLT changes. 
`cast_int_to_timestamp.slt` and `cast_float_to_timestamp.slt` still pass.
   
   ## Are there any user-facing changes?
   
   There are new public items: `SparkError`, `SparkCastOptions`, the 
`spark_cast` kernel, and the now-public `function::conversion::cast` module. 
The behavior of SQL functions doesn't change.
   
   _This PR was written by an LLM (Claude) from the Comet and Spark sources, 
and reviewed before opening._
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to