andygrove opened a new issue, #5103: URL: https://github.com/apache/datafusion-comet/issues/5103
### What is the problem the feature request solves? Beyond arrow kernels, two areas duplicate functionality now available upstream in DataFusion / datafusion-spark (54.1): 1. **Date/timestamp truncation**: `native/spark-expr/src/kernels/temporal.rs` implements `trunc(date, fmt)` and `date_trunc(fmt, ts)` by hand. Arrow has no truncation kernel, but `datafusion_functions::datetime::date_trunc` is timezone-aware (chrono-tz), supports all the granularities Spark needs with Monday-based weeks, and preserves the input tz annotation. Upstream datafusion-spark's `trunc` already composes it with the same alias map (yy/yyyy to year, mm/mon to month) via a cast sandwich. 2. **xxhash64**: datafusion-spark 54.1 ships a Spark-compatible `SparkXxhash64` (twox-hash based, with its own type-dispatch machinery). Comet's `hash_funcs/xxhash64.rs` could potentially be rewired to it. Note datafusion-spark has no murmur3, so Comet's `murmur3.rs` stays. ### Describe the potential solution - Truncation: wire `date_trunc` at the physical level for the scalar-format paths (datafusion-spark's `trunc`/`date_diff` are simplify-only UDFs whose `invoke_with_args` returns an internal error, so they cannot be called from Comet's physical-plan pipeline directly). Keep custom code for: per-row-format variants (DataFusion requires scalar granularity), dictionary inputs, and the Spark session-timezone pre-shift. - Requires differential DST-boundary testing first: Comet resolves ambiguous local times to the earliest occurrence and falls back to the original instant on gaps (`test_timestamp_trunc_dst_boundary`); DataFusion's fine-versus-coarse granularity handling needs case-by-case verification. - xxhash64: verify type coverage before rewiring (Comet's macro additionally handles the small/large Decimal128 split, Map, FixedSizeList, and Struct fast paths). ### Additional context Related in spirit: `string_funcs/contains.rs` and `array_funcs/array_slice.rs` each document an upstream DataFusion issue that, once fixed, lets the Comet file be dropped entirely. Found during an audit of native code that replicates existing arrow-rs kernels. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
