mbutrovich commented on code in PR #332:
URL: https://github.com/apache/datafusion-spark/pull/332#discussion_r4602653062
##########
crates/spark/src/function/math/greatest_least.rs:
##########
@@ -0,0 +1,1173 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! Spark-compatible `greatest` and `least`.
+//!
+//! Spark evaluates the arguments in order, skips NULLs, and replaces its
running result only
+//! with a value that is strictly greater (or smaller) in its SQL ordering, so
the result is NULL
+//! only when every argument is NULL and the first of equal arguments wins.
The SQL ordering
+//! (`SQLOrderingUtil.compareDoubles`) treats `-0.0` as equal to `0.0` and NaN
as greater than
+//! every other value, at any depth of arrays and structs, and orders a NULL
inside an array or
+//! struct before every other value.
+//!
+//! DataFusion's built-in `greatest` and `least` differ: they compare floats
by IEEE 754 total
+//! order (so `-0.0 < 0.0`, and a NaN with the sign bit set is smaller than
`-Infinity`), let a
+//! later argument win a tie, compare the constant arguments before the
columns, accept a single
+//! argument, order a NULL inside an array or struct after every other value
in `least`, and
+//! coerce their arguments with DataFusion's comparison coercion, which for
example compares a
+//! number with a string. Spark rejects that.
+
+use crate::function::array::float_normalize::{contains_float, normalize_array};
+use arrow::array::{
+ Array, ArrayRef, BooleanArray, Datum, Scalar, make_comparator,
new_null_array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::{gt, lt};
+use arrow::compute::kernels::zip::zip;
+use arrow::compute::{SortOptions, cast};
+use arrow::datatypes::{DataType, Field, FieldRef, Fields, TimeUnit};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{DataFusionError, Result, ScalarValue, plan_err};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs,
ScalarUDF,
+ ScalarUDFImpl, Signature, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::sync::Arc;
+
+/// Spark-compatible `greatest` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#greatest>
+#[user_doc(
+ doc_section(label = "Math Functions"),
+ description = "Returns the greatest value of all arguments, skipping
NULLs. The result is \
+ NULL only if every argument is NULL. At least two arguments are
required. The \
+ arguments are cast to a common type as in Spark: numbers widen (an
integer and a FLOAT \
+ widen to DOUBLE with `datafusion.execution.enable_ansi_mode` on and to
FLOAT with it \
+ off), integers and decimals widen to a decimal that holds both, a
decimal and a float \
+ widen to DOUBLE, a date and a timestamp widen to the timestamp, and
arrays and structs \
+ widen element by element. Strings are never promoted: a string and a
number, a string \
+ and a date, or a boolean and a number are an error, as are maps, which
cannot be \
+ ordered. NaN is greater than every other value and `-0.0` equals
`0.0`, also inside \
+ arrays and structs, and of equal values the first argument wins, so \
+ `greatest(-0.0, 0.0)` is `-0.0`. A NULL inside an array or struct is
smaller than \
+ every other value. DataFusion's built-in `greatest` orders floats by
IEEE 754 total \
+ order, lets a later argument win a tie, accepts a single argument, and
compares \
+ numbers with strings.",
+ syntax_example = "greatest(expr1, expr2[, ..., expr_n])",
+ sql_example = r#"```sql
+> SELECT greatest(10, 9, 2, 4, 3);
+10
+> SELECT greatest(1, NULL, 3);
+3
+> SELECT greatest(1.0, 'NaN'::double);
+NaN
+```"#,
+ argument(
+ name = "expr1, expr2, ..., expr_n",
+ description = "Expressions of an orderable type that share a common
type. NULLs are \
+ skipped."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkGreatest {
+ inner: GreatestLeast,
+}
+
+impl Default for SparkGreatest {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkGreatest {
+ pub fn new() -> Self {
+ Self {
+ inner: GreatestLeast::new(true, false),
+ }
+ }
+
+ /// Creates a `greatest` that coerces its arguments as Spark does with
+ /// `datafusion.execution.enable_ansi_mode` set to `enable_ansi_mode`.
+ pub fn new_with_ansi_mode(enable_ansi_mode: bool) -> Self {
+ Self {
+ inner: GreatestLeast::new(true, enable_ansi_mode),
+ }
+ }
+}
+
+/// Spark-compatible `least` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#least>
+#[user_doc(
+ doc_section(label = "Math Functions"),
+ description = "Returns the least value of all arguments, skipping NULLs.
The result is \
+ NULL only if every argument is NULL. At least two arguments are
required. The \
+ arguments are cast to a common type as in `greatest`. NaN is greater
than every other \
+ value and `-0.0` equals `0.0`, also inside arrays and structs, and of
equal values the \
+ first argument wins, so `least(0.0, -0.0)` is `0.0`. A NULL inside an
array or struct \
+ is smaller than every other value, so `least(array(1, NULL), array(1,
0))` is \
+ `[1, NULL]`. DataFusion's built-in `least` orders floats by IEEE 754
total order, lets a \
+ later argument win a tie, orders a NULL inside an array or struct
after every other \
+ value, accepts a single argument, and compares numbers with strings.",
+ syntax_example = "least(expr1, expr2[, ..., expr_n])",
+ sql_example = r#"```sql
+> SELECT least(10, 9, 2, 4, 3);
+2
+> SELECT least(NULL, 2, NULL);
+2
+> SELECT least(1.0, 'NaN'::double);
+1.0
+```"#,
+ argument(
+ name = "expr1, expr2, ..., expr_n",
+ description = "Expressions of an orderable type that share a common
type. NULLs are \
+ skipped."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLeast {
+ inner: GreatestLeast,
+}
+
+impl Default for SparkLeast {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLeast {
+ pub fn new() -> Self {
+ Self {
+ inner: GreatestLeast::new(false, false),
+ }
+ }
+
+ /// Creates a `least` that coerces its arguments as Spark does with
+ /// `datafusion.execution.enable_ansi_mode` set to `enable_ansi_mode`.
+ pub fn new_with_ansi_mode(enable_ansi_mode: bool) -> Self {
+ Self {
+ inner: GreatestLeast::new(false, enable_ansi_mode),
+ }
+ }
+}
+
+macro_rules! impl_greatest_least_udf {
+ ($udf:ident) => {
+ impl ScalarUDFImpl for $udf {
+ fn name(&self) -> &str {
+ self.inner.name()
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.inner.signature
+ }
+
+ fn coerce_types(&self, arg_types: &[DataType]) ->
Result<Vec<DataType>> {
+ self.inner.coerce_types(arg_types)
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ self.inner.check_arg_count(arg_types.len(), true)?;
+ Ok(arg_types[0].clone())
+ }
+
+ fn return_field_from_args(&self, args: ReturnFieldArgs) ->
Result<FieldRef> {
+ self.inner.check_arg_count(args.arg_fields.len(), true)?;
+ // As in Spark, the result is nullable only if every argument
is.
+ let nullable = args.arg_fields.iter().all(|f| f.is_nullable());
+ Ok(Arc::new(Field::new(
+ self.name(),
+ args.arg_fields[0].data_type().clone(),
+ nullable,
+ )))
+ }
+
+ fn with_updated_config(&self, config: &ConfigOptions) ->
Option<ScalarUDF> {
+ Some(ScalarUDF::from(Self::new_with_ansi_mode(
+ config.execution.enable_ansi_mode,
+ )))
+ }
+
+ fn invoke_with_args(
+ &self,
+ args: ScalarFunctionArgs,
+ ) -> Result<ColumnarValue> {
+ self.inner.invoke(args)
+ }
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.doc()
+ }
+ }
+ };
+}
+
+impl_greatest_least_udf!(SparkGreatest);
+impl_greatest_least_udf!(SparkLeast);
+
+/// The implementation shared by [`SparkGreatest`] and [`SparkLeast`].
+#[derive(Debug, PartialEq, Eq, Hash)]
+struct GreatestLeast {
+ signature: Signature,
+ greatest: bool,
+ /// Whether to coerce as Spark's `AnsiTypeCoercion` does, rather than
`TypeCoercion`.
+ ansi_mode: bool,
+}
+
+/// Spark orders a NULL inside an array or struct before every other value.
+const SORT_OPTIONS: SortOptions = SortOptions {
+ descending: false,
+ nulls_first: true,
+};
+
+impl GreatestLeast {
+ fn new(greatest: bool, ansi_mode: bool) -> Self {
+ Self {
+ signature: Signature::user_defined(Volatility::Immutable),
+ greatest,
+ ansi_mode,
+ }
+ }
+
+ fn name(&self) -> &'static str {
+ if self.greatest { "greatest" } else { "least" }
+ }
+
+ fn check_arg_count(&self, count: usize, planning: bool) -> Result<()> {
+ if count > 1 {
+ return Ok(());
+ }
+ let message = format!(
+ "[WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `{}` requires > 1
parameters but the \
+ actual number is {count}. Please, refer to \
+ 'https://spark.apache.org/docs/latest/sql-ref-functions.html' for
a fix.",
+ self.name()
+ );
+ Err(if planning {
+ DataFusionError::Plan(message)
+ } else {
+ DataFusionError::Execution(message)
+ })
+ }
+
+ /// Spark's `TypeCoercion` rule for `Greatest` and `Least`: every argument
is cast to
+ /// `findWiderTypeWithoutStringPromotion` of all the argument types, and
the result must be
+ /// orderable.
+ fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ self.check_arg_count(arg_types.len(), true)?;
+ let common = arg_types
+ .iter()
+ .skip(1)
+ .try_fold(arg_types[0].clone(), |common, data_type| {
+ wider_type(&common, data_type, self.ansi_mode)
+ });
+ let Some(common) = common else {
+ return plan_err!(
+ "[DATATYPE_MISMATCH.DATA_DIFF_TYPES] Input to `{}` should all
be the same \
+ type, but it's [{}]",
+ self.name(),
+ arg_types
+ .iter()
+ .map(|t| format!("\"{}\"", spark_sql_type(t)))
+ .collect::<Vec<_>>()
+ .join(", ")
+ );
+ };
+ if !is_orderable(&common) {
+ return plan_err!(
+ "[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] The `{}` does not
support ordering \
+ on type \"{}\"",
+ self.name(),
+ spark_sql_type(&common)
+ );
+ }
+ Ok(vec![common; arg_types.len()])
+ }
+
+ fn invoke(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ self.check_arg_count(args.args.len(), false)?;
+ let data_type = args.return_field.data_type();
+ let all_scalars = args
+ .args
+ .iter()
+ .all(|arg| matches!(arg, ColumnarValue::Scalar(_)));
+ let rows = if all_scalars { 1 } else { args.number_rows };
+
+ if data_type.is_null() {
+ return Ok(if all_scalars {
+ ColumnarValue::Scalar(ScalarValue::Null)
+ } else {
+ ColumnarValue::Array(new_null_array(&DataType::Null, rows))
+ });
+ }
+
+ // The planner casts every argument to the common type, but a field
name or the
+ // nullability of a nested field can still differ when the function is
called directly,
+ // and `zip` needs identical types.
+ let to_array = |arg: &ColumnarValue, rows: usize| -> Result<ArrayRef> {
+ let array = arg.to_array(rows)?;
+ Ok(if array.data_type() == data_type {
+ array
+ } else {
+ cast(&array, data_type)?
+ })
+ };
+ // Floats are compared in a copy where `-0.0` is `0.0` and every NaN
is the canonical
+ // NaN, where Arrow's total order agrees with Spark's. The result
keeps the original.
+ let normalize = contains_float(data_type);
+ let comparable = |array: &ArrayRef| -> Result<ArrayRef> {
+ if normalize {
+ normalize_array(array)
+ } else {
+ Ok(Arc::clone(array))
+ }
+ };
+
+ let mut result = to_array(&args.args[0], rows)?;
+ let mut result_cmp = comparable(&result)?;
+ for arg in &args.args[1..] {
+ let candidate = match arg {
+ // A NULL never replaces the result.
+ ColumnarValue::Scalar(scalar) if scalar.is_null() => continue,
+ ColumnarValue::Scalar(_) => {
+ let value = to_array(arg, 1)?;
+ let cmp = comparable(&value)?;
+ Candidate {
+ value,
+ cmp,
+ scalar: true,
+ }
+ }
+ ColumnarValue::Array(_) => {
+ let value = to_array(arg, rows)?;
+ let cmp = comparable(&value)?;
+ Candidate {
+ value,
+ cmp,
+ scalar: false,
+ }
+ }
+ };
+ let take = self.replaces(&candidate, &result, &result_cmp)?;
+ let take = BooleanArray::new(take, None);
+ let new_result = zip(&take, candidate.value_datum().as_ref(),
&result)?;
+ result_cmp = if normalize {
+ zip(&take, candidate.cmp_datum().as_ref(), &result_cmp)?
+ } else {
+ Arc::clone(&new_result)
+ };
+ result = new_result;
+ }
+
+ if all_scalars {
+ Ok(ColumnarValue::Scalar(ScalarValue::try_from_array(
+ &result, 0,
+ )?))
+ } else {
+ Ok(ColumnarValue::Array(result))
+ }
+ }
+
+ /// For each row, whether `candidate` replaces the current result: it is
not NULL, and
+ /// either the result is NULL or the candidate ranks strictly before it.
+ fn replaces(
+ &self,
+ candidate: &Candidate,
+ current: &ArrayRef,
+ current_cmp: &ArrayRef,
+ ) -> Result<BooleanBuffer> {
+ let len = current.len();
+ let current_nulls = current.logical_nulls();
+ let candidate_nulls = if candidate.scalar {
+ // A NULL constant is skipped before this is called.
+ None
+ } else {
+ candidate.value.logical_nulls()
+ };
+
+ if !current.data_type().is_nested() {
+ let candidate_cmp = candidate.cmp_datum();
+ let ordered = if self.greatest {
+ gt(candidate_cmp.as_ref(), current_cmp)?
+ } else {
+ lt(candidate_cmp.as_ref(), current_cmp)?
+ };
+ // Where either side is NULL the comparison's value is arbitrary,
but such rows are
+ // decided by the null masks below.
+ let mut take = ordered.values().clone();
+ if let Some(nulls) = ¤t_nulls {
+ take = &take | &!nulls.inner();
+ }
+ if let Some(nulls) = &candidate_nulls {
+ take = &take & nulls.inner();
+ }
+ return Ok(take);
+ }
+
+ let compare =
+ make_comparator(candidate.cmp.as_ref(), current_cmp.as_ref(),
SORT_OPTIONS)?;
+ let scalar = candidate.scalar;
+ Ok(BooleanBuffer::collect_bool(len, |row| {
+ let candidate_row = if scalar { 0 } else { row };
+ if candidate_nulls
+ .as_ref()
+ .is_some_and(|nulls| nulls.is_null(candidate_row))
+ {
+ return false;
+ }
+ if current_nulls
+ .as_ref()
+ .is_some_and(|nulls| nulls.is_null(row))
+ {
+ return true;
+ }
+ let ordering = compare(candidate_row, row);
+ if self.greatest {
+ ordering.is_gt()
+ } else {
+ ordering.is_lt()
+ }
+ }))
+ }
+}
+
+/// An argument after the first, with its values as they are and as they are
compared. A
+/// constant is a one-row array.
+struct Candidate {
+ value: ArrayRef,
+ cmp: ArrayRef,
+ scalar: bool,
+}
+
+impl Candidate {
+ fn datum(&self, array: &ArrayRef) -> Box<dyn Datum> {
+ if self.scalar {
+ Box::new(Scalar::new(Arc::clone(array)))
+ } else {
+ Box::new(Arc::clone(array))
+ }
+ }
+
+ fn value_datum(&self) -> Box<dyn Datum> {
+ self.datum(&self.value)
+ }
+
+ fn cmp_datum(&self) -> Box<dyn Datum> {
+ self.datum(&self.cmp)
+ }
+}
+
+/// Spark's `findWiderTypeWithoutStringPromotionForTwo`: the tightest common
type, then a
+/// decimal wide enough for both, then the same for the elements of arrays and
the fields of
+/// structs. Strings are never promoted. `ansi_mode` picks between
`AnsiTypeCoercion` and
+/// `TypeCoercion`, which differ only in that ANSI widens an integer and a
FLOAT to DOUBLE.
+fn wider_type(left: &DataType, right: &DataType, ansi_mode: bool) ->
Option<DataType> {
+ tightest_common_type(left, right, ansi_mode)
+ .or_else(|| wider_decimal_type(left, right))
+ .or_else(|| complex_type(left, right, ansi_mode))
+}
+
+/// Spark's `findTightestCommonType`, on the Arrow types that Spark's types
map to. Arrays and
+/// structs are handled by [`complex_type`] with [`wider_type`], which gives
the same result.
+fn tightest_common_type(
+ left: &DataType,
+ right: &DataType,
+ ansi_mode: bool,
+) -> Option<DataType> {
+ use DataType::*;
+ if left == right {
+ return Some(left.clone());
+ }
+ if let (Some(l), Some(r)) = (numeric_precedence(left),
numeric_precedence(right)) {
+ let wider = if l >= r { left } else { right };
+ // With ANSI mode on, an integer and a FLOAT widen to DOUBLE, which
holds every INT
+ // exactly.
+ return Some(if ansi_mode && *wider == Float32 {
+ Float64
+ } else {
+ wider.clone()
+ });
+ }
+ match (left, right) {
+ (Null, other) | (other, Null) => Some(other.clone()),
+ // Spark has one string type and one binary type.
+ (l, r) if is_string(l) && is_string(r) => Some(wider_physical_type(l,
r)),
+ (l, r) if is_binary(l) && is_binary(r) => Some(wider_physical_type(l,
r)),
+ // The decimal must hold every value of the integral type.
+ (int, Decimal128(p, s)) | (Decimal128(p, s), int) => {
+ let digits = integral_digits(int)?;
+ (*p as i16 - *s as i16 >= digits).then_some(Decimal128(*p, *s))
+ }
+ (Date32, ts @ Timestamp(_, _)) | (ts @ Timestamp(_, _), Date32) => {
+ Some(ts.clone())
+ }
+ // TIMESTAMP_NTZ and TIMESTAMP widen to TIMESTAMP.
+ (Timestamp(lu, ltz), Timestamp(ru, rtz)) => Some(Timestamp(
+ finer_unit(*lu, *ru),
+ ltz.clone().or_else(|| rtz.clone()),
+ )),
+ _ => None,
+ }
+}
+
+/// Spark's `findWiderTypeForDecimal`, with
`DecimalType.boundedPreferIntegralDigits`, which
+/// gives up fractional digits rather than integral ones when the precision
exceeds 38.
+fn wider_decimal_type(left: &DataType, right: &DataType) -> Option<DataType> {
+ use DataType::*;
+ let (p1, s1, p2, s2) = match (left, right) {
+ (Decimal128(p1, s1), Decimal128(p2, s2)) => {
+ (*p1 as i16, *s1 as i16, *p2 as i16, *s2 as i16)
+ }
+ (int, Decimal128(p, s)) | (Decimal128(p, s), int)
+ if integral_digits(int).is_some() =>
+ {
+ (integral_digits(int)?, 0, *p as i16, *s as i16)
+ }
+ (Float32 | Float64, Decimal128(_, _)) | (Decimal128(_, _), Float32 |
Float64) => {
+ return Some(Float64);
+ }
+ _ => return None,
+ };
+ let scale = s1.max(s2);
+ let range = (p1 - s1).max(p2 - s2);
+ let precision = scale + range;
+ const MAX_PRECISION: i16 = 38;
+ if precision <= MAX_PRECISION {
+ Some(Decimal128(precision as u8, scale as i8))
+ } else {
+ let scale = (scale - (precision - MAX_PRECISION)).max(0);
+ Some(Decimal128(MAX_PRECISION as u8, scale as i8))
+ }
+}
+
+/// Spark's `findTypeForComplex` with [`wider_type`] for the elements and
fields. A nested field
+/// is nullable if it is on either side. Struct fields must have the same
names in the same
+/// order. Spark compares them ignoring case, but this requires an exact match.
+fn complex_type(left: &DataType, right: &DataType, ansi_mode: bool) ->
Option<DataType> {
+ use DataType::*;
+ match (left, right) {
+ (
+ List(lf) | LargeList(lf) | FixedSizeList(lf, _),
+ List(rf) | LargeList(rf) | FixedSizeList(rf, _),
+ ) => {
+ let item = wider_type(lf.data_type(), rf.data_type(), ansi_mode)?;
+ let field = Arc::new(Field::new(
+ lf.name(),
+ item,
+ lf.is_nullable() || rf.is_nullable(),
+ ));
+ Some(match (left, right) {
+ (FixedSizeList(_, ln), FixedSizeList(_, rn)) if ln == rn => {
+ FixedSizeList(field, *ln)
+ }
+ (LargeList(_), _) | (_, LargeList(_)) => LargeList(field),
+ _ => List(field),
+ })
+ }
+ (Struct(lf), Struct(rf)) if lf.len() == rf.len() => {
+ let fields = lf
+ .iter()
+ .zip(rf.iter())
+ .map(|(a, b)| {
+ if a.name() != b.name() {
+ return None;
+ }
+ let data_type = wider_type(a.data_type(), b.data_type(),
ansi_mode)?;
+ Some(Field::new(
+ a.name(),
+ data_type,
+ a.is_nullable() || b.is_nullable(),
+ ))
+ })
+ .collect::<Option<Vec<_>>>()?;
+ Some(Struct(Fields::from(fields)))
+ }
+ // Spark finds a common map type, which is then rejected as not
orderable.
+ (Map(lf, sorted), Map(rf, _)) => {
+ let entries = wider_type(lf.data_type(), rf.data_type(),
ansi_mode)?;
+ Some(Map(
+ Arc::new(Field::new(lf.name(), entries, false)),
+ *sorted,
+ ))
+ }
+ _ => None,
+ }
+}
+
+/// Whether Spark can order values of this type. It cannot order maps, at any
depth.
+fn is_orderable(data_type: &DataType) -> bool {
+ match data_type {
+ DataType::Map(_, _) | DataType::Union(_, _) => false,
Review Comment:
Spark can't order `CalendarIntervalType`. This crate's `make_interval`
returns it as `Interval(MonthDayNano)`, and Comet maps it to the same Arrow
type
([`serde.rs`](https://github.com/apache/datafusion-comet/blob/ba16e1a53127c90ba00fdc7058703945a8bb9f34/native/core/src/execution/serde.rs#L106-L108)).
[`OrderUtils.isOrderable`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/api/src/main/scala/org/apache/spark/sql/catalyst/expressions/OrderUtils.scala#L26-L38)
accepts only atomic types and arrays and structs of them, and
`CalendarIntervalType` isn't atomic. For `SELECT greatest(make_interval(1),
make_interval(2))`, Spark 4.2.0 raises:
```
[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] ... The `greatest` does not
support ordering on type "INTERVAL".
```
Here it returns `24 mons`. `greatest(make_interval(0, 0, 0, 40),
make_interval(0, 1))` returns `1 mons`, because Arrow orders by months, then
days, then nanoseconds. Could `is_orderable` reject it, with `spark_sql_type`
rendering it as `INTERVAL` and an slt error case for a flat and a nested
interval? This needs `IntervalUnit` imported.
```suggestion
DataType::Map(_, _)
| DataType::Union(_, _)
| DataType::Interval(IntervalUnit::MonthDayNano) => false,
```
##########
crates/spark/src/function/math/greatest_least.rs:
##########
@@ -0,0 +1,1173 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! Spark-compatible `greatest` and `least`.
+//!
+//! Spark evaluates the arguments in order, skips NULLs, and replaces its
running result only
+//! with a value that is strictly greater (or smaller) in its SQL ordering, so
the result is NULL
+//! only when every argument is NULL and the first of equal arguments wins.
The SQL ordering
+//! (`SQLOrderingUtil.compareDoubles`) treats `-0.0` as equal to `0.0` and NaN
as greater than
+//! every other value, at any depth of arrays and structs, and orders a NULL
inside an array or
+//! struct before every other value.
+//!
+//! DataFusion's built-in `greatest` and `least` differ: they compare floats
by IEEE 754 total
+//! order (so `-0.0 < 0.0`, and a NaN with the sign bit set is smaller than
`-Infinity`), let a
+//! later argument win a tie, compare the constant arguments before the
columns, accept a single
+//! argument, order a NULL inside an array or struct after every other value
in `least`, and
+//! coerce their arguments with DataFusion's comparison coercion, which for
example compares a
+//! number with a string. Spark rejects that.
+
+use crate::function::array::float_normalize::{contains_float, normalize_array};
+use arrow::array::{
+ Array, ArrayRef, BooleanArray, Datum, Scalar, make_comparator,
new_null_array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::{gt, lt};
+use arrow::compute::kernels::zip::zip;
+use arrow::compute::{SortOptions, cast};
+use arrow::datatypes::{DataType, Field, FieldRef, Fields, TimeUnit};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{DataFusionError, Result, ScalarValue, plan_err};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs,
ScalarUDF,
+ ScalarUDFImpl, Signature, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::sync::Arc;
+
+/// Spark-compatible `greatest` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#greatest>
+#[user_doc(
+ doc_section(label = "Math Functions"),
+ description = "Returns the greatest value of all arguments, skipping
NULLs. The result is \
+ NULL only if every argument is NULL. At least two arguments are
required. The \
+ arguments are cast to a common type as in Spark: numbers widen (an
integer and a FLOAT \
+ widen to DOUBLE with `datafusion.execution.enable_ansi_mode` on and to
FLOAT with it \
+ off), integers and decimals widen to a decimal that holds both, a
decimal and a float \
+ widen to DOUBLE, a date and a timestamp widen to the timestamp, and
arrays and structs \
+ widen element by element. Strings are never promoted: a string and a
number, a string \
+ and a date, or a boolean and a number are an error, as are maps, which
cannot be \
+ ordered. NaN is greater than every other value and `-0.0` equals
`0.0`, also inside \
+ arrays and structs, and of equal values the first argument wins, so \
+ `greatest(-0.0, 0.0)` is `-0.0`. A NULL inside an array or struct is
smaller than \
+ every other value. DataFusion's built-in `greatest` orders floats by
IEEE 754 total \
+ order, lets a later argument win a tie, accepts a single argument, and
compares \
+ numbers with strings.",
+ syntax_example = "greatest(expr1, expr2[, ..., expr_n])",
+ sql_example = r#"```sql
+> SELECT greatest(10, 9, 2, 4, 3);
+10
+> SELECT greatest(1, NULL, 3);
+3
+> SELECT greatest(1.0, 'NaN'::double);
+NaN
+```"#,
+ argument(
+ name = "expr1, expr2, ..., expr_n",
+ description = "Expressions of an orderable type that share a common
type. NULLs are \
+ skipped."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkGreatest {
+ inner: GreatestLeast,
+}
+
+impl Default for SparkGreatest {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkGreatest {
+ pub fn new() -> Self {
+ Self {
+ inner: GreatestLeast::new(true, false),
+ }
+ }
+
+ /// Creates a `greatest` that coerces its arguments as Spark does with
+ /// `datafusion.execution.enable_ansi_mode` set to `enable_ansi_mode`.
+ pub fn new_with_ansi_mode(enable_ansi_mode: bool) -> Self {
+ Self {
+ inner: GreatestLeast::new(true, enable_ansi_mode),
+ }
+ }
+}
+
+/// Spark-compatible `least` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#least>
+#[user_doc(
+ doc_section(label = "Math Functions"),
+ description = "Returns the least value of all arguments, skipping NULLs.
The result is \
+ NULL only if every argument is NULL. At least two arguments are
required. The \
+ arguments are cast to a common type as in `greatest`. NaN is greater
than every other \
+ value and `-0.0` equals `0.0`, also inside arrays and structs, and of
equal values the \
+ first argument wins, so `least(0.0, -0.0)` is `0.0`. A NULL inside an
array or struct \
+ is smaller than every other value, so `least(array(1, NULL), array(1,
0))` is \
+ `[1, NULL]`. DataFusion's built-in `least` orders floats by IEEE 754
total order, lets a \
+ later argument win a tie, orders a NULL inside an array or struct
after every other \
+ value, accepts a single argument, and compares numbers with strings.",
+ syntax_example = "least(expr1, expr2[, ..., expr_n])",
+ sql_example = r#"```sql
+> SELECT least(10, 9, 2, 4, 3);
+2
+> SELECT least(NULL, 2, NULL);
+2
+> SELECT least(1.0, 'NaN'::double);
+1.0
+```"#,
+ argument(
+ name = "expr1, expr2, ..., expr_n",
+ description = "Expressions of an orderable type that share a common
type. NULLs are \
+ skipped."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLeast {
+ inner: GreatestLeast,
+}
+
+impl Default for SparkLeast {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLeast {
+ pub fn new() -> Self {
+ Self {
+ inner: GreatestLeast::new(false, false),
+ }
+ }
+
+ /// Creates a `least` that coerces its arguments as Spark does with
+ /// `datafusion.execution.enable_ansi_mode` set to `enable_ansi_mode`.
+ pub fn new_with_ansi_mode(enable_ansi_mode: bool) -> Self {
+ Self {
+ inner: GreatestLeast::new(false, enable_ansi_mode),
+ }
+ }
+}
+
+macro_rules! impl_greatest_least_udf {
+ ($udf:ident) => {
+ impl ScalarUDFImpl for $udf {
+ fn name(&self) -> &str {
+ self.inner.name()
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.inner.signature
+ }
+
+ fn coerce_types(&self, arg_types: &[DataType]) ->
Result<Vec<DataType>> {
+ self.inner.coerce_types(arg_types)
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ self.inner.check_arg_count(arg_types.len(), true)?;
+ Ok(arg_types[0].clone())
+ }
+
+ fn return_field_from_args(&self, args: ReturnFieldArgs) ->
Result<FieldRef> {
+ self.inner.check_arg_count(args.arg_fields.len(), true)?;
+ // As in Spark, the result is nullable only if every argument
is.
+ let nullable = args.arg_fields.iter().all(|f| f.is_nullable());
+ Ok(Arc::new(Field::new(
+ self.name(),
+ args.arg_fields[0].data_type().clone(),
+ nullable,
+ )))
+ }
+
+ fn with_updated_config(&self, config: &ConfigOptions) ->
Option<ScalarUDF> {
+ Some(ScalarUDF::from(Self::new_with_ansi_mode(
+ config.execution.enable_ansi_mode,
+ )))
+ }
+
+ fn invoke_with_args(
+ &self,
+ args: ScalarFunctionArgs,
+ ) -> Result<ColumnarValue> {
+ self.inner.invoke(args)
+ }
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.doc()
+ }
+ }
+ };
+}
+
+impl_greatest_least_udf!(SparkGreatest);
+impl_greatest_least_udf!(SparkLeast);
+
+/// The implementation shared by [`SparkGreatest`] and [`SparkLeast`].
+#[derive(Debug, PartialEq, Eq, Hash)]
+struct GreatestLeast {
+ signature: Signature,
+ greatest: bool,
+ /// Whether to coerce as Spark's `AnsiTypeCoercion` does, rather than
`TypeCoercion`.
+ ansi_mode: bool,
+}
+
+/// Spark orders a NULL inside an array or struct before every other value.
+const SORT_OPTIONS: SortOptions = SortOptions {
+ descending: false,
+ nulls_first: true,
+};
+
+impl GreatestLeast {
+ fn new(greatest: bool, ansi_mode: bool) -> Self {
+ Self {
+ signature: Signature::user_defined(Volatility::Immutable),
+ greatest,
+ ansi_mode,
+ }
+ }
+
+ fn name(&self) -> &'static str {
+ if self.greatest { "greatest" } else { "least" }
+ }
+
+ fn check_arg_count(&self, count: usize, planning: bool) -> Result<()> {
+ if count > 1 {
+ return Ok(());
+ }
+ let message = format!(
+ "[WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `{}` requires > 1
parameters but the \
+ actual number is {count}. Please, refer to \
+ 'https://spark.apache.org/docs/latest/sql-ref-functions.html' for
a fix.",
+ self.name()
+ );
+ Err(if planning {
+ DataFusionError::Plan(message)
+ } else {
+ DataFusionError::Execution(message)
+ })
+ }
+
+ /// Spark's `TypeCoercion` rule for `Greatest` and `Least`: every argument
is cast to
+ /// `findWiderTypeWithoutStringPromotion` of all the argument types, and
the result must be
+ /// orderable.
+ fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ self.check_arg_count(arg_types.len(), true)?;
+ let common = arg_types
+ .iter()
+ .skip(1)
+ .try_fold(arg_types[0].clone(), |common, data_type| {
+ wider_type(&common, data_type, self.ansi_mode)
+ });
+ let Some(common) = common else {
+ return plan_err!(
+ "[DATATYPE_MISMATCH.DATA_DIFF_TYPES] Input to `{}` should all
be the same \
+ type, but it's [{}]",
+ self.name(),
+ arg_types
+ .iter()
+ .map(|t| format!("\"{}\"", spark_sql_type(t)))
+ .collect::<Vec<_>>()
+ .join(", ")
+ );
+ };
+ if !is_orderable(&common) {
+ return plan_err!(
+ "[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] The `{}` does not
support ordering \
+ on type \"{}\"",
+ self.name(),
+ spark_sql_type(&common)
+ );
+ }
+ Ok(vec![common; arg_types.len()])
+ }
+
+ fn invoke(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ self.check_arg_count(args.args.len(), false)?;
+ let data_type = args.return_field.data_type();
+ let all_scalars = args
+ .args
+ .iter()
+ .all(|arg| matches!(arg, ColumnarValue::Scalar(_)));
+ let rows = if all_scalars { 1 } else { args.number_rows };
+
+ if data_type.is_null() {
+ return Ok(if all_scalars {
+ ColumnarValue::Scalar(ScalarValue::Null)
+ } else {
+ ColumnarValue::Array(new_null_array(&DataType::Null, rows))
+ });
+ }
+
+ // The planner casts every argument to the common type, but a field
name or the
+ // nullability of a nested field can still differ when the function is
called directly,
+ // and `zip` needs identical types.
+ let to_array = |arg: &ColumnarValue, rows: usize| -> Result<ArrayRef> {
+ let array = arg.to_array(rows)?;
+ Ok(if array.data_type() == data_type {
+ array
+ } else {
+ cast(&array, data_type)?
+ })
+ };
+ // Floats are compared in a copy where `-0.0` is `0.0` and every NaN
is the canonical
+ // NaN, where Arrow's total order agrees with Spark's. The result
keeps the original.
+ let normalize = contains_float(data_type);
+ let comparable = |array: &ArrayRef| -> Result<ArrayRef> {
+ if normalize {
+ normalize_array(array)
+ } else {
+ Ok(Arc::clone(array))
+ }
+ };
Review Comment:
Every Float32 or Float64 argument gets a normalized copy per batch here,
before the `gt`/`lt` kernel runs. #331 has a `spark_float_cmp` that compares in
place with `partial_cmp`, and its description says normalizing and comparing
with `total_cmp` was slower there (393 us against 298 us for `array_max` over
Float64). Comet's `greatest_least` also compares flat floats in place
(`float_gt`/`float_lt`). #125 plans for Comet to use this version and remove
its copy, so could we add a criterion bench for `greatest` over two or three
Float64 columns at 8192 rows, like #331's `benches/array_extrema.rs`? That
would show whether the flat float path should use `spark_float_cmp` instead of
a copy. Nested types would keep `normalize_array` plus `make_comparator`, as in
#331's generic path.
##########
crates/spark/src/function/math/greatest_least.rs:
##########
@@ -0,0 +1,1173 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! Spark-compatible `greatest` and `least`.
+//!
+//! Spark evaluates the arguments in order, skips NULLs, and replaces its
running result only
+//! with a value that is strictly greater (or smaller) in its SQL ordering, so
the result is NULL
+//! only when every argument is NULL and the first of equal arguments wins.
The SQL ordering
+//! (`SQLOrderingUtil.compareDoubles`) treats `-0.0` as equal to `0.0` and NaN
as greater than
+//! every other value, at any depth of arrays and structs, and orders a NULL
inside an array or
+//! struct before every other value.
+//!
+//! DataFusion's built-in `greatest` and `least` differ: they compare floats
by IEEE 754 total
+//! order (so `-0.0 < 0.0`, and a NaN with the sign bit set is smaller than
`-Infinity`), let a
+//! later argument win a tie, compare the constant arguments before the
columns, accept a single
+//! argument, order a NULL inside an array or struct after every other value
in `least`, and
+//! coerce their arguments with DataFusion's comparison coercion, which for
example compares a
+//! number with a string. Spark rejects that.
+
+use crate::function::array::float_normalize::{contains_float, normalize_array};
+use arrow::array::{
+ Array, ArrayRef, BooleanArray, Datum, Scalar, make_comparator,
new_null_array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::{gt, lt};
+use arrow::compute::kernels::zip::zip;
+use arrow::compute::{SortOptions, cast};
+use arrow::datatypes::{DataType, Field, FieldRef, Fields, TimeUnit};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{DataFusionError, Result, ScalarValue, plan_err};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs,
ScalarUDF,
+ ScalarUDFImpl, Signature, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::sync::Arc;
+
+/// Spark-compatible `greatest` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#greatest>
+#[user_doc(
+ doc_section(label = "Math Functions"),
+ description = "Returns the greatest value of all arguments, skipping
NULLs. The result is \
+ NULL only if every argument is NULL. At least two arguments are
required. The \
+ arguments are cast to a common type as in Spark: numbers widen (an
integer and a FLOAT \
+ widen to DOUBLE with `datafusion.execution.enable_ansi_mode` on and to
FLOAT with it \
+ off), integers and decimals widen to a decimal that holds both, a
decimal and a float \
+ widen to DOUBLE, a date and a timestamp widen to the timestamp, and
arrays and structs \
+ widen element by element. Strings are never promoted: a string and a
number, a string \
+ and a date, or a boolean and a number are an error, as are maps, which
cannot be \
+ ordered. NaN is greater than every other value and `-0.0` equals
`0.0`, also inside \
+ arrays and structs, and of equal values the first argument wins, so \
+ `greatest(-0.0, 0.0)` is `-0.0`. A NULL inside an array or struct is
smaller than \
+ every other value. DataFusion's built-in `greatest` orders floats by
IEEE 754 total \
+ order, lets a later argument win a tie, accepts a single argument, and
compares \
+ numbers with strings.",
+ syntax_example = "greatest(expr1, expr2[, ..., expr_n])",
+ sql_example = r#"```sql
+> SELECT greatest(10, 9, 2, 4, 3);
+10
+> SELECT greatest(1, NULL, 3);
+3
+> SELECT greatest(1.0, 'NaN'::double);
+NaN
+```"#,
+ argument(
+ name = "expr1, expr2, ..., expr_n",
+ description = "Expressions of an orderable type that share a common
type. NULLs are \
+ skipped."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkGreatest {
+ inner: GreatestLeast,
+}
+
+impl Default for SparkGreatest {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkGreatest {
+ pub fn new() -> Self {
+ Self {
+ inner: GreatestLeast::new(true, false),
+ }
+ }
+
+ /// Creates a `greatest` that coerces its arguments as Spark does with
+ /// `datafusion.execution.enable_ansi_mode` set to `enable_ansi_mode`.
+ pub fn new_with_ansi_mode(enable_ansi_mode: bool) -> Self {
+ Self {
+ inner: GreatestLeast::new(true, enable_ansi_mode),
+ }
+ }
Review Comment:
Should `new_with_ansi_mode` be public? Its only caller is
`with_updated_config`, and DataFusion calls that whenever the session config
changes, so a session gets the right coercion without it. The flag only changes
`coerce_types`, which callers that build the physical expression directly
(Comet) never run. A public constructor is permanent once released, so could it
be private? The same applies to `SparkLeast::new_with_ansi_mode` at lines
147-153.
##########
crates/spark/src/function/math/greatest_least.rs:
##########
@@ -0,0 +1,1173 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! Spark-compatible `greatest` and `least`.
+//!
+//! Spark evaluates the arguments in order, skips NULLs, and replaces its
running result only
+//! with a value that is strictly greater (or smaller) in its SQL ordering, so
the result is NULL
+//! only when every argument is NULL and the first of equal arguments wins.
The SQL ordering
+//! (`SQLOrderingUtil.compareDoubles`) treats `-0.0` as equal to `0.0` and NaN
as greater than
+//! every other value, at any depth of arrays and structs, and orders a NULL
inside an array or
+//! struct before every other value.
+//!
+//! DataFusion's built-in `greatest` and `least` differ: they compare floats
by IEEE 754 total
+//! order (so `-0.0 < 0.0`, and a NaN with the sign bit set is smaller than
`-Infinity`), let a
+//! later argument win a tie, compare the constant arguments before the
columns, accept a single
+//! argument, order a NULL inside an array or struct after every other value
in `least`, and
+//! coerce their arguments with DataFusion's comparison coercion, which for
example compares a
+//! number with a string. Spark rejects that.
+
+use crate::function::array::float_normalize::{contains_float, normalize_array};
+use arrow::array::{
+ Array, ArrayRef, BooleanArray, Datum, Scalar, make_comparator,
new_null_array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::{gt, lt};
+use arrow::compute::kernels::zip::zip;
+use arrow::compute::{SortOptions, cast};
+use arrow::datatypes::{DataType, Field, FieldRef, Fields, TimeUnit};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{DataFusionError, Result, ScalarValue, plan_err};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs,
ScalarUDF,
+ ScalarUDFImpl, Signature, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::sync::Arc;
+
+/// Spark-compatible `greatest` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#greatest>
+#[user_doc(
+ doc_section(label = "Math Functions"),
+ description = "Returns the greatest value of all arguments, skipping
NULLs. The result is \
+ NULL only if every argument is NULL. At least two arguments are
required. The \
+ arguments are cast to a common type as in Spark: numbers widen (an
integer and a FLOAT \
+ widen to DOUBLE with `datafusion.execution.enable_ansi_mode` on and to
FLOAT with it \
+ off), integers and decimals widen to a decimal that holds both, a
decimal and a float \
+ widen to DOUBLE, a date and a timestamp widen to the timestamp, and
arrays and structs \
+ widen element by element. Strings are never promoted: a string and a
number, a string \
+ and a date, or a boolean and a number are an error, as are maps, which
cannot be \
+ ordered. NaN is greater than every other value and `-0.0` equals
`0.0`, also inside \
+ arrays and structs, and of equal values the first argument wins, so \
+ `greatest(-0.0, 0.0)` is `-0.0`. A NULL inside an array or struct is
smaller than \
+ every other value. DataFusion's built-in `greatest` orders floats by
IEEE 754 total \
+ order, lets a later argument win a tie, accepts a single argument, and
compares \
+ numbers with strings.",
+ syntax_example = "greatest(expr1, expr2[, ..., expr_n])",
+ sql_example = r#"```sql
+> SELECT greatest(10, 9, 2, 4, 3);
+10
+> SELECT greatest(1, NULL, 3);
+3
+> SELECT greatest(1.0, 'NaN'::double);
+NaN
+```"#,
+ argument(
+ name = "expr1, expr2, ..., expr_n",
+ description = "Expressions of an orderable type that share a common
type. NULLs are \
+ skipped."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkGreatest {
+ inner: GreatestLeast,
+}
+
+impl Default for SparkGreatest {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkGreatest {
+ pub fn new() -> Self {
+ Self {
+ inner: GreatestLeast::new(true, false),
+ }
+ }
+
+ /// Creates a `greatest` that coerces its arguments as Spark does with
+ /// `datafusion.execution.enable_ansi_mode` set to `enable_ansi_mode`.
+ pub fn new_with_ansi_mode(enable_ansi_mode: bool) -> Self {
+ Self {
+ inner: GreatestLeast::new(true, enable_ansi_mode),
+ }
+ }
+}
+
+/// Spark-compatible `least` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#least>
+#[user_doc(
+ doc_section(label = "Math Functions"),
+ description = "Returns the least value of all arguments, skipping NULLs.
The result is \
+ NULL only if every argument is NULL. At least two arguments are
required. The \
+ arguments are cast to a common type as in `greatest`. NaN is greater
than every other \
+ value and `-0.0` equals `0.0`, also inside arrays and structs, and of
equal values the \
+ first argument wins, so `least(0.0, -0.0)` is `0.0`. A NULL inside an
array or struct \
+ is smaller than every other value, so `least(array(1, NULL), array(1,
0))` is \
+ `[1, NULL]`. DataFusion's built-in `least` orders floats by IEEE 754
total order, lets a \
+ later argument win a tie, orders a NULL inside an array or struct
after every other \
+ value, accepts a single argument, and compares numbers with strings.",
+ syntax_example = "least(expr1, expr2[, ..., expr_n])",
+ sql_example = r#"```sql
+> SELECT least(10, 9, 2, 4, 3);
+2
+> SELECT least(NULL, 2, NULL);
+2
+> SELECT least(1.0, 'NaN'::double);
+1.0
+```"#,
+ argument(
+ name = "expr1, expr2, ..., expr_n",
+ description = "Expressions of an orderable type that share a common
type. NULLs are \
+ skipped."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLeast {
+ inner: GreatestLeast,
+}
+
+impl Default for SparkLeast {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLeast {
+ pub fn new() -> Self {
+ Self {
+ inner: GreatestLeast::new(false, false),
+ }
+ }
+
+ /// Creates a `least` that coerces its arguments as Spark does with
+ /// `datafusion.execution.enable_ansi_mode` set to `enable_ansi_mode`.
+ pub fn new_with_ansi_mode(enable_ansi_mode: bool) -> Self {
+ Self {
+ inner: GreatestLeast::new(false, enable_ansi_mode),
+ }
+ }
+}
+
+macro_rules! impl_greatest_least_udf {
+ ($udf:ident) => {
+ impl ScalarUDFImpl for $udf {
+ fn name(&self) -> &str {
+ self.inner.name()
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.inner.signature
+ }
+
+ fn coerce_types(&self, arg_types: &[DataType]) ->
Result<Vec<DataType>> {
+ self.inner.coerce_types(arg_types)
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ self.inner.check_arg_count(arg_types.len(), true)?;
+ Ok(arg_types[0].clone())
+ }
+
+ fn return_field_from_args(&self, args: ReturnFieldArgs) ->
Result<FieldRef> {
+ self.inner.check_arg_count(args.arg_fields.len(), true)?;
+ // As in Spark, the result is nullable only if every argument
is.
+ let nullable = args.arg_fields.iter().all(|f| f.is_nullable());
+ Ok(Arc::new(Field::new(
+ self.name(),
+ args.arg_fields[0].data_type().clone(),
+ nullable,
+ )))
+ }
+
+ fn with_updated_config(&self, config: &ConfigOptions) ->
Option<ScalarUDF> {
+ Some(ScalarUDF::from(Self::new_with_ansi_mode(
+ config.execution.enable_ansi_mode,
+ )))
+ }
+
+ fn invoke_with_args(
+ &self,
+ args: ScalarFunctionArgs,
+ ) -> Result<ColumnarValue> {
+ self.inner.invoke(args)
+ }
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.doc()
+ }
+ }
+ };
+}
+
+impl_greatest_least_udf!(SparkGreatest);
+impl_greatest_least_udf!(SparkLeast);
+
+/// The implementation shared by [`SparkGreatest`] and [`SparkLeast`].
+#[derive(Debug, PartialEq, Eq, Hash)]
+struct GreatestLeast {
+ signature: Signature,
+ greatest: bool,
+ /// Whether to coerce as Spark's `AnsiTypeCoercion` does, rather than
`TypeCoercion`.
+ ansi_mode: bool,
+}
+
+/// Spark orders a NULL inside an array or struct before every other value.
+const SORT_OPTIONS: SortOptions = SortOptions {
+ descending: false,
+ nulls_first: true,
+};
+
+impl GreatestLeast {
+ fn new(greatest: bool, ansi_mode: bool) -> Self {
+ Self {
+ signature: Signature::user_defined(Volatility::Immutable),
+ greatest,
+ ansi_mode,
+ }
+ }
+
+ fn name(&self) -> &'static str {
+ if self.greatest { "greatest" } else { "least" }
+ }
+
+ fn check_arg_count(&self, count: usize, planning: bool) -> Result<()> {
+ if count > 1 {
+ return Ok(());
+ }
+ let message = format!(
+ "[WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `{}` requires > 1
parameters but the \
+ actual number is {count}. Please, refer to \
+ 'https://spark.apache.org/docs/latest/sql-ref-functions.html' for
a fix.",
+ self.name()
+ );
+ Err(if planning {
+ DataFusionError::Plan(message)
+ } else {
+ DataFusionError::Execution(message)
+ })
+ }
+
+ /// Spark's `TypeCoercion` rule for `Greatest` and `Least`: every argument
is cast to
+ /// `findWiderTypeWithoutStringPromotion` of all the argument types, and
the result must be
+ /// orderable.
+ fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ self.check_arg_count(arg_types.len(), true)?;
+ let common = arg_types
+ .iter()
+ .skip(1)
+ .try_fold(arg_types[0].clone(), |common, data_type| {
+ wider_type(&common, data_type, self.ansi_mode)
+ });
+ let Some(common) = common else {
+ return plan_err!(
+ "[DATATYPE_MISMATCH.DATA_DIFF_TYPES] Input to `{}` should all
be the same \
+ type, but it's [{}]",
+ self.name(),
+ arg_types
+ .iter()
+ .map(|t| format!("\"{}\"", spark_sql_type(t)))
+ .collect::<Vec<_>>()
+ .join(", ")
+ );
+ };
+ if !is_orderable(&common) {
+ return plan_err!(
+ "[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] The `{}` does not
support ordering \
+ on type \"{}\"",
+ self.name(),
+ spark_sql_type(&common)
+ );
Review Comment:
Spark 4.2.0 ends these messages with a period
([`DATA_DIFF_TYPES`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/common/utils/src/main/resources/error/error-conditions.json#L1706-L1709),
[`INVALID_ORDERING_TYPE`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/common/utils/src/main/resources/error/error-conditions.json#L1751-L1754))
and then its SQLSTATE: `... but it's ["INT", "STRING"]. SQLSTATE: 42K09` and
`... for a fix. SQLSTATE: 42605`. The hand-built messages on `main` include the
SQLSTATE, for example
[`encode.rs`](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/spark/src/function/string/encode.rs#L118).
Could we end all three messages the way Spark does?
##########
crates/sqllogictest/test_files/math/greatest.slt:
##########
@@ -10,5 +10,233 @@
## Original Query: SELECT greatest(10, 9, 2, 4, 3);
## PySpark 3.5.5 Result: {'greatest(10, 9, 2, 4, 3)': 10, 'typeof(greatest(10,
9, 2, 4, 3))': 'int', 'typeof(10)': 'int', 'typeof(9)': 'int', 'typeof(2)':
'int', 'typeof(4)': 'int', 'typeof(3)': 'int'}
-#query
-#SELECT greatest(10::int, 9::int, 2::int, 4::int, 3::int);
+query I
+SELECT greatest(10::int, 9::int, 2::int, 4::int, 3::int);
+----
+10
+
+# NULLs are skipped; the result is NULL only if every argument is NULL
+query III
+SELECT greatest(1::int, NULL::int, 3::int), greatest(NULL::int, 2::int),
greatest(NULL::int, NULL::int);
+----
+3 2 NULL
+
+query ?
+SELECT greatest(NULL, NULL);
+----
+NULL
+
+# Integers widen to the widest integer type
+query I
+SELECT greatest(1::tinyint, 2::smallint);
+----
+2
+
+query T
+SELECT arrow_typeof(greatest(1::tinyint, 2::smallint));
+----
+Int16
+
+query I
+SELECT greatest(1::int, 3000000000::bigint);
+----
+3000000000
+
+# An integer and a DOUBLE widen to DOUBLE
+query R
+SELECT greatest(1::int, 2.5::double);
+----
+2.5
+
+query T
+SELECT arrow_typeof(greatest(1::int, 2.5::double));
+----
+Float64
+
+# NaN is greater than every other value, including Infinity
+query RRR
+SELECT greatest('NaN'::double, 'Infinity'::double), greatest(1::double,
'NaN'::double), greatest('-Infinity'::double, NULL::double);
+----
+NaN NaN -Infinity
+
+query R
+SELECT greatest('NaN'::float, 1::float);
+----
+NaN
+
+# -0.0 equals 0.0 and the first of equal values wins. The sign of a zero shows
in its string
+# form, which starts with '-' only for -0.0.
+query BB
+SELECT CAST(greatest('-0.0'::double, 0.0::double) AS STRING) LIKE '-%',
CAST(greatest(0.0::double, '-0.0'::double) AS STRING) LIKE '-%';
+----
+true false
+
+# Decimals
+query R
+SELECT greatest(1.5::decimal(2,1), 2::int);
+----
+2.0
+
+query T
+SELECT arrow_typeof(greatest(1.5::decimal(2,1), 2::int));
+----
+Decimal128(11, 1)
+
+query T
+SELECT arrow_typeof(greatest(1::decimal(5,2), 1::bigint));
+----
+Decimal128(22, 2)
+
+query T
+SELECT arrow_typeof(greatest(1::decimal(38,10), 1::decimal(38,0)));
+----
+Decimal128(38, 0)
+
+# The fractional digits that do not fit are rounded away
+query R
+SELECT greatest(1.5::decimal(38,10), 1::decimal(38,0));
+----
+2
+
+query R
+SELECT greatest(1::decimal(38,10),
'12345678901234567890123456789'::decimal(38,0));
+----
+12345678901234567890123456789
+
+# A decimal and a DOUBLE widen to DOUBLE
+query R
+SELECT greatest(1.5::decimal(2,1), 2.5::double);
+----
+2.5
+
+query T
+SELECT arrow_typeof(greatest(1.5::decimal(2,1), 2.5::double));
+----
+Float64
+
+# Strings
+query TTT
+SELECT greatest('abc', 'abd', 'ab'), greatest('a', 'b', NULL), greatest('',
'a');
+----
+abd b a
+
+# Booleans
+query B
+SELECT greatest(true, false);
+----
+true
+
+# Binary
+query ?
+SELECT greatest(X'01', X'0102');
+----
+0102
+
+# Dates and timestamps; a date and a timestamp widen to the timestamp
+query D
+SELECT greatest(DATE '2024-01-01', DATE '2023-12-31', NULL::date);
+----
+2024-01-01
+
+query P
+SELECT greatest(DATE '2024-01-01', TIMESTAMP '2023-01-01 10:00:00');
+----
+2024-01-01T00:00:00
+
+query P
+SELECT greatest(TIMESTAMP '2024-01-01 10:00:00', TIMESTAMP '2024-01-01
10:00:01');
+----
+2024-01-01T10:00:01
+
+# Arrays compare element by element and then by length, a NULL element is
smaller than any
+# value, and elements widen
+query ??
+SELECT greatest(array(1, 2), array(1, 2, 3)), greatest(array(1, NULL),
array(1, 0));
+----
+[1, 2, 3] [1, 0]
+
+query ?
+SELECT greatest(array(1), array(1.5::double));
+----
+[1.5]
+
+query ?
+SELECT greatest(array('NaN'::double), array(1::double));
+----
+[NaN]
+
+# Structs compare field by field
+query ?
+SELECT greatest(named_struct('a', 1, 'b', 2), named_struct('a', 1, 'b', 3));
+----
+{a: 1, b: 3}
+
+# Columns
+statement ok
+CREATE TABLE t AS VALUES
+ (1::int, 2::int, 'NaN'::double, 1::double, 'b'),
+ (5::int, NULL::int, -1::double, NULL::double, 'a'),
+ (NULL::int, NULL::int, NULL::double, NULL::double, NULL),
+ (NULL::int, 7::int, 'Infinity'::double, 'NaN'::double, 'c');
+
+query IIRRT
+SELECT greatest(column1, column2), greatest(column1, 3::int, column2),
greatest(column3, column4), greatest(column3, 0::double), greatest(column5,
'b') FROM t;
Review Comment:
Could we add queries where the first argument is a constant and a later one
is a column, such as `greatest(3::int, column1)` and `greatest(NULL::int,
column1)`, here and in `least.slt`? The first argument takes its own path in
`invoke`. It is broadcast with `to_array`, and a NULL first argument isn't
skipped the way later NULL constants are. No slt query or unit test puts a
constant first.
##########
crates/spark/src/function/math/greatest_least.rs:
##########
@@ -0,0 +1,1173 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! Spark-compatible `greatest` and `least`.
+//!
+//! Spark evaluates the arguments in order, skips NULLs, and replaces its
running result only
+//! with a value that is strictly greater (or smaller) in its SQL ordering, so
the result is NULL
+//! only when every argument is NULL and the first of equal arguments wins.
The SQL ordering
+//! (`SQLOrderingUtil.compareDoubles`) treats `-0.0` as equal to `0.0` and NaN
as greater than
+//! every other value, at any depth of arrays and structs, and orders a NULL
inside an array or
+//! struct before every other value.
+//!
+//! DataFusion's built-in `greatest` and `least` differ: they compare floats
by IEEE 754 total
+//! order (so `-0.0 < 0.0`, and a NaN with the sign bit set is smaller than
`-Infinity`), let a
+//! later argument win a tie, compare the constant arguments before the
columns, accept a single
+//! argument, order a NULL inside an array or struct after every other value
in `least`, and
+//! coerce their arguments with DataFusion's comparison coercion, which for
example compares a
+//! number with a string. Spark rejects that.
+
+use crate::function::array::float_normalize::{contains_float, normalize_array};
+use arrow::array::{
+ Array, ArrayRef, BooleanArray, Datum, Scalar, make_comparator,
new_null_array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::{gt, lt};
+use arrow::compute::kernels::zip::zip;
+use arrow::compute::{SortOptions, cast};
+use arrow::datatypes::{DataType, Field, FieldRef, Fields, TimeUnit};
+use datafusion_common::config::ConfigOptions;
+use datafusion_common::{DataFusionError, Result, ScalarValue, plan_err};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs,
ScalarUDF,
+ ScalarUDFImpl, Signature, Volatility,
+};
+use datafusion_macros::user_doc;
+use std::sync::Arc;
+
+/// Spark-compatible `greatest` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#greatest>
+#[user_doc(
+ doc_section(label = "Math Functions"),
+ description = "Returns the greatest value of all arguments, skipping
NULLs. The result is \
+ NULL only if every argument is NULL. At least two arguments are
required. The \
+ arguments are cast to a common type as in Spark: numbers widen (an
integer and a FLOAT \
+ widen to DOUBLE with `datafusion.execution.enable_ansi_mode` on and to
FLOAT with it \
+ off), integers and decimals widen to a decimal that holds both, a
decimal and a float \
+ widen to DOUBLE, a date and a timestamp widen to the timestamp, and
arrays and structs \
+ widen element by element. Strings are never promoted: a string and a
number, a string \
+ and a date, or a boolean and a number are an error, as are maps, which
cannot be \
+ ordered. NaN is greater than every other value and `-0.0` equals
`0.0`, also inside \
+ arrays and structs, and of equal values the first argument wins, so \
+ `greatest(-0.0, 0.0)` is `-0.0`. A NULL inside an array or struct is
smaller than \
+ every other value. DataFusion's built-in `greatest` orders floats by
IEEE 754 total \
+ order, lets a later argument win a tie, accepts a single argument, and
compares \
+ numbers with strings.",
+ syntax_example = "greatest(expr1, expr2[, ..., expr_n])",
+ sql_example = r#"```sql
+> SELECT greatest(10, 9, 2, 4, 3);
+10
+> SELECT greatest(1, NULL, 3);
+3
+> SELECT greatest(1.0, 'NaN'::double);
+NaN
+```"#,
+ argument(
+ name = "expr1, expr2, ..., expr_n",
+ description = "Expressions of an orderable type that share a common
type. NULLs are \
+ skipped."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkGreatest {
+ inner: GreatestLeast,
+}
+
+impl Default for SparkGreatest {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkGreatest {
+ pub fn new() -> Self {
+ Self {
+ inner: GreatestLeast::new(true, false),
+ }
+ }
+
+ /// Creates a `greatest` that coerces its arguments as Spark does with
+ /// `datafusion.execution.enable_ansi_mode` set to `enable_ansi_mode`.
+ pub fn new_with_ansi_mode(enable_ansi_mode: bool) -> Self {
+ Self {
+ inner: GreatestLeast::new(true, enable_ansi_mode),
+ }
+ }
+}
+
+/// Spark-compatible `least` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#least>
+#[user_doc(
+ doc_section(label = "Math Functions"),
+ description = "Returns the least value of all arguments, skipping NULLs.
The result is \
+ NULL only if every argument is NULL. At least two arguments are
required. The \
+ arguments are cast to a common type as in `greatest`. NaN is greater
than every other \
+ value and `-0.0` equals `0.0`, also inside arrays and structs, and of
equal values the \
+ first argument wins, so `least(0.0, -0.0)` is `0.0`. A NULL inside an
array or struct \
+ is smaller than every other value, so `least(array(1, NULL), array(1,
0))` is \
+ `[1, NULL]`. DataFusion's built-in `least` orders floats by IEEE 754
total order, lets a \
+ later argument win a tie, orders a NULL inside an array or struct
after every other \
+ value, accepts a single argument, and compares numbers with strings.",
+ syntax_example = "least(expr1, expr2[, ..., expr_n])",
+ sql_example = r#"```sql
+> SELECT least(10, 9, 2, 4, 3);
+2
+> SELECT least(NULL, 2, NULL);
+2
+> SELECT least(1.0, 'NaN'::double);
+1.0
+```"#,
+ argument(
+ name = "expr1, expr2, ..., expr_n",
+ description = "Expressions of an orderable type that share a common
type. NULLs are \
+ skipped."
+ )
+)]
+#[derive(Debug, PartialEq, Eq, Hash)]
+pub struct SparkLeast {
+ inner: GreatestLeast,
+}
+
+impl Default for SparkLeast {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkLeast {
+ pub fn new() -> Self {
+ Self {
+ inner: GreatestLeast::new(false, false),
+ }
+ }
+
+ /// Creates a `least` that coerces its arguments as Spark does with
+ /// `datafusion.execution.enable_ansi_mode` set to `enable_ansi_mode`.
+ pub fn new_with_ansi_mode(enable_ansi_mode: bool) -> Self {
+ Self {
+ inner: GreatestLeast::new(false, enable_ansi_mode),
+ }
+ }
+}
+
+macro_rules! impl_greatest_least_udf {
+ ($udf:ident) => {
+ impl ScalarUDFImpl for $udf {
+ fn name(&self) -> &str {
+ self.inner.name()
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.inner.signature
+ }
+
+ fn coerce_types(&self, arg_types: &[DataType]) ->
Result<Vec<DataType>> {
+ self.inner.coerce_types(arg_types)
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ self.inner.check_arg_count(arg_types.len(), true)?;
+ Ok(arg_types[0].clone())
+ }
+
+ fn return_field_from_args(&self, args: ReturnFieldArgs) ->
Result<FieldRef> {
+ self.inner.check_arg_count(args.arg_fields.len(), true)?;
+ // As in Spark, the result is nullable only if every argument
is.
+ let nullable = args.arg_fields.iter().all(|f| f.is_nullable());
+ Ok(Arc::new(Field::new(
+ self.name(),
+ args.arg_fields[0].data_type().clone(),
+ nullable,
+ )))
+ }
+
+ fn with_updated_config(&self, config: &ConfigOptions) ->
Option<ScalarUDF> {
+ Some(ScalarUDF::from(Self::new_with_ansi_mode(
+ config.execution.enable_ansi_mode,
+ )))
+ }
+
+ fn invoke_with_args(
+ &self,
+ args: ScalarFunctionArgs,
+ ) -> Result<ColumnarValue> {
+ self.inner.invoke(args)
+ }
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.doc()
+ }
+ }
+ };
+}
+
+impl_greatest_least_udf!(SparkGreatest);
+impl_greatest_least_udf!(SparkLeast);
+
+/// The implementation shared by [`SparkGreatest`] and [`SparkLeast`].
+#[derive(Debug, PartialEq, Eq, Hash)]
+struct GreatestLeast {
+ signature: Signature,
+ greatest: bool,
+ /// Whether to coerce as Spark's `AnsiTypeCoercion` does, rather than
`TypeCoercion`.
+ ansi_mode: bool,
+}
+
+/// Spark orders a NULL inside an array or struct before every other value.
+const SORT_OPTIONS: SortOptions = SortOptions {
+ descending: false,
+ nulls_first: true,
+};
+
+impl GreatestLeast {
+ fn new(greatest: bool, ansi_mode: bool) -> Self {
+ Self {
+ signature: Signature::user_defined(Volatility::Immutable),
+ greatest,
+ ansi_mode,
+ }
+ }
+
+ fn name(&self) -> &'static str {
+ if self.greatest { "greatest" } else { "least" }
+ }
+
+ fn check_arg_count(&self, count: usize, planning: bool) -> Result<()> {
+ if count > 1 {
+ return Ok(());
+ }
+ let message = format!(
+ "[WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `{}` requires > 1
parameters but the \
+ actual number is {count}. Please, refer to \
+ 'https://spark.apache.org/docs/latest/sql-ref-functions.html' for
a fix.",
+ self.name()
+ );
+ Err(if planning {
+ DataFusionError::Plan(message)
+ } else {
+ DataFusionError::Execution(message)
+ })
+ }
+
+ /// Spark's `TypeCoercion` rule for `Greatest` and `Least`: every argument
is cast to
+ /// `findWiderTypeWithoutStringPromotion` of all the argument types, and
the result must be
+ /// orderable.
+ fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ self.check_arg_count(arg_types.len(), true)?;
+ let common = arg_types
+ .iter()
+ .skip(1)
+ .try_fold(arg_types[0].clone(), |common, data_type| {
+ wider_type(&common, data_type, self.ansi_mode)
+ });
+ let Some(common) = common else {
+ return plan_err!(
+ "[DATATYPE_MISMATCH.DATA_DIFF_TYPES] Input to `{}` should all
be the same \
+ type, but it's [{}]",
+ self.name(),
+ arg_types
+ .iter()
+ .map(|t| format!("\"{}\"", spark_sql_type(t)))
+ .collect::<Vec<_>>()
+ .join(", ")
+ );
+ };
+ if !is_orderable(&common) {
+ return plan_err!(
+ "[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] The `{}` does not
support ordering \
+ on type \"{}\"",
+ self.name(),
+ spark_sql_type(&common)
+ );
+ }
+ Ok(vec![common; arg_types.len()])
+ }
+
+ fn invoke(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ self.check_arg_count(args.args.len(), false)?;
+ let data_type = args.return_field.data_type();
+ let all_scalars = args
+ .args
+ .iter()
+ .all(|arg| matches!(arg, ColumnarValue::Scalar(_)));
+ let rows = if all_scalars { 1 } else { args.number_rows };
+
+ if data_type.is_null() {
+ return Ok(if all_scalars {
+ ColumnarValue::Scalar(ScalarValue::Null)
+ } else {
+ ColumnarValue::Array(new_null_array(&DataType::Null, rows))
+ });
+ }
+
+ // The planner casts every argument to the common type, but a field
name or the
+ // nullability of a nested field can still differ when the function is
called directly,
+ // and `zip` needs identical types.
+ let to_array = |arg: &ColumnarValue, rows: usize| -> Result<ArrayRef> {
+ let array = arg.to_array(rows)?;
+ Ok(if array.data_type() == data_type {
+ array
+ } else {
+ cast(&array, data_type)?
+ })
+ };
+ // Floats are compared in a copy where `-0.0` is `0.0` and every NaN
is the canonical
+ // NaN, where Arrow's total order agrees with Spark's. The result
keeps the original.
+ let normalize = contains_float(data_type);
+ let comparable = |array: &ArrayRef| -> Result<ArrayRef> {
+ if normalize {
+ normalize_array(array)
+ } else {
+ Ok(Arc::clone(array))
+ }
+ };
+
+ let mut result = to_array(&args.args[0], rows)?;
+ let mut result_cmp = comparable(&result)?;
+ for arg in &args.args[1..] {
+ let candidate = match arg {
+ // A NULL never replaces the result.
+ ColumnarValue::Scalar(scalar) if scalar.is_null() => continue,
+ ColumnarValue::Scalar(_) => {
+ let value = to_array(arg, 1)?;
+ let cmp = comparable(&value)?;
+ Candidate {
+ value,
+ cmp,
+ scalar: true,
+ }
+ }
+ ColumnarValue::Array(_) => {
+ let value = to_array(arg, rows)?;
+ let cmp = comparable(&value)?;
+ Candidate {
+ value,
+ cmp,
+ scalar: false,
+ }
+ }
+ };
+ let take = self.replaces(&candidate, &result, &result_cmp)?;
+ let take = BooleanArray::new(take, None);
+ let new_result = zip(&take, candidate.value_datum().as_ref(),
&result)?;
+ result_cmp = if normalize {
+ zip(&take, candidate.cmp_datum().as_ref(), &result_cmp)?
+ } else {
+ Arc::clone(&new_result)
+ };
+ result = new_result;
+ }
+
+ if all_scalars {
+ Ok(ColumnarValue::Scalar(ScalarValue::try_from_array(
+ &result, 0,
+ )?))
+ } else {
+ Ok(ColumnarValue::Array(result))
+ }
+ }
+
+ /// For each row, whether `candidate` replaces the current result: it is
not NULL, and
+ /// either the result is NULL or the candidate ranks strictly before it.
+ fn replaces(
+ &self,
+ candidate: &Candidate,
+ current: &ArrayRef,
+ current_cmp: &ArrayRef,
+ ) -> Result<BooleanBuffer> {
+ let len = current.len();
+ let current_nulls = current.logical_nulls();
+ let candidate_nulls = if candidate.scalar {
+ // A NULL constant is skipped before this is called.
+ None
+ } else {
+ candidate.value.logical_nulls()
+ };
+
+ if !current.data_type().is_nested() {
+ let candidate_cmp = candidate.cmp_datum();
+ let ordered = if self.greatest {
+ gt(candidate_cmp.as_ref(), current_cmp)?
+ } else {
+ lt(candidate_cmp.as_ref(), current_cmp)?
+ };
+ // Where either side is NULL the comparison's value is arbitrary,
but such rows are
+ // decided by the null masks below.
+ let mut take = ordered.values().clone();
+ if let Some(nulls) = ¤t_nulls {
+ take = &take | &!nulls.inner();
+ }
+ if let Some(nulls) = &candidate_nulls {
+ take = &take & nulls.inner();
+ }
+ return Ok(take);
+ }
+
+ let compare =
+ make_comparator(candidate.cmp.as_ref(), current_cmp.as_ref(),
SORT_OPTIONS)?;
+ let scalar = candidate.scalar;
+ Ok(BooleanBuffer::collect_bool(len, |row| {
+ let candidate_row = if scalar { 0 } else { row };
+ if candidate_nulls
+ .as_ref()
+ .is_some_and(|nulls| nulls.is_null(candidate_row))
+ {
+ return false;
+ }
+ if current_nulls
+ .as_ref()
+ .is_some_and(|nulls| nulls.is_null(row))
+ {
+ return true;
+ }
+ let ordering = compare(candidate_row, row);
+ if self.greatest {
+ ordering.is_gt()
+ } else {
+ ordering.is_lt()
+ }
+ }))
+ }
+}
+
+/// An argument after the first, with its values as they are and as they are
compared. A
+/// constant is a one-row array.
+struct Candidate {
+ value: ArrayRef,
+ cmp: ArrayRef,
+ scalar: bool,
+}
+
+impl Candidate {
+ fn datum(&self, array: &ArrayRef) -> Box<dyn Datum> {
+ if self.scalar {
+ Box::new(Scalar::new(Arc::clone(array)))
+ } else {
+ Box::new(Arc::clone(array))
+ }
+ }
+
+ fn value_datum(&self) -> Box<dyn Datum> {
+ self.datum(&self.value)
+ }
+
+ fn cmp_datum(&self) -> Box<dyn Datum> {
+ self.datum(&self.cmp)
+ }
+}
+
+/// Spark's `findWiderTypeWithoutStringPromotionForTwo`: the tightest common
type, then a
+/// decimal wide enough for both, then the same for the elements of arrays and
the fields of
+/// structs. Strings are never promoted. `ansi_mode` picks between
`AnsiTypeCoercion` and
+/// `TypeCoercion`, which differ only in that ANSI widens an integer and a
FLOAT to DOUBLE.
+fn wider_type(left: &DataType, right: &DataType, ansi_mode: bool) ->
Option<DataType> {
+ tightest_common_type(left, right, ansi_mode)
+ .or_else(|| wider_decimal_type(left, right))
+ .or_else(|| complex_type(left, right, ansi_mode))
+}
+
+/// Spark's `findTightestCommonType`, on the Arrow types that Spark's types
map to. Arrays and
+/// structs are handled by [`complex_type`] with [`wider_type`], which gives
the same result.
+fn tightest_common_type(
+ left: &DataType,
+ right: &DataType,
+ ansi_mode: bool,
+) -> Option<DataType> {
+ use DataType::*;
+ if left == right {
+ return Some(left.clone());
+ }
+ if let (Some(l), Some(r)) = (numeric_precedence(left),
numeric_precedence(right)) {
+ let wider = if l >= r { left } else { right };
+ // With ANSI mode on, an integer and a FLOAT widen to DOUBLE, which
holds every INT
+ // exactly.
+ return Some(if ansi_mode && *wider == Float32 {
+ Float64
+ } else {
+ wider.clone()
+ });
+ }
+ match (left, right) {
+ (Null, other) | (other, Null) => Some(other.clone()),
+ // Spark has one string type and one binary type.
+ (l, r) if is_string(l) && is_string(r) => Some(wider_physical_type(l,
r)),
+ (l, r) if is_binary(l) && is_binary(r) => Some(wider_physical_type(l,
r)),
+ // The decimal must hold every value of the integral type.
+ (int, Decimal128(p, s)) | (Decimal128(p, s), int) => {
+ let digits = integral_digits(int)?;
+ (*p as i16 - *s as i16 >= digits).then_some(Decimal128(*p, *s))
+ }
+ (Date32, ts @ Timestamp(_, _)) | (ts @ Timestamp(_, _), Date32) => {
+ Some(ts.clone())
+ }
+ // TIMESTAMP_NTZ and TIMESTAMP widen to TIMESTAMP.
+ (Timestamp(lu, ltz), Timestamp(ru, rtz)) => Some(Timestamp(
+ finer_unit(*lu, *ru),
+ ltz.clone().or_else(|| rtz.clone()),
+ )),
+ _ => None,
+ }
+}
Review Comment:
`tightest_common_type`, `numeric_precedence`, `integral_digits`,
`finer_unit`, `is_string`, and `is_binary` are copies of the helpers in
[`array_position.rs`](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/spark/src/function/array/array_position.rs#L203-L349),
with an ANSI flag added. The copies already differ: for two string layouts,
`array_position` picks the left type, and this one picks the wider layout. #330
reuses `array_position`'s `tightest_common_type` and `contains_map` by making
them `pub(super)`. #70 adds a private `spark_type_name` that renders the same
Spark type names as `spark_sql_type` here.
Could we move these into one module under `function/` that `array` and
`math` both use? That module would hold `tightest_common_type` with the ANSI
flag as a parameter, the orderability check, and the Spark type names. Spark's
`ArrayPosition` calls the non-ANSI
[`TypeCoercion.findTightestCommonType`](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L2510-L2513),
so it would pass `false`. `float_normalize` could move there too, instead of
becoming `pub(crate)` under `array`. Two fixes would then land once for every
caller: rejecting `Interval(MonthDayNano)` as unorderable (see the comment on
line 602), and widening map keys and values in `tightest_common_type`, whose
`Map` arm on `main` [requires identical entry
types](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/spark/src/function/array/array_position.rs#L2
67-L273) where Spark [widens
both](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala#L149-L163).
#70, #330, and #331 would use the module too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]