andygrove commented on code in PR #336:
URL: https://github.com/apache/datafusion-spark/pull/336#discussion_r4588485613
##########
crates/spark/src/function/datetime/date_part.rs:
##########
@@ -140,41 +154,21 @@ impl ScalarUDFImpl for SparkDatePart {
) -> Result<ExprSimplifyResult> {
let [part_expr, date_expr] = take_function_args(self.name(), args)?;
- let part = match part_expr.as_literal() {
- Some(ScalarValue::Utf8(Some(v)))
- | Some(ScalarValue::Utf8View(Some(v)))
- | Some(ScalarValue::LargeUtf8(Some(v))) => {
- let part = v.to_lowercase();
- if !SPARK_FIELDS.iter().any(|g| g.split(' ').any(|f| f ==
part)) {
- let data_type = info.get_data_type(&date_expr)?;
- return plan_err!(
- "Cannot extract `{v}` from {data_type}. SQLSTATE:
42601"
- );
- }
- part
- }
+ let part = match check_field(part_expr.as_literal()) {
+ Ok(ExtractField::Part(part)) => part,
// A NULL field returns NULL, like Spark
- Some(value) if value.is_null() => {
+ Ok(ExtractField::Null) => {
return Ok(ExprSimplifyResult::Simplified(Expr::Literal(
ScalarValue::Int32(None),
None,
)));
}
- _ => {
- return internal_err!(
- "First argument of `DATE_PART` must be non-null scalar
Utf8"
- );
+ Err(e) => {
Review Comment:
`simplify` still turns `FieldError::NotConstant` into a planning error, so
the new runtime check in `invoke_with_args` is never reached for a field that
isn't a literal at planning time. For example, with a prepared statement
`SELECT date_part($1, d) FROM t`, `simplify_expressions` runs before `$1` is
bound and fails with "must be a constant string". Returning
`Ok(ExprSimplifyResult::Original(...))` for `NotConstant` would leave that case
to `invoke_with_args`.
Related: `info.get_data_type(&date_expr)?` runs for `NotConstant` too, even
though that message doesn't use the type. If the type lookup fails, its error
replaces the intended one. Looking up the type only for `FieldError::Invalid`
avoids that.
##########
crates/spark/src/function/datetime/date_part.rs:
##########
@@ -185,3 +179,230 @@ impl ScalarUDFImpl for SparkDatePart {
Ok(ExprSimplifyResult::Simplified(date_part_expr))
}
}
+
+/// A checked `field` argument
+enum ExtractField {
+ /// The name of the field in DataFusion's `date_part`
+ Part(&'static str),
+ Null,
+}
+
+/// Why a `field` argument is rejected
+enum FieldError<'a> {
+ Invalid(&'a str),
+ NotConstant,
+}
+
+impl FieldError<'_> {
+ fn message(&self, source_type: &DataType) -> String {
+ match self {
+ FieldError::Invalid(name) => {
+ format!("Cannot extract `{name}` from {source_type}. SQLSTATE:
42601")
+ }
+ FieldError::NotConstant => {
+ "First argument of `DATE_PART` must be a constant
string".to_string()
+ }
+ }
+ }
+}
+
+/// Checks the `field` argument against Spark's field names. The caller
formats the
+/// error with the source type and picks the error kind.
+fn check_field(
+ field: Option<&ScalarValue>,
+) -> std::result::Result<ExtractField, FieldError<'_>> {
+ match field {
+ Some(ScalarValue::Utf8(Some(name)))
Review Comment:
This only accepts `Utf8`, `Utf8View` and `LargeUtf8`, but the signature
(`logical_string()`) also accepts dictionary- and run-end-encoded strings, so a
constant such as `ScalarValue::Dictionary(Int32, Utf8("year"))` is rejected as
"not constant". `ScalarValue::try_as_str()` handles all of these (it's what
DataFusion's own `date_part` uses for this argument) and returns
`Some(Some(s))` for a string, `Some(None)` for null and `None` for a
non-string, so it could replace this match.
##########
crates/sqllogictest/test_files/datetime/date_part.slt:
##########
@@ -45,13 +45,8 @@ SELECT date_part('QTR'::string, '2000-01-01'::date);
----
1
-query error
+query error Cannot extract `QUARTERS` from Date32. SQLSTATE: 42601
Review Comment:
These invalid-field cases used to assert that the error comes from
`simplify_expressions` at planning time. The new pattern also matches the
execution-time error, so if `simplify` ever stopped validating the field, all
three cases would still pass. Could we keep the full expected output (or at
least match on `Error during planning`) so the planning-time path stays
covered? Also, `.` in `Date32. SQLSTATE` is an unescaped regex wildcard (minor).
##########
crates/spark/src/function/datetime/date_part.rs:
##########
@@ -129,8 +129,22 @@ impl ScalarUDFImpl for SparkDatePart {
self.doc()
}
- fn invoke_with_args(&self, _args: ScalarFunctionArgs) ->
Result<ColumnarValue> {
- internal_err!("spark date_part should have been simplified to standard
date_part")
+ fn invoke_with_args(&self, mut args: ScalarFunctionArgs) ->
Result<ColumnarValue> {
+ let field = match &args.args[0] {
+ ColumnarValue::Scalar(field) => Some(field),
+ ColumnarValue::Array(_) => None,
Review Comment:
Nit: there's no unit test for a field passed as an array (the "must be a
constant string" execution error here), or for an invalid field with a
timestamp source. Worth adding since callers that skip `simplify`, such as
Comet, will hit this branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]