james-willis commented on code in PR #27143:
URL: https://github.com/apache/datafusion/pull/27143#discussion_r4605033618
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -685,11 +688,35 @@ impl ExprSchemable for Expr {
))),
}
}
+ Expr::Case(case) => {
+ let fields = case
+ .when_then_expr
+ .iter()
+ .map(|(_, then)| then)
+ .chain(case.else_expr.iter())
+ .map(|expr| expr.to_field(schema).map(|(_, field)| field))
+ .collect::<Result<Vec<_>>>()?;
+ let metadata =
+ coerced_fields_metadata(fields.iter().map(|f|
f.as_ref()))?;
+ // Same as `get_type`: the first non-NULL branch type. Reading
it from
+ // `fields` avoids a second walk of the branches, which
multiplies with
+ // the depth of nested CASE and function calls.
+ let data_type = fields
+ .iter()
+ .map(|f| f.data_type())
+ .find(|t| !t.is_null())
+ .cloned()
+ .unwrap_or(DataType::Null);
+ Ok(Arc::new(metadata.add_to_field(Field::new(
+ &schema_name,
+ data_type,
+ self.nullable(schema)?,
Review Comment:
Current main is exponential on this shape too. I bisected between this PR’s
merge base and main and landed on
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139),
merged Nov 6. It changed CASE get_type to read every branch, and main’s
to_field calls get_type and then nullable(), so the ELSE is read twice per
level. At 18 levels, the commit before
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139)
plans your query in 0.06 s and
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139)
takes 22 s.
While looking at
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139), I
noticed it also conflicts with this PR. Merged as-is, our to_field would undo
it: DESCRIBE SELECT CASE WHEN false THEN 1 ELSE 2.5 END reports Int64 instead
of Float64. A test already on main catches this
(union_infers_types_before_analysis), but this PR’s CI skips the full test
jobs, so it would first have failed in the merge queue.
So I merged main and moved
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139)’s
type rule into one helper that get_type and our to_field both call, so each
branch is still read only once. Your case_nullable change is in too.
I also timed 17 nesting shapes, including this one. Against current main, 10
take the same time, and 6 that are exponential on main are linear here. The
last one, COALESCE nested in its first argument, was about 1.4x slower than
main because the constant-WHEN simplification read the branches several times.
It now reads each once and is about 1.06x main. That shape is still exponential
on both, because the simplifier rewrites COALESCE to a CASE that repeats the
argument.
Pushed in afd47cb39, and the description’s performance section has the full
numbers.
I didn’t appreciate how complex this part of the code base was when I
started working on this 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]