Re: [PR] Add PhysicalExpr optimizer and cast unwrapping [datafusion]
alamb merged PR #16530: URL: https://github.com/apache/datafusion/pull/16530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Add PhysicalExpr optimizer and cast unwrapping [datafusion]
adriangb commented on PR #16530: URL: https://github.com/apache/datafusion/pull/16530#issuecomment-3029145280 Thanks @alamb! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Add PhysicalExpr optimizer and cast unwrapping [datafusion]
adriangb commented on PR #16530: URL: https://github.com/apache/datafusion/pull/16530#issuecomment-3021857622 @alamb thank you for the review. I think I've addressed all of the feedback 🙏🏻 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Add PhysicalExpr optimizer and cast unwrapping [datafusion]
adriangb commented on code in PR #16530:
URL: https://github.com/apache/datafusion/pull/16530#discussion_r2176469355
##
datafusion/physical-expr/src/simplifier/unwrap_cast.rs:
##
@@ -0,0 +1,621 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Unwrap casts in binary comparisons for physical expressions
+//!
+//! This module provides optimization for physical expressions similar to the
logical
+//! optimizer's unwrap_cast module. It attempts to remove casts from
comparisons to
+//! literals by applying the casts to the literals if possible.
+//!
+//! The optimization improves performance by:
+//! 1. Reducing runtime cast operations on column data
+//! 2. Enabling better predicate pushdown opportunities
+//! 3. Optimizing filter expressions in physical plans
+//!
+//! # Example
+//!
+//! Physical expression: `cast(column as INT64) > INT64(10)`
+//! Optimized to: `column > INT32(10)` (assuming column is INT32)
+
+use std::sync::Arc;
+
+use arrow::datatypes::{DataType, Schema};
+use datafusion_common::{
+tree_node::{Transformed, TreeNode},
+Result, ScalarValue,
+};
+use datafusion_expr::Operator;
+use datafusion_expr_common::casts::try_cast_literal_to_type;
+
+use crate::expressions::{lit, BinaryExpr, CastExpr, Literal, TryCastExpr};
+use crate::PhysicalExpr;
+
+/// Attempts to unwrap casts in comparison expressions.
+pub(crate) fn unwrap_cast_in_comparison(
+expr: Arc,
+schema: &Schema,
+) -> Result>> {
+expr.transform_down(|e| {
+if let Some(binary) = e.as_any().downcast_ref::() {
+if let Some(unwrapped) = try_unwrap_cast_binary(binary, schema)? {
+return Ok(Transformed::yes(unwrapped));
+}
+}
+Ok(Transformed::no(e))
+})
+}
+
+/// Try to unwrap casts in binary expressions
+fn try_unwrap_cast_binary(
+binary: &BinaryExpr,
+schema: &Schema,
+) -> Result>> {
+// Case 1: cast(left_expr) op literal
+if let (Some((inner_expr, _cast_type)), Some(literal)) = (
+extract_cast_info(binary.left()),
+binary.right().as_any().downcast_ref::(),
+) {
+if let Some(unwrapped) = try_unwrap_cast_comparison(
+Arc::clone(inner_expr),
+literal.value(),
+*binary.op(),
+schema,
+)? {
+return Ok(Some(unwrapped));
+}
+}
+
+// Case 2: literal op cast(right_expr)
+if let (Some(literal), Some((inner_expr, _cast_type))) = (
+binary.left().as_any().downcast_ref::(),
+extract_cast_info(binary.right()),
+) {
+// For literal op cast(expr), we need to swap the operator
+let swapped_op = swap_operator(*binary.op());
+if let Some(unwrapped) = try_unwrap_cast_comparison(
+Arc::clone(inner_expr),
+literal.value(),
+swapped_op,
+schema,
+)? {
+// unwrapped is already the correct expression: inner_expr
swapped_op casted_literal
+return Ok(Some(unwrapped));
+}
+}
+
+Ok(None)
+}
+
+/// Extract cast information from a physical expression
+fn extract_cast_info(
+expr: &Arc,
+) -> Option<(&Arc, &DataType)> {
+if let Some(cast) = expr.as_any().downcast_ref::() {
+Some((cast.expr(), cast.cast_type()))
+} else if let Some(try_cast) = expr.as_any().downcast_ref::()
{
+Some((try_cast.expr(), try_cast.cast_type()))
+} else {
+None
+}
+}
+
+/// Try to unwrap a cast in comparison by moving the cast to the literal
+fn try_unwrap_cast_comparison(
+inner_expr: Arc,
+literal_value: &ScalarValue,
+op: Operator,
+schema: &Schema,
+) -> Result>> {
+// Get the data type of the inner expression
+let inner_type = inner_expr.data_type(schema)?;
+
+// Try to cast the literal to the inner expression's type
+if let Some(casted_literal) = try_cast_literal_to_type(literal_value,
&inner_type) {
+let literal_expr = lit(casted_literal);
+let binary_expr = BinaryExpr::new(inner_expr, op, literal_expr);
+return Ok(Some(Arc::new(binary_expr)));
+}
+
+Ok(None)
+}
+
+/// Swap comparison operators for right-side cast unwrapping
+fn swap_opera
Re: [PR] Add PhysicalExpr optimizer and cast unwrapping [datafusion]
adriangb commented on code in PR #16530:
URL: https://github.com/apache/datafusion/pull/16530#discussion_r2176459049
##
datafusion/pruning/src/pruning_predicate.rs:
##
@@ -468,6 +469,9 @@ impl PruningPredicate {
&mut required_columns,
&unhandled_hook,
);
+let predicate_schema = required_columns.schema();
+let predicate_expr =
+
PhysicalExprSimplifier::new(&predicate_schema).simplify(predicate_expr)?;
Review Comment:
```suggestion
// Simplify the newly created predicate to get rid of redundant
casts, comparisons, etc.
let predicate_expr =
PhysicalExprSimplifier::new(&predicate_schema).simplify(predicate_expr)?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Add PhysicalExpr optimizer and cast unwrapping [datafusion]
adriangb commented on code in PR #16530:
URL: https://github.com/apache/datafusion/pull/16530#discussion_r2176447484
##
datafusion/expr-common/src/casts.rs:
##
@@ -0,0 +1,1227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for casting scalar literals to different data types
+//!
+//! This module contains functions for casting ScalarValue literals
+//! to different data types, originally extracted from the optimizer's
+//! unwrap_cast module to be shared between logical and physical layers.
+
+use std::cmp::Ordering;
+
+use arrow::datatypes::{
+DataType, TimeUnit, MAX_DECIMAL128_FOR_EACH_PRECISION,
+MIN_DECIMAL128_FOR_EACH_PRECISION,
+};
+use arrow::temporal_conversions::{MICROSECONDS, MILLISECONDS, NANOSECONDS};
+use datafusion_common::ScalarValue;
+
+/// Convert a literal value from one data type to another
+pub fn try_cast_literal_to_type(
Review Comment:
Makes sense! There's already a `ScalarValue::cast_to` which may do what we
want but I'm hesitant to change it in this PR since that might have unintended
consequences if the implementations differ. I'd rather do that in it's own PR.
I opened https://github.com/apache/datafusion/issues/16635 to track.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Add PhysicalExpr optimizer and cast unwrapping [datafusion]
alamb commented on code in PR #16530:
URL: https://github.com/apache/datafusion/pull/16530#discussion_r2176033909
##
datafusion/datasource-parquet/src/opener.rs:
##
@@ -233,7 +234,13 @@ impl FileOpener for ParquetOpener {
)
.rewrite(p)
.map_err(ArrowError::from)
+.map(|p| {
+PhysicalExprSimplifier::new(&physical_file_schema)
Review Comment:
I think it is worth worth mentioning here that we are optimizing the
physical expression here as it may have been altered due to differences in
table and file schema
Perhaps something like this
```suggestion
// After rewriting to the file schema, further
simplifications may
// be possible.
PhysicalExprSimplifier::new(&physical_file_schema)
```
##
datafusion/expr-common/src/casts.rs:
##
@@ -0,0 +1,1227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for casting scalar literals to different data types
Review Comment:
👍
##
datafusion/physical-expr/src/simplifier/unwrap_cast.rs:
##
@@ -0,0 +1,621 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Unwrap casts in binary comparisons for physical expressions
+//!
+//! This module provides optimization for physical expressions similar to the
logical
+//! optimizer's unwrap_cast module. It attempts to remove casts from
comparisons to
+//! literals by applying the casts to the literals if possible.
+//!
+//! The optimization improves performance by:
+//! 1. Reducing runtime cast operations on column data
+//! 2. Enabling better predicate pushdown opportunities
+//! 3. Optimizing filter expressions in physical plans
+//!
+//! # Example
+//!
+//! Physical expression: `cast(column as INT64) > INT64(10)`
+//! Optimized to: `column > INT32(10)` (assuming column is INT32)
+
+use std::sync::Arc;
+
+use arrow::datatypes::{DataType, Schema};
+use datafusion_common::{
+tree_node::{Transformed, TreeNode},
+Result, ScalarValue,
+};
+use datafusion_expr::Operator;
+use datafusion_expr_common::casts::try_cast_literal_to_type;
+
+use crate::expressions::{lit, BinaryExpr, CastExpr, Literal, TryCastExpr};
+use crate::PhysicalExpr;
+
+/// Attempts to unwrap casts in comparison expressions.
+pub(crate) fn unwrap_cast_in_comparison(
+expr: Arc,
+schema: &Schema,
+) -> Result>> {
+expr.transform_down(|e| {
+if let Some(binary) = e.as_any().downcast_ref::() {
+if let Some(unwrapped) = try_unwrap_cast_binary(binary, schema)? {
+return Ok(Transformed::yes(unwrapped));
+}
+}
+Ok(Transformed::no(e))
+})
+}
+
+/// Try to unwrap casts in binary expressions
+fn try_unwrap_cast_binary(
+binary: &BinaryExpr,
+schema: &Schema,
+) -> Result>> {
+// Case 1: cast(left_expr) op literal
+if let (Some((inner_expr, _cast_type)), Some(literal)) = (
+extract_cast_info(binary.left()),
+binary.right().as_any().downcast_ref::(),
+) {
+if let Some(unwrapped) = try_unwrap_cast_comparison(
+Arc::clone(inner_expr),
+literal.value(),
+*binary.op(),
+schema,
+)? {
+return Ok(Some(unwrapped));
+}
+}
+
+// Case 2: literal op cast(right_expr)
+if let (Some(li
Re: [PR] Add PhysicalExpr optimizer and cast unwrapping [datafusion]
alamb commented on PR #16530: URL: https://github.com/apache/datafusion/pull/16530#issuecomment-3001900336 I will try and find time to review this tomorrow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
