alamb commented on PR #7775:
URL:
https://github.com/apache/arrow-datafusion/pull/7775#issuecomment-1856322304
> What did turn out to have a _huge_ perf impact on our workloads, was the
asymptotic behavior of the logical plan constructors. Specifically, many
methods in `LogicalPlanBuilder`, e.g. `project` and `join`, perform input
sanitization which is (at least) O(n) in the size of the parent plan(s), and as
a result using `LogicalPlanBuilder` to construct logical plans takes O(n^2)
time in the size of the input query.
Thank you @sadboy this is great feedback. I wonder if we could / should
make "don't error check" type constructors for this kind of optimization
Perhaps something like
```rust
impl ProjectionExec {
// Creates a new projection exec without any error checking. Use this only
// if you know the correct arguments
pub fn try_new_unchecked(
expr: Vec<(Arc<dyn PhysicalExpr>, String)>,
input: Arc<dyn ExecutionPlan>
) -> Result<ProjectionExec, DataFusionError> {
...
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
