mustafasrepo commented on code in PR #9690:
URL: https://github.com/apache/arrow-datafusion/pull/9690#discussion_r1531850260
##########
datafusion/optimizer/src/optimize_projections.rs:
##########
@@ -162,14 +163,37 @@ fn optimize_projections(
.map(|input|
((0..input.schema().fields().len()).collect_vec(), false))
.collect::<Vec<_>>()
}
+ LogicalPlan::Extension(extension) => {
+ let children = extension.node.inputs();
+ if children.len() != 1 {
+ // TODO: Add support for `LogicalPlan::Extension` with multi
children.
+ return Ok(None);
+ }
+ // has single child
+ let exprs = plan.expressions();
+ let child = children[0];
+ let node_schema = extension.node.schema();
+ let child_schema = child.schema();
+ if let Some(parent_required_indices_mapped) =
Review Comment:
> I am thinking of an API like
> ```
> trait UserDefinedLogicalNode {
> ...
> /// If only the specified indices of the output of node are needed
> /// by the consumer, optionally returns a new UserDefinedNode that
> /// computes only those indices, in the specified order.
> ///
> /// Return Ok(None), the default, if no such pushdown is possible
> ///
> /// This is used to implement projection pushdown for user defined nodes
> fn try_project(&self, indices: &[usize]) -> Result<Option<Arc<dyn Self>>> {
> Ok(None)
> }
> ```
I think, implementers of the `try_project` needs to insert
`LogicalPlan::Projection` on top of their input to pushdown projection. This
might not be the case, if I am missing something. If that is the case, this
might be cumbersome for the users. What about an API something like
```
/// Returns the necessary expressions at each child to calculate argument.
/// assuming exprs refers to valid fields at the output of the operator.
/// Return Ok(None), the default, Cannot determine necessary expressions at
the children to calculate given exprs.
fn necessary_children_exprs(&self, exprs: &[Expr]) -> Option<Vec<Vec<Expr>>>
{
Ok(None)
}
```
with an API like above, we can handle projection insertion outside the trait
method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
