Tpt commented on code in PR #18993:
URL: https://github.com/apache/datafusion/pull/18993#discussion_r2615534158
##########
datafusion/physical-plan/src/work_table.rs:
##########
@@ -196,11 +206,22 @@ impl ExecutionPlan for WorkTableExec {
0,
"WorkTableExec got an invalid partition {partition} (expected 0)"
);
- let batch = self.work_table.take()?;
+ let ReservedBatches {
+ mut batches,
+ reservation,
+ } = self.work_table.take()?;
+ if let Some(projection) = &self.projection {
+ // We apply the projection
+ // TODO: it would be better to apply it as soon as possible and
not only here
+ // TODO: an aggressive projection makes the memory reservation
smaller, even if we do not edit it
Review Comment:
`WorkTableExec` is used for recursive CTEs. They are evaluated by executing
the same query plan multiple time. The output of the previous iteration is
injected as the value of the "work table" in the next iteration.
Here "as soon as possible" means that when we execute the previous iteration
we don't need to save all the columns of the output of this iteration to feed
them into the "work table" of the next iteration but only the columns that are
actually needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
