adriangb commented on code in PR #18993:
URL: https://github.com/apache/datafusion/pull/18993#discussion_r2615519750
##########
datafusion/physical-plan/src/work_table.rs:
##########
@@ -196,11 +206,22 @@ impl ExecutionPlan for WorkTableExec {
0,
"WorkTableExec got an invalid partition {partition} (expected 0)"
);
- let batch = self.work_table.take()?;
+ let ReservedBatches {
+ mut batches,
+ reservation,
+ } = self.work_table.take()?;
+ if let Some(projection) = &self.projection {
+ // We apply the projection
+ // TODO: it would be better to apply it as soon as possible and
not only here
+ // TODO: an aggressive projection makes the memory reservation
smaller, even if we do not edit it
Review Comment:
Could you expand on this a bit? Where would "as soon as possible" be? I'm
not familiar with what WorkTableExec is or how it is used 🥲
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
