adriangb commented on issue #17334:
URL: https://github.com/apache/datafusion/issues/17334#issuecomment-3570503658
I was thinking about this a bit today. The thought I had was of some
structure like (note this is a half baked idea):
```rust
trait ExecutionPlan {
async fn spill(&self, request_bytes: usize) -> Result<Option<usize>>> {
Ok(None) }
// existing stuff
}
struct CooperativeMemoryPool {
plan: Arc<dyn ExecutionPlan>
pool: Arc<dyn MemoryPool>
}
impl MemoryPool for CooperativeMemoryPool {
async fn try_grow(&self, reservation: &MemoryReservation, additional:
usize) -> Result<()> {
// Maybe some sort of re-entrant lock here?
if let Ok(()) = self.pool.try_reserve(reservation, additional) {
return Ok(())
}
// Do a top-down traversal through the plan asking each node to
spill the required memory
// until we've spilled enough or we realize we can't spill in which
case error
// Now try again?
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
