ariel-miculas commented on issue #22526: URL: https://github.com/apache/datafusion/issues/22526#issuecomment-4623942588
> Not sure I follow. Why would this sliced record batch require more memory than what agg released? 1. the memory is reserved before the large record batch is created, so there's no guarantee that the released memory is sufficient for reserving the large record batch (well, a slice of this large record batch, but for memory accounting purposes it doesn't matter) 2. it also depends on how memory is accounted: `RecordBatch::get_array_memory_size` / [get_record_batch_memory_size](https://github.com/apache/datafusion/blob/7c05b208e115c29d1f014d2e318b974984cea1a6/datafusion/common/src/utils/memory.rs#L133) or whatever the downstream operator decides to use 3. since there's no way to transfer a memory reservation from one operator to another, other memory pool operations could happen in-between, so there's no guarantee that if you free N bytes from a reservation in an operator you could reserve the same N bytes in another operator -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
