LiaCastaneda commented on issue #8938: URL: https://github.com/apache/arrow-rs/issues/8938#issuecomment-4288369021
Hi @ariel-miculas, I’m not familiar with the wrapper APIs. The approach I considered was just wrapping the Arrow Rust (arrow-rs) used by DataFusion to use Arrow’s memory pool (whenever an array was created on each DataFusion ExecutionPlan node) while also accounting for it in the DataFusion memory pool. I have a rough draft of my attempt on this [branch](https://github.com/LiaCastaneda/datafusion/commit/c66668190256c64eaf8667e4964efd6d7eee1785#diff-e2ae56ec969246ffa18ca878e011cf923dedf7053a0db07afdb89848310bb402). The main blocker was getting both the Arrow memory pool and DataFusion’s MemoryPool, MemoryReservation, and memory registration to work together in a sound way. I ended up dropping the implementation since it was becoming burdensome and would require deeper thought into DataFusion’s memory accounting design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
