andygrove opened a new issue, #6047: URL: https://github.com/apache/datafusion-comet/issues/6047
## What is the problem the feature request solves? Comet's tracing counters cover native allocations (`native_allocated`, `jemalloc_allocated`), memory pool reservations (`thread_NNN_comet_memory_reserved`) and the JVM heap (`jvm_heap_used`). Arrow buffers allocated on the JVM are off-heap, so none of these counters see them. When a traced run shows native memory growing faster than the pools account for, there is currently no way to tell how much Arrow memory the JVM itself is holding. A single gauge over `CometArrowAllocator` would not answer the question either. Comet imports batches from native over the Arrow C Data Interface, and Arrow charges an imported buffer to whichever allocator wraps it (`BaseAllocator.wrapForeignAllocation` calls `allocateBytes`). So the root allocator's total mixes memory the JVM allocated with native memory that `native_allocated` already counts, and the two cannot be separated after the fact. ## Describe the potential solution Hold buffers imported over the C Data Interface in a dedicated child allocator, and report two counters: - `jvm_arrow_allocated`: everything the root allocator accounts for - `jvm_arrow_imported`: the imported portion, which is native memory and so is also counted by `native_allocated` Because the child reserves nothing, every byte still escalates to the parent and the root keeps reporting the total, which makes the difference between the two counters the Arrow memory the JVM allocated itself. ## Describe alternatives you've considered Reporting only the root allocator's total. That is simpler, but it double counts imported native buffers against `native_allocated` with no way to tell how much of the overlap there is, which defeats the purpose of comparing the two. ## Additional context Comes out of an investigation into what a traced TPC-H run can and cannot account for. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
