FelixYBW opened a new pull request, #13415:
URL: https://github.com/apache/gluten/pull/13415
## What changes are proposed in this pull request?
Fixes #13381.
`ColumnarBatches.load` imports the Arrow buffers exported from a native
batch into Gluten's task allocator (`ArrowBufferAllocators.contextInstance()`).
Arrow's `wrapForeignAllocation` calls the allocator's listener for imported
buffers too, so `ManagedAllocationListener` reserves Spark memory for them. But
the buffers stay owned by the native batch, whose memory is already reserved:
by the Velox memory pool, or by the Java allocator the batch was offloaded
from. While the loaded batch is alive, its memory is counted twice.
- **`ArrowBufferAllocators.contextUntrackedInstance()`** (new): a child of
the task allocator with a no-op listener. Only the allocator that creates a
buffer reports it to its listener, so it doesn't reserve Spark memory. Being in
the same allocator tree, its buffers can still be moved to or shared with
vectors of the task allocator. A separate `RootAllocator` doesn't work: the
Python runner's `VectorLoader` loads the loaded vectors into a root of the task
allocator, which Arrow only allows within one allocator tree. The child is a
task resource with priority 11, so it's closed before its parent (10). If it
still holds memory at task end, it logs a leak and stays open, and the parent
reports the leak too.
- **`ColumnarBatches.load`** imports into that allocator. The C structs and
the schema export still use the allocator passed in.
- **Docs**: `ArrowColumnarConversion.md` describes the import allocator. The
PlantUML source of the release diagram is updated, but
`docs/image/ArrowBufReleaseDispatch.png` is not regenerated yet.
The Java → native direction (`offload`) isn't changed.
## How was this patch tested?
- `ColumnarBatchTest.testLoadDoesNotReserveNativeMemoryTwice` (new): builds
a native Velox batch of 4M `bigint` rows (32 MiB), loads it, and checks the
task's reserved memory (`TaskResources.getSharedUsage()`, shared by the Arrow
and Velox reservation listeners) before and after the load. It also checks that
the loaded vectors share the task allocator's root.
- Before the fix: `Loading 33554432 bytes of native data reserved 41943040
more bytes`.
- With the fix: passes.
- `ColumnarBatchTest` and `ArrowColumnVectorTest` pass (Spark 3.5).
- `ArrowEvalPythonExecSuite`, `UDFPartialProjectSuite` and
`VeloxTransitionSuite`: still running.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-5-5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
