sezruby commented on PR #12958: URL: https://github.com/apache/gluten/pull/12958#issuecomment-6904672648
@FelixYBW a follow-up on the copy in `ArrowColumnVectors.adopt` (issue 2 in my earlier comments), now that #13648 moves the Lance scan onto lance-spark's Arrow batches. lance-spark allocates those batches under Spark's root allocator. lance-format/lance-spark#908 settled on that in review, so that consumers under Spark's root can transfer the vectors and keep them after the reader closes. Gluten's task allocator is its own `RootAllocator`, so `adopt` copies every batch. Without pruning, that makes the Arrow-batch path up to 21% slower than `LanceScanTransformer` in my emulated benchmark. The "Ongoing work" section of `ArrowColumnarConversion.md` suggests making Gluten's task allocator a child of Spark's root, which would turn that copy into a move. That alone breaks Gluten's memory accounting, though. Arrow charges a transfer to the target allocator with `forceAllocate`, which doesn't call its `AllocationListener`, but releasing the buffer later calls the target listener's `onRelease`. With Arrow 15, two child allocators of one root, and a 1 MiB buffer moved from one to the other: ``` after transfer: target listener onPreAllocation=0 onAllocation=0 onRelease=0, target.getAllocatedMemory=1048576 after release: target listener onRelease=1048576; source listener onAllocation=1048576 onRelease=0 ``` With `ManagedAllocationListener` as the target, every adopted batch would repay Spark memory that Gluten never borrowed. What I'd propose: 1. Create the task allocators as children of `ArrowUtils.rootAllocator`, keeping `ManagedAllocationListener`. 2. When `adopt` moves a vector, reserve the moved bytes on the target listener itself: `onPreAllocation` before the transfer and `onAllocation` after it, for the accounted size of the buffers whose ownership moves. The later `onRelease` then balances. Batches from Spark's root carry no listener on their side, so nothing is left unbalanced there. Longer term, Arrow could notify listeners on transfers. Does this match the allocator change you had in mind, or should I open a PR for it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
