sezruby opened a new pull request, #13652: URL: https://github.com/apache/gluten/pull/13652
## What changes are proposed in this pull request? `ColumnarBatches.offload` adopts the vectors of a batch of plain Spark `ArrowColumnVector`s into a Gluten batch and then offloads that batch. Two failure paths left memory in the allocator until the task ended: - **The export fails after adopting**, e.g. the allocator refuses memory for the Arrow C structs (Gluten's task allocator throws when Spark has no memory left). The adopted batch was never closed. It's now closed when the offload fails. - **Adopting a later column fails**, e.g. a copy runs out of memory. The columns adopted before it were never closed. They're now released. ## How was this patch tested? Two new tests in `ArrowColumnVectorTest` offload into a child of the task allocator that has no room left: - `testOffloadReleasesAdoptedVectorsWhenExportFails`: the column moves in, then exporting it fails. - `testOffloadReleasesAdoptedVectorsWhenACopyFails`: the first column moves in, then copying the second, from another allocator root, fails. Both expect `OutOfMemoryException` and an empty allocator afterwards. - On main, each leaves 64 bytes behind. - With this change, both pass. `ArrowColumnVectorTest` (8) and `ColumnarBatchTest` (6) pass on Spark 3.5, run on amd64 Linux. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
