sezruby opened a new pull request, #13652:
URL: https://github.com/apache/gluten/pull/13652

   ## What changes are proposed in this pull request?
   
   `ColumnarBatches.offload` adopts the vectors of a batch of plain Spark 
`ArrowColumnVector`s into a Gluten batch and then offloads that batch. Two 
failure paths left memory in the allocator until the task ended:
   
   - **The export fails after adopting**, e.g. the allocator refuses memory for 
the Arrow C structs (Gluten's task allocator throws when Spark has no memory 
left). The adopted batch was never closed. It's now closed when the offload 
fails.
   - **Adopting a later column fails**, e.g. a copy runs out of memory. The 
columns adopted before it were never closed. They're now released.
   
   ## How was this patch tested?
   
   Two new tests in `ArrowColumnVectorTest` offload into a child of the task 
allocator that has no room left:
   - `testOffloadReleasesAdoptedVectorsWhenExportFails`: the column moves in, 
then exporting it fails.
   - `testOffloadReleasesAdoptedVectorsWhenACopyFails`: the first column moves 
in, then copying the second, from another allocator root, fails.
   
   Both expect `OutOfMemoryException` and an empty allocator afterwards.
   - On main, each leaves 64 bytes behind.
   - With this change, both pass.
   
   `ArrowColumnVectorTest` (8) and `ColumnarBatchTest` (6) pass on Spark 3.5, 
run on amd64 Linux.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Opus 5.5
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to