luomh0910 opened a new issue, #13596:
URL: https://github.com/apache/gluten/issues/13596

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   ColumnarBuildSideRelation.buildHashTable deserializes all broadcast batches 
before starting native hash-table construction.
   For large broadcast relations, this creates significant memory pressure 
because the following allocations can coexist during the build:
   
   - The serialized broadcast data.
   - All deserialized build-side batches.
   - The hash table being constructed.
   
   The deserialized batches can occupy substantially more memory than their 
serialized representation, especially for wide schemas and variable-length 
columns. Retaining the entire deserialized input alongside the growing hash 
table increases peak memory usage and can lead to out-of-memory failures, even 
when the final hash table alone could fit in memory.
   
   ### Expected Behavior
   Support consuming broadcast batches incrementally: deserialize a batch, feed 
it into the hash-table builder, and release the input batch when it is no 
longer needed.
   
   This issue was written with the assistance of AI (Codex with GPT-6).
   
   ### Gluten version
   
   _No response_
   
   ### Spark version
   
   None
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   ```bash
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to