luomh0910 opened a new pull request, #13616:
URL: https://github.com/apache/gluten/pull/13616

   ## What changes are proposed in this pull request?
   
   This PR adds an optional streaming input path for Velox broadcast hash-table 
construction. Instead of deserializing and retaining all input batches before 
building the hash table, the new path consumes batches on demand and feeds them 
incrementally to the native hash-table builder.
   - Add spark.gluten.velox.broadcastBuild.inputStreaming.enabled, disabled by 
default.
   - Pass broadcast build input through ColumnarBatchInIterator and a native 
batch callback.
   - Force single-threaded construction when streaming is enabled, including a 
native-side fallback when the requested thread count differs.
   - Support both on-heap and off-heap broadcast relations, with driver-side 
and executor-side hash-table construction.
   - Keep the materialized input path and existing serial/parallel build 
behavior when streaming is disabled.
   The change avoids retaining all deserialized input batches simultaneously; 
it does not change the final join-table preparation.
   
   ## How was this patch tested?
   
   - Added native HashTableBuilderTest coverage for incremental input 
consumption, batch lifetime, empty input, null-aware anti-join early 
termination, error handling, and join semantics.
   - Extended VeloxHashJoinSuite to cover streaming enabled/disabled, 
on-heap/off-heap relations, driver-side/executor-side builds, residual join 
conditions, broadcast reuse, and dump/replay.
   - Added VeloxBroadcastBuildThreadSuite to verify thread selection with 
streaming enabled and disabled.
   - Local validation passed: Scala Spotless, C++ formatting with clang-format 
15.0.7
   
   ## Was this patch authored or co-authored using generative AI tooling?
   Generated-by: Codex <GPT-6-Astra>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to