Ma77Ball opened a new pull request, #8560: URL: https://github.com/apache/texera/pull/8560
### What changes were proposed in this PR? - Third PR of the columnar series: adds the columnar join, offloads the sink decode, lets terminal operators consume columns, and passes Arrow straight through to Python. - `HashJoinProbeOpExec` probes the hash table directly from the Arrow batch (selective probe), decoding only the matched rows. - Sink offload: `OutputManager.saveArrowBatchToStorageIfNeeded` hands the Arrow batch to the result-writer thread (`OutputPortStorageWriterThread`), moving the decode off the DP thread. - Terminal/sink operators return `Consumed` for a columnar batch instead of forcing a row decode. - Python passthrough: a `ColumnarFrame` reaching `PythonProxyClient` is streamed straight to Arrow Flight with no tuple round-trip (Python already speaks Arrow). - Contract: `ColumnarResult` gains `EmitRows` and `processColumnarBatch` takes a `port`, so an operator can consume columns but emit rows. ### Any related issues, documentation, discussions? Part of: #8556 ### How was this PR tested? - `sbt "testOnly *ColumnarShuffleCorrectnessSpec"`: 2-worker shuffle, row output == columnar output through the join and aggregate. - `sbt "testOnly *OutputPortStorageWriterThreadSpec"`: the Arrow batch write path stores the same rows as the row path. - `sbt "testOnly *ProjectionOpExecSpec *VectorizedFilterCorrectnessSpec"`: operator output unchanged under the new `port` contract. ### Was this PR authored or co-authored using generative AI tooling? Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
