Ma77Ball opened a new pull request, #8558: URL: https://github.com/apache/texera/pull/8558
### What changes were proposed in this PR? - First PR of a stacked series adding an opt-in columnar (Apache Arrow) execution path to Amber, behind a flag and defaulting to the existing row path. - `ColumnarFrame` payload: a batch encoded as Arrow IPC bytes, alongside the existing `DataFrame`. - `ColumnarOperatorExecutor` / `ColumnarResult` (`Emit`/`Consumed`/`Unsupported`): an opt-in interface to consume an Arrow batch directly, with automatic fall back to the row path so results never differ. - Engine plumbing: `DataProcessor` dispatches a `ColumnarFrame` to the columnar or row path; `OutputManager`, `Partitioner`/`NetworkOutputBuffer`, `InputManager`, `DPThread`, and `EndChannelHandler` carry the batch end to end. - Operators: `CSVScanSourceOpExec` produces Arrow batches at the source (no per-row `Tuple`), and `SpecializedFilterOpExec` filters an Arrow batch directly. ### Any related issues, documentation, discussions? Part of: #8556 ### How was this PR tested? - `sbt "testOnly *ArrowIpcRoundTripSpec"`: a batch survives serialize to Arrow IPC to deserialize unchanged across all types. - `sbt "testOnly *VectorizedFilterCorrectnessSpec"`: the vectorized filter returns exactly the row path's survivors across every type, operator, nulls, IS NULL, and multi-predicate OR. - `sbt "testOnly *ColumnarReproSpec"`: scan to filter end to end, results read back from storage. - Run `DataProcessingSpec` with `COLUMNAR_WIRE=1` and `COLUMNAR_WIRE=0`: expect identical results (no regression). ### Was this PR authored or co-authored using generative AI tooling? Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
