Ma77Ball opened a new pull request, #8559: URL: https://github.com/apache/texera/pull/8559
### What changes were proposed in this PR? - Second PR of the columnar series: makes the columnar path survive multi-worker shuffles, adds two more columnar operators, and moves the flags into config. - Partition-aware emit: `OutputManager.emitColumnarBatch` splits an Arrow batch per receiver using each link's partitioner (one-to-one/broadcast ship the whole batch; hash/range/round-robin slice it), so a shuffle edge stays columnar instead of silently falling back. A one-time per-worker log makes any fall back visible. - `AggregateOpExec` consumes Arrow directly, decoding only the group-key and aggregated columns, feeding the existing COUNT/SUM/MIN/MAX/AVERAGE accumulation. - `ProjectionOpExec` selects, reorders, renames, or drops columns on the Arrow batch with no per-row decode. - Config: flags move to `application.conf` (`columnar.enable-columnar-wire`, `enable-vectorized-operators`) via `ApplicationConfig`; the `COLUMNAR_WIRE` / `FILTER_VECTORIZED` env vars still override. - New `ArrowUtils` helpers: `selectColumns`, `projectionIndices`, `getProjectedTuple`, and a schema-passing `getTexeraTuple` overload. ### Any related issues, documentation, discussions? Part of: #8556 ### How was this PR tested? - `sbt "testOnly *ColumnarShuffleCorrectnessSpec"`: on a 2-worker hash shuffle, row output == columnar output for the aggregate (COUNT/SUM/MIN/MAX/AVERAGE) and for a projection feeding a downstream filter and aggregate. - `sbt "testOnly *ProjectionOpExecSpec"`: columnar rename/reorder and drop match the row path. - Run `DataProcessingSpec` with `COLUMNAR_WIRE=1` and `COLUMNAR_WIRE=0`: expect identical results. ### Was this PR authored or co-authored using generative AI tooling? Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
