Ma77Ball opened a new pull request, #8561: URL: https://github.com/apache/texera/pull/8561
### What changes were proposed in this PR? - Final PR of the columnar series: hardens the wire format, adds optional wire compression and the vectorized Iceberg sink, and speeds up decode. - Wire versioning: `ColumnarFrame` carries a format version; a worker rejects an unsupported version instead of misreading bytes. - Optional wire compression (LZ4_FRAME/ZSTD) via `arrow-compression`, config-gated; the reader auto-detects compressed vs uncompressed. - Vectorized sink: `IcebergTableWriter.writeArrowBatch` writes an Arrow batch straight to Iceberg through a lazy `ArrowRecordView` (reads values from the Arrow columns), gated by `enable-vectorized-sink`. - Config: `columnar.enable-vectorized-sink` and `columnar.wire-compression` in `application.conf` via `ApplicationConfig`, with env overrides. - Faster decode: schema-once and projected decode helpers in `ArrowUtils`. ### Any related issues, documentation, discussions? Closes: #8556 ### How was this PR tested? - `sbt "testOnly *ArrowIpcRoundTripSpec"`: a batch round-trips unchanged, including compressed (LZ4_FRAME/ZSTD) and versioned frames. - `sbt "testOnly *ColumnarShuffleCorrectnessSpec"`: 2-worker shuffle, row output == columnar output with the vectorized sink enabled. - Run a scan to filter to sink workflow with `COLUMNAR_SINK=1`: expect the same stored results as the row sink. ### Was this PR authored or co-authored using generative AI tooling? Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
