qzyu999 opened a new issue, #3733: URL: https://github.com/apache/parquet-java/issues/3733
### Feature Request Add a public API to write Arrow `VectorSchemaRoot` data to Parquet files without per-row object construction — achieving parity with what `pyarrow.parquet.write_table()` and Arrow C++ `parquet::WriteTable()` already provide. ### Background In C++/Python, writing Arrow columnar data to Parquet is a single call that transfers column buffers directly into Parquet page encoding. In Java, the only supported write path is row-at-a-time via `ParquetWriter<T>.write(T value)` which requires callers to construct row objects and hand them to the writer one by one. This has been raised before: - #2264 — "Arrow read write support" (open since 2019, no progress) - #3353 — "Why no bulk Arrow-to-Parquet write API in Java?" (2025, community confirms the gap) ### Demand Signal Multiple major Apache projects are blocked or working around this: - **Apache Iceberg** — `FileAppender<Record>` forces row-object construction for every record written. Iceberg has vectorized *reads* via their `arrow` module but no equivalent write path. Every engine writing to Iceberg tables (Flink, Spark, Trino) pays this cost. - **Apache Fluss** — streaming storage that uses Arrow as its internal columnar format. When tiering data to Iceberg/Parquet, must materialize row objects from Arrow vectors. See [fluss#4047](https://github.com/apache/fluss/issues/4047). - **Apache Paimon** — built their own `paimon-arrow` module with `ArrowBundleRecords` that bypasses parquet-java's row API entirely, writing Arrow batches to Parquet through their own internal writer. - **PyIceberg / iceberg-rust** — Arrow-native Iceberg clients (Python via DataFusion/Polars, Rust) need Java interop or pure-Java paths. - **[parquetforge](https://github.com/Earnix/parquetforge)** — a community project created specifically because this capability is missing from parquet-java. ### Proposed Approach Building on the excellent #3530 performance series, which is already adding batch read APIs (`readIntegers()`, `readLongs()`) to `ValuesReader`: **Phase 1: Batch write methods on ValuesWriter** (mirrors the read-side pattern from #3535) - `writeIntegers(int[] values, int offset, int count)` - `writeLongs(long[] values, int offset, int count)` - `writeFloats(float[] values, int offset, int count)` - `writeDoubles(double[] values, int offset, int count)` - `writeBytes(Binary[] values, int offset, int count)` This is the "level write batching" mentioned as planned work in #3530. Default implementations loop, but specialized encoders (PLAIN, DELTA_BINARY_PACKED, BYTE_STREAM_SPLIT) can optimize bulk encoding. **Phase 2: Arrow VectorSchemaRoot writer** - New class in `parquet-arrow` module (or a new `parquet-arrow-writer` module) that implements `ParquetWriter<VectorSchemaRoot>` or a new `ParquetBatchWriter` interface - Takes a `VectorSchemaRoot`, iterates columns, calls Phase 1 batch APIs - Handles nullability via Arrow validity buffers mapped to Parquet definition levels - For flat schemas (no nested types), definition/repetition levels are trivial (0 or 1) ### Why This Matters The current row-at-a-time API introduces: 1. **Object allocation overhead** — constructing Record/Row objects for data already in columnar form 2. **Virtual dispatch per value** — each `write()` call goes through the writer hierarchy 3. **Lost locality** — data in Arrow is column-contiguous in memory; destructuring it into rows and re-columnarizing it is fundamentally wasteful For streaming ingestion workloads (Fluss, Flink, Kafka Connect into Iceberg), this overhead is paid on every record of every batch, continuously. Batch write APIs would eliminate the structural inefficiency. ### Compatibility - Parquet file format is unchanged — this is purely a writer-side API addition - Existing `ParquetWriter<T>` API is unaffected - Default loop-based implementations ensure no encoder breaks - The Arrow dependency would be optional (in `parquet-arrow` module, not core) ### Component(s) parquet-column, parquet-arrow, parquet-hadoop -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
