qzyu999 opened a new pull request, #3734:
URL: https://github.com/apache/parquet-java/pull/3734
## Summary
Adds `ArrowParquetWriter` to the `parquet-arrow` module — a page-level
writer that accepts Arrow `VectorSchemaRoot` batches and produces valid Parquet
files without per-row object construction.
Closes #3733. Related: #2264, #3353.
## Design
Bypasses `RecordConsumer`/`ColumnWriter` entirely. Writes assembled pages
directly to `PageWriter` with per-column strategy selection:
- **ZeroCopyPlainWriter** (required fixed-width): wraps Arrow data buffer as
page `BytesInput` directly. O(1) level encoding, O(N) stats scan, zero
per-value dispatch.
- **NullablePlainWriter** (optional fixed-width): single-pass compaction of
non-null values with inline statistics.
- **VarWidthPlainWriter** (STRING/BINARY): single-pass transformation from
Arrow offset+data buffers to Parquet length-prefixed format.
- **BooleanPlainWriter** (BOOLEAN): bit-pack compaction with null handling.
Does not extend `ParquetWriter<T>` because `ParquetWriter.write(T)`
increments its internal record count by 1 per call, which is incompatible with
batch semantics.
## Key properties
- Page format: V1 with length-prefixed RLE-encoded RL/DL sections
- Buffer safety: `PageWriter.writePage()` copies bytes immediately via
`ConcatenatingByteBufferCollector.collect()` — Arrow buffers can be freed after
`writeBatch()` returns
- Thread-safe statistics: `StatsResult` return object, no shared mutable
state
- NaN tracking for float/double columns
- Page-size chunking: large batches split into ~1MB pages for column-index
effectiveness
- Row group flush: based on actual `PageWriter.getMemSize()`, not heuristic
- Defensive null-buffer checks on all nullable writers
## Types supported
INT32, INT64, FLOAT, DOUBLE, BOOLEAN, BINARY (string), FIXED_LEN_BYTE_ARRAY.
Nullable and required. Covers common streaming workload schemas.
## Not included (follow-up PRs)
- Dictionary encoding (requires two-pass algorithm — build dict, then write
indices)
- Nested types (requires recursive RL/DL computation)
Both throw clear `UnsupportedOperationException`.
## Tests
10 round-trip tests (write via ArrowParquetWriter, read via standard
ParquetReader):
- Required and nullable integers
- Floats/doubles including NaN
- Strings with nulls and empty values
- Booleans with nulls
- Mixed-type schema
- Multi-batch writes
- Large batch page splitting (500K rows)
- Footer row count verification
- Zero-copy path with manually-constructed REQUIRED schema
## Dependencies added to parquet-arrow POM
- `parquet-hadoop` (compile) — for ParquetFileWriter,
ColumnChunkPageWriteStore
- `arrow-memory-netty` (test) — Arrow allocator runtime
- `hadoop-common` (test) — for ParquetReader in round-trip tests
- `hadoop-mapreduce-client-core` (test) — required by ParquetReader
- `parquet-hadoop` test-jar (test) — GroupReadSupport
## Discussion point
Adding `parquet-hadoop` as a compile dependency to `parquet-arrow` changes
the module's weight. Alternative: create a new `parquet-arrow-hadoop` module.
Open to guidance from maintainers.
## How to run tests
```
mvn test -pl parquet-arrow \
-Dsurefire.argLine="--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
