kszucs opened a new issue, #3817: URL: https://github.com/apache/parquet-java/issues/3817
### Describe the enhancement requested Add content-defined chunking (CDC) to the Parquet writer. It is already available in Parquet C++ (apache/arrow#45750, PyArrow >= 21) and parquet-rs (apache/arrow-rs#9450). With CDC, data page boundaries come from a rolling hash of the column values rather than fixed sizes. Pages then stay stable when rows are inserted or appended, so content-addressable storage (e.g. Hugging Face Xet) can deduplicate them. The output is a regular Parquet file, with no format change. It would help to use the same parameters as the other implementations: - `min_chunk_size` (default 256 KiB) - `max_chunk_size` (default 1 MiB) - `norm_level` (default 0) Motivation: Iceberg is adding `write.parquet.content-defined-chunking.*` table properties (apache/iceberg-rust#2375, apache/iceberg-python#3889). Iceberg Java cannot honor them until parquet-java supports CDC. ### Component(s) Core -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
