goutamadwant commented on code in PR #24155: URL: https://github.com/apache/datafusion/pull/24155#discussion_r3755180895
########## docs/source/user-guide/parquet-content-defined-chunking.md: ########## @@ -0,0 +1,149 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Parquet Content-Defined Chunking + +Content-defined chunking (CDC) is an experimental Parquet writer feature that +makes data page boundaries depend on column values rather than fixed row or byte +counts. This makes unchanged regions more likely to produce identical pages when +closely related versions of a dataset are written with the same settings. + +CDC is useful when the resulting files are stored or transferred through a +content-addressable or block-deduplicating system. Such a system can reuse the +identical pages instead of storing or transferring them again. For example, a +small insertion near the beginning of a dataset can change one page while later +page boundaries converge back to those of the previous version. + +CDC does not itself deduplicate data or provide a page store. On a conventional +filesystem or object store, each Parquet file is still stored in full. The output +is a normal Parquet file and requires no CDC-specific reader support. + +## When to enable CDC + +Consider CDC when all of the following apply: + +- You regularly write similar versions of the same dataset. +- Your storage or transfer layer detects and reuses duplicate byte ranges. +- Reducing storage or network transfer is more important than maximizing write + parallelism for an individual file. + +Leave CDC disabled for ordinary Parquet output unless you have measured a benefit +in the system that stores or transfers the files. CDC is disabled by default. + +When CDC is enabled, DataFusion uses the sequential Arrow writer for each output +file because the chunker's state must persist across row groups. This can reduce +write throughput compared with DataFusion's parallel writer path. Writing +different output files can still proceed concurrently. + +## Enable CDC with SQL + +Set CDC for one [`COPY`](sql/dml.md#copy) operation with Parquet format options: + +```sql +COPY ( + SELECT + value AS id, + CONCAT('event-', CAST(value AS VARCHAR)) AS event + FROM generate_series(1, 100000) +) TO 'cdc-output' +STORED AS PARQUET +OPTIONS ( + 'format.content_defined_chunking.enabled' 'true' +); +``` + +The default chunking parameters are a good starting point. To tune them for one +write: + +```sql +COPY source_table TO 'cdc-output' Review Comment: Yes, confirmed @neilconway..directory output distributes RecordBatches round-robin across parallel output files, with `minimum_parallel_output_files` defaulting to four, while CDC state is maintained independently per file. I updated both the SQL and Rust examples to use explicit single-file output and added guidance about keeping input ordering and file layout stable, or using stable partition keys when multiple files are required. Let me know. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
