neilconway commented on code in PR #24155:
URL: https://github.com/apache/datafusion/pull/24155#discussion_r3754416336
##########
docs/source/index.rst:
##########
@@ -122,6 +122,7 @@ To get started, see
user-guide/cli/index
user-guide/dataframe
user-guide/arrow-introduction
+ user-guide/parquet-content-defined-chunking
Review Comment:
This places the file right next to "A Gentle Arrow Introduction" and
"Expressions", which seems an odd place for a more advanced/esoteric topic
guide. Maybe move it down, next to "Reading Explain Plans"?
##########
docs/source/user-guide/parquet-content-defined-chunking.md:
##########
@@ -0,0 +1,149 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Parquet Content-Defined Chunking
+
+Content-defined chunking (CDC) is an experimental Parquet writer feature that
+makes data page boundaries depend on column values rather than fixed row or
byte
+counts. This makes unchanged regions more likely to produce identical pages
when
+closely related versions of a dataset are written with the same settings.
+
+CDC is useful when the resulting files are stored or transferred through a
+content-addressable or block-deduplicating system. Such a system can reuse the
+identical pages instead of storing or transferring them again. For example, a
+small insertion near the beginning of a dataset can change one page while later
+page boundaries converge back to those of the previous version.
+
+CDC does not itself deduplicate data or provide a page store. On a conventional
+filesystem or object store, each Parquet file is still stored in full. The
output
+is a normal Parquet file and requires no CDC-specific reader support.
+
+## When to enable CDC
+
+Consider CDC when all of the following apply:
+
+- You regularly write similar versions of the same dataset.
+- Your storage or transfer layer detects and reuses duplicate byte ranges.
+- Reducing storage or network transfer is more important than maximizing write
+ parallelism for an individual file.
+
+Leave CDC disabled for ordinary Parquet output unless you have measured a
benefit
+in the system that stores or transfers the files. CDC is disabled by default.
+
+When CDC is enabled, DataFusion uses the sequential Arrow writer for each
output
+file because the chunker's state must persist across row groups. This can
reduce
+write throughput compared with DataFusion's parallel writer path. Writing
+different output files can still proceed concurrently.
+
+## Enable CDC with SQL
+
+Set CDC for one [`COPY`](sql/dml.md#copy) operation with Parquet format
options:
+
+```sql
+COPY (
+ SELECT
+ value AS id,
+ CONCAT('event-', CAST(value AS VARCHAR)) AS event
+ FROM generate_series(1, 100000)
+) TO 'cdc-output'
+STORED AS PARQUET
+OPTIONS (
+ 'format.content_defined_chunking.enabled' 'true'
+);
+```
+
+The default chunking parameters are a good starting point. To tune them for one
+write:
+
+```sql
+COPY source_table TO 'cdc-output'
+STORED AS PARQUET
+OPTIONS (
+ 'format.content_defined_chunking.enabled' 'true',
+ 'format.content_defined_chunking.min_chunk_size' '262144',
+ 'format.content_defined_chunking.max_chunk_size' '1048576',
+ 'format.content_defined_chunking.norm_level' '0'
+);
Review Comment:
These are the default values; I might clarify that, as the wording above
might suggest this is an example of changing the behavior for this specific
query.
##########
docs/source/user-guide/parquet-content-defined-chunking.md:
##########
@@ -0,0 +1,149 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Parquet Content-Defined Chunking
+
+Content-defined chunking (CDC) is an experimental Parquet writer feature that
+makes data page boundaries depend on column values rather than fixed row or
byte
+counts. This makes unchanged regions more likely to produce identical pages
when
+closely related versions of a dataset are written with the same settings.
+
+CDC is useful when the resulting files are stored or transferred through a
+content-addressable or block-deduplicating system. Such a system can reuse the
+identical pages instead of storing or transferring them again. For example, a
+small insertion near the beginning of a dataset can change one page while later
+page boundaries converge back to those of the previous version.
+
+CDC does not itself deduplicate data or provide a page store. On a conventional
+filesystem or object store, each Parquet file is still stored in full. The
output
+is a normal Parquet file and requires no CDC-specific reader support.
+
+## When to enable CDC
+
+Consider CDC when all of the following apply:
+
+- You regularly write similar versions of the same dataset.
+- Your storage or transfer layer detects and reuses duplicate byte ranges.
+- Reducing storage or network transfer is more important than maximizing write
+ parallelism for an individual file.
+
+Leave CDC disabled for ordinary Parquet output unless you have measured a
benefit
+in the system that stores or transfers the files. CDC is disabled by default.
+
+When CDC is enabled, DataFusion uses the sequential Arrow writer for each
output
+file because the chunker's state must persist across row groups. This can
reduce
+write throughput compared with DataFusion's parallel writer path. Writing
+different output files can still proceed concurrently.
+
+## Enable CDC with SQL
+
+Set CDC for one [`COPY`](sql/dml.md#copy) operation with Parquet format
options:
+
+```sql
+COPY (
+ SELECT
+ value AS id,
+ CONCAT('event-', CAST(value AS VARCHAR)) AS event
+ FROM generate_series(1, 100000)
+) TO 'cdc-output'
+STORED AS PARQUET
+OPTIONS (
+ 'format.content_defined_chunking.enabled' 'true'
+);
+```
+
+The default chunking parameters are a good starting point. To tune them for one
+write:
+
+```sql
+COPY source_table TO 'cdc-output'
Review Comment:
By default, this will use round-robin partitioning to split batches across
files in this directory, I believe? If so, that would seem to defeat some/all
of the value of CDC, no? This might be worth discussing in more depth in the
guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
