kszucs opened a new pull request, #1469:
URL: https://github.com/apache/arrow-go/pull/1469
### Rationale for this change
Content-defined chunking (CDC) makes the Parquet writer cut data pages at
boundaries derived from the data instead of from a byte count. When a table is
edited (rows inserted, deleted or updated) and written again, most data pages
come out byte-identical to the previous version, so content-addressable storage
such as Hugging Face Xet only has to store the pages that changed.
Arrow C++/PyArrow (apache/arrow#45360) and arrow-rs (apache/arrow-rs#9450)
already support it. This PR ports the same algorithm to arrow-go.
### What changes are included in this PR?
- **Writer option:** `parquet.CDCOptions`, `parquet.DefaultCDCOptions()`
(256 KiB min, 1 MiB max, norm level 0), `parquet.WithContentDefinedChunking`
and `WriterProperties.ContentDefinedChunking()`. CDC is off by default.
- **Chunker:** a new internal package, `parquet/internal/cdc`, ports the C++
`ContentDefinedChunker`. It keeps:
- the gearhash rolling hash over (def level, rep level, value);
- the same 8 gear tables and mask formula;
- a cut after 8 mask matches, one per gear table, or at the max size.
Nested columns are cut only at record boundaries. Fixed-width values are
hashed as their little-endian bytes, so boundaries don't depend on the host's
byte order. The gear tables are checked in as `gearhash_table.go`, generated by
`gearhash_gen.go` through `go generate`.
- **pqarrow integration:** `pqarrow.FileWriter` keeps one chunker per leaf
column for the life of the file. Page boundaries therefore don't depend on how
rows are split across row groups or across `Write`, `WriteBuffered`,
`WriteTable` and `WriteColumnChunked`/`WriteColumnData` calls.
- Each leaf write is sliced per chunk, zero-copy, and the current page is
closed between chunks.
- Invalid options make `NewFileWriter` return an error before anything is
written to the sink.
- **`file` package:** `ColumnChunkWriter` gains `FlushCurrentPage() error`,
which every column writer already implements. `FlushCurrentPage` no longer
writes an empty data page when nothing is buffered.
- **Dictionary writes:** slices of one dictionary array share their
dictionary data, so `pqarrow` now skips the dictionary comparison for them.
Behaviour that differs from Arrow C++:
- **State across row groups:** chunker state carries over across row groups.
C++ does the same since GH-51684 (Arrow 27.0.0); earlier versions restart at
every row group.
- **Nested columns:** a boundary at the very start of a write call closes
the page left open by the previous call. C++ drops that boundary, which makes
nested pages depend on how the data was split across write calls.
- **Dictionary fallback:** falling back from dictionary to plain encoding
doesn't start a new page, because Go's fallback re-encodes the buffered page.
- **Writer scope:** only `pqarrow.FileWriter` chunks.
`pqarrow.WriteArrowToColumn` returns `arrow.ErrNotImplemented` when CDC is
enabled, and the low-level `file` column writers ignore the option. C++ throws
from `WriteBatch`.
Page boundaries are not compared against files written by PyArrow or
arrow-rs. The algorithm and gear tables are the same, but there are no
cross-implementation fixtures.
Of the ~5.3k added lines, about 2.1k are the generated gear table and about
2.5k are tests.
### Are these changes tested?
Yes.
- **`parquet/internal/cdc`:**
- The mask and option-validation tables and the gear-table values are
taken from the C++ tests.
- Chunk invariants are checked for required, nullable and nested columns,
along with the forced cut at the max size and determinism.
- Chunking a column in one call gives the same boundaries as chunking it
across two calls, including when a boundary falls exactly at the split.
- Type dispatch: dictionaries chunk like their indices, extensions like
their storage, all binary-like types alike, and unsupported types return an
error.
- Hashing a fixed-width value word by word matches hashing its
little-endian bytes.
- `TestChunksStableBoundaries` pins exact boundaries, so a hashing change
can't silently break deduplication between versions.
- Invariant checks run in builds with `-tags assert`.
- **`parquet/pqarrow`:** a port of the C++ CDC test suite, including its
data generator and the `PagesFollowContent` page assertion.
- **Single row group:** inserts, deletes, updates, prepend, append, an
empty table and array offsets. The type matrix covers dictionary on and off and
data page V1/V2.
- **Multiple row groups:** edits across row groups.
- **Independence:** pages don't depend on row-group boundaries, write
calls, Arrow chunking, `WriteBuffered`, column-by-column writes or
dictionary-typed input.
- **Error paths:** invalid options, write errors, missing levels and
`WriteArrowToColumn`.
- Under `-race` the type matrix shrinks to the subset C++ runs under
sanitizers.
- **`parquet/file`:** a `FlushCurrentPage` call with nothing buffered writes
no page.
- **Benchmarks:** `BenchmarkWriteContentDefinedChunking` covers int64 and
string columns with CDC off and on. `BenchmarkChunks` covers the chunker on its
own.
Local runs:
- `go test -tags assert` and `-tags assert,noasm` over `./parquet/...`;
- `-race` on the `cdc`, `pqarrow` and `file` packages;
- a `-tags pqarrow_read_only` build;
- `go vet` of the changed packages for 386 and s390x;
- golangci-lint over `parquet`.
### Are there any user-facing changes?
Yes. CDC is a new, experimental writer option:
```go
props := parquet.NewWriterProperties(
parquet.WithContentDefinedChunking(parquet.DefaultCDCOptions()),
)
fw, err := pqarrow.NewFileWriter(schema, w, props,
pqarrow.DefaultWriterProps())
```
**This PR includes breaking changes to public APIs:**
- `file.ColumnChunkWriter` has a new method, `FlushCurrentPage() error`, so
implementations outside this module must add it.
- Calling `FlushCurrentPage` with nothing buffered no longer writes an empty
data page.
- `pqarrow.WriteArrowToColumn` returns an error when the column writer's
properties enable CDC.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
