rustyconover opened a new pull request, #53994: URL: https://github.com/apache/arrow/pull/53994
### Rationale for this change In IPC a dictionary-encoded column is a single `Field` whose type is the value type. The only metadata slot was `Field.custom_metadata`, so `dictionary<values=extension<X>>` and `extension<X, storage=dictionary<...>>` serialized identically. Arrow C++ reads both as the latter, so pyarrow cannot read back its own stream of `pa.dictionary(pa.int32(), pa.uuid())`: `Invalid storage type for UuidType: dictionary<...>`. The dev@ thread (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding KeyValue metadata for the value type to `DictionaryEncoding` (option 1). This is a format change. It needs a `[VOTE]` on dev@ and two reference implementations, so this PR is a draft. The Go implementation is in the companion arrow-go PR linked below. ### What changes are included in this PR? - **Format:** an optional `custom_metadata: [KeyValue]` appended to `DictionaryEncoding`. It holds the metadata of the dictionary value type, and is the IPC counterpart of `ArrowSchema.dictionary->metadata` in the C Data Interface. - Writers should put a value extension type's `ARROW:extension:name`/`ARROW:extension:metadata` there rather than in `Field.custom_metadata`. - Readers apply it to the value type before wrapping it in the dictionary type, then apply `Field.custom_metadata` as before. Readers may ignore keys they do not understand. - Format version 1.6, recorded in the `Schema.fbs` history and the Columnar, Versioning and Integration docs. `MetadataVersion` stays V5, and the file footer and Flight need no other change. - **Generated code:** `Schema_generated.h` regenerated with flatc 24.3.6, limited to the new member. - **C++:** the IPC writer and reader, and the integration JSON reader and writer (`"dictionary": {..., "metadata": [...]}`). - **Archery:** a new `dictionary_extension` case, run for C++ only for now. - Go is skipped until the companion arrow-go PR lands, because integration CI builds arrow-go from main. With that PR, C++ ⇄ Go passes over IPC. - Rust is skipped because arrow-rs cannot represent a dictionary of an extension type (apache/arrow-rs#7982). It reads the case correctly but cannot produce it, and archery has no producer-only skip. - **Not included:** the Ruby flatbuffers bindings (`ruby/red-arrow-format/.../dictionary_encoding.rb`) were not regenerated, because no Ruby toolchain was available. They need `rake flat_buffers:generate`. - **Follow-ups:** - `DictionaryFieldMapper` and `DictionaryCollector` (`ipc/dictionary.cc`) don't look through extension value types, so dictionaries nested inside extension dictionary values are not supported. - A Parquet `ARROW:schema` round-trip test. ### Are these changes tested? Yes. - **C++ IPC tests:** - schema and record batch round trips, including nested types and the file footer; - the flatbuffer layout; - delta and replacement dictionaries of an extension value type; - unregistered names and other keys being dropped on read; - legacy field-level metadata, which behaves as before. - **Also:** JSON integration tests, archery C++ IPC and C Data runs, and a pyarrow regression test. - **Cross-implementation, run manually:** C++ ⇄ Go passes over IPC with the companion arrow-go PR, and arrow-rs reads the C++ and Go output in every format. ### Are there any user-facing changes? Yes: - C++ and pyarrow now round-trip a dictionary whose value type is an extension type (e.g. `pa.dictionary(pa.int32(), pa.uuid())`) through IPC. - The C++/pyarrow IPC output for such columns changes: the extension keys move from `Field.custom_metadata` to `DictionaryEncoding.custom_metadata`. - Readers that predate this change, including older C++/pyarrow, read such columns as a plain dictionary of the storage type and lose the extension type. - That includes types like `arrow.opaque`, which older readers used to keep, though with the wrong nesting (as an extension type with dictionary storage). - Extension types whose storage type is a dictionary are serialized as before. - There is deliberately no fallback for data written by existing writers. A field-level `arrow.uuid` on a dictionary still fails to read, as it does today. ### Was AI used for this PR? **PR code and description written by:** - [ ] Human - [x] AI (Claude Code) **Reviewed before submission by:** - [x] Human (maintainer) - [x] AI (AI reviewer) - [ ] Not reviewed * GitHub Issue: #49704 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
