rustyconover opened a new pull request, #54959: URL: https://github.com/apache/arrow/pull/54959
### Rationale for this change This PR was split out of #53994 at review request, so that the specification change can be reviewed on its own. In IPC, a dictionary-encoded column is a single `Field` whose type is the value type. The only metadata slot was `Field.custom_metadata`, so `dictionary<values=extension<X>>` and `extension<X, storage=dictionary<...>>` serialized identically. Arrow C++ reads both as the latter, so pyarrow cannot read back its own stream of `pa.dictionary(pa.int32(), pa.uuid())`. The dev@ thread (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding KeyValue metadata for the value type to `DictionaryEncoding` (option 1). This is a format change, so it needs a `[VOTE]` on dev@ and two reference implementations (see below). ### What changes are included in this PR? - **`format/Schema.fbs`:** an optional `custom_metadata: [KeyValue]` appended to `DictionaryEncoding`. It is the `custom_metadata` of the dictionary value type, with the same semantics as `Field.custom_metadata` (including the reserved `ARROW:extension:*` keys) but applied to the dictionary values. It is the IPC counterpart of `ArrowSchema.dictionary->metadata` in the C Data Interface. - **`Columnar.rst`:** a new "Dictionary-encoded extension types" section. The format version is bumped to 1.6. - Writers should put a value extension type's `ARROW:extension:name`/`ARROW:extension:metadata` there rather than in `Field.custom_metadata`. - Readers apply it to the value type before wrapping it in the dictionary type, then apply `Field.custom_metadata` as before. - Extension types whose storage type is a dictionary keep using `Field.custom_metadata`, as today. - **`Versioning.rst`:** the version 1.6 entry. - **`CDataInterface.rst`:** a note that `ArrowSchema.dictionary->metadata` holds the metadata of the dictionary value type, and how it maps to the two IPC `custom_metadata` slots. - **`Integration.rst`:** a matching optional `"metadata"` member in the `"dictionary"` object of the integration JSON format, with an example. The member is appended, so it is backward compatible. `MetadataVersion` stays V5, and the file footer and Flight need no other change. Readers that predate it ignore it and read a dictionary of the storage type. Generated flatbuffers code is not included. Each implementation regenerates its bindings in its own PR. ### Are these changes tested? This PR changes only the specification. It is exercised by the implementations and a new `dictionary_extension` archery integration case: - C++: #53994, stacked on this PR - Go, the second reference implementation: apache/arrow-go#1381. C++ ⇄ Go passes the new case over IPC. - Rust, which reads the new member: apache/arrow-rs#11622 - nanoarrow, which maps the member to `ArrowSchema.dictionary->metadata`: apache/arrow-nanoarrow#972 ### Are there any user-facing changes? Yes, this is a format addition (version 1.6). Writers that adopt it move a dictionary value type's extension keys from `Field.custom_metadata` to `DictionaryEncoding.custom_metadata`. Readers that predate it read such columns as a plain dictionary of the storage type and lose the extension type. ### Was AI used for this PR? **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed * GitHub Issue: #54958 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
