Hi all, Following up on the consensus for option 1 (an optional `custom_metadata` on `DictionaryEncoding` for the dictionary value type), there are now draft PRs:
- Format + C++: https://github.com/apache/arrow/pull/53994 - Go: https://github.com/apache/arrow-go/pull/1381 - Rust: https://github.com/apache/arrow-rs/pull/11622 - nanoarrow: https://github.com/apache/arrow-nanoarrow/pull/972 The new member has the same semantics as `Field.custom_metadata`, including the reserved `ARROW:extension:*` keys, but applies to the dictionary values rather than to the field. It is the IPC counterpart of `ArrowSchema.dictionary->metadata` in the C Data Interface. It is appended to `DictionaryEncoding` as format version 1.6, and `MetadataVersion` stays V5. Readers that predate it see `dictionary<storage type>`. With the drafts, C++ and Go produce and consume a new archery case, `dictionary_extension`, over IPC and C Data, and arrow-rs and nanoarrow read it. One implementation limitation: the C++ and Go dictionary types can only attach a registered extension type to the values, so they drop other value-type metadata on read. nanoarrow keeps all of it. arrow-rs maps it into the field's metadata until it can represent metadata on dictionary values (apache/arrow-rs#7982). Readers get no fallback for files written by existing writers, where the extension keys sit on the Field, so those read the same as today. If there are no objections, I'd like to start a vote on the format change in a week or so. Have a great weekend! Rusty
