Thank you Rusty for taking this on! The fact that IPC can't represent this is a bit of a hole in Arrow's type model...in addition to theoretical completeness, I believe Rusty found that this came up exporting DuckDB types via Arrow where this can come up. In the spatial world, there are certain types of geometry columns that contain a lot of repeated points and we've wondered if dictionary encoding (e.g., a dictionary-encoded geoarrow.point extension type) would be a way to handle that.
I will work through reviewing these next week! Cheers, -dewey On Fri, Oct 16, 2026 at 12:33 PM Rusty Conover via dev <[email protected]> wrote: > > Hi all, > > Following up on the consensus for option 1 (an optional `custom_metadata` on > `DictionaryEncoding` for the dictionary value type), there are now draft PRs: > > - Format + C++: https://github.com/apache/arrow/pull/53994 > - Go: https://github.com/apache/arrow-go/pull/1381 > - Rust: https://github.com/apache/arrow-rs/pull/11622 > - nanoarrow: https://github.com/apache/arrow-nanoarrow/pull/972 > > The new member has the same semantics as `Field.custom_metadata`, including > the reserved `ARROW:extension:*` keys, but applies to the dictionary values > rather than to the field. It is the IPC counterpart of > `ArrowSchema.dictionary->metadata` in the C Data Interface. It is appended to > `DictionaryEncoding` as format version 1.6, and `MetadataVersion` stays V5. > Readers that predate it see `dictionary<storage type>`. > > With the drafts, C++ and Go produce and consume a new archery case, > `dictionary_extension`, over IPC and C Data, and arrow-rs and nanoarrow read > it. One implementation limitation: the C++ and Go dictionary types can only > attach a registered extension type to the values, so they drop other > value-type metadata on read. nanoarrow keeps all of it. arrow-rs maps it into > the field's metadata until it can represent metadata on dictionary values > (apache/arrow-rs#7982). > > Readers get no fallback for files written by existing writers, where the > extension keys sit on the Field, so those read the same as today. > > If there are no objections, I'd like to start a vote on the format change in > a week or so. > > Have a great weekend! > > Rusty
