rustyconover commented on issue #49704: URL: https://github.com/apache/arrow/issues/49704#issuecomment-6218521327
We hit this in DuckDB's nanoarrow extension (duckdb/duckdb-nanoarrow#129), which is adding dictionary-encoded columns to its Arrow IPC stream writer. DuckDB's C Data export puts `arrow.uuid` on the dictionary's value schema (`dictionary<values=extension<arrow.uuid>>`). nanoarrow's IPC encoder only writes the index field's metadata, so the extension name is dropped. Moving it to the `Field`, as Arrow C++ does, gives the ambiguous form described above, which pyarrow 26 can't read back (`Invalid storage type for UuidType: dictionary<values=fixed_size_binary[16], indices=int32, ordered=0>`). For now we reject dictionary encoding for columns written as extension types. We'd like to help get option 1 from the dev@ thread implemented: an optional `custom_metadata: [KeyValue]` on `DictionaryEncoding` that holds the value type's metadata. @paleolimbot, would you welcome help with this, or have you already started? Since a format change needs two reference implementations with integration tests, we could put together draft PRs for the `Schema.fbs` and Columnar docs change, with C++, Go (arrow-go) and Rust (arrow-rs) implementations and an archery integration case. We'd follow your lead on naming and semantics, and don't want to duplicate work you have in progress. The semantics we'd propose for the drafts: - **Writers** put the value type's extension metadata (`ARROW:extension:name` and `ARROW:extension:metadata`) in `DictionaryEncoding.custom_metadata`, and keep `Field.custom_metadata` for the field's own type, i.e. an extension type with dictionary storage. - **Readers** apply `DictionaryEncoding.custom_metadata` to the value type before wrapping it in the dictionary type, then apply `Field.custom_metadata` to the result as they do today. A reader that doesn't know the new member still gets `dictionary<storage type>`, which is the graceful degradation Antoine described. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
