rustyconover opened a new pull request, #1381: URL: https://github.com/apache/arrow-go/pull/1381
### Rationale for this change Closes #1380. Part of apache/arrow#49704. In IPC a dictionary-encoded field is one `Field` whose type is the value type, so `dictionary<values=extension<X>>` and `extension<X, storage=dictionary<...>>` serialize identically. In arrow-go: - a dictionary of `arrow.uuid` values is written with the UUID keys on the `Field`, and reading it back fails (`invalid storage type for UUIDType: dictionary`); - writing the schema of an extension type with dictionary storage panics. The dev@ discussion (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding `custom_metadata: [KeyValue]` to `DictionaryEncoding` for the dictionary value type's metadata. This depends on the format change in apache/arrow#53994 (`Schema.fbs`, columnar spec, archery integration case), which is a draft pending a dev@ vote. It is the second reference implementation for that vote, so it should not be merged before the format change is accepted. ### What changes are included in this PR? - **Generated code:** `arrow/internal/flatbuf/DictionaryEncoding.go` regenerated (flatc 25.12.19) with the new `DictionaryEncoding.custom_metadata` member. - **IPC writer:** - The `ARROW:extension:name`/`ARROW:extension:metadata` keys of a dictionary's extension value type are written to `DictionaryEncoding.custom_metadata` instead of `Field.custom_metadata`. - An extension type with dictionary storage keeps writing its keys to `Field.custom_metadata`, and its schema no longer panics on write. - **IPC reader:** - Applies `DictionaryEncoding.custom_metadata` to the value type, wraps the result in the dictionary type, then applies `Field.custom_metadata` as before. - Only registered extension types are applied; other keys and unregistered names are ignored. - Removing the extension keys from field metadata no longer mutates slices shared with the caller's `arrow.Metadata`. - **Integration JSON:** an optional `"metadata"` list in a field's `"dictionary"` object. The reader also now honours `"isOrdered"`. - **`dictutils.Mapper`:** finds nested dictionaries inside extension value types. - **New `arrow/internal/integration` package:** registers the canonical extension types. It is imported by `arrow-json-integration-test`, `arrow-file-to-stream`, `arrow-stream-to-file` and the Flight integration client and server. The C Data integration library doesn't import it, because its importer does not use the extension registry. ### Are these changes tested? Yes. - **IPC unit tests:** - schema serialization, checking the keys sit in `DictionaryEncoding` and not `Field`; - stream and file data round trips of dictionaries of UUID and of an extension type with metadata; - extension types with dictionary storage, including a dictionary of extension values as storage; - a dictionary nested inside an extension value type; - delta dictionaries of `dict<uuid>`; - unregistered, non-extension and wrong-storage dictionary metadata on read; - legacy field-level metadata. - **Integration JSON round trips:** dictionary metadata, `opaque<dict<uuid>>`, `list<dict<uuid>>` and ordered dictionaries. - **Archery, run manually:** - C++ (apache/arrow#53994) ⇄ Go passes over IPC and C Data arrays for the `extension`, `dictionary_extension`, `dictionary`, `nested_dictionary` and `extension_union` cases. - Go's C schema import is skipped there, because the C Data importer does not use the extension registry. - Go-to-Go Flight integration passes for the same cases. ### Are there any user-facing changes? Yes: - **Changed IPC output:** for a dictionary whose values are an extension type, the extension keys move from `Field.custom_metadata` to the new `DictionaryEncoding.custom_metadata`. - Such columns now round-trip as `dictionary<extension<...>>` in Go and in other implementations that support the new member. - Readers that predate the format change ignore the member and see a plain dictionary of the storage type. Previously they saw the keys on the field, which typically failed or produced an extension type with dictionary storage. - **Integration JSON format:** gains an optional `"metadata"` key in the `"dictionary"` object. - **Integration programs:** the IPC JSON integration, file/stream conversion and Flight integration client/server now register the canonical extension types. This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer and an AI reviewer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
