rustyconover opened a new pull request, #11622: URL: https://github.com/apache/arrow-rs/pull/11622
# Which issue does this PR close? - Closes #11621. - Related to #7982. # Rationale for this change In Arrow IPC, a dictionary whose values are an extension type and an extension type whose storage is a dictionary serialize identically, because `Field.custom_metadata` is the only metadata slot. apache/arrow#49704 adds an optional `custom_metadata: [KeyValue]` member to `DictionaryEncoding` holding metadata for the dictionary value type. That change has lazy consensus on dev@arrow, targets format version 1.6, and is drafted in apache/arrow#53994, with a Go implementation in apache/arrow-go#1381. arrow-rs needs to read schemas that use it. This PR is a draft until the format change is voted on in apache/arrow. # What changes are included in this PR? - **Schema:** `format/Schema.fbs` gains `DictionaryEncoding.custom_metadata` and the version 1.6 history line. `arrow-ipc/src/gen/Schema.rs` is regenerated; only the `DictionaryEncoding` additions change. - **IPC readers:** `DataType::Dictionary` cannot carry metadata for its value type (#7982). Until it can, the reader copies the value type metadata into the `Field`'s metadata when the field has no `ARROW:extension:name` of its own; keys already in the field metadata win. Otherwise the value metadata is dropped. This keeps today's behaviour for arrow-rs users, who see e.g. `arrow.uuid` on dictionary fields, and that metadata survives read-then-write. The mapping is documented in the `arrow_ipc::reader` module docs and on the schema conversion functions. - **Integration JSON:** `field_from_json` validates the optional `"dictionary"."metadata"` attribute and maps it the same way, so the JSON and IPC readings of an integration file agree. - **C Data Interface import (arrow-schema):** `Field::try_from(&FFI_ArrowSchema)` maps the metadata of an imported dictionary schema's `dictionary` (its value type) the same way, so the IPC, integration JSON and C Data readers agree. The C Data export is unchanged. - **IPC writer:** unchanged. It never writes the new member, and dictionary field metadata is written to `Field.custom_metadata`, the pre-1.6 position. # Are these changes tested? Yes. Unit tests build schema flatbuffers containing the new member with the generated builders. They cover: - copying the value metadata to the field; - field keys taking precedence; - dropping the value metadata when the field has its own extension type; - an empty but present vector; - nesting in list and struct; - rejection of non-UTF-8 strings; - reading through `StreamReader` and through the `FileReader` footer. Integration JSON tests cover the same mapping and the validation errors, and C Data import tests cover the mapping, including nesting in list and struct. With the C++ and Go drafts, archery reads their `dictionary_extension` output correctly in arrow-rs, over IPC and C Data. # Are there any significant user-facing changes? - **Copied metadata:** value type metadata is copied into arrow-rs `Field` metadata as described above. This applies to the arrow-ipc readers and to the arrow-schema C Data import (`Field::try_from(&FFI_ArrowSchema)`), which previously dropped the metadata of an imported schema's `dictionary`. Malformed metadata on that `dictionary` is now an import error. - **Output position:** files and schemas written or exported by arrow-rs keep dictionary field metadata on the field itself, the pre-1.6 position. Readers implementing format 1.6 read it as metadata of the field's own type, so they do not read a dictionary of an extension type written by arrow-rs back as one. - **UTF-8 check:** the flatbuffers verifier now also checks that the strings in `DictionaryEncoding.custom_metadata` are valid UTF-8. A schema with non-UTF-8 bytes there, which Arrow C++ accepts, is now rejected; previously the member was ignored as an unknown field. - **Breaking API change:** `arrow_ipc::DictionaryEncodingArgs` gains a public field, `custom_metadata`. Code that builds it with a struct literal must add the field or use `..Default::default()`. This needs the `api-change` label. The code and this description were written by AI (Claude Code). The generated flatbuffers code was verified byte-identical against flatc 25.2.10 output, and the change was reviewed by a human maintainer and an AI reviewer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
