rustyconover opened a new issue, #11621: URL: https://github.com/apache/arrow-rs/issues/11621
### Is your feature request related to a problem or challenge? apache/arrow#49704 proposes (Arrow format version 1.6, with lazy consensus on dev@: https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) an optional `custom_metadata: [KeyValue]` member on the IPC `DictionaryEncoding` table. It holds the metadata of the dictionary *value* type, such as the extension keys of a dictionary of `arrow.uuid` values. Today a dictionary of an extension type and an extension type whose storage is a dictionary serialize identically, because `Field.custom_metadata` is the only metadata slot. Once C++ and Go writers adopt it, arrow-rs needs to read schemas that use the new member. arrow-rs cannot represent metadata on a dictionary's value type (#7982). ### Describe the solution you'd like - Add the member to `format/Schema.fbs` and regenerate `arrow-ipc/src/gen`. - Until #7982 is resolved, map the value type metadata into the arrow-rs `Field` metadata when the field has no extension type of its own. This keeps today's behaviour, where users see e.g. `arrow.uuid` on dictionary fields. Apply the same mapping in the IPC readers, the integration JSON reader (`"dictionary"."metadata"`) and the C Data Interface import (`ArrowSchema.dictionary->metadata`). - The writer would stay unchanged. ### Describe alternatives you've considered Ignoring the new member, so the field is read as a dictionary of the storage type. That is spec-conformant, but arrow-rs users would silently lose extension names once writers move them to the new member. ### Additional context The C++ and Go implementations of the format change are in progress as drafts. `DictionaryEncodingArgs` gains a public field, so this is an API change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
