rustyconover commented on issue #49704: URL: https://github.com/apache/arrow/issues/49704#issuecomment-6220094897
Draft PRs for option 1 are up: - **Format + C++** (`Schema.fbs`, Columnar/Versioning/Integration docs, archery `dictionary_extension` case): #53994 - **Go**, the second reference implementation: apache/arrow-go#1381 - **Rust**: apache/arrow-rs#11622 They add `custom_metadata: [KeyValue]` at the end of `DictionaryEncoding` (format 1.6, `MetadataVersion` stays V5). Writers should put a value extension type's keys there. Readers apply it to the value type before the dictionary wrap, then apply `Field.custom_metadata` as before. Readers that predate the change see `dictionary<storage>`. With the drafts, C++ ⇄ Go pass archery over IPC and C Data arrays. arrow-rs reads both correctly, but can't produce the case, because it can't represent a dictionary of an extension type (apache/arrow-rs#7982). Until it can, it maps the value metadata into the field's metadata when the field has no extension of its own. One point we'd like feedback on before a vote: **what readers should do with value metadata they can't apply.** C++ and Go can only attach a *registered* extension type to a dictionary's values. Their `DictionaryType` has nowhere to keep other keys or unregistered extension names, so those are dropped on read, and pass-through tools (Flight servers, file↔stream converters) lose them. At the field level, the same keys survive in `Field.custom_metadata`. The draft spec text currently says: > The `custom_metadata` of the `DictionaryEncoding` holds the metadata of the dictionary value type. It is the IPC counterpart of the metadata of the dictionary value `ArrowSchema` in the C Data Interface (`ArrowSchema.dictionary->metadata`) … Readers may ignore keys they do not understand. An implementation whose dictionary type cannot hold metadata for the value type may be unable to preserve the `DictionaryEncoding`'s `custom_metadata`, so unregistered extension names and other keys can be lost when such an implementation passes the data through. Is that acceptable, or should the spec restrict the member to the `ARROW:extension:*` keys, or require more? We also deliberately left out a reader fallback for files written by existing writers (field-level `arrow.uuid` on a dictionary still fails to read, as today), to keep the old form unambiguous. Happy to revisit that if people prefer it. @paleolimbot, nanoarrow would need the same encoder/decoder change once the format lands. Happy to help there too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
