rustyconover opened a new pull request, #972:
URL: https://github.com/apache/arrow-nanoarrow/pull/972
**Draft until the Arrow format vote.** This implements the proposed Arrow
format 1.6 change from apache/arrow#49704. The companion drafts are
apache/arrow#53994 (format and C++), apache/arrow-go#1381 (Go) and
apache/arrow-rs#11622 (Rust).
`DictionaryEncoding` gains an optional `custom_metadata` that holds the
metadata of the dictionary value type. It is the IPC counterpart of
`ArrowSchema.dictionary->metadata`, while `Field.custom_metadata` corresponds
to the parent `ArrowSchema.metadata`. Because nanoarrow works on `ArrowSchema`
directly, the mapping is lossless. One open question on #49704 is whether the
member should be limited to the `ARROW:extension:*` keys. This PR writes all
keys of `ArrowSchema.dictionary->metadata`, which is easy to narrow if the vote
decides otherwise.
- **Regenerated header:** both commits regenerate `flatcc_generated.h` with
`dev/update_fbs.sh` (flatcc e3e44836).
- The first commit regenerates from apache/arrow main as committed, so it
only changes comments: upstream docstring fixes and the Tensor/SparseTensor
deprecation notes.
- The second commit regenerates from apache/arrow main plus the draft
Schema.fbs change. Its header diff is only the new member.
- **Encoder:** writes `schema->dictionary->metadata` (all keys) to
`DictionaryEncoding.custom_metadata`, omitted when NULL or empty, and keeps
writing `schema->metadata` to `Field.custom_metadata`. Previously the value
metadata was silently dropped.
- **Decoder:** keeps `Field.custom_metadata` on the parent schema, as
before, and now sets `schema->dictionary->metadata` from
`DictionaryEncoding.custom_metadata`.
- Messages without the member decode exactly as before. There is no
fallback for older files, per the spec.
- This covers dictionaries nested in struct/list children and footer
decoding.
- **Footer encoding:** carries the member when the caller fills
`footer.dictionaries`. nanoarrow's file writer doesn't fill it (#622), so
nanoarrow doesn't write IPC files with dictionaries today.
- **Integration JSON:** the reader and writer support the optional
`"metadata"` of a field's `"dictionary"` object. `TestingJSONComparison` now
compares dictionary value metadata at any depth, treating empty as absent.
**Tests:**
- **nanoarrow-only (run in CI):**
- encode/decode round trips that check the flatbuffer directly: extension
and arbitrary keys, non-UTF-8 values, nesting in struct and list, alongside
field metadata, empty metadata omitted;
- a footer encode/decode round trip with the dictionary encodings filled
in;
- decoding hand-built messages with and without the member;
- a writer/reader stream round trip;
- JSON reader/writer/comparison tests.
- **Arrow C++ interop:** with `NANOARROW_BUILD_TESTS_WITH_ARROW`,
`dictionary(int32, uuid)` cases are enabled, including Arrow C++ reading
nanoarrow-encoded schemas.
- They skip at runtime unless the linked Arrow C++ writes the new member,
so they skip on released Arrow C++.
- I ran them locally against an Arrow C++ build of apache/arrow#53994:
888/888 (886 tests plus the two integration tests), also under ASan/UBSan.
**Archery:** with the `dictionary_extension` case enabled for nanoarrow, C
Data passes between C++ and nanoarrow in both directions. On main,
nanoarrow→C++ fails because the value metadata is lost. IPC integration for
dictionary cases is still blocked by existing writer limitations tracked in
#622, which also affect the existing `dictionary` case.
This PR's code and description were written by AI (Claude Code) and reviewed
by a human maintainer and an AI reviewer.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
