CurtHagenlocher opened a new issue, #475:
URL: https://github.com/apache/arrow-dotnet/issues/475
### Describe the bug, including details regarding any error messages,
version, and platform.
In IPC, a dictionary-encoded field has no separate schema node for its value
type, so `ARROW:extension:name` can only go in the field's custom metadata.
That leaves it ambiguous whether the extension's storage is the whole
dictionary or only the dictionary values. Apache.Arrow and Arrow C++ resolve
the ambiguity in opposite directions.
- **Apache.Arrow:** `MessageSerializer.FieldFromFlatbuffer` resolves the
extension against the value type *before* wrapping it in a `DictionaryType`
("Resolve extension type from metadata (before dictionary wrapping)"). So the
metadata is read as applying to the values: `dictionary<int32, arrow.uuid>`.
- **Arrow C++ (pyarrow 24.0.0):** the metadata applies to the whole
dictionary type. An extension type whose storage is `dictionary<int32, string>`
round-trips through IPC. But a `dictionary<int32, arrow.uuid>` that pyarrow
itself writes can't be read back:
```python
u = pa.array([uuid.uuid4().bytes], type=pa.uuid())
arr = pa.DictionaryArray.from_arrays(pa.array([0], pa.int32()), u)
# write with pa.ipc.new_stream, then read back:
# ArrowInvalid: Invalid storage type for UuidType:
dictionary<values=fixed_size_binary[16], indices=int32, ordered=0>
```
**Effect on exchanging IPC between Apache.Arrow and Arrow C++**
- An extension type over a dictionary written by Arrow C++ is read by
Apache.Arrow as an extension over the dictionary's *values*. That either fails
to resolve, or resolves to the wrong type if a definition happens to accept the
value type.
- A `dictionary<int32, arrow.uuid>` written by either implementation can be
read by Apache.Arrow, but not by Arrow C++.
The C Data Interface doesn't have this ambiguity, because the dictionary has
its own child `ArrowSchema`. #474 aligns Apache.Arrow's C Data import and
export with Arrow C++ there.
**Possible directions**
1. Follow Arrow C++: try resolving the field-level extension against the
whole `DictionaryType` first, and fall back to the value type. That would keep
reading existing Apache.Arrow-written files, and is what the C Data importer
now does for metadata on the field.
2. Raise the ambiguity upstream, since even Arrow C++ can't read back a
`dictionary<extension>` it writes to IPC.
### Component(s)
C#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
