CurtHagenlocher opened a new issue, #473:
URL: https://github.com/apache/arrow-dotnet/issues/473
### Describe the bug, including details regarding any error messages,
version, and platform.
For a dictionary-encoded extension type, such as `dictionary<int32,
arrow.uuid>`, the C Data Interface exporter and importer disagree with Arrow
C++ about which `ArrowSchema` carries `ARROW:extension:name`. As a result, such
a field can't be exchanged in either direction, and it doesn't round-trip
within .NET either.
**Where the metadata goes**
| Producer | Field schema (format `i`) | `dictionary` child (format `w:16`) |
|---|---|---|
| pyarrow 24.0.0 | none | `ARROW:extension:name=arrow.uuid` |
| Apache.Arrow (`main`) | `ARROW:extension:name=arrow.uuid` | none |
Arrow C++ treats the dictionary's value type as the extension type, so the
metadata belongs on the `dictionary` child. `CArrowSchemaExporter.ExportField`
instead moves a dictionary value's extension type up to the field
(`src/Apache.Arrow/C/CArrowSchemaExporter.cs`, the `field.DataType is
DictionaryType dt && dt.ValueType is ExtensionType dext` branch). That matches
IPC, where a dictionary-encoded field's metadata describes its value type, but
not the C Data Interface.
**Effects** (with `GuidExtensionDefinition` registered)
1. **.NET → pyarrow:** `pa.Field._import_from_c` fails with `Invalid storage
type for UuidType: dictionary<values=fixed_size_binary[16], indices=int32,
ordered=0>`. C++ tries to apply the extension to the field's whole type, which
is the dictionary.
2. **pyarrow → .NET:** `CArrowSchemaImporter.ImportField(ptr, registry)`
returns `dictionary<int32, fixed_size_binary[16]>`, and the imported
`DictionaryArray.Dictionary` is a `FixedSizeBinaryArray`. The importer never
reads the `dictionary` child's metadata.
3. **.NET → .NET:** this also comes back as `dictionary<int32,
fixed_size_binary[16]>` with the extension metadata left on the field.
`ImportedArrowSchema.GetAsField` calls `TryCreateType` with the
`DictionaryType` as the storage type, which `GuidExtensionDefinition` rejects.
Repro for (1):
```csharp
var field = new Field("f", new DictionaryType(Int32Type.Default,
GuidType.Default, false), true);
CArrowSchema* c = CArrowSchema.Create();
CArrowSchemaExporter.ExportField(field, c);
// c->metadata has ARROW:extension:name; c->dictionary->metadata is null
// pyarrow: pa.Field._import_from_c(ptr) -> ArrowInvalid: Invalid storage
type for UuidType
```
Repro for (2):
```python
pa.field("f", pa.dictionary(pa.int32(), pa.uuid()))._export_to_c(ptr)
# then CArrowSchemaImporter.ImportField(ptr, registry) -> value type is
FixedSizeBinaryType
```
**Suggested fix**
- **Export:** write the value type's extension metadata on the `dictionary`
child rather than the field. Any field-level metadata the user supplied should
stay on the field.
- **Import:** resolve the `dictionary` child's own extension metadata
against the registry, so its value type becomes the extension type. For
compatibility with schemas already exported by older Apache.Arrow versions, the
importer could also accept the extension name on the field when the field is
dictionary-encoded and the definition rejects the dictionary as storage.
#472 doesn't fix this. It passes the registry into the dictionary's nested
fields, but it doesn't read the `dictionary` child's own metadata.
Found while working on #466 / #472.
### Component(s)
C#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
