CurtHagenlocher commented on code in PR #474:
URL: https://github.com/apache/arrow-dotnet/pull/474#discussion_r4309335632
##########
src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -59,17 +67,25 @@ public static unsafe void ExportType(IArrowType datatype,
CArrowSchema* schema)
throw new ArgumentNullException(nameof(schema));
}
+ // A dictionary-encoded field's extension metadata describes its
value type. IPC
+ // keeps it on the field, but the C Data Interface puts it on the
dictionary child.
+ IReadOnlyDictionary<string, string> dictionaryMetadata = null;
+ if (datatype is DictionaryType)
+ {
+ metadata = SplitExtensionMetadata(metadata, out
dictionaryMetadata);
Review Comment:
Agreed, thanks. This was a real problem. On a dictionary field, C Data
extension metadata means the extension's storage is the whole dictionary;
that's how Arrow C++ exports an extension over `dictionary<int32, string>`.
Moving unresolved keys to the dictionary child changed that meaning.
Fixed in 4ad0d8b:
- The exporter moves field-level extension keys to the dictionary child only
when they name the resolved `ExtensionType` value type, which is how the IPC
reader leaves them. Unresolved keys stay on the field.
- The importer no longer copies dictionary child extension metadata to the
field, since that would cause the same misplacement in reverse. Like Arrow C++,
an unresolved value type's extension metadata is dropped.
- Added `CDataDictionaryStorageExtensionPassThroughWithoutRegistry`, which
imports an extension over `dictionary<int32, string>` without its definition,
re-exports it, and checks that the metadata stays on the field. Also added
`CDataExportKeepsUnresolvedDictionaryFieldExtensionMetadata`.
IPC has the same ambiguity, and our IPC reader resolves it the opposite way
from Arrow C++. That's filed as #475.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
