JosephLenton opened a new issue, #2929:
URL: https://github.com/apache/iceberg-rust/issues/2929
### Apache Iceberg Rust version
None
### Describe the bug
Arrow supports a UUID extension type. When calling `arrow_schema_to_schema`,
this extension is ignored. This adds friction when using UUID fields with
Iceberg.
I propose that `arrow_schema_to_schema` takes the UUID extension into
account, and when present, converts items to the Iceberg UUID type.
This adds friction because in practice we often need both Arrow and Iceberg
schemas for fields. Which means either defining the fields twice by hand, or
adding a wrapper around `arrow_schema_to_schema` to fixup the conversion (this
is what I have had to do in my own work).
### To Reproduce
Here is a test for `crates/iceberg/src/arrow/schema.rs` which recreates the
behaviour I would expect:
```rust
use arrow_schema::extension::Uuid as DataTypeUuidExt;
#[test]
fn test_arrow_schema_to_schema_should_convert_uuids() {
let converted_schema = arrow_schema_to_schema(&ArrowSchema::new(vec![
simple_field("uuid_field", DataType::FixedSizeBinary(16), false, "1")
.with_extension_type(DataTypeUuidExt),
]))
.unwrap();
let expected = Schema::builder()
.with_fields([NestedField::required(
1,
"uuid_field",
Type::Primitive(PrimitiveType::Uuid),
)
.into()])
.build()
.unwrap();
pretty_assertions::assert_eq!(expected, converted_schema);
}
```
### Expected behavior
* When I pass a field with the Arrow UUID type to `arrow_schema_to_schema`,
the field is converted to the Iceberg `PrimitiveType::Uuid`.
* The Arrow UUID type is a `DataType::FixedSizeBinary(16)` with the Arrow
UUID metadata of `ARROW:extension:metadata=arrow.uuid` set to it. This is based
on the Arrow docs:
https://arrow.apache.org/docs/format/CanonicalExtensions.html#uuid
* _Optional_ -- If a field has an Arrow `DataType` which is **not**
`DataType::FixedSizeBinary(16)`, but does have the UUID metadata set, then it
should raise an error.
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
