JosephLenton opened a new issue, #2913:
URL: https://github.com/apache/iceberg-rust/issues/2913
### Apache Iceberg Rust version
None
### Describe the bug
When making a table with a UUID partition, using the UUID type, transactions
to that table crash.
I'm not sure if this is a case that UUIDs just aren't supported. However the
type system suggests I should be able to have a UUID type, and for my needs
where I work this would end up being extremely common. As a workaround I'm
currently working around this by using a String type instead.
If it helps:
* I have this issue against both version `0.10.0` and `main`.
* I am using the S3 backend.
Btw I am able to work on this issue full time to help get this resolved. I
will need some guidance on where, primarily to avoid going off on tangents.
Also thank you guys for this library. It's already been a huge help!
### To Reproduce
I don't have a full reproducible example I'm afraid, as the code I am using
is split across a large project.
The steps are:
1. Create a table with a UUID field, which uses the type
`Type::Primitive(PrimitiveType::Uuid))`.
2. Add a partition using the UUID field
3. When writing data, use a UUID partition key.
4. Start a transaction.
5. Commit the transaction.
## Pseudo code
The pseudo code for this is along the lines of:
```rust
// 1. Create a table with a UUID field
let table = TableCreation::builder()
.name("ExampleTable".to_string())
.schema(
IcebergSchema::builder()
.with_fields([
// This is a UUID here
NestedField::required(1, "uuid",
Type::Primitive(PrimitiveType::Uuid))
.into(),
NestedField::optional(2, "value",
Type::Primitive(PrimitiveType::String))
.into(),
])
.build()
.unwrap(),
)
// 2. Add a partition using the UUID field
.partition_spec(
UnboundPartitionSpec::builder()
.with_spec_id(0)
.add_partition_fields([UnboundPartitionField::builder()
.source_id(1)
.field_id(1)
.name("uuid".to_string())
.transform(Transform::Identity)
.build()])?
.build(),
)
.build();
// 3. When writing data, use a UUID partition key.
let uuid_key = Some(Literal::uuid(Uuid::new_v4()));
let data = Struct::from_iter([uuid_key]);
let partition_key = PartitionKey::new(spec, schema, data);
let rolling_builder = todo!(" ... omitted ...");
let data_writer = DataFileWriterBuilder::new(rolling_builder)
.build(Some(partition_key))
.await?;
// 4. Start a transaction.
let data_files = todo!(" ... omitted ...");
let catalog = todo!(" ... omitted ...");
let transaction = Transaction::new(&table);
transaction
.fast_append()
.add_data_files(data_files)
.apply(transaction)?
// 5. Start a transaction.
.commit(catalog.as_ref())
.await?; // <-- it is failing here
```
## Stack trace
Here is part of the stack trace. I have excluded the work code stack trace
below, and only included the Iceberg parts:
```
called `Result::unwrap()` on an `Err` value: Iceberg(DataInvalid => Failure
in conversion with avro
Source: Could not find matching type in UnionSchema { schemas: [Null, Uuid],
variant_index: {Null: 0, Uuid: 1} } for Bytes([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0])
Backtrace:
0: std::backtrace_rs::backtrace::libunwind::trace
at
/rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
1:
std::backtrace_rs::backtrace::trace_unsynchronized::<<std::backtrace::Backtrace>::create::{closure#0}>
at
/rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
2: <std::backtrace::Backtrace>::create
at
/rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/backtrace.rs:331:13
3: <iceberg::error::Error>::new::<&str>
at /iceberg-rust/crates/iceberg/src/error.rs:234:24
4: <iceberg::error::Error as
core::convert::From<apache_avro::error::Error>>::from
at ~/iceberg-rust/crates/iceberg/src/error.rs:340:17
5:
<core::result::Result<iceberg::spec::manifest_list::manifest_file::ManifestFile,
iceberg::error::Error> as
core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,
apache_avro::error::Error>>>::from_residual
at
~/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:2189:27
6:
<iceberg::spec::manifest::writer::ManifestWriter>::write_manifest_file::{closure#0}
at
~/iceberg-rust/crates/iceberg/src/spec/manifest/writer.rs:498:21
7:
<iceberg::transaction::snapshot::SnapshotProducer>::write_added_manifest::{closure#0}
at
~/iceberg-rust/crates/iceberg/src/transaction/snapshot.rs:340:38
8:
<iceberg::transaction::snapshot::SnapshotProducer>::produce_manifests::<iceberg::transaction::append::FastAppendOperation,
iceberg::transaction::snapshot::DefaultManifestProcess>::{closure#0}
at
~/iceberg-rust/crates/iceberg/src/transaction/snapshot.rs:367:62
9:
<iceberg::transaction::snapshot::SnapshotProducer>::commit::<iceberg::transaction::append::FastAppendOperation,
iceberg::transaction::snapshot::DefaultManifestProcess>::{closure#0}
at
~/iceberg-rust/crates/iceberg/src/transaction/snapshot.rs:494:14
10: <iceberg::transaction::append::FastAppendAction as
iceberg::transaction::action::TransactionAction>::commit::{closure#0}
at
~/iceberg-rust/crates/iceberg/src/transaction/append.rs:108:14
11: <core::pin::Pin<alloc::boxed::Box<dyn
core::future::future::Future<Output =
core::result::Result<iceberg::transaction::action::ActionCommit,
iceberg::error::Error>> + core::marker::Send>> as
core::future::future::Future>::poll
at
~/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/future/future.rs:133:9
12: <iceberg::transaction::Transaction>::do_commit::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/mod.rs:233:75
13:
<iceberg::transaction::Transaction>::commit::{closure#0}::{closure#0}::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/mod.rs:195:48
14:
<backon::retry_with_context::RetryWithContext<backon::backoff::exponential::ExponentialBackoff,
iceberg::table::Table, iceberg::error::Error,
iceberg::transaction::Transaction,
<iceberg::transaction::Transaction>::commit::{closure#0}::{closure#0}::{closure#0},
<iceberg::transaction::Transaction>::commit::{closure#0}::{closure#0},
tokio::time::sleep::sleep,
<iceberg::transaction::Transaction>::commit::{closure#0}::{closure#1}, for<'a>
fn(&'a iceberg::error::Error, core::time::Duration)> as
core::future::future::Future>::poll
at
~/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/backon-1.6.0/src/retry_with_context.rs:327:58
15: <iceberg::transaction::Transaction>::commit::{closure#0}
at ~/iceberg-rust/crates/iceberg/src/transaction/mod.rs:202:10
16:
my_library::iceberg::iceberg_data_writer::IcebergDataWriter>::flush::{closure#0}
at ... this and below is our code ...
```
I would be happy to write a failing test within the Iceberg repo. I'm not
100% sure where the best place would be to recreate this, so some guidance on
that would help.
### Expected behavior
This is what I would expect:
* It should not error
* I should get a partition on S3 with a hyphen formatted UUID. i.e.
`s3://.../uuid=4bc971ed-7590-4552-8408-d60928ba76f0/...`
### Willingness to contribute
I would be willing to contribute a fix for this bug with guidance from the
Iceberg community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
