ziting-openai commented on code in PR #5262:
URL: https://github.com/apache/datafusion-comet/pull/5262#discussion_r3930092772
##########
native/Cargo.toml:
##########
@@ -62,8 +62,8 @@ object_store = { version = "0.13.2", features = ["gcp",
"azure", "aws", "http"]
url = "2.2"
aws-config = "1.8.18"
aws-credential-types = "1.2.13"
-iceberg = { git = "https://github.com/apache/iceberg-rust";, rev =
"8adaa872f31549dd5ad8255848715758228038bc" }
-iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust";,
rev = "8adaa872f31549dd5ad8255848715758228038bc", features = ["opendal-memory",
"opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
+iceberg = { git = "https://github.com/apache/iceberg-rust";, rev =
"665c64e48e8d33797ecb1a421f327edd9b024879" }
Review Comment:
Rechecked `598f5e979bdaf82b929596dfcae6ec116bae265e`: the compilation sites
are migrated, but builder validation breaks a valid partition-evolution case.
On a format-v1 table partitioned by `[identity(region), identity(category)]`,
remove the trailing `category` partition field (retained as `void`), write a
file under that spec, then drop the `category` source column. The JVM
serializer filters the now-unknown trailing field from partition values but
keeps the full spec because `region` remains. `.build()` at `planner.rs:4540`
rejects this one-value/two-field task. The previous ordinary read could read
the identity value at index 0 and skip the trailing void field. Please keep the
serialized spec and values consistent for this mixed live/dropped case and add
a regression.
The existing
`test_unified_partition_type_merges_specs_by_descending_spec_id` fixture also
selects a two-field spec at `planner.rs:7097` without `partition_data_idx`; the
builder rejects missing partition values, so `.expect("parse tasks")` at line
7103 fails. Supply matching partition data and its index there.
These remain part of this Iceberg migration P2. Source-traced against the
pinned Rust dependency and Java 1.11; not executed locally.
CI update: the [exact-head Rust
job](https://github.com/apache/datafusion-comet/actions/runs/33824701130/job/100875289278)
now reproduces the fixture failure at `planner.rs:7103` with `FileScanTask
with a partitioned spec requires partition values`. The evolved-table runtime
case remains source-traced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
