andygrove commented on code in PR #7028:
URL: https://github.com/apache/datafusion-comet/pull/7028#discussion_r4395397066
##########
native/Cargo.toml:
##########
@@ -85,8 +85,8 @@ aws-sdk-sts = { version = "1.114.0", default-features = false
}
# Pinned to a revision rather than a release because no published iceberg-rust
# version is on our arrow: 0.10.1 requires arrow/parquet ^58. Move to a plain
# version requirement once a release ships on arrow 59. Bump policy: #5645.
-iceberg = { git = "https://github.com/apache/iceberg-rust";, rev =
"1f3bc34058e3f3e45336bc5428f37b92649a608b" }
-iceberg-storage-opendal = { git = "https://github.com/apache/iceberg-rust";,
rev = "1f3bc34058e3f3e45336bc5428f37b92649a608b", features = ["opendal-memory",
"opendal-fs", "opendal-s3", "opendal-gcs", "opendal-oss", "opendal-azdls"] }
+iceberg = { git = "https://github.com/apache/iceberg-rust";, rev =
"b154603a710d730d2f43f8af4db257cc137237ef" }
Review Comment:
Agreed. Comet's native tasks were built case-insensitive, and the
equality-delete predicate is the only thing that reads that flag. Its names
come from the task schema, so they match exactly. Tasks are now case-sensitive
in bc89e2cc37, with a test that fails without it.
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -850,18 +851,17 @@ case class CometScanRule(session: SparkSession)
} else {
resolvedProjectedFieldIds
}
- val changed = schemas
- .flatMap { schema =>
- IcebergReflection.nestedFieldsAddedOrRenamed(metadata.table,
schema, fieldIds)
- }
+ val disjoint = metadata.taskSchema.toSeq
+
.flatMap(IcebergReflection.structsWithNoFieldsInCommon(metadata.table, _,
fieldIds))
Review Comment:
Good catch. With nested pruning off the task schema holds nested fields the
query's schema doesn't, and iceberg-rust errors on a nested default the file
lacks. The default check now also covers the projected columns of the task
schema, with a test, in bc89e2cc37.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
