ziting-openai commented on code in PR #5262:
URL: https://github.com/apache/datafusion-comet/pull/5262#discussion_r3935733976
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala:
##########
@@ -1082,6 +1082,19 @@ object CometIcebergNativeScan extends
CometOperatorSerde[CometBatchScanExec] wit
}
}
+ // iceberg-rust validates a FileScanTask by resolving its
partition spec against
+ // the task schema, so a task carrying a partition spec needs
that spec's source
+ // columns present even when the query projects them out (e.g.
selecting only
+ // _spec_id / _partition). Union them in. project_field_ids
still drives the read,
+ // so these columns are not materialized into the output.
+ val schema: AnyRef =
+ IcebergReflection
+ .schemaWithRequiredFields(
+ baseSchema,
+ metadata.table,
+ IcebergReflection.partitionSourceFieldIds(task,
fileScanTaskClass))
Review Comment:
[P2] Check nested field IDs before adding partition sources
This new call also runs when the partition source already exists inside a
struct. For a table with `id` and `s:struct<region:string>` partitioned by
`identity(s.region)`, an ordinary `SELECT id` without deletes uses the full
table schema here. `schemaWithRequiredFields` checks presence through
`buildFieldIdMapping`, which only enumerates top-level columns, so it considers
the nested `region` ID missing and appends the same field at the top level. The
schema then contains that ID twice (`s.region` and `region`); Iceberg 1.11's
`Schema` constructor rejects the duplicate while building its ID-to-name map.
This runs in `serializePartitions` at execution time and propagates out of
`CometIcebergNativeScanExec`, so the scan fails instead of falling back. Please
use recursive field-ID lookup and preserve nesting when restoring genuinely
missing fields, with a nested-partition regression test. This is source-traced;
no local test was run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
