ajsquared commented on code in PR #5262:
URL: https://github.com/apache/datafusion-comet/pull/5262#discussion_r3936986159
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala:
##########
@@ -1082,6 +1082,19 @@ object CometIcebergNativeScan extends
CometOperatorSerde[CometBatchScanExec] wit
}
}
+ // iceberg-rust validates a FileScanTask by resolving its
partition spec against
+ // the task schema, so a task carrying a partition spec needs
that spec's source
+ // columns present even when the query projects them out (e.g.
selecting only
+ // _spec_id / _partition). Union them in. project_field_ids
still drives the read,
+ // so these columns are not materialized into the output.
+ val schema: AnyRef =
+ IcebergReflection
+ .schemaWithRequiredFields(
+ baseSchema,
+ metadata.table,
+ IcebergReflection.partitionSourceFieldIds(task,
fileScanTaskClass))
Review Comment:
The nested partition-source concern remains active at
`3ee7fa65b2f5e7018b5c60f4088a57eba2b9934f`. The new `schemaWithRequiredFields`
call runs even without deletes, using the full table schema. Its
`buildFieldIdMapping` checks only `schema.columns()`, so an existing nested
source such as `s.region` is treated as missing; `findFieldObject` retrieves
that same nested field and the helper appends it at the top level, duplicating
its ID. Please check IDs recursively and preserve nesting when adding missing
sources. This is source-confirmed; I did not run tests or inspect CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
