anoopj commented on code in PR #2869:
URL: https://github.com/apache/iceberg-rust/pull/2869#discussion_r3632122850
##########
crates/iceberg/src/scan/cache.rs:
##########
@@ -69,21 +69,46 @@ impl PartitionFilterCache {
format!("Could not find partition spec for id {spec_id}"),
))?;
- let partition_type = partition_spec.partition_type(schema)?;
- let partition_fields = partition_type.fields().to_owned();
- let partition_schema = Arc::new(
- Schema::builder()
- .with_schema_id(partition_spec.spec_id())
- .with_fields(partition_fields)
- .build()?,
- );
+ // A historical spec may reference a source column that was later
dropped from the
+ // schema, which is a legitimate v2+ state. Such a spec cannot be
resolved to a
+ // partition type, so it falls back to an always-true filter: files
under the spec
+ // are not partition-pruned but still receive the row filter. Any
other resolution
+ // failure is unexpected and propagates. The fallback is cached by
spec id like any
+ // other filter; this is safe only because the cache lives per-scan in
`PlanContext`
+ // with a fixed schema and predicate. Hoisting it to table or catalog
scope would
+ // pin a spec to always-true even for a later scan whose schema could
resolve it.
+ // TODO(https://github.com/apache/iceberg-rust/issues/2844): derive
partition types from
+ // transforms where possible to restore pruning on a historical spec's
still-live fields,
+ // which also makes the fallback per-term (dropped fields only)
instead of per-spec.
+ let dropped_source_column = partition_spec
Review Comment:
renamed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
