A while back I bumped into an issue with what seems to be an inconsistency
in the partition spec API or maybe it's just an implementation bug.
Attempting to have multiple partitions specs on the same schema field I
bumped into an issue regarding the fact that while the API allows for
multiple partitions spec defined for same field, internally this conflicts
with the assumption that there is only one partition spec per field.
Given this partition spec:
PartitionSpec spec = PartitionSpec.builderFor(schema)
.withSpecId(0)
.year("timestamp")
.month("timestamp")
.day("timestamp")
.hour("timestamp")
.build();
Trying to validate partition pruning with similar code to:
UnboundPredicate<Object> match = Expressions.equal("timestamp",
Literal.of("2019-01-11T00:00:00.000000").to(TimestampType.withoutZone()).value());
Assert.assertTrue(
new InclusiveManifestEvaluator(spec,
match).eval(table.currentSnapshot().manifests().get(0));
I get an unexpected google collection exception:
java.lang.IllegalArgumentException: Multiple entries with same key:
1=org.apache.iceberg.PartitionField@da8cdda7 and
1=org.apache.iceberg.PartitionField@e5c6fddb
at
com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:215)
at
com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:209)
at
com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:147)
at
com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:110)
at
com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:393)
at
org.apache.iceberg.PartitionSpec.lazyFieldsBySourceId(PartitionSpec.java:232)
at
org.apache.iceberg.PartitionSpec.getFieldBySourceId(PartitionSpec.java:95)
at
org.apache.iceberg.expressions.Projections$InclusiveProjection.predicate(Projections.java:208)
at
org.apache.iceberg.expressions.Projections$InclusiveProjection.predicate(Projections.java:200)
at
org.apache.iceberg.expressions.Projections$BaseProjectionEvaluator.predicate(Projections.java:185)
at
org.apache.iceberg.expressions.Projections$BaseProjectionEvaluator.predicate(Projections.java:136)
at
org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:152)
at
org.apache.iceberg.expressions.Projections$BaseProjectionEvaluator.project(Projections.java:152)
at
org.apache.iceberg.expressions.InclusiveManifestEvaluator.<init>(InclusiveManifestEvaluator.java:63)
at
org.apache.iceberg.expressions.InclusiveManifestEvaluator.<init>(InclusiveManifestEvaluator.java:56)
at
org.apache.iceberg.TestScansAndSchemaEvolution.testMultiPartitionPerFieldTransform(TestScansAndSchemaEvolution.java:177)
I was wondering if this issue is tracked so maybe I could help out.
Thanks,
/Filip
