rdblue commented on a change in pull request #2284:
URL: https://github.com/apache/iceberg/pull/2284#discussion_r642710220
##########
File path: core/src/main/java/org/apache/iceberg/TableMetadata.java
##########
@@ -703,6 +705,35 @@ public TableMetadata removeSnapshotLogEntries(Set<Long>
snapshotIds) {
snapshots, newSnapshotLog, addPreviousFile(file, lastUpdatedMillis));
}
+ private PartitionSpec reassignPartitionIds(PartitionSpec partitionSpec,
AtomicInteger lastPartitionId) {
+ if (formatVersion > 1) {
+ Map<Pair<Integer, String>, Integer> transformToFieldId = specs.stream()
+ .flatMap(spec -> spec.fields().stream())
+ .collect(Collectors.toMap(
+ field -> Pair.of(field.sourceId(), field.transform().toString()),
+ PartitionField::fieldId,
+ (n1, n2) -> n2));
+
+ PartitionSpec.Builder specBuilder =
PartitionSpec.builderFor(partitionSpec.schema())
+ .withSpecId(partitionSpec.specId());
+
+ for (PartitionField field : partitionSpec.fields()) {
+ // reassign the partition field ids
+ Integer fieldId = transformToFieldId.computeIfAbsent(
+ Pair.of(field.sourceId(), field.transform().toString()), k ->
lastPartitionId.incrementAndGet());
+ specBuilder.add(
+ field.sourceId(),
+ fieldId,
+ field.name(),
+ field.transform());
+ }
+ return specBuilder.build();
+ } else {
+ // noop for v1 table
+ return partitionSpec;
Review comment:
@jun-he, you can't drop a column if it is referenced by the partition
spec. I don't think we could get into the case you mentioned because of that
limitation. When this method runs, the column was referenced by the previous
version of the field. So we know that the column exists. After this method
runs, the void partition should prevent the column from being removed. Am I
missing a case where the column can be dropped but still referenced by a void
transform?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
