aokolnychyi commented on code in PR #5665:
URL: https://github.com/apache/iceberg/pull/5665#discussion_r960097179
##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkDataFile.java:
##########
@@ -84,10 +93,31 @@ public SparkDataFile(Types.StructType type, StructType
sparkType) {
sortOrderIdPosition = positions.get("sort_order_id");
}
+ private void wrapPartitionSpec(GenericRowWithSchema specRow) {
+ // We get all the partition fields, but want to project to the current one
+ StructType wrappedPartitionStruct = specRow.schema();
+
+ if (!wrappedPartitionStruct.equals(currentWrappedPartitionStruct)) {
+ this.currentWrappedPartitionStruct = wrappedPartitionStruct;
+
+ // The original IDs are lost in translation, therefore we apply the ones
that we know
Review Comment:
I think it may not work in all cases (e.g. rewriting manifests for an older
spec if the union type had forced renames).
Let me think a bit.
##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkDataFile.java:
##########
@@ -84,10 +93,31 @@ public SparkDataFile(Types.StructType type, StructType
sparkType) {
sortOrderIdPosition = positions.get("sort_order_id");
}
+ private void wrapPartitionSpec(GenericRowWithSchema specRow) {
+ // We get all the partition fields, but want to project to the current one
+ StructType wrappedPartitionStruct = specRow.schema();
+
+ if (!wrappedPartitionStruct.equals(currentWrappedPartitionStruct)) {
+ this.currentWrappedPartitionStruct = wrappedPartitionStruct;
+
+ // The original IDs are lost in translation, therefore we apply the ones
that we know
Review Comment:
I think it may not work in all cases (e.g. rewriting manifests for an older
spec if the union type had forced renames). Let me think a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
