lvyanquan opened a new issue, #5399:
URL: https://github.com/apache/iceberg/issues/5399
error message:
```
Caused by: java.lang.NullPointerException: Cannot find source column: 3
at
org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:953)
~[iceberg-bundled-guava-0.13.2.jar:na]
at org.apache.iceberg.PartitionSpec$Builder.add(PartitionSpec.java:503)
~[iceberg-api-0.13.2.jar:na]
at
org.apache.iceberg.PartitionSpecParser.buildFromJsonFields(PartitionSpecParser.java:155)
~[iceberg-core-0.13.2.jar:na]
at
org.apache.iceberg.PartitionSpecParser.fromJson(PartitionSpecParser.java:78)
~[iceberg-core-0.13.2.jar:na]
at
org.apache.iceberg.TableMetadataParser.fromJson(TableMetadataParser.java:357)
~[iceberg-core-0.13.2.jar:na]
at
org.apache.iceberg.TableMetadataParser.fromJson(TableMetadataParser.java:288)
~[iceberg-core-0.13.2.jar:na]
```
json of metadata file contains information of
schemas/partition-specs/sort-orders.
But there is no link between schemas and partition-specs, thus deleting a
partitioned column will raise error while building history partition-specs,
because source-id could not be found in current schema. I think that schema-id
should be add to json of partition-specs.
part of metadata file:
```
"last-column-id":3,
"current-schema-id":1,
"schemas":[
{
"type":"struct",
"schema-id":0,
"fields":[
{
"id":1,
"name":"name1",
"required":false,
"type":"string"
},
{
"id":2,
"name":"name2",
"required":false,
"type":"string"
},
{
"id":3,
"name":"name3",
"required":false,
"type":"string"
}
]
},
{
"type":"struct",
"schema-id":1,
"fields":[
{
"id":1,
"name":"name1",
"required":false,
"type":"string"
},
{
"id":2,
"name":"name2",
"required":false,
"type":"string"
}
]
}
],
"default-spec-id":1,
"partition-specs":[
{
"spec-id":0,
"fields":[
{
"name":"name3",
"transform":"identity",
"source-id":3,
"field-id":1000
}
]
},
{
"spec-id":1,
"fields":[
]
}
],
"last-partition-id":1000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
