dotjdk opened a new issue, #5676:
URL: https://github.com/apache/iceberg/issues/5676
### Apache Iceberg version
0.14.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
On a format version 2 table, dropping an old partition column on an iceberg
table causes a `NullPointerException` in `PartitionSpecBuilder`, and every
subsequent operation on the table throws the same exception.
```java
java.lang.NullPointerException: Cannot find source column: 2
at
org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:963)
at org.apache.iceberg.PartitionSpec$Builder.add(PartitionSpec.java:517)
at
org.apache.iceberg.UnboundPartitionSpec.copyToBuilder(UnboundPartitionSpec.java:56)
at
org.apache.iceberg.UnboundPartitionSpec.bind(UnboundPartitionSpec.java:44)
at
org.apache.iceberg.PartitionSpecParser.fromJson(PartitionSpecParser.java:87)
```
On a v1 table, the table is still accessible after, but the `ALTER TABLE`
throws the same NPE.
The issue is easily reproducible using the following script in a Spark Shell:
```sql
CREATE TABLE data.test_table (ts timestamp not null, day_of_ts date) USING
iceberg PARTITIONED BY (day_of_ts);
ALTER TABLE data.test_table SET TBLPROPERTIES ('format-version' = '2');
ALTER TABLE data.test_table REPLACE PARTITION FIELD day_of_ts WITH days(ts);
ALTER TABLE data.test_table DROP COLUMN day_of_ts;
REFRESH TABLE data.test_table;
SELECT * FROM data.test_table;
```
On closer inspection of the metadata, I see that on a v1 table, the metadata
is not updated when dropping the old partition field, which explains why the
table is still working on v1 after, but I also don't see what the issue is with
the v2 metadata.
I am using Spark Shell on Spark 3.3.0 with Iceberg 0.14.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
