rdblue commented on code in PR #11947:
URL: https://github.com/apache/iceberg/pull/11947#discussion_r1917395711
##########
core/src/main/java/org/apache/iceberg/TableMetadataParser.java:
##########
@@ -372,6 +373,7 @@ public static TableMetadata fromJson(String
metadataLocation, JsonNode node) {
formatVersion == 1, "%s must exist in format v%s", SCHEMAS,
formatVersion);
schema = SchemaParser.fromJson(JsonUtil.get(SCHEMA, node));
+ Schema.checkCompatibility(schema, formatVersion);
Review Comment:
@HonahX, when I introduced this compatibility check, I purposely called it
at write time rather than at read time. I've also pushed back on strictness in
the read path like this in the REST catalog and other places. The reason for it
is that we don't want to immediately reject metadata on read because that
prevents the library from being used to debug or fix metadata issues.
For example, this check validates that initial default values are not used
in v2 tables. But what if a table is created by another library that way?
Checking compatibility here prevents loading the table at all, let alone using
this library to drop the problematic column from the schema using SQL DDL or
direct calls. Failing to load the table has much broader consequences, like
failing existence checks because `tableExists` calls `loadTable`, failing to
run `information_schema` queries that are unrelated, or failing to run
`expireSnapshots` and remove old data -- which can cause compliance problems.
I think that a better time to fail is when the actual problem caused by the
compatibility issue happens. For instance, if there is a default that can't be
applied, then the library should fail to read from the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
