nevzheng opened a new pull request, #13331: URL: https://github.com/apache/gravitino/pull/13331
### What changes were proposed in this pull request? Add two Iceberg catalog properties that set the format version of new tables: - `table-format-version.default`: the format version of a new table that does not set `format-version`. Defaults to `2`, today's behaviour. - `table-format-version.max`: the highest format version a table may be created at. Defaults to `4`, the build ceiling below. Both accept the values `format-version` accepts (`1`-`4`). The Iceberg catalog applies them to tables created through the Gravitino API: a table without `format-version` gets the catalog default, and a create above the maximum fails with `IllegalArgumentException` (HTTP 400): `Iceberg format-version N exceeds the catalog limit M set by 'table-format-version.max'`. Existing tables are not changed. **Build ceiling.** `IcebergConstants.SUPPORTED_TABLE_FORMAT_VERSIONS` (`1`-`4`, the bundled Iceberg's maximum) is the highest version this Gravitino build accepts, and `table-format-version.max` can only lower it. A request above it fails with HTTP 400 and `Iceberg format-version 5 is not supported by this Gravitino (supports 1-4)`, whether or not the maximum is set. A guard test, `TestGravitinoTableFormatVersionCeiling`, fails when the ceiling and Iceberg's `TableMetadata.SUPPORTED_TABLE_FORMAT_VERSION` differ, so an Iceberg upgrade forces a review of the ceiling. **Validation at load.** A catalog whose default exceeds the maximum, or whose `table-default.format-version` (set directly or with `gravitino.bypass.`) conflicts with them, still loads as metadata, but every schema and table operation on it fails with `IllegalArgumentException` (HTTP 400) naming both properties. When the default is set, `IcebergConfig` also writes it to Iceberg's `table-default.format-version`. The shared parsing and validation live in `IcebergPropertiesUtils` and `IcebergConfig`, so the Iceberg REST catalog can reuse them in a follow-up PR, which also documents the upgrade limit. ### Why are the changes needed? Today every catalog creates tables at format version `2` and allows any version the bundled Iceberg writes. Operators need a per-catalog default and ceiling so they control which versions their engines must read and every new table gets a predictable version. Part of #13330 ### Does this PR introduce _any_ user-facing change? Yes, two new optional catalog properties, `table-format-version.default` and `table-format-version.max`, and the build ceiling, documented in `docs/lakehouse-iceberg-catalog.md`. A catalog that sets neither behaves as before, except that a version above `4` is refused with the ceiling message. ### How was this patch tested? Unit tests only. Coverage for the Gravitino API (the Iceberg REST catalog is covered in the follow-up PR): | Case | Gravitino API create | Covered in | |---|---|---| | No version, both unset | version 2 | `TestIcebergTable` | | No version, default=3 | version 3 | `TestIcebergTable` | | Requested up to max | kept | `TestIcebergTable` | | Requested above a set max | 400, max message, not created | `TestIcebergTable`, `TestIcebergPropertiesUtils` | | Requested 5, max unset | 400, ceiling message, not created | `TestIcebergTable`, `TestIcebergPropertiesUtils` | | Bad property value (0, 5, abc, blank) | refused | `TestIcebergCatalogPropertiesMetadata`, `TestIcebergConfig` | | Default above max, or conflicting `table-default.format-version`, at load | every operation fails with 400 | `TestIcebergConfig`, `TestIcebergTable`, `TestIcebergCatalog` | | Ceiling equals the bundled Iceberg's max | guard | `TestGravitinoTableFormatVersionCeiling` | `TestGravitinoTableFormatVersionCeiling` lives in `iceberg-common` in the `org.apache.iceberg` package, because Iceberg's constant is package-private. `./gradlew :catalogs:catalog-common:test :catalogs:catalog-lakehouse-iceberg:test :iceberg:iceberg-common:test :iceberg:iceberg-rest-server:test -PskipITs`: 690 tests, 0 failures, 0 errors, 0 skipped (catalog-common 40, catalog-lakehouse-iceberg 124, iceberg-common 82, iceberg-rest-server 444). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
