bbiiaaoo opened a new pull request, #12383:
URL: https://github.com/apache/gravitino/pull/12383
### What changes were proposed in this pull request?
This pull request adds support for `TableChange.AddColumn` on Lance tables
in the Generic Lakehouse Catalog through the Gravitino table API and Java
client.
The implementation:
- Batches multiple `AddColumn` changes into a single Lance schema commit.
- Supports nullable, top-level columns appended to the schema.
- Uses Lance-native schema evolution so existing rows expose new columns as
`NULL`.
- Validates column options, duplicate names, and schema consistency before
modifying the dataset.
- Verifies the resulting Lance schema before updating Gravitino metadata.
- Protects metadata updates with optimistic locking.
- Attempts to remove added columns when result validation or metadata
persistence fails.
- Adds documentation for the supported behavior and limitations.
### Why are the changes needed?
Lance table schema evolution is not currently available through the
Gravitino API. Users must modify the Lance dataset through an external client,
which may cause the physical Lance schema and Gravitino metadata to become
inconsistent.
Supporting `AddColumn` through Gravitino provides a catalog-managed schema
evolution path while avoiding data rewrites for historical rows.
Fix: #12382
### Does this PR introduce _any_ user-facing change?
Yes.
Lance tables now support adding one or more columns through the Gravitino
table API and Java client.
The first phase has the following limitations:
- Only top-level columns are supported.
- New columns must be nullable.
- New columns are appended to the end of the schema.
- Default values and auto-increment columns are not supported.
- `AddColumn` cannot be combined with other table-change types in one
request.
- The Lance REST `/add_columns` endpoint is not included in this change.
No new configuration properties are introduced.
### How was this patch tested?
The following checks were run:
```shell
./gradlew spotlessApply
./gradlew :catalogs:catalog-lakehouse-generic:check -PskipITs
./gradlew :catalogs:catalog-lakehouse-generic:test \
--tests
org.apache.gravitino.catalog.lakehouse.lance.integration.test.CatalogGenericCatalogLanceIT.testAddNullableColumnsBackfillNullInSingleLanceVersion
\
-PskipTests -PskipDockerTests=false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
