bbiiaaoo opened a new issue, #12382: URL: https://github.com/apache/gravitino/issues/12382
### Describe the feature Support `TableChange.AddColumn` for Lance tables in the Generic Lakehouse Catalog through the Gravitino table API and Java client. Multiple `AddColumn` changes in one request should be applied in a single Lance schema commit. Existing rows should expose the newly added nullable columns as `NULL`. ### Motivation Schema evolution is a common requirement for Lance tables. Currently, users cannot add Lance columns through Gravitino and must modify the dataset through an external Lance client, which may cause the Lance schema and Gravitino metadata to become inconsistent. Providing this capability through Gravitino keeps schema changes and table metadata under the same catalog operation. ### Describe the solution - Validate all changes before modifying the Lance dataset. - Initially support only top-level, nullable columns appended to the end of the schema. - Reject default values, auto-increment columns, custom positions, and nested columns. - Reject requests that combine `AddColumn` with other table-change types. - Convert all new columns to Arrow fields and call `Dataset.addColumns` once. - Verify the Lance schema before and after the operation. - Persist the new columns and Lance dataset version in Gravitino with optimistic-lock protection. - Attempt to remove the added Lance columns if result validation or Gravitino metadata persistence fails. - Add unit and integration tests covering batching, conflicts, failure recovery, and `NULL` backfill. - Document the supported behavior and limitations. ### Additional context The first phase does not include: - Nested columns - Non-nullable columns - Default-value backfill - `FIRST` or `AFTER` positions - Auto-increment columns - A Lance REST `/add_columns` endpoint Lance and the Gravitino entity store do not share a distributed transaction, so failure compensation is best-effort. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
