yuqi1129 opened a new pull request, #12895:
URL: https://github.com/apache/gravitino/pull/12895
### What changes were proposed in this pull request?
- Re-examine the Lance dataset schema whenever the stored columns are empty,
even when `lance.version` is already recorded.
- Apply this recovery behavior to both `DECLARED_AND_EMPTY` and
`VERSION_CHECK`.
- Avoid redundant metadata updates when the dataset remains empty and its
version is unchanged.
- Update the schema refresh documentation and regression tests.
### Why are the changes needed?
The existing logic treats `lance.version` as proof that an empty stored
schema is complete. However, the version may already have been recorded while
the Gravitino columns are incomplete.
This leaves the table permanently stuck with empty columns because
subsequent loads skip reading the actual Lance schema.
Fix: #12407
### Does this PR introduce _any_ user-facing change?
Yes. Lance tables with empty stored columns can now recover their schema
from the underlying dataset even when `lance.version` is already present.
There are no API, configuration, property-key, or data-format changes.
Genuinely zero-column datasets are re-examined on load but do not trigger
redundant metadata writes.
### How was this patch tested?
- Added regression coverage for repairing empty stored columns with a
matching recorded version in both refresh modes.
- Added coverage for genuinely empty datasets avoiding repeated metadata
updates.
- Added coverage for falling back to stored metadata when the dataset cannot
be opened.
- Ran:
- `./gradlew :catalogs:catalog-lakehouse-generic:build -PskipWeb=true
-PskipDockerTests=true`
- `./gradlew :docs:build -PskipWeb=true -PskipDockerTests=true`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
