FANNG1 commented on issue #11027:
URL: https://github.com/apache/gravitino/issues/11027#issuecomment-4456443708
Good point on the version lookup cost. Let me clarify how Lance version
resolution actually works, as the cost depends on which naming scheme is in use.
**Two different "version" queries**
| API | Cost |
|---|---|
| Currently checked-out version | Zero I/O — reads `manifest.version` from
the in-memory manifest |
| Latest version on storage | I/O required — explained below |
**Manifest naming scheme matters**
Lance has two manifest naming schemes under `_versions/`:
- **V1**: `_versions/{version}.manifest` — files are named by ascending
version number, so finding the latest requires listing all files and taking the
max.
- **V2**: `_versions/{u64::MAX - version:020}.manifest` — version numbers
are *inverted* and zero-padded, so the latest version always sorts *first*
lexicographically.
V2 was [specifically
designed](https://github.com/lancedb/lance/blob/main/rust/lance-table/src/io/commit.rs)
to make latest-version lookup O(1) on object stores: one `LIST` call returns
the first entry, which is the latest manifest. V2 paths are the default for all
new datasets (`enable_v2_manifest_paths=true`).
For V1 naming on object stores (S3, GCS, etc.), or any store where `LIST` is
not lexically ordered, Lance must scan all files in `_versions/` and take the
maximum — that is the expensive O(n) case @yuqi1129 described.
**Schema is free once you have the manifest path**
The schema is embedded inside the manifest file itself. So "resolve latest
version" and "read schema" are not two separate I/O operations — once
`current_manifest_path()` returns the manifest file path, a single file read
gives you both the version number and the full schema.
**Caching design**
For the cache invalidation question: the lightweight staleness check is
exactly `latest_version_id()` — on V2 + S3 this is one LIST call returning the
first key, which is cheap. The cache can be keyed by version number (or
manifest path). On a cache hit the version hasn't changed, no re-read needed.
On a miss, read the manifest once to get the new schema.
This is analogous to Iceberg's `metadata-location` pointer check, just using
a directory list instead of a pointer file.
**Namespace-managed versioning path**
When `namespaceClientManagedVersioning=true`, `latest_version_id()` calls
`namespace.list_table_versions(limit=1, desc=true)` instead of touching the
object store directly. The namespace server returns the latest version in one
API call, completely bypassing the file scan. This is another reason to prefer
namespace-managed versioning for catalog integrations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
