This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 157c06a70c6 [docs](lance) Document time travel, tags, branches and
managed versioning (#4172)
157c06a70c6 is described below
commit 157c06a70c606b85501911f04dd53e94204c9593
Author: zy-kkk <[email protected]>
AuthorDate: Tue Sep 29 14:54:09 2026 +0800
[docs](lance) Document time travel, tags, branches and managed versioning
(#4172)
Documents Lance time travel, tags, branches and REST Namespace managed
versioning, added by apache/doris#68453 (issue apache/doris#66491).
Changes to `lakehouse/catalogs/lance-catalog.mdx` (4.x, English and
Chinese):
- Feature table: Time Travel is now supported (`FOR VERSION AS OF`, `FOR
TIME AS OF`, `@tag(...)`, `@branch(...)`), including tables whose
versions are managed by a REST Namespace.
- Version and compatibility: the FE uses `org.lance:lance-core:12.0.0`;
the BE reader is unchanged.
- REST catalog: the caution that managed-versioning tables are
unsupported is replaced by a pointer to the new section.
- New "Time Travel" section: version and timestamp semantics, tags and
branches, and the managed-versioning behavior and limitations.
- Removed the limitation that time travel is not supported.
## Versions
- [ ] dev
- [x] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [x] Checked by AI
- [ ] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not: the dev docs have no Lance catalog page yet, so only 4.x
changes.
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
---
.../lakehouse/catalogs/lance-catalog.mdx | 66 +++++++++++++++++++---
.../lakehouse/catalogs/lance-catalog.mdx | 66 +++++++++++++++++++---
2 files changed, 116 insertions(+), 16 deletions(-)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index f2f6ca1bbd4..7e6328025a7 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -45,12 +45,12 @@ Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lan
| Lance Cache | BE 内共享索引、元数据缓存及本地磁盘数据缓存 |
| TopN 两阶段读取 | 对向量和全文检索中可延迟的输出列默认开启，由 `enable_lance_lazy_materialization` 独立控制
|
| 写入 Lance | 暂不支持 |
-| Time Travel | 暂不支持 |
+| Time Travel | 支持 `FOR VERSION AS OF`、`FOR TIME AS OF`、`@tag(...)` 和
`@branch(...)`，包括由 REST Namespace 管理版本的表 |
| Hybrid Search | 暂不支持 |
## Lance 版本与兼容性
-Doris BE 数据读取器使用 `lance-c v0.1.9` 加 Doris 补丁构建，补丁包含多向量检索支持，并将其内置的 Lance Rust
crates 升级到 `11.0.0`（commit `ab6b5bbe`）。Doris FE 使用
`org.lance:lance-core:11.0.0` 读取 Namespace 和数据集元数据，FE 与 BE 的 Lance 版本统一为
`11.0.0`。Dataset 必须同时满足 FE 元数据加载和 BE 数据读取的兼容性要求。这些实现版本与数据集中记录的 Lance
`data_storage_version` 不是同一个概念。
+Doris BE 数据读取器使用 `lance-c v0.1.9` 加 Doris 补丁构建，补丁包含多向量检索支持，并将其内置的 Lance Rust
crates 升级到 `11.0.0`（commit `ab6b5bbe`）。Doris FE 使用
`org.lance:lance-core:12.0.0` 读取 Namespace 和数据集元数据；FE 只读元数据、不写数据集，因此 FE 采用较新的
Lance 版本不改变 BE 能读取的范围。Dataset 必须同时满足 FE 元数据加载和 BE 数据读取的兼容性要求。这些实现版本与数据集中记录的
Lance `data_storage_version` 不是同一个概念。
当前读取器的文件格式兼容情况如下：
@@ -92,7 +92,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
| `lance.namespace.parent` | 否 | 空 | 仅访问指定 Lance Namespace 及其子
Namespace。例如默认分隔符下，`production$analytics` 表示两级 Namespace。 |
| `lance.namespace.delimiter` | 否 | `$` | 用于解析 `lance.namespace.parent`，同时会传递给
REST Namespace 客户端。该配置不改变 Doris 中多级 Namespace 的展示方式。 |
| `lance.namespace.root_database` | 否 | `default` | Lance 根 Namespace 在 Doris
中映射的数据库名。 |
-| `lance.table_access_cache_ttl_seconds` | 否 | `60` | 非负整数，表示 FE
表访问缓存的最长有效期，单位为秒；`0` 表示禁用。适用于 Filesystem 和 REST Catalog。仅缓存不含下发存储配置或携带凭证 URI
的结果，详见 [FE 表访问缓存](#fe-表访问缓存)。 |
+| `lance.table_access_cache_ttl_seconds` | 否 | `60` | 非负整数，表示 FE
表访问缓存的最长有效期，单位为秒；`0` 表示禁用。适用于 Filesystem 和 REST Catalog。managed versioning
表、含下发存储配置或携带凭证 URI 的结果不缓存，详见 [FE 表访问缓存](#fe-表访问缓存)。 |
### Filesystem Catalog
@@ -314,9 +314,7 @@ REST 服务认证和对象存储认证是两套独立配置：`lance.rest.*` 用
如果不同表位于不同 Bucket，不要让它们共用上例中固定为 `my-bucket` 的 S3 虚拟主机 endpoint。应由 Namespace
为每张表下发匹配的 endpoint 和访问方式，或为不同 Bucket 分别配置 Catalog。对于支持 Path Style 的服务，也可以使用不带
Bucket 的服务 endpoint，配合 `use_path_style=true`，由客户端从各表 URI 读取 Bucket。
-:::caution
-当前 BE Reader 不支持由 REST Namespace 管理版本的 Lance 表（Managed Versioning）。
-:::
+如果 REST Namespace 管理该表的版本（`DescribeTable` 返回 `managed_versioning =
true`），Doris 会通过 Namespace 的版本接口解析最新版本和 Time Travel 指定的版本，再从存储读取对应的 manifest。详见
[Time Travel](#time-travel)。
## Namespace 映射
@@ -384,6 +382,59 @@ FROM lance_catalog.default.user_profiles;
普通 Catalog 查询会在规划阶段固定一个 Lance 数据集版本，并按 Fragment 生成扫描任务。因此，同一条查询读取一致的快照，同时可以由多个
Scanner 并行扫描不同 Fragment，不会让每个 Scanner 重复扫描整个数据集。
+## Time Travel
+
+Lance 数据集的每次提交都会产生一个新版本，版本号从 1 开始。查询可以读取某个历史版本，而不是最新版本：
+
+```sql
+-- 读取表的版本 2
+SELECT * FROM lance_catalog.db.tbl FOR VERSION AS OF 2;
+
+-- 读取指定时间点之前（含）提交的最新版本
+SELECT * FROM lance_catalog.db.tbl FOR TIME AS OF '2026-09-19 13:06:10';
+```
+
+- `FOR VERSION AS OF` 接受正整数版本号；与 Paimon 表以及 Iceberg 的 tag 名用法一样，也可以写带引号的 tag
名（`FOR VERSION AS OF 'v2'` 等同于 `@tag(v2)`）。
+- `FOR TIME AS OF` 接受会话时区下的时间戳，支持秒或毫秒精度（`yyyy-MM-dd HH:mm:ss` 或 `yyyy-MM-dd
HH:mm:ss.SSS`）。Doris 按 manifest
记录的提交时间，在不晚于该时间戳的版本中选择提交时间最晚的一个（提交时间相同的取版本号大的）。和 Iceberg
按时间选快照一样，提交时间按记录的原值比较，不假设它随版本号递增。Lance 记录的提交时间精确到毫秒以下，比较时保留全部精度：Doris
不选在请求的那一毫秒内、晚于请求时间提交的版本。cleanup 删掉版本时，提交时间也一起删掉了，所以 Doris 只在 cleanup
删掉的最新一个版本之后选择，和 Iceberg 在过期快照处截断 snapshot log
的做法一致。时间戳早于这些版本时会报错：它可能早于第一个版本，也可能落在 cleanup 删掉的那段历史里（例如落在被 tag
保留的版本和其后仍存在的版本之间），这时无法确定表在那个�
��的状态。
+- 选中的版本在整条语句内固定：schema、Fragment 规划、谓词下推和 BE
扫描都使用该版本。同一条语句里对同一张表的两次引用可以选择不同版本。`EXPLAIN` 中的 `lanceVersion` 显示选中的版本。
+- 被 Lance `cleanup_old_versions` 清理掉的版本无法读取，报错与从未存在的版本相同。由 REST Namespace
管理版本的表见下文“REST Namespace 管理的版本”。
+- `vector_search()`、`full_text_search()` 和索引查看总是作用于 main 的最新版本，它们的 `table`
参数中不能指定版本、tag 或 branch。
+
+Lance 的 tag 和 branch 使用与 Iceberg、Paimon 表相同的语法：
+
+```sql
+-- 读取 tag "v2" 指向的版本
+SELECT * FROM lance_catalog.db.tbl@tag(v2);
+
+-- 读取 branch "dev" 的最新版本
+SELECT * FROM lance_catalog.db.tbl@branch(dev);
+
+-- 读取 branch "dev" 的版本 2
+SELECT * FROM lance_catalog.db.tbl@branch(dev) FOR VERSION AS OF 2;
+```
+
+- tag 指向某个 branch 上的某个版本，读取时选中的就是该 branch 的该版本：建在 `dev` 上的 tag 读的是 `dev`，而不是
main 上同号的版本。`@tag` 不能再与 `FOR VERSION AS OF` 或 `FOR TIME AS OF` 同时使用。
+- `@branch(main)` 读取 main，与不写 `@branch` 相同。
+- 每个 branch 有自己的版本序列，从创建它时所基于的版本开始。`@branch` 后的 `FOR VERSION AS OF` / `FOR
TIME AS OF` 只在该 branch 的版本中选择，因此早于 branch 创建时间的时间戳在 branch 上选不到版本。`@branch`
后（包括 `@branch(main)`）的 `FOR VERSION AS OF` 不能写 tag 名，因为 tag 本身已经确定了 branch。
+- branch 存放在 `<table>/tree/<branch>/` 下，是表的 shallow clone，manifest 以绝对 URI
记录表的位置。因此 branch 只能在创建它的位置读取：把数据集复制或迁移到别处后，main 仍可读，branch 不可读。
+- branch 的元数据已删除，或创建中途失败时，只要 `tree/` 下的目录还没清理，仍可以通过 `@branch` 读到。原因是 Lance SDK
按目录 checkout branch。由 REST Namespace 管理版本的表，branch 是否存在由 Namespace 决定，读取 branch
时直接读它的目录，不读取 `main`。
+- `EXPLAIN` 中的 `lanceBranch` 显示读取的 branch。不存在的 tag 或 branch 会报错。
+
+### REST Namespace 管理的版本（Managed Versioning）
+
+REST Namespace 可以自行管理表的版本（`DescribeTable` 返回 `managed_versioning =
true`）。对这类表，哪些版本存在、哪个是最新版本由 Namespace 决定：最新版本和 `FOR VERSION AS OF` 以 Namespace
的记录为准（`ListTableVersions`、`DescribeTableVersion`），`FOR TIME AS OF` 按 manifest
记录的提交时间选择，并且只在 Namespace 列出的版本中选择；tag 与普通 Lance 表一样从数据集的 `_refs/tags/`
读取，它指向的版本必须是 Namespace 记录过的；branch 的版本同样以 Namespace 为准。选定版本后，FE 和 BE
都按表的位置和版本号从存储读取，和其他 Lance 表一样。
+
+- Namespace 没有记录的版本报告为不存在，即使它的 manifest 仍在存储中；最新版本以 Namespace 为准，即使存储中已有更新的
manifest。
+- Namespace 不再记录、但前后版本仍有记录的版本，在 `FOR TIME AS OF` 中视为已删除的历史，和 cleanup
删掉的版本一样：时间戳可能落在这个版本上时报错，不会选择更早的版本。
+- Namespace 对某张表或某个 branch 没有记录任何版本时会报错，即使存储中有它的 manifest。
+- 这类表在 `EXPLAIN` 中显示 `lanceManagedVersioning=true`。
+
+限制：
+
+- Doris 以 `DescribeTable` 返回的 `table_uri` 读取数据集，没有 `table_uri` 时用
`location`，它们必须是完整的存储 URI。`location` 为空，或同时返回的 `table_uri`
指向别的位置时（查询串不算，例如预签名凭证），查询直接报错，不读取数据。`s3+ddb` URI 同样报错：它的 DynamoDB
提交逻辑会在读取时登记并定稿版本。managed 表每次读取都会重新调用 `DescribeTable`，不做缓存；FE 和 BE
都用这次返回的位置和存储配置读取。
+- Doris 按版本号读取版本，读的是表目录（branch 则是 `tree/<branch>/`）下的规范路径 `_versions/<u64::MAX
- version>.manifest`，Lance 旧的命名方式下是 `_versions/<version>.manifest`。Namespace
为选中版本记录的 manifest 必须是这个路径，或者它旁边的 staged manifest（`<规范路径>-<id>`）；执行 `FOR TIME AS
OF` 时，对应 `main` 或 branch
上记录的每个版本都要满足这一点，因为选择时要比较它们的提交时间。记录在其他位置的版本无法读取，查询会报错，报错信息给出记录的路径和规范路径。查询过程中
Namespace 移动了表时，查询也可能这样报错，重试即可。
+- Namespace 只在 staged manifest 上记录、规范路径上又没有 manifest 的版本无法读取：它的提交没有定稿，或者
cleanup 删掉了这个版本。查询会报错，并说明这两种原因。这个版本的提交时间未知，所以在对应的 `main` 或 branch 上执行 `FOR TIME
AS OF` 时，时间戳可能落在它上面就会报错；如果它是最新版本，在写入端或其他使用 Namespace 的读者把它定稿之前，任何时间戳都会报错，不带
`FOR VERSION AS OF` 读取表、加载表结构也会报错。Lance 自带的 Namespace 实现会先定稿再记录版本，只有提交中途中断，或
Namespace 记录的是 staged manifest 时，才会出现未定稿的版本。Doris 读取时不会向表写入任何数据。
+
## 查看 Lance 索引
对于 Filesystem Catalog 中的表，`SHOW INDEX` 可以查看 Lance 数据集中记录的逻辑标量索引（包括 FTS
倒排索引）和向量索引。变种语法 `SHOW INDEXES`、`SHOW KEY` 和 `SHOW KEYS` 返回相同结果：
@@ -878,7 +929,7 @@ ORDER BY _distance ASC, user_id;
每个 FE 可以在 Catalog 内缓存表解析后的 Dataset URI 和访问配置，避免查询规划时重复执行文件系统发现或 REST
`describeTable` 请求。每个 Catalog 客户端最多保留 10,000
个条目。`lance.table_access_cache_ttl_seconds` 默认为 `60` 秒，命中缓存不会延长有效期。此缓存不保存
Dataset 版本：每次读取仍会打开 Dataset 并选择快照。
-如果响应包含下发的存储配置，即使提供了 `expires_at_millis`，也会在每次读取时重新解析。包含用户信息、查询参数（包括预签名或 SAS
凭证）或片段的 URI 同样不缓存。固定的凭证到期余量无法保证凭证在整个扫描期间有效，BE 也无法在扫描过程中更新这些凭证。响应不含这些字段的 REST
Catalog 可以使用缓存。
+由 Namespace 管理版本（managed versioning）的表在每次读取时重新解析，FE 和 BE
都使用这次返回的位置和存储配置。如果响应包含下发的存储配置，即使提供了
`expires_at_millis`，也会在每次读取时重新解析。包含用户信息、查询参数（包括预签名或 SAS 凭证）或片段的 URI
同样不缓存。固定的凭证到期余量无法保证凭证在整个扫描期间有效，BE 也无法在扫描过程中更新这些凭证。响应不含这些字段的 REST Catalog 可以使用缓存。
显式刷新表或数据库会清除该 Catalog 的全部表访问缓存；Follower FE
重放刷新时，即使对应的本地数据库或表对象已被淘汰，也会清除缓存。普通的本地数据库对象淘汰不会清除此缓存。启用缓存失效的 Catalog
刷新、Namespace 移除或 Catalog 配置变更也会使访问缓存失效。索引检查和索引任务目标校验始终重新解析访问配置。
@@ -1119,7 +1170,6 @@ ORDER BY _score DESC, document_id;
## 当前限制和建议
- Lance Catalog 和 Lance TVF 当前仅支持读取，不支持 `CREATE
TABLE`、`INSERT`、`UPDATE`、`DELETE`、`TRUNCATE TABLE` 或写回 Lance。
-- 查询总是读取规划时选择的当前版本，不支持通过 SQL 指定 Version 或执行 Time Travel。
- 使用不支持的列类型时，建议显式列出需要读取的列，避免 `SELECT *` 投影到不支持的列。
- 对普通扫描使用 `EXPLAIN` 检查 `lancePushdownPredicate`，确认目标条件是否已下推。
- 向量检索前应在 Lance 中创建与查询方式匹配的索引；小数据集或验证场景可以设置 `"use_index" = "false"` 使用 Flat
Search。
diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index 6f3c13df07b..bf1eb3474ec 100644
--- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -45,12 +45,12 @@ Doris currently provides read-only access to Lance.
Creating, writing, updating,
| Lance Cache | Shares index and metadata caches and a local disk data cache
within each BE |
| Two-phase TopN read | Enabled by default for eligible output columns in
vector and full-text searches; controlled independently by
`enable_lance_lazy_materialization` |
| Writing to Lance | Not supported |
-| Time Travel | Not supported |
+| Time Travel | Supports `FOR VERSION AS OF`, `FOR TIME AS OF`, `@tag(...)`
and `@branch(...)`, including tables whose versions are managed by a REST
Namespace |
| Hybrid Search | Not supported |
## Lance Version and Compatibility
-The Doris BE data reader is built with `lance-c v0.1.9` plus Doris patches,
which add multi-vector search support and upgrade its embedded Lance Rust
crates to `11.0.0` (commit `ab6b5bbe`). The Doris FE reads Namespace and
dataset metadata with `org.lance:lance-core:11.0.0`, aligning the FE and BE on
Lance `11.0.0`. A Dataset must be compatible with both FE metadata loading and
BE data reading. These implementation versions are different from the Lance
`data_storage_version` recorded in [...]
+The Doris BE data reader is built with `lance-c v0.1.9` plus Doris patches,
which add multi-vector search support and upgrade its embedded Lance Rust
crates to `11.0.0` (commit `ab6b5bbe`). The Doris FE reads Namespace and
dataset metadata with `org.lance:lance-core:12.0.0`; the FE only reads metadata
and never writes a dataset, so its newer Lance release does not change what the
BE can read. A Dataset must be compatible with both FE metadata loading and BE
data reading. These implementa [...]
The following table describes the file-format compatibility of this reader:
@@ -92,7 +92,7 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
| `lance.namespace.parent` | No | Empty | Limits access to the specified Lance
Namespace and its child Namespaces. With the default delimiter, for example,
`production$analytics` represents a two-level Namespace. |
| `lance.namespace.delimiter` | No | `$` | Delimiter used to parse
`lance.namespace.parent`. It is also passed to the REST Namespace client. This
property does not change how multilevel Namespaces are displayed in Doris. |
| `lance.namespace.root_database` | No | `default` | Doris database name to
which the root Lance Namespace is mapped. |
-| `lance.table_access_cache_ttl_seconds` | No | `60` | Non-negative integer.
Maximum FE table-access cache lifetime in seconds; `0` disables it. Applies to
Filesystem and REST Catalogs. Only results without vended storage options or
credential-bearing URIs are cached; see [FE Table Access
Cache](#fe-table-access-cache). |
+| `lance.table_access_cache_ttl_seconds` | No | `60` | Non-negative integer.
Maximum FE table-access cache lifetime in seconds; `0` disables it. Applies to
Filesystem and REST Catalogs. Tables with namespace-managed versioning, and
results with vended storage options or credential-bearing URIs, are not cached;
see [FE Table Access Cache](#fe-table-access-cache). |
### Filesystem Catalog
@@ -314,9 +314,7 @@ For native OSS access, the Namespace may vend
`oss_endpoint`, `oss_access_key_id
If tables reside in different buckets, do not share the S3
virtual-hosted-style endpoint fixed to `my-bucket` in the example. Have the
Namespace vend a matching endpoint and access style for each table, or
configure separate Catalogs for different buckets. For services supporting
path-style access, you can also use a service endpoint without a bucket and set
`use_path_style=true`, allowing the client to read the bucket from each table
URI.
-:::caution
-The current BE Reader does not support Lance tables whose versions are managed
by REST Namespace (Managed Versioning).
-:::
+If the REST Namespace manages the table's versions (`managed_versioning =
true` in `DescribeTable`), Doris resolves the latest version and any
time-travel selector through the Namespace's version APIs and reads the
corresponding manifest from storage. See [Time Travel](#time-travel) for the
details and limitations.
## Namespace Mapping
@@ -384,6 +382,59 @@ FROM lance_catalog.default.user_profiles;
For a regular Catalog query, Doris pins a Lance dataset version during
planning and generates scan tasks by Fragment. A query therefore reads a
consistent snapshot, while multiple Scanners can read different Fragments in
parallel without every Scanner repeatedly scanning the entire dataset.
+## Time Travel
+
+Every commit to a Lance dataset produces a new version, numbered from 1. A
query can read a historical version instead of the latest one:
+
+```sql
+-- Read version 2 of the table.
+SELECT * FROM lance_catalog.db.tbl FOR VERSION AS OF 2;
+
+-- Read the latest version committed at or before the given time.
+SELECT * FROM lance_catalog.db.tbl FOR TIME AS OF '2026-09-19 13:06:10';
+```
+
+- `FOR VERSION AS OF` takes a positive integer version, or, as for Paimon
tables and for tag names in Iceberg, a quoted tag name (`FOR VERSION AS OF
'v2'` is the same as `@tag(v2)`).
+- `FOR TIME AS OF` takes a timestamp in the session time zone, with second or
millisecond precision (`yyyy-MM-dd HH:mm:ss` or `yyyy-MM-dd HH:mm:ss.SSS`).
Among the versions committed at or before the timestamp, Doris selects the one
with the latest commit time as recorded in its manifest (of versions committed
at the same instant, the newer one). As in Iceberg's selection of a snapshot by
time, commit times are compared as recorded and are not assumed to grow with
version numbers. Lance [...]
+- The selected version is fixed for the whole statement: schema, Fragment
planning, predicate pushdown, and the BE scan all use it. Two references to the
same table in one statement can select different versions. `EXPLAIN` shows the
selected version as `lanceVersion`.
+- Versions removed by Lance's `cleanup_old_versions` cannot be read; they are
reported the same way as versions that never existed. For a table whose
versions a REST Namespace manages, see REST Namespace Managed Versioning below.
+- `vector_search()`, `full_text_search()` and index inspection always use the
latest version of `main`; their `table` argument cannot select a version, tag
or branch.
+
+Lance tags and branches are selected with the same syntax as for Iceberg and
Paimon tables:
+
+```sql
+-- Read the version the tag "v2" points at.
+SELECT * FROM lance_catalog.db.tbl@tag(v2);
+
+-- Read the latest version of the branch "dev".
+SELECT * FROM lance_catalog.db.tbl@branch(dev);
+
+-- Read version 2 of the branch "dev".
+SELECT * FROM lance_catalog.db.tbl@branch(dev) FOR VERSION AS OF 2;
+```
+
+- A tag points at a version on a branch, and reading the tag selects that
version of that branch: a tag created on `dev` reads `dev`, not the version
with the same number on `main`. `@tag` cannot be combined with `FOR VERSION AS
OF` or `FOR TIME AS OF`.
+- `@branch(main)` reads `main`, the same as leaving `@branch` out.
+- Each branch has its own sequence of versions, starting at the version it was
created from. `FOR VERSION AS OF` / `FOR TIME AS OF` after `@branch` select
among that branch's versions only, so a timestamp before the branch was created
selects nothing on it. After `@branch`, `@branch(main)` included, `FOR VERSION
AS OF` cannot name a tag, because a tag already determines its branch.
+- A branch is stored under `<table>/tree/<branch>/` as a shallow clone of the
table whose manifests record the table's location as an absolute URI. A branch
is therefore readable only at the location it was created at: after the dataset
is copied or moved elsewhere, `main` stays readable but its branches do not.
+- A branch whose metadata was deleted, or whose creation stopped partway,
stays readable through `@branch` until its directory under `tree/` is cleaned
up, because the Lance SDK checks a branch out by its directory. For a table
whose versions a REST Namespace manages, the Namespace decides whether the
branch exists, and the branch is read from its directory without reading `main`.
+- `EXPLAIN` shows the branch read as `lanceBranch`. A tag or branch that does
not exist is an error.
+
+### REST Namespace Managed Versioning
+
+A REST Namespace can manage a table's versions itself (`managed_versioning =
true` in the `DescribeTable` response). For such a table, the Namespace decides
which versions exist and which is the latest: the latest version and `FOR
VERSION AS OF` follow the Namespace's records (`ListTableVersions`,
`DescribeTableVersion`), `FOR TIME AS OF` is resolved from the commit times the
manifests record, choosing only among the versions the Namespace lists, a tag
is read from the dataset's `_refs/t [...]
+
+- A version the Namespace does not record is reported as not found, even if
its manifest still exists in storage; the latest version is the Namespace's
latest, even when storage already holds a newer manifest.
+- A version the Namespace no longer records, between versions it records, is
removed history for `FOR TIME AS OF`, as a version removed by cleanup is: a
timestamp that version may cover is an error instead of selecting an older
version.
+- A table, or a branch, for which the Namespace records no version is an
error, even if storage holds manifests for it.
+- `EXPLAIN` shows `lanceManagedVersioning=true` for these tables.
+
+Limitations:
+
+- Doris reads the dataset at the `table_uri` returned by `DescribeTable`, or
at its `location` when there is no `table_uri`; either must be a complete
storage URI. If `location` is empty, or a `table_uri` returned alongside it
names another place (a query string, such as presigned credentials, aside), the
query fails with an error and reads nothing. An `s3+ddb` URI is rejected as
well: its DynamoDB commit handler would record and finalize versions on read.
Doris describes a managed table [...]
+- Doris reads a version by its number, at its canonical path
`_versions/<u64::MAX - version>.manifest` in the table directory (or in
`tree/<branch>/` for a branch), or `_versions/<version>.manifest` in Lance's
older naming scheme. The manifest the Namespace records for the selected
version must be that path, or a staged manifest beside it (`<canonical
path>-<id>`); for `FOR TIME AS OF`, so must the manifest of every version it
records on that `main` or branch, since their commit times de [...]
+- A version the Namespace records only at a staged manifest, whose canonical
manifest does not exist, cannot be read: its commit was not finalized, or
cleanup removed the version. The query fails and names both causes. `FOR TIME
AS OF` on that `main` or branch fails for any timestamp the version may cover,
because its commit time is unknown; for the newest version that is every
timestamp, until the writer, or another reader that uses the Namespace,
finalizes it. Reading the table without [...]
+
## Inspect Lance Indexes
For a table in a Filesystem Catalog, `SHOW INDEX` displays the logical scalar
indexes (including FTS inverted indexes) and vector indexes recorded in the
Lance dataset. The variant statements `SHOW INDEXES`, `SHOW KEY`, and `SHOW
KEYS` produce the same result:
@@ -878,7 +929,7 @@ This setting requires a Doris build containing [the table
access cache change](h
Each FE can cache a table's resolved Dataset URI and access options within a
Catalog, avoiding repeated filesystem discovery or REST `describeTable`
requests during query planning. Each Catalog client holds at most 10,000
entries. `lance.table_access_cache_ttl_seconds` defaults to `60`; cache hits do
not extend the lifetime. This cache does not store Dataset versions: each read
still opens the Dataset and selects its snapshot.
-A response containing vended storage options is resolved again on every read,
even if it reports `expires_at_millis`. URIs containing userinfo, query
parameters (including presigned/SAS credentials), or fragments are also not
cached. A fixed credential-expiry margin cannot guarantee that credentials
remain valid throughout a scan, and the BE cannot renew these credentials
during the scan. REST Catalogs without these response fields can use the cache.
+A table with namespace-managed versioning is resolved again on every read, so
both readers use the location and storage options the namespace returns for
that read. A response containing vended storage options is also resolved again
on every read, even if it reports `expires_at_millis`. URIs containing
userinfo, query parameters (including presigned/SAS credentials), or fragments
are also not cached. A fixed credential-expiry margin cannot guarantee that
credentials remain valid througho [...]
An explicit table or database refresh clears all table-access entries in that
Catalog, including on follower FEs replaying the refresh when the corresponding
local database/table object has been evicted. Routine local database-object
eviction does not clear the access cache. Catalog refresh with cache
invalidation, namespace removal, or Catalog configuration changes also
invalidate access entries. Index inspection and index-job target validation
always resolve access afresh.
@@ -1121,7 +1172,6 @@ Pinned dataset snapshot
## Current Limitations and Recommendations
- Lance Catalogs and Lance TVFs are read-only. `CREATE TABLE`, `INSERT`,
`UPDATE`, `DELETE`, `TRUNCATE TABLE`, and writing data back to Lance are not
supported.
-- Queries always read the current version selected during planning. SQL cannot
select a Version or perform Time Travel.
- For tables containing unsupported column types, explicitly list the columns
to read instead of projecting unsupported columns through `SELECT *`.
- For regular scans, inspect `lancePushdownPredicate` in `EXPLAIN` to verify
which conditions have been pushed down.
- Create a vector index in Lance that matches the intended query before
running indexed vector search. For small datasets or validation, set
`"use_index" = "false"` to perform Flat Search.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
