morningman opened a new pull request, #70830:
URL: https://github.com/apache/doris/pull/70830
### Background
On master, `lance` catalogs are served by the `fe-connector-lance` plugin.
To plan a scan, FE asks the Lance namespace (a warehouse directory or a REST
server) to describe the table, which gives the dataset URI and its storage
options, then opens the dataset at the statement's version and reads its
schema, fragments and, for a pushed filter, its indexes; BE reads the data
through lance-c. Lance keeps the manifests and index metadata it has read in a
`Session`.
This is the last of four PRs porting the Lance read path (#70680).
### What problem does this PR solve?
Issue Number: close #70680
Related PR: #70827 (pushdown, the third PR; this builds on it), #70725 (REST
catalogs and OSS, the second PR), #70681 (the plugin, the first PR), #68078,
#68305 (the branch-4.x PRs this ports)
PR Type: Enhancement
Problem Summary:
On master every statement describes the table again and opens each dataset
with a Session of its own, so a repeated query repeats the namespace round trip
and re-reads the manifest from storage, and its profile shows none of this
time. 4.1.5 and 4.2 avoid both (#68078, #68305). This PR ports them: each
catalog keeps one Lance Session, a described table is reused for the catalog
property `lance.table_access_cache_ttl_seconds` (default 60, 0 off; vended
credentials are always described again), and each Lance scan's profile gets a
`Lance Metadata Metrics` entry. Most of the diff is tests.
Ten identical `EXPLAIN SELECT int32_value FROM doris.predicate_pushdown
WHERE int32_value = 10` on a filesystem catalog over S3, one statement each,
with only the plugin swapped:
| Plugin | S3 requests from FE | Per statement |
| --- | --- | --- |
| before (#70827) | 100 | 4 HEAD + 2 GET on `__manifest` (the describe), 1
LIST, 1 HEAD + 2 GET of the table's manifest |
| after | 40 | 2 HEAD on `__manifest` (the existence check), 1 LIST + 1 HEAD
(finding the newest version) |
A query's profile:
```
Lance Metadata Metrics:
Table Scan (doris.predicate_pushdown):
- MetadataReadTime: 6.046ms
- MetadataReadCalls: 2
- MetadataReadFailures: 0
- SplitPlanningTime: 229.875us
- SplitPlanningCalls: 1
- SplitPlanningFailures: 0
- TableAccessResolveTime: 47.708us
- TableAccessResolveCalls: 2
- DatasetOpenTime: 5.374ms
- DatasetOpenCalls: 2
...
```
Original authors: @zhangstar333 and @Gabriel39.
### Details
- **Change map:** +2215 −147 in 27 files: production code +778 −140 (10
files), tests +1415 −7 (16 files: FE unit tests, two regression checks), build
+22 (Caffeine 2.9.3 in the plugin zip, Apache-2.0, the version the other
connector plugins bundle). The production code, all in the plugin, in reading
order:
1. Catalog generations and the shared Session (+297 −116):
`LanceCatalogClient` opens a generation (Arrow allocator, namespace, Session,
namespace client) on the first call, runs each call on one generation from
start to end, and retires it on `refresh()`; the last call on a retired
generation releases it. `LanceConnector` routes the engine's `invalidateAll` /
`invalidateDb` / `invalidateTable` to it.
2. Table-access cache (+135 −7): `LanceNamespaceClient` (Caffeine, each
entry expiring a fixed time after its describe) and the property in
`LanceCatalogProperties`.
3. Metadata metrics (+338 −14): `LanceMetadataMetrics`, kept per statement
and table in the statement scope and published through `collectScanProfiles`;
the timing in the client, `LanceIndexMetadata` and `LanceScanPlanProvider`.
4. The `lance.conf` wording (+8 −3): the Arrow limit applies per
generation, and the Session's caches are not counted.
- **Before and after:**
| | Before | After |
| --- | --- | --- |
| Describing the table | every statement | once per table per TTL (default
60 s); every statement when the namespace vends credentials or the URI may
carry them |
| Manifest, fragment and index metadata | re-read by every statement |
from the catalog's Session after the first read |
| REFRESH CATALOG | nothing cached | new namespace and Session; reads in
flight finish on the old ones |
| REFRESH TABLE / DATABASE | nothing cached | the next read describes
again; the Session is kept |
| Query profile | nothing about Lance metadata | `Lance Metadata Metrics`
per scanned table: `MetadataRead` and `SplitPlanning` time, calls, failures,
and time and calls per stage (`TableAccessResolve`, `DatasetOpen`,
`VersionResolve`, `SchemaRead`, `FragmentMetadataRead`, `FieldIdsRead`,
`IndexMetadataRead`) |
- **Configs and defaults:** catalog property
`lance.table_access_cache_ttl_seconds`: seconds, default 60, 0 describes on
every read, negative or non-integer values rejected by CREATE and ALTER
CATALOG, a new value applies at once (ALTER rebuilds the connector). Same key
and rules as 4.1.5 and 4.2. The Session's capacities are fixed (64 MiB of
metadata, 128 MiB of index metadata), as on 4.x.
- **Impact on existing users:** Catalogs upgraded from 4.1.5 or 4.2 keep
their caching and their value of the key. Unlike 4.x, the profile group has one
child per scan with formatted values, and Lance time is not added to `External
Table Get Table Meta Time`, which master's other plugin connectors leave empty
too. As on 4.x, a moved table is read at its old URI for up to the TTL unless
refreshed, a dataset replaced at the same URI needs REFRESH CATALOG, and each
catalog's Session may hold up to 192 MiB of native memory (twice that while a
REFRESH CATALOG overlaps two generations).
- **Results:** The table above, and the eight Lance regression suites on the
same cluster: 9,254 S3 requests with #70827's plugin, 4,780 with this one (FE
and BE together). No latency was measured.
### Testing
- `fe-connector-lance` unit tests, locally on Mac arm64 with the JNI
libraries loaded: 254 pass, none skipped (213 before). New:
`LanceTableAccessCacheTest` (TTL from the describe, what is never cached,
invalidation during a describe, hits without the namespace lock),
`LanceCatalogClientTest` (close and refresh with a call in flight, failed
opening, shared Arrow limit), `LanceSharedSessionNativeTest` (real Sessions:
cache hits, newest version still found, refresh closes the Session),
`LanceMetadataMetricsTest`, `LanceIndexMetadataTest`; and cases in
`LanceRestCatalogTest` (describe requests counted across statements and
refreshes), `LanceConnectorNativeTest` (profile counts end to end) and three
more classes. 19 mutations of the new behavior each turn their target tests red.
- The test session's statement scope now rejects a loader that uses the
scope, as the engine's does (`Recursive update`): the cluster run caught such a
loader in an earlier revision of this PR, on which 11 unit tests fail with the
stricter scope.
- Regression, locally on an FE and BE built from #70827's head with this
plugin, against a host S3 substitute and the REST stub instead of MinIO: the
eight `external_table_p0/lance` suites pass, with two added checks: the scan
profile in `test_lance_array_predicate_pushdown` (failing against #70827's
plugin) and a negative TTL in `test_lance_catalog_all_types`.
### Notes for Reviewers
- Stacked on #70827: review the last commit; until #70827 merges, the diff
also shows its commit.
- Caffeine rather than fe-connector-cache's `MetaCache`, which expires
entries after their last access: #68305's TTL counts from the describe, so a
busy table must still be described again.
- REFRESH TABLE drops the whole access cache, as on 4.2: swapping it never
waits for a describe in flight, invalidating one Caffeine key would.
- Generations open on the first call, so a refresh builds nothing, and 4.2's
slow or failed replacements cannot happen.
- Metrics use the existing `collectScanProfiles`, so no SPI change. The test
seams (`LanceCatalogClient.ResourceFactory`, a package-private `LanceConnector`
constructor, `LanceIndexMetadata.read` over suppliers) stand in for 4.2's
Mockito mocks, which connector modules do not allow.
### Release note
Lance catalogs keep reusing table metadata across queries, as in 4.1 and
4.2, with `lance.table_access_cache_ttl_seconds` (default 60 seconds). In the
query profile, `Lance Metadata Metrics` now has one entry per scanned table,
and Lance metadata time is no longer included in `External Table Get Table Meta
Time`.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No (the property and its refresh behavior
are documented by apache/doris-website#4159)
- Pre-submission status: Partially verified: the regression suites ran
locally against a host S3 substitute and the REST stub, not the External
pipeline's MinIO; the native unit tests ran on Mac arm64 only
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: the code, the tests and this description
- Human review: The author reviewed part of the changes.
- Prompts: Open a new branch on #70827's branch and continue with the next
task of the Lance port (4.2's session cache, table-access cache and metadata
profile), and open the PR once development is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]