morningman opened a new pull request, #70830:
URL: https://github.com/apache/doris/pull/70830

   ### Background
   
   On master, `lance` catalogs are served by the `fe-connector-lance` plugin. 
To plan a scan, FE asks the Lance namespace (a warehouse directory or a REST 
server) to describe the table, which gives the dataset URI and its storage 
options, then opens the dataset at the statement's version and reads its 
schema, fragments and, for a pushed filter, its indexes; BE reads the data 
through lance-c. Lance keeps the manifests and index metadata it has read in a 
`Session`.
   
   This is the last of four PRs porting the Lance read path (#70680).
   
   ### What problem does this PR solve?
   
   Issue Number: close #70680
   Related PR: #70827 (pushdown, the third PR; this builds on it), #70725 (REST 
catalogs and OSS, the second PR), #70681 (the plugin, the first PR), #68078, 
#68305 (the branch-4.x PRs this ports)
   PR Type: Enhancement
   
   Problem Summary:
   
   On master every statement describes the table again and opens each dataset 
with a Session of its own, so a repeated query repeats the namespace round trip 
and re-reads the manifest from storage, and its profile shows none of this 
time. 4.1.5 and 4.2 avoid both (#68078, #68305). This PR ports them: each 
catalog keeps one Lance Session, a described table is reused for the catalog 
property `lance.table_access_cache_ttl_seconds` (default 60, 0 off; vended 
credentials are always described again), and each Lance scan's profile gets a 
`Lance Metadata Metrics` entry. Most of the diff is tests.
   
   Ten identical `EXPLAIN SELECT int32_value FROM doris.predicate_pushdown 
WHERE int32_value = 10` on a filesystem catalog over S3, one statement each, 
with only the plugin swapped:
   
   | Plugin | S3 requests from FE | Per statement |
   | --- | --- | --- |
   | before (#70827) | 100 | 4 HEAD + 2 GET on `__manifest` (the describe), 1 
LIST, 1 HEAD + 2 GET of the table's manifest |
   | after | 40 | 2 HEAD on `__manifest` (the existence check), 1 LIST + 1 HEAD 
(finding the newest version) |
   
   A query's profile:
   
   ```
   Lance Metadata Metrics:
     Table Scan (doris.predicate_pushdown):
        - MetadataReadTime: 6.046ms
        - MetadataReadCalls: 2
        - MetadataReadFailures: 0
        - SplitPlanningTime: 229.875us
        - SplitPlanningCalls: 1
        - SplitPlanningFailures: 0
        - TableAccessResolveTime: 47.708us
        - TableAccessResolveCalls: 2
        - DatasetOpenTime: 5.374ms
        - DatasetOpenCalls: 2
        ...
   ```
   
   Original authors: @zhangstar333 and @Gabriel39.
   
   ### Details
   
   - **Change map:** +2215 −147 in 27 files: production code +778 −140 (10 
files), tests +1415 −7 (16 files: FE unit tests, two regression checks), build 
+22 (Caffeine 2.9.3 in the plugin zip, Apache-2.0, the version the other 
connector plugins bundle). The production code, all in the plugin, in reading 
order:
     1. Catalog generations and the shared Session (+297 −116): 
`LanceCatalogClient` opens a generation (Arrow allocator, namespace, Session, 
namespace client) on the first call, runs each call on one generation from 
start to end, and retires it on `refresh()`; the last call on a retired 
generation releases it. `LanceConnector` routes the engine's `invalidateAll` / 
`invalidateDb` / `invalidateTable` to it.
     2. Table-access cache (+135 −7): `LanceNamespaceClient` (Caffeine, each 
entry expiring a fixed time after its describe) and the property in 
`LanceCatalogProperties`.
     3. Metadata metrics (+338 −14): `LanceMetadataMetrics`, kept per statement 
and table in the statement scope and published through `collectScanProfiles`; 
the timing in the client, `LanceIndexMetadata` and `LanceScanPlanProvider`.
     4. The `lance.conf` wording (+8 −3): the Arrow limit applies per 
generation, and the Session's caches are not counted.
   - **Before and after:**
   
     | | Before | After |
     | --- | --- | --- |
     | Describing the table | every statement | once per table per TTL (default 
60 s); every statement when the namespace vends credentials or the URI may 
carry them |
     | Manifest, fragment and index metadata | re-read by every statement | 
from the catalog's Session after the first read |
     | REFRESH CATALOG | nothing cached | new namespace and Session; reads in 
flight finish on the old ones |
     | REFRESH TABLE / DATABASE | nothing cached | the next read describes 
again; the Session is kept |
     | Query profile | nothing about Lance metadata | `Lance Metadata Metrics` 
per scanned table: `MetadataRead` and `SplitPlanning` time, calls, failures, 
and time and calls per stage (`TableAccessResolve`, `DatasetOpen`, 
`VersionResolve`, `SchemaRead`, `FragmentMetadataRead`, `FieldIdsRead`, 
`IndexMetadataRead`) |
   - **Configs and defaults:** catalog property 
`lance.table_access_cache_ttl_seconds`: seconds, default 60, 0 describes on 
every read, negative or non-integer values rejected by CREATE and ALTER 
CATALOG, a new value applies at once (ALTER rebuilds the connector). Same key 
and rules as 4.1.5 and 4.2. The Session's capacities are fixed (64 MiB of 
metadata, 128 MiB of index metadata), as on 4.x.
   - **Impact on existing users:** Catalogs upgraded from 4.1.5 or 4.2 keep 
their caching and their value of the key. Unlike 4.x, the profile group has one 
child per scan with formatted values, and Lance time is not added to `External 
Table Get Table Meta Time`, which master's other plugin connectors leave empty 
too. As on 4.x, a moved table is read at its old URI for up to the TTL unless 
refreshed, a dataset replaced at the same URI needs REFRESH CATALOG, and each 
catalog's Session may hold up to 192 MiB of native memory (twice that while a 
REFRESH CATALOG overlaps two generations).
   - **Results:** The table above, and the eight Lance regression suites on the 
same cluster: 9,254 S3 requests with #70827's plugin, 4,780 with this one (FE 
and BE together). No latency was measured.
   
   ### Testing
   
   - `fe-connector-lance` unit tests, locally on Mac arm64 with the JNI 
libraries loaded: 254 pass, none skipped (213 before). New: 
`LanceTableAccessCacheTest` (TTL from the describe, what is never cached, 
invalidation during a describe, hits without the namespace lock), 
`LanceCatalogClientTest` (close and refresh with a call in flight, failed 
opening, shared Arrow limit), `LanceSharedSessionNativeTest` (real Sessions: 
cache hits, newest version still found, refresh closes the Session), 
`LanceMetadataMetricsTest`, `LanceIndexMetadataTest`; and cases in 
`LanceRestCatalogTest` (describe requests counted across statements and 
refreshes), `LanceConnectorNativeTest` (profile counts end to end) and three 
more classes. 19 mutations of the new behavior each turn their target tests red.
   - The test session's statement scope now rejects a loader that uses the 
scope, as the engine's does (`Recursive update`): the cluster run caught such a 
loader in an earlier revision of this PR, on which 11 unit tests fail with the 
stricter scope.
   - Regression, locally on an FE and BE built from #70827's head with this 
plugin, against a host S3 substitute and the REST stub instead of MinIO: the 
eight `external_table_p0/lance` suites pass, with two added checks: the scan 
profile in `test_lance_array_predicate_pushdown` (failing against #70827's 
plugin) and a negative TTL in `test_lance_catalog_all_types`.
   
   ### Notes for Reviewers
   
   - Stacked on #70827: review the last commit; until #70827 merges, the diff 
also shows its commit.
   - Caffeine rather than fe-connector-cache's `MetaCache`, which expires 
entries after their last access: #68305's TTL counts from the describe, so a 
busy table must still be described again.
   - REFRESH TABLE drops the whole access cache, as on 4.2: swapping it never 
waits for a describe in flight, invalidating one Caffeine key would.
   - Generations open on the first call, so a refresh builds nothing, and 4.2's 
slow or failed replacements cannot happen.
   - Metrics use the existing `collectScanProfiles`, so no SPI change. The test 
seams (`LanceCatalogClient.ResourceFactory`, a package-private `LanceConnector` 
constructor, `LanceIndexMetadata.read` over suppliers) stand in for 4.2's 
Mockito mocks, which connector modules do not allow.
   
   ### Release note
   
   Lance catalogs keep reusing table metadata across queries, as in 4.1 and 
4.2, with `lance.table_access_cache_ttl_seconds` (default 60 seconds). In the 
query profile, `Lance Metadata Metrics` now has one entry per scanned table, 
and Lance metadata time is no longer included in `External Table Get Table Meta 
Time`.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No (the property and its refresh behavior 
are documented by apache/doris-website#4159)
   - Pre-submission status: Partially verified: the regression suites ran 
locally against a host S3 substitute and the REST stub, not the External 
pipeline's MinIO; the native unit tests ran on Mac arm64 only
   
   ### AI Disclosure
   
   - Tool: Claude Code (Claude Opus 5.5)
   - Generated: the code, the tests and this description
   - Human review: The author reviewed part of the changes.
   - Prompts: Open a new branch on #70827's branch and continue with the next 
task of the Lance port (4.2's session cache, table-access cache and metadata 
profile), and open the PR once development is done.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to