u70b3 opened a new pull request, #66671:
URL: https://github.com/apache/doris/pull/66671
### What problem does this PR solve?
Issue Number: #66497
Related PR: #66637
Problem Summary:
This is PR2 of the Lance index lifecycle work and depends on #66637. Do not
merge this PR before #66637. After PR1 is merged, this branch will be rebased
onto `branch-4.1` with only the four PR2 commits retained.
This PR adds bounded, read-only inspection of Lance logical and physical
index metadata:
- `lance_indexes("table" = "catalog.db.table")` exposes the strict logical
view.
- `lance_index_entries("table" = "catalog.db.table")` exposes physical
diagnostic entries.
- `SHOW INDEX` now uses the same provider-neutral logical snapshot for
Directory and REST catalogs.
- Directory reads schema, logical descriptions, and physical entries from
one `Dataset` snapshot and never calls `getIndexStatistics()`.
- REST reads a pinned table version, bounded paginated index lists, and at
most four rolling stats requests with one shared absolute deadline.
- Both analysis and master execution enforce `SHOW` privilege before catalog
initialization or provider access.
- A bounded FE executor provides global/per-catalog admission, stable
overload/timeout/failure tokens, metrics, and retains task ownership after
caller timeout without claiming native cancellation.
The feature is independently releasable and strictly read-only. Lance index
mutation remains disabled and no mutation route is added.
User-visible schemas:
- `lance_indexes`: `CatalogName`, `DatabaseName`, `TableName`, `IndexName`,
`Columns`, `IndexType`, nullable `RowsIndexed`, `PhysicalEntryCount`,
`Properties`.
- `lance_index_entries`: `CatalogName`, `DatabaseName`, `TableName`,
`IndexName`, `IndexUuid`, `Columns`, nullable `IndexType`, nullable
`DatasetVersion`, nullable `ProviderStatus`, `MetadataConsistency`.
- `Columns` is a deterministic JSON string array so commas, quotes, and
Unicode column names remain unambiguous.
Safety bounds include 256 logical names, 10,000 physical entries, 64 columns
per index, 1,000 REST entries per page, 100 pages, four stats requests in
flight, bounded external strings/properties, exact-case grouping,
repeated-token detection, and all-or-error logical reads.
### Release note
Add read-only Lance logical and physical index inspection through
`lance_indexes`, `lance_index_entries`, and REST-backed `SHOW INDEX`.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test
- Added Directory and REST regression coverage, including pagination
and privilege denial.
- Focused FE suite passed 69/69 before final audit fixes.
- Final executor/REST tests passed 28/28; Directory loader tests
passed 24/24.
- BE ASAN `meta_scanner.cpp.o` compilation passed.
- REST mock passed Python compilation and local describe/list
pagination/stats/401 probes.
- The full Docker regression suite and full `buildall` have not been
run yet.
- Behavior changed:
- [ ] No.
- [x] Yes. Adds the two read-only TVFs and enables REST `SHOW INDEX`
through the shared logical snapshot.
- Does this need documentation?
- [x] No. The staged SQL contract and current limitations are recorded
in this PR; broader lifecycle documentation remains a later phase of #66497.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm #66637 is merged and this PR has been rebased to contain only
PR2 commits.
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
