u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5214045343
# Revised Design Proposal v4 — Lance Index Lifecycle, scoped to the 4.2 boundary > Draft for apache/doris#66497, prepared 2026-08-07. Incorporates @Gabriel39's round-3 review (issuecomment-5211673675) and its follow-up (issuecomment-5212293290): target **branch-4.1 only** (the confirmed 4.2 base), **no functionality requiring lance-c extension** in this release, query consumption stays on the existing `vector_search()` TVF, and the v3 distributed-build machinery moves to a separately tracked future-work item. v4 is intentionally a *subtraction* from v3: everything removed is named in §9 with its future-work tracking point. /cc @Gabriel39 @zhangstar333 ### Major changes from v3 1. **Scope cut to the one-shot pinned APIs.** Directory builds execute the existing lance-c v0.1.2 one-shot lifecycle calls on a selected BE (`lance_dataset_create_vector_index` / `create_scalar_index` / `drop_index` / `index_count` / `index_list_json`, [email protected] L490-541); REST keeps the v3 capability matrix on the pinned Namespace 0.7.7 model. No new native capability is required anywhere. 2. **`BUILD INDEX` (incremental) removed** from 4.2 — it was added during review and is not required by the issue; its intended implementation needs APIs outside the 4.2 boundary. The statement is rejected for Lance tables with an explicit error (§2). 3. **Provenance rewritten honestly (§6).** The one-shot C calls commit internally and carry no client operation ID and no pre-assigned segment UUID set. v3's deterministic Directory provenance is unavailable in 4.2; reconciliation after a lost response is explicitly bounded to `OUTCOME_UNKNOWN` + diagnostics. This is the proposed 4.2 trade-off, with the upgrade path tracked in §9. 4. **CANCEL is best-effort and documented as such** (§5): it fences Doris-side tracking and refresh; it does not interrupt the native build. 5. **Query consumption is verification-only**: acceptance proves an index built by this lifecycle is consumed by the existing `vector_search(..., "use_index"="true")` path (branch-4.1 `VectorSearchTableValuedFunction.java:81,158-159`) with compatible metric and correct results, and that scalar indexes are consumed by the existing Lance predicate path (#65730). No query-syntax or optimizer work. 6. **Q2 resolved per round-3**: property key is Lance-native `metric` with values `l2|cosine|dot`, matching the existing query surface (option c). `USING ANN` remains DDL category reuse only. 7. **`lance_index_segments` TVF deferred**: its distinct value was per-segment coverage, which is exactly the deferred capability; the remaining columns duplicate SHOW INDEX's grouped output. It returns with the future-work track (§9). 8. **FE-side experimental build fallback removed**: it existed to de-risk the distributed path. With the one-shot BE path as the production form, keeping it would add a second, non-production execution mode against the "simplified substantially" direction. ## 1. Goals and Phase-1 boundary (4.2) Provide the issue's lifecycle over the pinned SDKs, nothing more: - `CREATE INDEX` for IVF_PQ vector indexes and BTREE/BITMAP scalar indexes (Directory full; REST per §4 matrix); - `CREATE OR REPLACE INDEX` = full rebuild via the existing `replace=true` behavior; - `SHOW INDEX` — logical definitions, names, UUIDs, columns, types, dataset versions (grouped by logical name where a table carries multiple physical entries per name; REST fields limited to the §4 stats model); - `DROP INDEX` by name (Directory and REST); - durable job status, admission limits, best-effort cancellation, metadata refresh, privileges, and honestly-bounded failure recovery; - Lance dataset manifests / the Namespace service remain the authoritative index metadata; Doris persists only its own job state. Explicit non-goals for 4.2 (each mapped in §9): fragment-scoped distributed builds, shared-model distribution, uncommitted segment transfer, coordinator commit of worker segments, physical segment merge, native progress/cancellation, per-segment coverage/statistics, incremental BUILD, FTS SQL semantics, internal-table ANN query syntax compatibility. Historical-version tables remain read-only for index mutation; inspection uses the selected snapshot, mutation targets only the latest writable state. ## 2. SQL surface - Vector: `CREATE INDEX idx ON t (embedding) USING ANN PROPERTIES ("index_type"="IVF_PQ", "metric"="l2", "num_partitions"="256", "num_sub_vectors"="16")`. - `"metric"` accepts `l2|cosine|dot` (Lance-native values, matching `vector_search()`). Unknown keys/values are rejected at analysis. - Scalar: `USING BTREE` / `USING BITMAP` (optional `"zone_size"` for BTREE per the pinned Lance default, unchanged from v3). - **Type and constraint matrix unchanged from v3 §6**: IVF_PQ on exactly one fixed-size-list column; element types FLOAT16/FLOAT32 only (UINT8/INT8/FLOAT64 deferred per the v3 corrections); field and elements non-null; `num_sub_vectors` must divide the dimension; `num_bits` fixed to 8; BTREE/BITMAP column-type lists per v3 §6. - `CREATE OR REPLACE INDEX` = full rebuild + atomic same-name replacement via `replace=true`. - `SHOW INDEX FROM t`; `DROP INDEX [IF EXISTS]`. - `SHOW BUILD INDEX FROM db WHERE ...`; `CANCEL BUILD INDEX ON ctl.db.tbl (job_id)` — retained (builds are async jobs), semantics per §5. - **`BUILD INDEX` is rejected for Lance tables in 4.2** with an explicit unsupported-operation error naming the deferral — not silently accepted, not partially implemented. Statement semantics: create/replace/drop each return after a durable job is accepted. `IF NOT EXISTS` / `IF EXISTS` reconciliation compares name + column + physical type (Directory) — definition fields not readable back through the pinned list JSON (metric, build parameters) do not participate, so a same-name/column/type index with different parameters is treated as existing; this limitation is documented; REST compares only stats-observable fields per §4. OR REPLACE remains convergent-but-not-idempotent. Single column per index. `USING ANN` is DDL/parser category reuse only — it implies nothing about query syntax. Carried over unchanged from prior revisions: conflict-retry decisions are never made by parsing native exception strings; `IF NOT EXISTS` and `OR REPLACE` are mutually exclusive; `USING` is required for Lance tables (no implicit default index type). ## 3. Neutral command model and dispatch Unchanged from v3: `ParsedIndexSpec` → resolve catalog/database/table → `InternalIndexProvider` or `LanceIndexProvider` → provider validation → internal schema change or Lance external-index job. Lance definitions never enter Doris internal `catalog.Index` serialization. Implementation targets **branch-4.1 only**. ## 4. Execution model and REST capability matrix ### Directory Catalog: one-shot build on a selected BE FE admits the job (§5), selects a BE (any healthy BE for object-storage datasets; the local/file restriction below constrains the choice for local datasets), and dispatches one task. The BE opens the dataset through lance-c v0.1.2 and executes the matching one-shot call, which trains, builds, and **commits internally** as a single dataset version. FE then refreshes metadata (§6). There is no FE-side commit in 4.2. Documented limitations of this execution form (honest, not solved in 4.2): no native progress reporting; no cooperative interruption; whole-dataset build in one BE process (memory/compute on a single node); commit carries no job-specific provenance (§6). These are exactly the capabilities tracked upstream in §9. **Local/file restriction (unchanged from v3)**: a local/file dataset is mutable only when the path is visible with identical contents to the selected executor and to every FE that may reconcile or finalize the operation; otherwise mutation is rejected. ### REST Namespace Catalog: capability matrix (rows unchanged from v3; the TVF row is subsumed by the global TVF deferral, change 7) Behavioral detection via typed `UnsupportedOperationException` (spec error code 0); unsupported combinations fail at analysis where statically known, otherwise with a clear capability error; Doris never falls back to direct Dataset mutation with vended credentials. | Operation | REST 4.2 (pinned Namespace 0.7.7) | |---|---| | `SHOW INDEX` | **Yes** — `ListTableIndices` + `DescribeTableIndexStats`; fields limited to the stats model (name, UUID, columns, status; type, distance, row counts where offered). | | `CREATE INDEX` | **Restricted** — only what `CreateTableIndexRequest` carries (column, index_type, name, distance_type; FTS fields unused). IVF build properties and BTREE `zone_size` are rejected for REST tables. | | `CREATE OR REPLACE INDEX` | **Rejected** — no replace semantics in the request model; a non-atomic drop+create pair is not offered. | | `BUILD INDEX` | **Rejected** (also deferred globally, §2). | | `DROP INDEX` | **Yes** — via `DropTableIndex`; durable job record applies. | Monitoring via the spec-sanctioned `ListTableIndices` / `DescribeTableIndexStats` polling; `transaction_id` persisted when returned (§6). ## 5. Durable jobs, SHOW/CANCEL integration, admission Job record (edit-log persisted; simplified from v3): job id, operation kind (CREATE / REPLACE / DROP), catalog/db/table identity, dataset URI or Namespace table id, logical index name and normalized definition, starting dataset version, state, timestamps, sanitized failure information, Namespace transaction id when returned. **No credentials** — obtained/refreshed at dispatch. States (simplified — the one-shot call fuses build and commit): `PENDING → RUNNING → REFRESHING → FINISHED`, plus `FAILED / CANCELLING / CANCELLED / COMMITTED_REFRESH_PENDING / OUTCOME_UNKNOWN`. `CANCELLING → CANCELLED` completes once the fence is persisted. FE failover reconstructs unfinished jobs from the edit log and resumes polling or refresh; it does not blindly re-dispatch a build (§6). SHOW/CANCEL BUILD INDEX integration as in v3 §4 (registry, proc union with `JobType`, catalog qualification, privileges: SHOW requires table-level SHOW, CANCEL requires table-level ALTER), minus any field that would require a new lance-c API; the `Progress` column for one-shot jobs is a coarse mapping from job state — no native build progress exists in 4.2. **Cancellation semantics (honest)**: CANCEL persists intent and fences Doris-side tracking, polling, and refresh. It **cannot interrupt** the native one-shot call. A fenced build may still complete and commit externally; the job's terminal state remains `CANCELLED`, and Doris performs no further observation of the fenced job — the externally committed index is picked up by the next reconciliation or metadata refresh (a subsequent DDL statement, manual `REFRESH`, or cache invalidation on access), and the `DROP INDEX` the operator issues performs that reconciliation itself. This is documented as best-effort cancellation, not abort. **Resource admission**: per-catalog and cluster-wide concurrency limits on running build jobs, unchanged in intent from v3. ## 6. Failure recovery, provenance, and metadata refresh (rewritten for 4.2) The v3 outcome classes are retained; the evidence standard is restated for what the pinned one-shot APIs can actually provide. - **Confirmed pre-commit failure** (BE/REST call returned an error): job `FAILED`; no refresh; unreferenced artifacts reclaimed by Lance cleanup per retention. - **Confirmed success**: Directory — the one-shot call returned success. REST — the request was accepted *and* completion observed through the §4 list/stats polling (REST index creation is asynchronous, spec.yaml:1633-1634: an accepted request is not a finished build). On confirmed success: invalidate/refresh metadata, emit the external-DDL refresh edit log; refresh failure → `COMMITTED_REFRESH_PENDING` with retry. In the failover flow (completion observation lost before the terminal state was persisted), REST confirmation additionally requires the persisted `transaction_id`; a list/stats postcondition match alone remains corroboration only, per the rule below. - **Lost response / post-dispatch failure**: reopen the latest snapshot / query the Namespace state. Because the one-shot commit carries **no** `operation_id` and Doris pre-assigns **no** segment UUID, a matching postcondition (index with expected name/column/type at a newer dataset version) is corroboration only — an external writer can independently create or replace the same name (Lance same-name CreateIndex is last-committer-wins). Without job-specific evidence the job remains **`OUTCOME_UNKNOWN`** with a diagnostic naming the ambiguity; it is never marked successful from a matching postcondition alone. For REST, the server-returned `transaction_id` (when present) remains the only job-specific evidence, matched via `DescribeTransaction` where supported. - **DROP** follows the same rule: postcondition "name absent" is corroboration only. - **Refresh-on-observation**: any reconciliation that observes newer external state — including a reconciled `IF NOT EXISTS` / `IF EXISTS` no-op or an `OUTCOME_UNKNOWN` diagnosis — still invalidates/refreshes Doris metadata, so a diagnosis never leaves a stale cache behind. - **Operator exit for `OUTCOME_UNKNOWN`**: such a job never retries automatically. The operator inspects actual state via `SHOW INDEX` (or the dataset directly), then either issues `DROP INDEX` if the mutation landed or re-issues the statement if it did not — `CREATE OR REPLACE INDEX` is the recommended re-issue form for create/replace ambiguities, converging manually to a known state. This is a deliberate, stated weakening relative to v3 (which carried deterministic UUID-set + transaction-properties evidence). It follows from the confirmed 4.2 scope: the only remedies live in the deferred capabilities, and §9 names the upgrade path. ## 7. Metadata surfaces and authorization - `SHOW INDEX`: unchanged 13-column schema (branch-4.1 `ShowIndexCommand.java:52-66`); one row per logical index/column, grouping same-named physical entries; `Index_type` reports the physical Lance type (`IVF_PQ`/`BTREE`/`BITMAP`); `Properties` carries bounded logical details only — physical type; segment count; metric and build parameters where known from the creating job record (on Directory these are not readable back through the pinned list JSON, so for externally created indexes they stay empty; on REST, `distance_type` is populated from the stats model where offered, and build parameters are always empty — they are inexpressible there) — never segment arrays, no coverage fields. The logical index count is the number of distinct grouped names, excluding system indexes. Field provenance is stated per catalog kind: **Directory** — UUID, columns, type, and dataset version come from the existing `index_list_json` and satisfy the issue's metadata scope; **REST** — the p inned stats model exposes name, UUID, columns, status, and (where offered) type/distance/row counts only — **no dataset version**, so the corresponding SHOW fields stay empty (spec.yaml:3162-3184, :3206-3234). - Authorization and identifiers unchanged from v3: mutation + CANCEL require table-level ALTER; SHOW surfaces require table-level SHOW; Doris-layer case-insensitive name resolution over Lance's case-sensitive exact-match storage, with explicit ambiguity errors on pre-existing external case-only collisions. - Versioning/concurrency unchanged in the retained parts: mutation commits are atomic new dataset versions; historical-version mutation rejected; Doris serializes same-name jobs internally and does not hold table metadata locks for the lifetime of a build; queries over a table whose index predates appended data combine indexed results with scans of unindexed data — the acceptance suite proves result equivalence with `use_index=false`. ## 8. Test plan (4.2) - Create/replace/show/drop for IVF_PQ + BTREE + BITMAP on Directory; REST capability matrix rejections (analysis-time vs capability error) for every rejected combination; - **Query consumption (acceptance)**: an IVF_PQ index created through this lifecycle is used by `vector_search(..., "use_index"="true")` with a compatible metric and returns correct results; scalar indexes are consumed by the existing Lance predicate path; equivalence with index use disabled; - IF NOT EXISTS / IF EXISTS reconciliation fields per §2/§4, including: same-name/column/type with divergent parameters treated as existing (documented Directory limitation); reconciled no-op refreshes metadata when newer external state is observed; - Job lifecycle: durable acceptance, failover reconstruction, refresh retry (`COMMITTED_REFRESH_PENDING`), admission limits enforced; - Cancellation honesty test: CANCEL fences tracking/refresh; a build that completes externally afterwards is picked up by the next reconciliation/refresh (§5) and is droppable; - Lost-response reconciliation stays `OUTCOME_UNKNOWN` (Directory and REST-without-transaction_id); - Local/file visibility restriction; job record contains no credentials (edit-log content test). ## 9. Moved to future work — tracked, not dropped All removals land in one tracked place: **[lance-format/lance-c#55](https://github.com/lance-format/lance-c/issues/55)** (upstream RFC, filed 2026-08-07) plus a Doris-side follow-up issue to be opened when the 4.2 branch cuts. **Every deferred item below is a Directory-path capability** — per round-3 follow-up, lance-c additions do not expand REST Catalog capabilities: REST `OR REPLACE`, incremental build, cancellation, and additional request parameters require Lance **Namespace spec evolution**, tracked separately as a spec-level follow-up with the Lance community; nothing in lance-c#55 expands the §4 REST matrix. The mapping: | Deferred item (round-3 list) | lance-c#55 extension | |---|---| | Fragment-scoped distributed builds | E1 (fragment-scoped uncommitted build) | | Shared IVF centroid/PQ-codebook distribution | E1 (training entries + model injection) | | BE→FE uncommitted segment metadata transfer | E2 (`pb::IndexMetadata` bytes; cross-SDK consumption requires an explicit Java decoder/adapter from bytes to `Index.Builder` per-field setters, plus a version-compatibility contract — the pinned SDK has no `parseFrom(byte[])`; per reviewer note, issuecomment-5212293290) | | Coordinator commit of worker-built segments | E5 / existing Java SDK path | | Physical segment merge | E5 open question | | Native progress / cooperative cancellation | E3 / E4 | | Per-segment coverage/statistics | E6 (also restores the TVF) | | Incremental `BUILD INDEX` | Composition of E1+E2+commit | Correction to my earlier cross-link comment (issuecomment-5212075800), per round-3 Q3: nothing in lance-c#55 gates this issue — the RFC continues as the post-4.2 distributed track, and its wave split was designed for exactly this decoupling. When the deferred track resumes, v3's provenance design (pre-assigned UUID sets + transaction-properties commits) is reinstated on top of E1/E2, closing the §6 gap. ## 10. Work items (4.2; branch-4.1; deliverable as separate PRs) 1. FE neutral command model + property validation (`metric` key; rejection of REST-inexpressible properties) + explicit `BUILD INDEX` rejection for Lance; 2. Durable job framework + SHOW/CANCEL integration (§5 states, admission, privileges); 3. BE one-shot execution task for Directory (lance-c v0.1.2 calls, selected-BE dispatch, credential handling at dispatch); 4. REST Namespace path per §4 matrix (create/drop/poll, transaction_id persistence); 5. Metadata surfaces (§7: SHOW INDEX grouping, physical type, bounded properties); 6. Failure recovery + refresh per §6; 7. Tests per §8 + documentation (syntax, lifecycle semantics, **limitations**: best-effort CANCEL, no native progress, `OUTCOME_UNKNOWN` boundaries, single-node whole-dataset build). ## Round-3 answers adopted - **Q1**: branch-4.1 only. ✔ - **Q2**: option (c) — `metric` with `l2|cosine|dot`, matching `vector_search()`; no internal ANN query/property syntax compatibility introduced. ✔ - **Q3**: lance-c extensions are not prerequisites; tracked as future work (§9). ✔ One clarification on scope wording: v3 had gated the *production* path on extended lance-c because its target was the distributed build form; with 4.2's production form redefined as the one-shot selected-BE execution, no gating remains. The §6 provenance weakening is the only deliberate technical downgrade, and it is stated rather than hidden. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
