Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5301826623 Thanks @Gabriel39. I updated the original [v5.1 design](https://github.com/apache/doris/issues/66497#issuecomment-5301314544) in place with only the two requested clarifications. No other design text changed. ```diff @@ Section 5.2 — Hard process isolation -The BE supervisor and queue are bounded. Pre-invocation busy rejection leaves the job `PENDING` for another admissible BE; durable `RUNNING` forbids any automatic redispatch. +The BE supervisor and queue are bounded. While a job is `PENDING`, BE selection and possible-live-slot reservation are strictly non-invoking: they do not send the execute request, launch a worker, initialize Lance/Rust, or enter FFI. FE durably records `RUNNING` before sending the single execute request. Once that request is sent, a complete trusted identity-matched busy or other pre-FFI rejection is terminal `NOT_COMMITTED` and is never automatically redispatched; a missing, partial, or untrusted result is `UNKNOWN`. + +Local/`file://` mutation has a separate disabled-by-default operator assertion, requires exactly one FE and BE, normalized absolute paths, pre/post version and identity checks, and rejection on topology change before dispatch. These checks do not prove a shared mount namespace. Object storage is the production mode. ``` The first change makes the `PENDING` boundary strictly non-invoking and makes every complete trusted post-send pre-FFI rejection terminal `NOT_COMMITTED`, with no automatic redispatch. The second restores the disabled-by-default single-node/test assertion, mandatory multi-node rejection, the unprovable shared-filesystem warning, and object storage as the production path. I will proceed with implementation against the updated v5.1 contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5301801004 Small v5.1 §3.1 clarification (no scope change): with pinned `lance-core 9.1.0-beta.3`, unfiltered `Dataset.describeIndices()` is not safe for Directory `SHOW INDEX`. The pinned JNI converts every returned `IndexDescription` by calling `details()`, while a valid system entry can have a details type with no scalar plugin; that entry can abort the bulk call before Doris can filter it ([pinned JNI source](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/java/lance-jni/src/index.rs#L15-L51); [observed PR case](https://github.com/apache/doris/pull/66637#discussion_r3781698221)). For this Directory SHOW slice, the one-snapshot rule therefore also permits one opened `Dataset` to call `listIndexes()`, filter the formal system names `__lance_frag_reuse` and `__lance_mem_wal`, deduplicate repeated physical entries by name, and call `describeIndices(IndexCriteria.hasName(name))` for each remaining exact name on that same `Dataset` ([physical-list source](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/java/lance-jni/src/blocking_dataset.rs#L1875-L1901); [logical-grouping source](https://github.com/lance-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/rust/lance/src/index.rs#L1270-L1311)). This Dataset index read MUST retain one deadline and separate finite raw/logical bounds, and fail closed unless every lookup returns exactly one non-null description with the requested name and the final logical metadata is well formed and unique. The formal pinned MemWAL name is `__lance_mem_wal` ([constant](https://github.com/la nce-format/lance/blob/e934cc2ceda2bd5f5aa37a953cc29f71d24bd5c0/rust/lance-table/src/system_index/mem_wal.rs#L13)). This clarification does not change REST, physical-inspection, or mutation scope. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5301637401 Thanks for the v5.1 revision. I read it against the full discussion history and the final Doris 4.2 scope clarification. I agree with the design overall: it preserves the accepted one-shot safety invariants, keeps the lifecycle compact, and correctly moves the broader reconciliation, REST, distributed, incremental, and query-architecture work out of the 4.2 release boundary. No further structural rewrite is needed. Please make only these two contained clarifications before proceeding: 1. **Restore the local/`file://` mutation restriction.** Earlier revisions correctly made local-path mutation disabled by default, allowed only under an explicit single-node/test operator assertion, and rejected it in multi-node deployments because identical path strings do not prove a shared filesystem namespace. V5.1 currently scopes mutation to Directory Catalog generally but no longer states this restriction. Please restore it and keep object storage as the production path. 2. **Remove the ambiguity around pre-invocation busy rejection.** Sections 5.1/5.3 require durable `RUNNING` before execute network I/O and prohibit redispatch once `RUNNING`, while Section 5.2 says a pre-invocation busy rejection leaves the job `PENDING` for another BE. Please make the boundary explicit: while `PENDING`, selection/reservation must be strictly non-invoking; once `RUNNING` is durable and the execute request is sent, a complete trusted pre-FFI rejection should become `NOT_COMMITTED` and must not be automatically redispatched. Alternatively, define a separate non-invoking reservation handshake and state clearly that it cannot enter FFI. With these two small corrections, the proposal matches the expected 4.2 boundary. Please go ahead with implementation after updating the design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5301314544 # Design Proposal v5.1 — Minimal One-Shot Lance Index Lifecycle for Doris 4.2 > Prepared for publication: 2026-08-15 (Asia/Shanghai) > Target: Apache Doris `branch-4.1` / Doris 4.2 > Issue: [apache/doris#66497](https://github.com/apache/doris/issues/66497) > Status: v5.1 final publication candidate; community confirmation requested below > Review boundary last checked: 2026-08-15, through > [the final 4.2 scope clarification](https://github.com/apache/doris/issues/66497#issuecomment-5290050744) ## 0. Authority and revision summary This standalone document defines the proposed capability and safety contract for Lance index lifecycle management in Doris 4.2. If accepted, it supersedes v5. The v5 proposal and its errata remain issue-history provenance only: they provide no implementation guidance, compatibility contract, or roadmap. Requirements exist only where stated here; the final 4.2 scope clarification controls if review-history statements conflict. ### Context for this revision During earlier review rounds, v5 incorporated the correctness and implementation constraints under discussion, and I interpreted some of them as Doris 4.2 delivery requirements. The latest review clarified a narrower final 4.2 boundary. v5.1 preserves the accepted safety invariants, explicitly defers the broader mechanisms, and supersedes v5 for implementation. Relative to v5, this revision retains the accepted correctness rules and four contained requirements—credential rotation, unresolved-work quotas, possible-live worker accounting, and a small versioned schema contract—while removing broader reconciliation, identity, metadata, REST, and distributed or incremental architecture from the 4.2 gates. The contract below restates every retained requirement without depending on an earlier proposal. The baseline is Doris commit [`e3289c1a5df7558cb8e63d80379d4edebf9c498c`](https://github.com/apache/doris/commit/e3289c1a5df7558cb8e63d80379d4edebf9c498c), with FE `lance-core` `9.1.0-beta.3` and BE `lance-c` `0.1.2` backed by Rust Lance `4.0.1`. No 4.2 guarantee depends on upgrading or extending those APIs. Normative terms in this document use **MUST**, **MUST NOT**, **SHOULD**, and **MAY** in their ordinary requirements sense. A section explicitly marked future work is non-normative for 4.2. > **Review focus: Does this document accurately reflect the final Doris 4.2 > implementation boundary?** The requested confirmation appears in Section 11. ## 1. Scope and release boundary ### 1.1 Core safety contract Doris 4.2 executes one one-shot Lance mutation without retrying after dispatch may have occurred, fabricating an outcome, or reusing its logical name while the result is uncertain. Such a result remains `UNKNOWN` until an authorized operator accepts the late-commit risk through audited `FORCE_RELEASE`. ### 1.2 Required in 4.2 The 4.2 deliverable includes: - Directory Catalog `CREATE INDEX` for the initially verified vector and scalar index types; - `CREATE OR REPLACE INDEX` as a full same-name rebuild; - `DROP INDEX` by logical name; - `SHOW INDEX` for logical index metadata; - one bounded physical inspection surface for index UUID and dataset version; - basic `IF NOT EXISTS` and `IF EXISTS` behavior based on authoritative Lance metadata; - table-level `SHOW` and `ALTER` authorization; - an asynchronous durable job created before dispatch; - at-most-once execution on one selected BE through a hard resource-isolated worker process; - the compact lifecycle `PENDING -> RUNNING -> COMMITTED | NOT_COMMITTED | UNKNOWN`; - independent metadata-refresh status; - a same-name correctness fence retained by `UNKNOWN`; - one explicit audited `FORCE_RELEASE` path; - a small versioned schema contract and dataset-version revalidation before native invocation; - bounded unresolved-job admission and hard possible-live worker accounting; - credential rotation while a job is active or `UNKNOWN`; - focused crash, OOM, panic, failover, replay, privilege, and end-to-end tests; - user documentation for syntax, lifecycle, failures, limitations, and operator recovery. Directory Lance manifests remain the authoritative index metadata. Doris persists mutation intent and control state, not a second copy of the current external index definition. ### 1.3 Explicitly deferred from 4.2 These exclusions are normative and cannot gate 4.2. Future work requires a separate issue and review and creates no roadmap here: - automatic `UNKNOWN` watchers or reconciliation daemons, including persisted backoff, jitter, observations, or metrics; -
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5290050744 This is a scope clarification and should be used as the 4.2 implementation boundary. For 4.2, the mutation requirement is intentionally small: > Safely execute one one-shot Lance mutation. If the result is uncertain, do not retry it, do not report a false success/failure/cancellation, retain the same-name fence, and allow an explicit audited manual release. The advanced mechanisms described in v5 are useful design material, but they are not requirements for the current 4.2 code or release gates. ## Required in 4.2 ### Original index lifecycle - Directory Catalog CREATE for the initial verified vector and scalar index types. - CREATE OR REPLACE as a full rebuild. - DROP by logical name. - `SHOW INDEX` for logical index metadata. - One bounded physical inspection surface only if needed to expose UUID and dataset version that do not fit the existing SHOW schema. - Basic `IF EXISTS` / `IF NOT EXISTS` behavior using authoritative Lance metadata. - Table SHOW/ALTER privileges, basic same-name serialization, metadata refresh, tests, and documentation. ### Minimum safe one-shot execution - Persist the job before dispatch. - Execute at most once on one selected BE through a resource-isolated worker. - Do not automatically retry after dispatch may have happened. - Do not offer running cancellation. - A complete typed result may report COMMITTED or NOT_COMMITTED. - A missing, partial, lost, crashed, timed-out, or otherwise ambiguous post-dispatch result becomes UNKNOWN. - UNKNOWN retains the same-name fence. - Provide one explicit audited FORCE RELEASE operation so an operator can accept the late-commit risk and unblock the name. - Keep mutation outcome and metadata refresh distinct so refresh success never fabricates a mutation result. - Revalidate the required dataset version and a small versioned schema contract before invocation. - Add simple unresolved-job quotas, allow credential rotation, and do not reuse a hard possible-live worker slot until termination proof or manual force release. - Include focused OOM/panic/failover tests and one end-to-end check that each created index type is consumable by the existing query path. No new query architecture is required. A compact lifecycle such as `PENDING -> RUNNING -> COMMITTED | NOT_COMMITTED | UNKNOWN`, plus refresh status and one manual force-release path, is sufficient for the current implementation. ## Move to follow-up issues ### Permanent UNKNOWN automation - automatic watcher/reconciliation daemons; - persisted backoff, jitter, periodic observations, and watcher metrics; - automatic long-term handling of permanently UNKNOWN jobs; - ACKNOWLEDGED plus multiple FORCE variants, including FORCE WITHOUT REFRESH; - a general immutable resolution-event framework; - permanent UNKNOWN tombstones and advanced archival/replay behavior beyond retaining the unresolved job and fence needed for the manual operation. ### External object identity and compatibility framework - provider-backed table/dataset incarnation; - detecting external deletion and recreation of another dataset at the same URI; - normalization-version migration and mixed-version fence protocols; - a general cross-release schema canonicalization framework beyond the small versioned schema contract needed by the current operation; - broad catalog-wide external-DDL transaction/guard machinery beyond preventing changes to the active target locator and same index name. For 4.2, external replacement of a dataset at the same URI may be documented as unsupported while a mutation is active or UNKNOWN. ### Extended metadata and REST surfaces - duplicate logical inspection APIs when `SHOW INDEX` already provides the logical view; - exact row counts, indexed/unindexed row coverage, fragment coverage, and FE `countRows()` work in normal SHOW paths; - rich physical consistency/provenance diagnostics beyond the fields required by the original issue; - REST inspection as a prerequisite for Directory mutation; - any REST mutation profile or REST mutation lifecycle. REST inspection may be delivered independently later, but it must not gate the Directory lifecycle. ### Distributed and advanced index/query capabilities - distributed index construction and segment commit; - distributed fragment-parallel vector TopN and candidate merging; - incremental BUILD or coverage repair; - build progress and running cancellation; - automatic retry or idempotency protocols; - continuously appended-table freshness SLA; - reusable query I/O recorder infrastructure; - additional index types, composite indexes, nullable-column support, and richer physical
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5289814826 One further scope clarification after comparing v5 with the original issue: metadata reads are needed here only to manage indexes. This issue should not grow into a query-engine, general external-transaction, or permanent reconciliation project for 4.2. Please narrow the 4.2 implementation and release gates to the original lifecycle requirements. **Keep for 4.2:** - Lance metadata remains the source of truth. - Read one pinned snapshot for the fields needed by index management. - `SHOW INDEX` exposes logical name, columns, type, and bounded properties. - At most one additional physical inspection surface exposes UUID and dataset version when they do not fit the existing SHOW schema. - `IF EXISTS`, `IF NOT EXISTS`, and REPLACE use the same authoritative name/type/column comparison. - Admission records and revalidates the required dataset version and schema contract. - Directory CREATE, CREATE OR REPLACE, and DROP run asynchronously on one selected BE in a resource-isolated worker. - There is no automatic mutation retry and no cancellation after dispatch. - A lost or ambiguous result becomes UNKNOWN and retains the same-name fence. - Provide one explicit audited force-release path and refresh metadata after known completion or operator resolution. - Keep the basic privilege, bounded-resource, quota, crash-safety, and vector/scalar end-to-end tests needed to prove the created indexes are usable. **Defer to follow-up issues unless already trivial and independently releasable:** - the duplicate logical `lance_indexes()` TVF when `SHOW INDEX` already represents logical indexes; - exact row counts, indexed/unindexed row or fragment coverage, and FE `countRows()` work in normal SHOW paths; - a permanent automatic UNKNOWN watcher/reconciliation daemon; - ACK plus multiple FORCE variants, permanent tombstones, and the full operator-resolution event framework beyond one force-release operation; - a full provider-backed table-incarnation subsystem and normalization-version upgrade protocol; - broad generic catalog-wide external-DDL transaction machinery beyond protecting the stable locator and same index name needed by this feature; - REST read support as a prerequisite for Directory mutation. REST inspection may ship independently, but it must not gate the original Directory lifecycle; - future REST mutation profiles; - query-side architecture such as distributed fragment-parallel vector TopN, candidate merging, coverage-freshness SLA, or a reusable query I/O recorder. One focused end-to-end test confirming that the existing query path consumes a newly created vector/scalar index is sufficient for this issue; - incremental BUILD, progress, running cancellation, distributed index construction, and other capabilities already listed as non-goals. The durable distinction between mutation outcome and metadata refresh, and the UNKNOWN same-name fence from the previous review, must remain. However, the rest of the five-dimensional lifecycle machinery should not automatically become a 4.2 delivery requirement when a smaller state model can satisfy the original issue safely. Please update the release boundary or provide a smaller implementation plan reflecting this split. The advanced design can remain useful future work, but the current deliverable should focus on create, inspect, replace/rebuild, drop, the minimum safe failure semantics, tests, and documentation requested by the original issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5289747725 Thanks for the comprehensive v5 revision. After reading the full discussion history and rechecking the branch-4.1 boundary, I agree that the three round-5 correctness issues are now addressed: UNKNOWN keeps its same-name fence, outcome and refresh are durable independent dimensions, and RESOLVE has an explicit audited/replayable protocol. For 4.2, I do not want to expand this into a complete external-object identity or distributed-query project. I am comfortable deferring the following, provided they are documented as explicit limitations: - external deletion and recreation of a different dataset at the same URI while a mutation is active or UNKNOWN; - a full provider-backed table-incarnation registry; - moving all Directory metadata reads into an isolated helper, unless the required JNI stress/failure evidence shows that the bounded FE executor is insufficient; - distributed fragment-parallel vector TopN. The current branch-4.1 path intentionally uses one whole-dataset split and one scanner/BE for global TopK, so the documentation should state that distributed vector search/merge is not provided in 4.2. Before approving the mutation part, I think only the following contained changes are required: 1. **Do not let an UNKNOWN job block credential rotation.** A credential-only catalog ALTER should not be classified as target-changing DDL. Locator/provider/version-selection changes must remain guarded, but watchers and refresh should be able to use newly rotated credentials; otherwise an expired session token makes reconciliation impossible and forces an emergency release without refresh. 2. **Add admission quotas for durable unresolved work.** Bound active+UNKNOWN jobs per table/catalog and globally, and reject before durable job creation when the bound is reached. Running concurrency and per-cycle reconciliation limits do not bound the number of permanent job records, fences, tombstones, or watcher entries. 3. **Keep possible-live worker accounting separate from the client/job deadline.** If catalog/cluster concurrency is documented as a hard bound, its slot cannot be automatically reused while an old child may still be running. Release the possible-live slot only after matching reap, BE process-epoch proof, or audited FORCE. The statement/job deadline may still expire independently. 4. **Define a small versioned schema-comparison contract.** This does not need to become a general Arrow canonicalization protocol. A bounded ordered representation of the fields relevant to admission—field ID, normalized name/type, nullability, and fixed-size-list dimension, plus any other intentionally guarded fields—is sufficient, as long as Java and Rust compare the same specified representation across replay and upgrade. For target identity in 4.2, a persisted catalog identity plus provider and normalized stable dataset locator is sufficient; same-URI external replacement can remain the documented unsupported race above. There is no need to add a large incarnation subsystem solely for that corner case. With these four changes and the deferred limitations stated clearly, I would consider the v5 design aligned with the practical 4.2 scope. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5279535778 @Gabriel39 friendly ping 🙂 When you have a moment, could you take a look at the v5 design above (the revised publication candidate posted on Aug 10)? It was rewritten specifically to pin down the three correctness issues from your round-5 review: 1. `OUTCOME_UNKNOWN` keeps the same-name serialization fence (including across normalization-version upgrades) until an audited force release — `RESOLVE LANCE INDEX JOB ... AS FORCE_RELEASE` requires a non-empty risk-acceptance note, so a late commit from an old executor can no longer slip through silently. 2. Mutation attribution and metadata refresh are now independent, durable decisions — e.g. a `CommitConflict` after invocation ends the job as `NOT_COMMITTED` but still requires an authoritative refresh, and `IF [NOT] EXISTS` resolves the race window by comparing authoritative readable fields. 3. Operator resolution is now a fully specified durable state transition: explicit `RESOLVE LANCE INDEX JOB` SQL, privilege requirements, persisted `LastResolvedBy`/`LastResolvedAt`, bounded immutable resolution events, and defined replay behavior — without claiming the original mutation outcome became known. The implementation constraints are pinned as well: native mutation runs in a hard-limited worker process (never in FE/BE), `file://` mutation is a separate disabled-by-default operator assertion with topology checks, and REST goes through one code-defined `generic-read-only-v1` profile with provider mutation profiles explicitly future work. I also posted a small implementation errata today as a comment above — logical/physical index-type vocabulary normalization (case/underscore-insensitive, umbrella `VECTOR` family accepted), INCONSISTENT results keep the `READ_FAILED` token, reserved system-index entries (`__lance_frag_reuse`/`__lance_mem_wal`) are filtered, and the physical-inspection TVF shares the 256-char name cap — recorded as comments rather than a v6 since none of these change the design semantics. Implementation has started in stages per @zhangstar333's suggestion — the first PR (#66637, read-only `SHOW INDEX`) is open — so your review or approval of v5 would unblock the next stages. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5275004917 Implementation errata for v5 (found during PR #66671 verification; recorded here instead of respinning the doc): 1. **Index type vocabulary (§8.1/§8.2/§7)**: on pinned Java SDK 9.1.0-beta.3, `getIndexes()` reports the umbrella manifest type (`VECTOR`) while `describeIndices()` derives the specific algorithm (`IVF_PQ`); scalar names differ in case/underscores (`BTREE` vs `BTree`). Verified with the regression fixture's exact `createIndex(IndexType.IVF_PQ)` call. Read literally, §8.1's "same physical type" invariant fails every healthy vector index. The implementation normalizes case/underscores and accepts the umbrella families (physical `VECTOR` ↔ vector algorithms, `SCALAR` ↔ scalar algorithms); internal types never match. The physical TVF shows the raw manifest value per §8.2; logical surfaces show the describe value. The mutation-side `IF NOT EXISTS` comparison (§7) must reuse this normalized rule. 2. **`LANCE_INDEX_METADATA_INCONSISTENT` scope (§8.1)**: orphan / missing-entry / mismatch / duplicate-UUID produce INCONSISTENT; normalize-time structural failures (unknown field id, exact duplicate name, malformed JSON, bound violations) surface as `LANCE_INDEX_METADATA_READ_FAILED`. Both typed and fail-closed. 3. **System indexes (§8.1)**: the pinned SDK list path returns `__lance_frag_reuse` / `__lance_mem_wal` unfiltered, and a fragment-reuse entry carries no fields. The implementation filters both names at the normalization boundary so compacted datasets stay inspectable. 4. Minor: the 256 exact-name bound also applies to the physical TVF (§8.1 scopes it to logical reads); intentional shared defensive bound. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497:
URL: https://github.com/apache/doris/issues/66497#issuecomment-5241658396
# Design Proposal v5 (Revised) — Lance Index Lifecycle for Doris 4.2
- Status: revised publication candidate
- Target: Apache Doris `branch-4.1` / Doris 4.2
- Issue: [apache/doris#66497](https://github.com/apache/doris/issues/66497)
- Source: full v5 proposal at commit `dec38e63f`
- Supersedes: proposals v1 through v4.1; supersedes the source v5 after
approval
## 0. Authority, evidence, and review request
This is a standalone revision of v5. It preserves the lifecycle, failure, and
acceptance contract of the source proposal while giving each requirement one
normative definition. Appendices in this document are normative;
implementation
notes are explicitly marked non-normative.
Statements have three meanings:
- **Baseline fact**: verified in the target Doris commit or pinned
dependency.
- **4.2 decision**: required behavior and acceptance evidence.
- **Future work**: behavior that 4.2 must not promise.
The baseline is Doris commit
[`e3289c1a5df7558cb8e63d80379d4edebf9c498c`](https://github.com/apache/doris/commit/e3289c1a5df7558cb8e63d80379d4edebf9c498c),
with FE `lance-core` `9.1.0-beta.3`, Lance Namespace `0.7.7`, and BE
`lance-c` `0.1.2` (Rust Lance `4.0.1`). No 4.2 guarantee depends on
upgrading or
extending those APIs.
Approval is requested for these decisions:
1. common read-only inspection lands before mutation;
2. mutation is Directory-only and uses the pinned one-shot APIs;
3. logical metadata/count and physical UUID/version are separate surfaces;
4. native mutation runs in a hard-limited worker process, never in FE or BE;
5. dispatched work has no automatic retry or running cancellation;
6. outcome, refresh, reconciliation, disposition, and resource accounting are
durable but independent;
7. an unknown job keeps its same-name fence until an audited force release;
8. mutation remains disabled until the release evidence in Section 10 passes.
## 1. Scope and release boundary
### 1.1 Goals
Doris 4.2 provides:
- authoritative Directory and REST `SHOW INDEX` support;
- an exact logical-index count and bounded physical UUID/version inspection;
- Directory CREATE, full same-name REPLACE, and DROP;
- IVF_PQ vector and BTREE/BITMAP scalar indexes;
- asynchronous durable jobs for every Directory mutation;
- explicit naming, version, privilege, concurrency, failure, refresh,
resource,
and operator-resolution behavior;
- cross-SDK and query-I/O evidence that the created indexes are readable and
actually consumed.
Lance manifests are authoritative for Directory metadata. The Namespace
service
is authoritative for REST metadata. Doris persists mutation requests and job
control, not a second copy of the current external index definition.
### 1.2 Non-goals
The following are not provided in 4.2:
- incremental `BUILD INDEX`, distributed fragment builds, segment merge, or
FE-side segment commit;
- native progress or reliable cancellation after dispatch;
- automatic retry of a dispatched mutation;
- attribution from a matching metadata postcondition;
- REST mutation under the generic Namespace contract;
- IVF_FLAT, HNSW, FTS, composite indexes, nullable indexed columns,
additional
vector dtypes, 4-bit PQ, or user-supplied scalar tuning JSON;
- fragment IDs, opaque index details, runtime hints, or unbounded arrays;
- proof that two processes share the same local mount namespace.
`BUILD INDEX` on a Lance table fails before job creation and names the
incremental-build deferral.
## 2. User-visible contract
### 2.1 SQL
```sql
CREATE INDEX [IF NOT EXISTS] idx
ON lance_ctl.db.tbl (embedding)
USING ANN
PROPERTIES (
"index_type" = "IVF_PQ",
"metric" = "l2",
"num_partitions" = "256",
"num_sub_vectors" = "16"
);
CREATE INDEX idx_btree ON lance_ctl.db.tbl (event_time) USING BTREE;
CREATE INDEX idx_bitmap ON lance_ctl.db.tbl (category) USING BITMAP;
CREATE OR REPLACE INDEX idx
ON lance_ctl.db.tbl (embedding)
USING ANN
PROPERTIES (
"index_type" = "IVF_PQ",
"metric" = "cosine",
"num_partitions" = "256",
"num_sub_vectors" = "16"
);
SHOW INDEX FROM lance_ctl.db.tbl;
SELECT COUNT(*) AS logical_index_count
FROM lance_indexes("table" = "lance_ctl.db.tbl");
SELECT *
FROM lance_index_entries("table" = "lance_ctl.db.tbl")
WHERE index_name = "idx";
DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;
```
`CREATE OR REPLACE` maps to `replace=true` and is a full rebuild. It is
convergent by name but not idempotent: another invocation may create a
different
UUID and dataset version. `IF NOT EXISTS` and `OR
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5238268312 > I also prefer a phased approach: > > 1. Implement the common SHOW INDEX support first. > 2. Then CREATE, REPLACE, and DROP INDEX for Directory Catalog. > 3. . >it's better to split parts of PRs Thanks, agreed. I will keep #66497 as the umbrella lifecycle issue, but the implementation will not be delivered as one large PR. The proposed order is: 1. Common read-only `SHOW INDEX` support for Lance external tables first. This PR will use authoritative FE metadata reads and will not include mutation jobs or BE worker changes. 2. Complete the remaining read-only inspection surfaces, such as the logical count and bounded UUID/version metadata, in the same phase or an immediate follow-up PR depending on review size. 3. Add Directory Catalog CREATE / REPLACE / DROP through smaller, feature-gated PRs: - neutral FE command dispatch and the durable job/state model; - isolated BE worker and IVF_PQ end-to-end mutation; - BTREE/BITMAP completion, failure tests, and documentation. The cross-SDK mutation spike will gate the mutation PRs, but it will not block the initial SHOW INDEX work. REST mutation and incremental BUILD remain out of scope for 4.2. I will update the v5 implementation section to make this PR ordering explicit. The lifecycle design remains the umbrella contract, not a plan for one monolithic PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
zhangstar333 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5237219932 I also prefer a phased approach: 1. Implement the common SHOW INDEX support first. 2. Then CREATE, REPLACE, and DROP INDEX for Directory Catalog. 3. . it's better to split parts of PRs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5226393984 Thanks for the v4.1 delta. It correctly addresses the four primary round-4 issues: cancellation is now pre-dispatch only, Directory failover no longer claims resumable polling without executor identity, ambiguous errors are separated from proven no-commit failures, and the IVF_PQ properties match the pinned lance-c implementation. I do not think the design is approval-ready yet. Three correctness issues remain. ## 1. `OUTCOME_UNKNOWN` must not automatically release the same-name serialization fence The proposal says that an unknown job no longer holds the same-name fence, even though the old executor may still be running and may commit later. This permits the following sequence: 1. job A for index `idx` loses contact and becomes `OUTCOME_UNKNOWN`; 2. its same-name fence is released; 3. the operator resolves or re-issues the operation, and job B establishes a new known state for `idx`; 4. the old executor for job A commits afterwards. The late commit can overwrite, remove, or reintroduce state after job B has finished. Holding a catalog/cluster admission slot does not prevent this when the configured limit is greater than one, and automatic retention expiry makes the same race possible without an operator decision. While executor termination cannot be proved, the table/index-name fence must remain. Retention expiry may archive the job record, but it must not silently release this correctness fence. If 4.2 needs an escape hatch, make it an explicit force-resolution operation requiring ALTER privilege, a non-empty note, and a warning that a late external commit remains possible. Please add a test in which the old executor commits after a force-release. ## 2. Proven no-commit and metadata refresh are independent decisions Section 3 currently maps typed commit conflict, already-exists, and not-found results to `FAILED` with no refresh. A commit conflict proves that this job did not commit, but it also proves that the external dataset advanced. Likewise, already-exists or not-found can reveal that the state changed between pre-dispatch reconciliation and execution. Doris must refresh authoritative metadata in these cases even if the mutation job itself ends as `FAILED`. The IF modifiers also need operation-specific handling in this race window: - `CREATE INDEX IF NOT EXISTS` receiving already-exists should re-read authoritative metadata and finish as a no-op when the documented comparison matches; - `DROP INDEX IF EXISTS` receiving not-found should refresh and finish as a no-op; - a mismatch may still fail, but it must not leave stale metadata behind. Please separate mutation attribution from cache-coherence behavior instead of making refresh conditional on a successful mutation. ## 3. Operator resolution is not yet a durable, implementable state transition The delta says that resolution records a note, refreshes metadata, and releases the slot, but it does not define: - the SQL or administrative API; - the required privilege; - the durable target state and persisted `resolved_by` / `resolved_at` information; - whether acknowledgement and resolution are distinct operations; - CAS behavior against concurrent resolution or DDL; - replay behavior if FE fails between refresh and slot release. Without these rules, the resolution and expiry cases in the acceptance suite cannot be implemented deterministically. Since there is no job-specific evidence, the resulting state should also avoid claiming that the original mutation outcome became known. ## Additional implementation constraints - **BE resource isolation:** catalog-wide and cluster-wide job counts do not prevent multiple one-shot builds from concentrating on one BE. The BE path needs a bounded dedicated executor, per-BE admission, and scheduler-aware selection. A potentially long lance-c call must not occupy a general RPC handler thread. The design should also state how native-memory pressure is guarded so one build cannot take down the BE. - **`file://` enforcement:** one registered FE and one registered BE does not prove that they share a filesystem namespace. Either restrict this to a verifiable co-located test deployment or define the flag explicitly as an operator assertion and document what Doris can and cannot check. - **REST provider profiles:** define how a code-defined profile is selected and versioned. The generic REST profile should remain read-only unless an allowlisted profile supplies the per-operation status, transaction, or synchronous-completion contract. Doris should neither infer a profile from the URI nor accept user-defined status strings. The FE Java-SDK metadata-read direction is feasible with the pinned SDK's
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5225638962 # Revised Design Proposal v4.1 — Round-4 Delta for Lance Index Lifecycle > Delta for [apache/doris#66497](https://github.com/apache/doris/issues/66497). This is not a standalone restatement of the design. The [v4 proposal](https://github.com/apache/doris/issues/66497#issuecomment-5214045343) remains authoritative except for the paragraphs explicitly replaced or supplemented below. v4 + this v4.1 delta is the current design; the first implementation PR will carry the consolidated full text. Thanks for the precise round-4 review. The four primary issues and five additional details are addressed below without reopening the accepted 4.2 framework. Section 5.2 also marks one deliberately new v4.1 safety decision—the REST terminal-evidence admission gate—rather than presenting it as a round-4 requirement. ## 1. CANCEL — accepted only before dispatch This replaces v4 §5 cancellation semantics. `CANCEL BUILD INDEX` is accepted only while the job is `PENDING`. It transitions: ```text PENDING -> CANCELLING -> CANCELLED ``` Nothing is dispatched in those states. Once `RUNNING` is durably persisted, CANCEL is rejected with a typed error explaining that the one-shot native call is not interruptible in 4.2. The job then runs to its real outcome: success enters refresh and `FINISHED`; proven no-commit failure enters `FAILED`; ambiguity enters `OUTCOME_UNKNOWN`. An accepted pre-dispatch CANCEL ends in `CANCELLED`, not `FAILED`. All state transitions use one FE job-registry writer in edit-log order, equivalently compare-and-swap on the expected durable state. The dispatcher's `PENDING -> RUNNING` transition and dispatch decision form one critical section: - if CANCEL wins first, the dispatcher observes `CANCELLING` and abandons dispatch; - if dispatch wins first, `RUNNING` is persisted before the RPC send and CANCEL is rejected; - a replayed `CANCELLING` job converges to `CANCELLED` and was never dispatched. The network send cannot be atomic with the edit log. If FE fails after persisting `RUNNING` but before or after the send, recovery follows §2 and never blindly re-dispatches the job. REST jobs use the same pre-dispatch-only rule; Namespace 0.7.7 exposes no cancel operation. ## 2. Directory `RUNNING` failover — no resumable polling in 4.2 This narrows v4 §5's general failover wording. The 4.2 Directory job record deliberately carries no selected-BE identity, BE task id, or queryable dispatch generation. Neither the Doris task path nor lance-c 0.1.2 provides a durable job-status API for the one-shot call. Therefore, loss of FE/BE task communication while a Directory job is `RUNNING`, including FE failover, BE process loss, and lost RPC response, transitions the job to `OUTCOME_UNKNOWN`. It cannot resume polling and is never automatically re-dispatched. `RUNNING` is persisted before the RPC is sent. Consequently, replayed states have these meanings: - `PENDING`: never dispatched; normal scheduling is safe; - `CANCELLING`: never dispatched; complete to `CANCELLED`; - Directory `RUNNING`: dispatch may or may not have begun; move to `OUTCOME_UNKNOWN`; - `REFRESHING` / `COMMITTED_REFRESH_PENDING`: retry refresh only, never mutation. For REST, polling may resume only when the returned `transaction_id` was durably persisted. If FE failed before persisting it, a matching list/stats postcondition remains corroboration only and the job becomes `OUTCOME_UNKNOWN`. The admission slot of an `OUTCOME_UNKNOWN` job is held conservatively until operator acknowledgement/resolution or expiry. The unknown job no longer holds the same-name serialization fence, but the held catalog/cluster admission slot may still block re-issue when the configured limit is exhausted; the design does not claim otherwise. `OUTCOME_UNKNOWN` is never retried automatically. Operator resolution records a non-empty resolution note, refreshes authoritative metadata, and releases the slot. Unacknowledged unknown jobs expire after the configurable terminal-job retention period, default seven days; expiry also releases the slot. Expiry is not proof that an old executor stopped and may therefore cause temporary over-admission—an explicit 4.2 limitation of not adding a durable/queryable BE task protocol. The same lifecycle applies to DROP jobs. ## 3. Typed error/outcome classification This replaces v4 §6's rule that any returned BE/REST error is a confirmed pre-commit failure. - **Proven not dispatched or not committed -> `FAILED`**: FE analysis/provider validation failure; BE proof that the native call was not invoked; typed unsupported/invalid-argument/not-found/already-exists result with a pinned no-commit guarantee; typed
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5214852990 Thanks for the v4 revision. The release scope is now substantially aligned with the confirmed 4.2 boundary: branch-4.1, the pinned one-shot lance-c APIs, no distributed/incremental build dependency, the existing `vector_search()` query contract, and a separate REST capability matrix. I do not think the design needs another structural rewrite. There are still four issues that should be corrected before approval. ## 1. A job must not become `CANCELLED` while its external mutation can still commit The proposed behavior persists a fence, marks the job `CANCELLED`, stops observing and refreshing, while the one-shot native operation may continue and commit. This makes the terminal state false: a CREATE, REPLACE, or even DROP can be reported as cancelled although its side effect succeeded. It also leaves Doris metadata stale and can release an admission slot while the BE is still consuming resources. For 4.2, please use one of these semantics: - allow cancellation only while the job is `PENDING`, before dispatch, and reject CANCEL once the one-shot call is `RUNNING`; this is my preferred option; or - keep a running job in `CANCEL_REQUESTED`/`CANCELLING`, continue observing it, and select the terminal state from the real outcome. It must not enter `CANCELLED` until non-execution/non-commit is known. If the operation may have committed, metadata refresh must not be deliberately suppressed. ## 2. The persisted job record is missing the identity required to recover a Directory task The v4 job record does not contain the selected BE, BE task ID, dispatch attempt/generation, or any queryable executor identity. Nevertheless, the failover section says a new FE resumes polling or refresh. Consider: FE1 dispatches the one-shot call to BE3 and fails while BE3 is still running. FE2 can replay a record saying only `RUNNING`; it does not know which BE/task to query, and the Directory path has neither a Namespace transaction nor a lance-c job-status API. It therefore cannot resume polling. Please either: - define a Doris-layer durable/queryable BE task protocol and persist at least `selected_be_id`, `be_task_id`, and dispatch generation before dispatch; or - state that loss of FE/BE task communication during a running Directory operation transitions the job to `OUTCOME_UNKNOWN` and is not recoverable by polling in 4.2. This is also necessary for correct admission accounting after FE failover. ## 3. A returned error is not always proof of a pre-commit failure The current classification treats any BE/REST call that returns an error as a confirmed pre-commit failure. That is only safe for errors that prove the operation never committed, such as local validation failures, unsupported operations, definite not-found/already-exists outcomes, or a definite commit conflict. RPC timeout/disconnect, object-store I/O around commit, BE process loss, and REST timeout/5xx/lost response may have an unknown outcome. Please define an error/outcome matrix based on typed error classes: - proven not dispatched/not committed → `FAILED`; - proven committed → refresh and finish; - transport loss, timeout, ambiguous commit-stage I/O, or lost executor state → `OUTCOME_UNKNOWN` plus refresh-on-observation. Do not classify all non-zero C returns or REST failures as confirmed pre-commit failures without an API guarantee for each error class. ## 4. IVF_PQ defaults do not match the actual lance-c 0.1.2 implementation The retained v3 property semantics make `num_partitions` and `num_sub_vectors` optional, but the pinned implementation calls `require_field` for both: `build_ivf` requires `num_partitions > 0`, and `build_pq` requires `num_sub_vectors > 0`. https://github.com/lance-format/lance-c/blob/v0.1.2/src/index.rs#L315-L357 Passing zero therefore fails even though the header comments suggest an internal default. The 4.2 contract must either require both properties for IVF_PQ or have Doris fill concrete, documented values. I prefer requiring them in the initial version rather than introducing an implicit data-dependent default policy. ## Additional details to tighten - `SHOW INDEX.Properties` should not recover metric/build parameters from a historical job record. The Lance manifest/Namespace is authoritative, the current job record does not establish that its definition still corresponds to the current UUID/version, and an external writer may have replaced the index. If 0.1.2 cannot read a property back, leave it empty rather than expose potentially stale auxiliary state. - Define how `OUTCOME_UNKNOWN` jobs are acknowledged/resolved or eventually retained/expired; otherwise they remain permanently visible even after an
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5214045343 # Revised Design Proposal v4 — Lance Index Lifecycle, scoped to the 4.2 boundary > Draft for apache/doris#66497, prepared 2026-08-07. Incorporates @Gabriel39's round-3 review (issuecomment-5211673675) and its follow-up (issuecomment-5212293290): target **branch-4.1 only** (the confirmed 4.2 base), **no functionality requiring lance-c extension** in this release, query consumption stays on the existing `vector_search()` TVF, and the v3 distributed-build machinery moves to a separately tracked future-work item. v4 is intentionally a *subtraction* from v3: everything removed is named in §9 with its future-work tracking point. /cc @Gabriel39 @zhangstar333 ### Major changes from v3 1. **Scope cut to the one-shot pinned APIs.** Directory builds execute the existing lance-c v0.1.2 one-shot lifecycle calls on a selected BE (`lance_dataset_create_vector_index` / `create_scalar_index` / `drop_index` / `index_count` / `index_list_json`, [email protected] L490-541); REST keeps the v3 capability matrix on the pinned Namespace 0.7.7 model. No new native capability is required anywhere. 2. **`BUILD INDEX` (incremental) removed** from 4.2 — it was added during review and is not required by the issue; its intended implementation needs APIs outside the 4.2 boundary. The statement is rejected for Lance tables with an explicit error (§2). 3. **Provenance rewritten honestly (§6).** The one-shot C calls commit internally and carry no client operation ID and no pre-assigned segment UUID set. v3's deterministic Directory provenance is unavailable in 4.2; reconciliation after a lost response is explicitly bounded to `OUTCOME_UNKNOWN` + diagnostics. This is the proposed 4.2 trade-off, with the upgrade path tracked in §9. 4. **CANCEL is best-effort and documented as such** (§5): it fences Doris-side tracking and refresh; it does not interrupt the native build. 5. **Query consumption is verification-only**: acceptance proves an index built by this lifecycle is consumed by the existing `vector_search(..., "use_index"="true")` path (branch-4.1 `VectorSearchTableValuedFunction.java:81,158-159`) with compatible metric and correct results, and that scalar indexes are consumed by the existing Lance predicate path (#65730). No query-syntax or optimizer work. 6. **Q2 resolved per round-3**: property key is Lance-native `metric` with values `l2|cosine|dot`, matching the existing query surface (option c). `USING ANN` remains DDL category reuse only. 7. **`lance_index_segments` TVF deferred**: its distinct value was per-segment coverage, which is exactly the deferred capability; the remaining columns duplicate SHOW INDEX's grouped output. It returns with the future-work track (§9). 8. **FE-side experimental build fallback removed**: it existed to de-risk the distributed path. With the one-shot BE path as the production form, keeping it would add a second, non-production execution mode against the "simplified substantially" direction. ## 1. Goals and Phase-1 boundary (4.2) Provide the issue's lifecycle over the pinned SDKs, nothing more: - `CREATE INDEX` for IVF_PQ vector indexes and BTREE/BITMAP scalar indexes (Directory full; REST per §4 matrix); - `CREATE OR REPLACE INDEX` = full rebuild via the existing `replace=true` behavior; - `SHOW INDEX` — logical definitions, names, UUIDs, columns, types, dataset versions (grouped by logical name where a table carries multiple physical entries per name; REST fields limited to the §4 stats model); - `DROP INDEX` by name (Directory and REST); - durable job status, admission limits, best-effort cancellation, metadata refresh, privileges, and honestly-bounded failure recovery; - Lance dataset manifests / the Namespace service remain the authoritative index metadata; Doris persists only its own job state. Explicit non-goals for 4.2 (each mapped in §9): fragment-scoped distributed builds, shared-model distribution, uncommitted segment transfer, coordinator commit of worker segments, physical segment merge, native progress/cancellation, per-segment coverage/statistics, incremental BUILD, FTS SQL semantics, internal-table ANN query syntax compatibility. Historical-version tables remain read-only for index mutation; inspection uses the selected snapshot, mutation targets only the latest writable state. ## 2. SQL surface - Vector: `CREATE INDEX idx ON t (embedding) USING ANN PROPERTIES ("index_type"="IVF_PQ", "metric"="l2", "num_partitions"="256", "num_sub_vectors"="16")`. - `"metric"` accepts `l2|cosine|dot` (Lance-native values, matching `vector_search()`). Unknown keys/values are rejected at analysis. - Scalar: `USING BTREE` / `USING BITMAP` (optional `"zone_size"` for BTREE per the pinned Lance default, unchanged from v3). -
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5212293290 Thanks for starting the upstream discussion. The Rust-core findings are useful for future distributed-index work, and lance-format/lance-c#55 can remain as that future track. However, this does not change the confirmed scope for Doris 4.2, and Q3 is no longer open: **lance-c extensions are not a prerequisite for #66497 in 4.2**. Features that depend on those extensions must be deferred. Therefore, Wave 1 should not be described as blocking the 4.2 implementation or as de-risking its current schedule; it is outside the release's critical path. For #66497, the next design revision should be simplified around the existing branch-4.1 baseline and the pinned lance-c 0.1.2 one-shot APIs: - `lance_dataset_create_vector_index` and `lance_dataset_create_scalar_index`; - full rebuild/replacement through the existing `replace=true` path; - `lance_dataset_index_count` and `lance_dataset_index_list_json` for the currently available metadata; - `lance_dataset_drop_index`; - execution of the heavy direct-Dataset call on a selected BE for Directory Catalog. The 4.2 design should explicitly exclude fragment-scoped distributed builds, shared-model distribution, BE-to-FE uncommitted segment transfer, coordinator segment commit, incremental `BUILD INDEX`, physical merge, native progress, cooperative cancellation, and per-segment coverage/statistics extensions. It should also avoid promising reliable `CANCEL BUILD INDEX` semantics when the underlying one-shot native call cannot be interrupted. The REST path is separate: lance-c additions do not expand REST Catalog capabilities. REST index lifecycle operations must continue to use the pinned Namespace request/response models and the server's implemented capabilities. In particular, newer lance-c APIs would not provide REST `OR REPLACE`, incremental build, cancellation, or additional request parameters. One future-facing note for lance-c#55: `pb::IndexMetadata` bytes are not directly accepted by the pinned Java SDK. `Index.Builder` takes the metadata as individual fields and exposes no public `parseFrom(byte[])` entry point, so the cross-SDK design still needs an explicit Java decoder/adapter and a version-compatibility contract. This does not need to be solved for #66497 because that distributed path is out of scope for 4.2. Please make the next revision a 4.2 design based only on the existing APIs, covering BE selection/execution, supported index types and properties, `replace=true` semantics, privileges, concurrency, versioning, failure behavior, metadata refresh, local-path visibility, and the restricted Namespace 0.7.7 REST capability matrix. Query-side work should remain limited to validating index consumption through the existing `vector_search()` and scalar predicate paths. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5212075800 Quick update on the prerequisite track (v3 §12 decision 2 / Q3): the upstream engagement has started — I've filed [lance-format/lance-c#55](https://github.com/lance-format/lance-c/issues/55), an RFC covering all six lance-c extensions, split into three independently-mergeable waves: - **Wave 1 (blocking)**: fragment-scoped uncommitted segment builds with shared-model injection (caller-supplied IVF centroids / PQ codebook) + segment metadata serialization (`pb::IndexMetadata` bytes as the BE→FE transfer format). - **Wave 2 (production quality)**: build progress reporting + cooperative cancellation. - **Wave 3 (symmetry/observability)**: commit-existing-segments C API + per-segment coverage/stats in listing. One finding worth surfacing here, since it de-risks the schedule: the Rust core (9.1.0-beta.3) already implements nearly everything behind the C boundary. `CreateIndexBuilder` already carries `fragments` / `index_uuid` / `train` / `progress` / `transaction_properties` and `execute_uncommitted() -> IndexMetadata` (`rust/lance/src/index/create.rs:50-141`), with scalar fragment-scoped builds supported as well (:270); the injection points `IvfBuildParams.centroids` and `PQBuildParams.codebook` both exist. The only genuine core-side gaps are the cancellation hook and per-segment listing stats — the rest is FFI exposure. This does not presume the Q3 answer: if round-3 prefers a different tracking form, the upstream issue adapts. Round-3 feedback welcome whenever convenient — no urgency imposed by this step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5211673675 Thanks for the v3 revision. After checking the confirmed 4.2 release boundary and re-reading the baseline implementation in #65730, I need to correct part of my previous review and narrow the proposal. There are two confirmed product constraints: 1. **Doris 4.2 is based on branch-4.1, not master.** Therefore this work should target branch-4.1. There is no need for a master-first or dual-track implementation in this issue. 2. **No functionality that requires extending lance-c will be included in this release.** Such work is explicitly outside the 4.2 scope. These constraints materially change the appropriate Phase-1 design. ## Query syntax: keep the existing vector_search() contract I rechecked #65730. The baseline deliberately exposes Lance vector search only through the `vector_search()` TVF: https://github.com/apache/doris/pull/65730 The implementation and regression tests use: ~~~sql SELECT row_id, _distance FROM vector_search( "table" = "catalog.db.table", "column" = "embedding", "query_vector" = "[0,0,0,0]", "metric" = "l2", "top_k" = "5" ); ~~~ - Implementation: https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java - Regression coverage: https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/regression-test/suites/external_table_p0/lance/test_lance_vector_search.groovy There is no implementation or stated plan in the baseline to translate internal-table ANN query forms such as `l2_distance_approximate(...)` or `inner_product_approximate(...)` into a Lance search. Therefore: - Lance queries should continue to use `vector_search()`. - This index-management issue should not add compatibility with the internal-table ANN query syntax. - Reusing `USING ANN` as a category in CREATE INDEX is only DDL/parser reuse; it does not imply query-syntax reuse. - For Q2, I now recommend the Lance-native property already used by the existing query surface: `"metric" = "l2|cosine|dot"` (option **c**), rather than introducing `metric_type` with internal-table values. Query behavior is relevant to this issue only as end-to-end verification: an index created by the lifecycle API must be consumed correctly by the existing `vector_search(..., "use_index"="true")` path, with a compatible metric and correct results. This issue should not redesign the query syntax or optimizer rules. ## Scope the implementation to the existing lance-c 0.1.2 APIs The pinned lance-c 0.1.2 already exposes the one-shot lifecycle APIs required by the original issue: - `lance_dataset_create_vector_index` - `lance_dataset_create_scalar_index` - `lance_dataset_drop_index` - `lance_dataset_index_count` - `lance_dataset_index_list_json` https://github.com/lance-format/lance-c/blob/v0.1.2/include/lance/lance.h#L490-L541 The 4.2 design should use those existing APIs, with the heavy direct-Dataset operation executed on a selected BE for Directory Catalog, and use the existing Namespace operations for the supported REST cases. The limitations of the one-shot API—no native progress reporting, cooperative cancellation, distributed fragment build, or job-specific transaction provenance—should be documented honestly rather than solved by expanding lance-c in this release. The following v3 items depend on new lance-c capabilities and should be removed from the 4.2 implementation scope: - fragment-scoped distributed builds; - shared IVF centroid/PQ-codebook distribution to BE workers; - BE-to-FE uncommitted segment metadata transfer; - coordinator commit of worker-built segments; - physical segment merge; - native progress and cooperative cancellation; - per-segment coverage/statistics extensions; - incremental BUILD over uncovered fragments. In particular, the original #66497 scope requires replacing or rebuilding an index with the same name. The existing `replace=true` path provides a full rebuild. Incremental `BUILD INDEX` was added during review, but it is not required by the original issue and should be deferred because the intended distributed/incremental implementation requires APIs outside the 4.2 boundary. ## Revised Phase-1 lifecycle For 4.2, I recommend limiting the lifecycle to: - `CREATE INDEX` for the verified vector and scalar types; - `CREATE OR REPLACE INDEX` mapped to the existing full-build `replace=true` behavior; - `SHOW INDEX`, using the existing list/count metadata and grouping physical entries by logical name where necessary; - `DROP INDEX`; - restricted REST behavior based only on the
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5208309962 Thank you @Gabriel39 for the patient and precise guidance across both review rounds — each round has made this design substantially stronger. Below is the v3 full revision, addressing every round-2 point directly and incorporating several self-identified corrections. # Revised Design Proposal v3 — Asynchronous Lance Index Lifecycle > Draft for apache/doris#66497, prepared 2026-08-07. Incorporates Gabriel39's round-2 review (2026-08-06 13:15 UTC) + self-identified errata. This revision has passed three independent adversarial reviews (research-consistency, review-coverage, source-level fact-check); every load-bearing claim carries pinned-source evidence, and the distributed build path is verified by an end-to-end PoC against lance-core 9.1.0-beta.3 (fragment-scoped uncommitted build → coordinator commit → k-NN consumption). Post as a full standalone revision. This revision incorporates the second-round review from @Gabriel39: REST is now an explicit capability matrix, SHOW/CANCEL BUILD INDEX get a concrete external-job integration design, reconciliation requires job-specific provenance, DROP uses the same durable job record, and the distributed segment contract is specified. It also contains several self-identified corrections to v2. /cc @Gabriel39 @zhangstar333 ### Major changes from v2 1. **REST capability matrix** replaces uniform-behavior language: a per-operation/per-property support table; unsupported combinations are rejected at analysis time where statically known, otherwise surfaced as a clear capability error. 2. **Job-specific provenance**: every mutation job persists a client-generated `operation_id` before dispatch. Reconciliation requires job-specific evidence — for Directory, the expected segment UUID set plus the `operation_id` recorded in the Lance transaction properties at commit time; for REST, the server-returned `transaction_id` (see the honest boundary in §10). A matching postcondition alone is never accepted as success — including for DROP. 3. **Concurrency semantics stated in both layers**: the Lance format documentation defines same-name concurrent CreateIndex as last-committer-wins; the pinned implementation enforces it via a retryable commit conflict + rebase on the first attempt. Either way, a name/definition match never proves attribution. 4. **DROP is durable**: it uses the same LanceIndexJob record (it usually transitions quickly), closing the post-commit failover gap — and its reconciliation follows the same provenance rules. 5. **Distributed segment contract**: a pinned lance-c/Java-SDK compatibility requirement, a worker-result validation checklist, full-final-segment-set commits for incremental BUILD, an explicit shared IVF model strategy, a defined BE→FE segment-metadata schema, and no credentials in the edit log. 6. **SHOW/CANCEL BUILD INDEX integration design** for external jobs: registry, proc source, catalog qualification, filtering, privileges, output schema, and state mapping (current implementations verified internal-only on both master and branch-4.1). 7. **Metadata**: SHOW INDEX.Index_type reports the physical type (IVF_PQ); REST SHOW fields are capability-dependent. 8. **Type matrix narrowed**: vector element types are FLOAT16/FLOAT32 only (see corrections). 9. **Execution model locked**: Directory = BE via extended lance-c; no external build service. The FE-side build remains an experimental, disabled-by-default development fallback using the Java SDK's existing uncommitted-build primitives. 10. **Provenance-carrying commits use the general Transaction path**: the one-shot `Dataset.createIndex` / `commitExistingIndexSegments` / `dropIndex` APIs do not carry transaction properties in the pinned SDK; commits that must record `operation_id` go through `Transaction.Builder.transactionProperties` + `operation.CreateIndex` + `CommitBuilder` (verified end-to-end through JNI in the pinned sources). ### Corrections to v2 (self-identified) - **Concurrent same-name CreateIndex — both layers, precisely.** The [Lance format transaction documentation](https://lance.org/format/table/transaction/) states: two concurrent CreateIndex operations are allowed; with the same name, "the second operation will win and replace the first" (`docs/src/format/table/transaction.md:172-173`). The pinned implementation reaches that semantics via conflict + rebase: `check_create_index_txn` returns a retryable conflict for a same-name regular index on the first attempt (`rust/lance/src/io/commit/conflict_resolver.rs:577-617`), and the Java `CommitBuilder.maxRetries` defaults to 0, so without retries the caller observes the conflict error. v2 saw only the implementation layer; the design must assume the doc layer for external writers (retried/
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5205105189 Thanks for the detailed revision. V2 addresses most of the architectural concerns from the first review: synchronous FE construction is no longer the production path, the neutral command model is clearer, BUILD INDEX is separated from full replacement, logical and physical metadata are separated, and the recovery states are substantially better defined. I think the Directory Catalog direction is now viable, but I do not think the design is implementation-ready yet. The following issues still need to be resolved. ### 1. The REST design currently promises more than the pinned Namespace API can express Doris pins Lance Namespace 0.7.7. Its `CreateTableIndexRequest` supports the column, index type, name, distance type, and FTS-related fields, but it does not carry `num_partitions`, `num_sub_vectors`, `zone_size`, replace semantics, or an uncovered-fragment/incremental-build request: https://github.com/lancedb/lance-catalog/blob/v0.7.7/java/lance-namespace-apache-client/docs/CreateTableIndexRequest.md There is also no separate Namespace BUILD INDEX operation. In addition, `ListTableIndices` exposes only name, UUID, columns, and status, while index stats expose only type, distance, indexed/unindexed row counts, and physical index count. That is not sufficient to verify that an existing REST index has the complete normalized definition required by the proposed `IF NOT EXISTS` reconciliation. Please either: - make the required Namespace specification/API changes an explicit prerequisite; or - define a Phase-1 REST capability matrix and reject unsupported replace, build, and property combinations instead of promising uniform behavior. “Advertised capabilities” should also be made concrete. In the pinned Java interface, optional operations are primarily represented by typed `UnsupportedOperationException`; there is no general capability-advertisement API. ### 2. Reusing SHOW/CANCEL BUILD INDEX requires a real external-job integration design The current `CancelBuildIndexCommand` explicitly rejects external catalogs and eventually looks up an internal `OlapTable` and `SchemaChangeHandler.IndexChangeJob`: https://github.com/apache/doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/CancelBuildIndexCommand.java Similarly, `SHOW BUILD INDEX` reads the internal `/jobs//build_index` proc data and has internal-index-specific columns such as `PartitionName` and `AlterInvertedIndexes`: https://github.com/apache/doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowBuildIndexCommand.java Extending the SQL vocabulary is reasonable, but the design should specify the common job registry/proc source, catalog qualification, filtering, privilege checks, output schema, and state mapping for internal and Lance jobs. Otherwise the SQL examples in this proposal cannot work through the existing implementation. ### 3. REST reconciliation needs job-specific provenance, not only a matching postcondition The remote `transaction_id` is optional. If the service commits the operation but FE fails before persisting the returned transaction ID, a later “same name and matching definition” observation does not prove that this Doris job performed the commit; an external writer may have produced the same postcondition. This is particularly important because Lance allows concurrent CreateIndex transactions. For two operations on the same logical name, the later operation can replace the earlier one rather than returning the retryable conflict assumed by this proposal: https://lance.org/format/table/transaction/ Please persist a client operation ID before dispatch and propagate it through Namespace transaction properties or another queryable server-side idempotency/provenance mechanism. For Directory builds, expected segment UUIDs can provide this proof. Without job-specific evidence, the REST job should remain `OUTCOME_UNKNOWN` instead of being marked successful merely because the logical definition matches. ### 4. DROP also needs a durable operation record The proposal creates durable jobs for create, replace, and build, but allows DROP to complete synchronously while still promising failover-safe post-commit reconciliation and refresh. If DROP commits externally and FE fails before refresh/edit-log completion, there is no durable state from which to resume. DROP should therefore use the same durable Lance job/operation record, even if it normally moves through the states very quickly. ### 5. Please make the distributed segment contract explicit The pinned Java SDK already exposes `mergeExistingIndexSegments` and
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5204040291 ## Revised Design Proposal v2 — Asynchronous Lance Index Lifecycle This revision supersedes the previous proposal and incorporates the review feedback from @Gabriel39. ### Major changes from v1 - Heavy index creation, replacement, and incremental maintenance no longer run synchronously in FE. Production execution is modeled as a durable asynchronous job, with FE orchestration and BE or external-service execution. - The SQL path introduces a neutral parsed index specification. Doris resolves the target table and catalog before choosing internal-index serialization or Lance-specific validation and dispatch. - Vector syntax reuses the Doris category-level surface: **USING ANN** with **index_type=IVF_PQ**. - **BUILD INDEX** is added to the Phase-1 lifecycle as the incremental operation for newly appended or otherwise uncovered fragments. - **SHOW INDEX** returns only bounded logical summaries. Physical UUIDs, dataset versions, and fragment membership are exposed through structured, filterable metadata surfaces. - REST behavior is capability-based. REST mutations go through Lance Namespace APIs and never bypass the metadata service with vended credentials. - The initial type set is narrowed to IVF_PQ, BTREE, and BITMAP. Lance INVERTED/FTS and IVF_FLAT are deferred until the Doris query path is verified end to end. - Index-name resolution, post-commit reconciliation, metadata refresh retries, and local-filesystem restrictions are defined explicitly. ## 1. Goals and Phase-1 boundary Phase 1 provides a production-oriented lifecycle for user-visible Lance indexes: - create IVF_PQ vector indexes and BTREE/BITMAP scalar indexes; - inspect logical index definitions, aggregate coverage, and logical index count; - inspect physical index segments through a separate metadata surface; - incrementally index newly appended or uncovered fragments; - atomically replace/rebuild an index by logical name; - drop an index by logical name; - provide durable status, cancellation, concurrency limits, retry/reconciliation, and metadata refresh; - support Directory Catalog operations and capability-based REST Namespace operations. Lance dataset manifests or the Namespace service remain the authoritative index metadata. Doris does not persist a second copy of Lance index definitions in its internal table/index metadata. Doris persists only its own asynchronous job state and the information needed to reconcile an external operation. Historical-version tables remain read-only for index mutation. Inspection uses the selected snapshot; mutation targets only the latest writable table state. An FE-side synchronous native build is not part of the production implementation. If retained temporarily for development, it must be experimental, disabled by default, and protected by strict dataset-size, fragment-count, and concurrency limits. It does not satisfy Phase-1 completion. ## 2. SQL surface ### Vector creation ~~~sql CREATE INDEX [IF NOT EXISTS] idx ON lance_ctl.db.tbl (vec_col) USING ANN PROPERTIES ( "index_type" = "IVF_PQ", "metric" = "l2", "num_partitions" = "256", "num_sub_vectors" = "16" ); ~~~ ### Scalar creation ~~~sql CREATE INDEX idx ON lance_ctl.db.tbl (scalar_col) USING BTREE PROPERTIES ( "zone_size" = "4096" ); CREATE INDEX idx_bitmap ON lance_ctl.db.tbl (category_col) USING BITMAP; ~~~ ### Full replacement and incremental maintenance ~~~sql CREATE OR REPLACE INDEX idx ON lance_ctl.db.tbl (vec_col) USING ANN PROPERTIES ( "index_type" = "IVF_PQ", "metric" = "l2", "num_partitions" = "256", "num_sub_vectors" = "16" ); BUILD INDEX idx ON lance_ctl.db.tbl; ~~~ **CREATE OR REPLACE INDEX** performs a full rebuild and atomically replaces the logical index. **BUILD INDEX** indexes only fragments not covered by the current logical index and maps to the Lance incremental optimization path. ### Inspection, status, cancellation, and drop ~~~sql SHOW INDEX FROM lance_ctl.db.tbl; SHOW BUILD INDEX FROM db WHERE TableName = "tbl"; CANCEL BUILD INDEX ON lance_ctl.db.tbl (job_id); DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl; ~~~ The existing Doris **SHOW BUILD INDEX** and **CANCEL BUILD INDEX** surfaces are extended to Lance external-index jobs instead of introducing a second job-control vocabulary. ### Statement semantics - **USING** is required for Lance tables. - **IF NOT EXISTS** and **OR REPLACE** are mutually exclusive. - Plain **CREATE INDEX** fails if the logical name exists. - **IF NOT EXISTS** verifies the visible logical
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5199781913 Thanks for the detailed proposal. The source-of-truth model, logical/physical index distinction, version semantics, and post-commit metadata refresh direction are sound. After comparing the proposal with the current Doris command path, the pinned Lance SDK/Namespace API, and the Lance Spark, StarRocks, ClickHouse, DuckDB, and Trino designs, I think the following points should be resolved before implementation. ## Blocking concerns ### 1. A synchronous native build in FE is not a safe production execution model The proposal already lists the essential problems: index creation performs data scans and native CPU/memory/object-store I/O in FE, while the pinned API has no reliable cancellation/progress mechanism and FE failover can leave the result ambiguous. This also bypasses Doris workload management and allows a user with ALTER to trigger unbounded native work in the control plane. The closest Lance integration distributes fragment builds to Spark executors and leaves the driver to coordinate/finalize the commit. ClickHouse similarly separates index metadata operations from the heavy MATERIALIZE INDEX mutation. - https://lance.org/integrations/spark/operations/ddl/create-index/ - https://clickhouse.com/docs/reference/statements/alter/skipping-index My recommendation is an asynchronous job with FE orchestration and BE or external-service execution, including status, cancellation, concurrency limits, and retry/reconciliation. If FE-side execution must exist temporarily, it should be explicitly experimental, disabled by default, and protected by dataset-size/fragment/concurrency limits. It should not be presented as the final production lifecycle implementation. ### 2. Reusing the current Doris index command chain requires a larger refactor than the proposal describes The current parser only accepts NGRAM_BF, INVERTED, and ANN; it does not accept IVF_PQ, BTREE, BITMAP, or OR REPLACE: https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4#L237-L240 More importantly, CreateIndexOp.validate() always translates the definition into the persisted internal catalog.Index, while AlterTableCommand currently rejects CreateIndexOp for external tables: https://github.com/apache/doris/blob/e3289c1a5df7558cb8e63d80379d4edebf9c498c/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateIndexOp.java#L73-L89 That conflicts with the stated goal of keeping Lance index DTOs separate from Doris internal index serialization. The design should explicitly introduce a neutral parsed index definition, resolve the target table/catalog first, and then choose internal-index validation/serialization or Lance-specific validation/dispatch. Adding only a LanceIndexOperator after the existing validation path will not be sufficient. For vector syntax, I would prefer reusing the Doris category-level surface: ~~~sql CREATE INDEX idx ON lance_ctl.db.tbl (vec_col) USING ANN PROPERTIES ( "index_type" = "IVF_PQ", ... ); ~~~ This also follows the StarRocks pattern of USING VECTOR plus an index_type property and avoids adding a grammar keyword for every Lance algorithm. ## Other required revisions ### 3. Incremental maintenance is necessary for a usable external index lifecycle External writers can append fragments immediately after an index is created. Coverage will then decay. CREATE OR REPLACE INDEX performs an expensive full rebuild and is not an adequate maintenance operation. The pinned Java SDK exposes Dataset.optimizeIndices(), and Lance documents it as the incremental path for indexing newly appended fragments. I suggest adding a separate operation such as: ~~~sql BUILD INDEX idx ON lance_ctl.db.tbl; -- or ALTER TABLE lance_ctl.db.tbl OPTIMIZE INDEX idx; ~~~ The tests must also prove that queries include unindexed fragments correctly. Otherwise partial coverage becomes a correctness issue, not merely a performance issue. ### 4. Do not put unbounded physical segment metadata in SHOW INDEX.Properties A JSON segments array can become extremely large for a fragmented or distributed index. It can create FE memory, MySQL result packet, and client usability problems. Lance Spark intentionally returns only logical summaries from SHOW INDEXES and excludes per-segment metadata: https://lance.org/integrations/spark/operations/ddl/show-indexes/ I recommend keeping the existing 13-column SHOW INDEX surface for logical name, columns, type, and aggregate coverage only. UUIDs, dataset versions, and fragment sets should be exposed through a structured and filterable
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497:
URL: https://github.com/apache/doris/issues/66497#issuecomment-5199612350
here is my design proposal for this issue. It builds on #65730 (merged), the
Lance Java SDK / lance-c APIs already
pinned in the repo, and two precedent studies: how Doris implements
mutations for other external catalogs (Iceberg/Hive/Paimon), and lance-spark's
index
DDL (the only landed cross-engine precedent). Open questions are numbered
at the end. /cc @Gabriel39 @zhangstar333
## 1. SQL surface
Reuse the existing index DDL grammar (`createIndex` / `dropIndex` /
`showIndex` in `DorisParser.g4`), extended to accept Lance external tables:
```sql
-- Vector index
CREATE INDEX [IF NOT EXISTS] idx ON lance_ctl.db.tbl (vec_col)
USING IVF_PQ PROPERTIES ("nlist" = "256", "metric" = "l2");
-- Scalar indexes
CREATE INDEX idx ON lance_ctl.db.tbl (col) USING BTREE; -- or BITMAP /
INVERTED
-- Replace / rebuild with the same name (atomic swap via Lance
replace=true)
CREATE OR REPLACE INDEX idx ON lance_ctl.db.tbl (vec_col) USING IVF_PQ;
-- Inspect
SHOW INDEX FROM lance_ctl.db.tbl;
DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;
```
Notes:
- `USING`: extend the enum with Lance concrete types. Vector: `IVF_FLAT |
IVF_SQ | IVF_PQ | IVF_HNSW_FLAT | IVF_HNSW_SQ | IVF_HNSW_PQ`; scalar: `BTREE |
BITMAP | INVERTED` (matching lance-c's `LanceVectorIndexType` /
`LanceScalarIndexType` and the #66340 matrix). PROPERTIES map to Lance
`IndexParams`
(`nlist`, `metric`, ...).
- Same-name semantics follow the Lance SDK's safe default: `CREATE INDEX`
fails if the name already exists; `IF NOT EXISTS` no-ops; `CREATE OR REPLACE
INDEX` maps to Lance `replace=true`, an atomic swap in a single commit.
This keeps lance-spark's replace capability in Doris's explicit style. (Q3)
- `SHOW INDEX` output columns for Lance tables: `Name | Column | Type |
UUID | DatasetVersion | IndexedFragments | UnindexedFragments` — covering the
issue's required "names, UUIDs, columns, types, dataset versions" plus
lance-spark-style coverage stats (backed by `listIndexes` /
`lance_dataset_index_list_json`).
- The number of indexes on a table is the row count of `SHOW INDEX`
(backed by `listIndexes().size()` / `lance_dataset_index_count`); no separate
statement is proposed.
- Index names are case-insensitive and stored lowercase, following Lance /
lance-spark convention.
- Single-column indexes only in the initial implementation.
## 2. Where index operations execute (Q1)
Precedent study: every mutable external catalog in Doris performs
metadata-level mutations inside the FE via an embedded SDK —
`IcebergMetadataOps`,
`HiveMetadataOps`, `PaimonMetadataOps`, `MaxComputeMetadataOps` all
implement the shared `ExternalMetadataOps` interface; a null `metadataOps` means
read-only (JDBC is the read-only example). BE never touches the external
system's metadata. `LanceExternalCatalog` currently has no metadataOps (=
read-only), so the conventional extension point is FE-side index methods
calling the Lance Java SDK.
**Proposal for the initial implementation — FE via Lance Java SDK (JNI).**
`createIndex` / `dropIndex` / `listIndexes` all exist in the pinned lance-core
9.1.0-beta.3. No thrift / RPC / BE changes; follows the
`ExternalMetadataOps` convention.
One honest caveat: unlike a pure metadata mutation, Lance index creation
bundles data-plane compute (column scan + IVF training) into a single monolithic
Rust call, so with this approach that compute runs in native threads
inside the FE process. Lance does expose the primitives for a distributed build
—
per-fragment segment building without commit, then a coordinator commit
(`create_index_uncommitted` / `commitExistingIndexSegments`; lance-spark's
`num_segments` mode uses this). A follow-up could offload building to BE
(the pinned lance-c v0.1.2 already ships the matching Phase-2 C APIs) while FE
keeps initiation and commit, mirroring the Iceberg INSERT pattern. The
initial implementation stays FE-only behind a `LanceIndexOperator` interface so
the later swap does not change SQL semantics.
## 3. Synchronous execution (Q2)
Proposal: synchronous DDL for the initial implementation. Precedents line
up: Doris's internal light `CREATE INDEX` is a synchronous metadata change; the
async `BUILD INDEX` framework (`IndexChangeJob` per partition) is bound to
Doris's tablet / replica / BE-agent model and does not transfer to Lance;
lance-spark's `CREATE INDEX` is synchronous as well.
Semantics: killing the statement before commit leaves nothing persisted
(Lance commits atomically at the end), so interruption is safe. Documentation
will note that build time scales with dataset size.
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497:
URL: https://github.com/apache/doris/issues/66497#issuecomment-5194018594
here is my design proposal for this issue. It builds on #65730 (merged), the
Lance Java SDK / lance-c APIs already
pinned in the repo, and two precedent studies: how Doris implements
mutations for other external catalogs (Iceberg/Hive/Paimon), and lance-spark's
index
DDL (the only landed cross-engine precedent). Open questions are numbered
at the end. /cc @Gabriel39 @zhangstar333
## 1. SQL surface
Reuse the existing index DDL grammar (`createIndex` / `dropIndex` /
`showIndex` in `DorisParser.g4`), extended to accept Lance external tables:
```sql
-- Vector index
CREATE INDEX [IF NOT EXISTS] idx ON lance_ctl.db.tbl (vec_col)
USING IVF_PQ PROPERTIES ("nlist" = "256", "metric" = "l2");
-- Scalar indexes
CREATE INDEX idx ON lance_ctl.db.tbl (col) USING BTREE; -- or BITMAP /
INVERTED
-- Replace / rebuild with the same name (atomic swap via Lance
replace=true)
CREATE OR REPLACE INDEX idx ON lance_ctl.db.tbl (vec_col) USING IVF_PQ;
-- Inspect
SHOW INDEX FROM lance_ctl.db.tbl;
DROP INDEX [IF EXISTS] idx ON lance_ctl.db.tbl;
```
Notes:
- `USING`: extend the enum with Lance concrete types. Vector: `IVF_FLAT |
IVF_SQ | IVF_PQ | IVF_HNSW_FLAT | IVF_HNSW_SQ | IVF_HNSW_PQ`; scalar: `BTREE |
BITMAP | INVERTED` (matching lance-c's `LanceVectorIndexType` /
`LanceScalarIndexType` and the #66340 matrix). PROPERTIES map to Lance
`IndexParams`
(`nlist`, `metric`, ...).
- Same-name semantics follow the Lance SDK's safe default: `CREATE INDEX`
fails if the name already exists; `IF NOT EXISTS` no-ops; `CREATE OR REPLACE
INDEX` maps to Lance `replace=true`, an atomic swap in a single commit.
This keeps lance-spark's replace capability in Doris's explicit style. (Q3)
- `SHOW INDEX` output columns for Lance tables: `Name | Column | Type |
UUID | DatasetVersion | IndexedFragments | UnindexedFragments` — covering the
issue's required "names, UUIDs, columns, types, dataset versions" plus
lance-spark-style coverage stats (backed by `listIndexes` /
`lance_dataset_index_list_json`).
- The number of indexes on a table is the row count of `SHOW INDEX`
(backed by `listIndexes().size()` / `lance_dataset_index_count`); no separate
statement is proposed.
- Index names are case-insensitive and stored lowercase, following Lance /
lance-spark convention.
- Single-column indexes only in the initial implementation.
## 2. Where index operations execute (Q1)
Precedent study: every mutable external catalog in Doris performs
metadata-level mutations inside the FE via an embedded SDK —
`IcebergMetadataOps`,
`HiveMetadataOps`, `PaimonMetadataOps`, `MaxComputeMetadataOps` all
implement the shared `ExternalMetadataOps` interface; a null `metadataOps` means
read-only (JDBC is the read-only example). BE never touches the external
system's metadata. `LanceExternalCatalog` currently has no metadataOps (=
read-only), so the conventional extension point is FE-side index methods
calling the Lance Java SDK.
**Proposal for the initial implementation — FE via Lance Java SDK (JNI).**
`createIndex` / `dropIndex` / `listIndexes` all exist in the pinned lance-core
9.1.0-beta.3. No thrift / RPC / BE changes; follows the
`ExternalMetadataOps` convention.
One honest caveat: unlike a pure metadata mutation, Lance index creation
bundles data-plane compute (column scan + IVF training) into a single monolithic
Rust call, so with this approach that compute runs in native threads
inside the FE process. Lance does expose the primitives for a distributed build
—
per-fragment segment building without commit, then a coordinator commit
(`create_index_uncommitted` / `commitExistingIndexSegments`; lance-spark's
`num_segments` mode uses this). A follow-up could offload building to BE
(the pinned lance-c v0.1.2 already ships the matching Phase-2 C APIs) while FE
keeps initiation and commit, mirroring the Iceberg INSERT pattern. The
initial implementation stays FE-only behind a `LanceIndexOperator` interface so
the later swap does not change SQL semantics.
## 3. Synchronous execution (Q2)
Proposal: synchronous DDL for the initial implementation. Precedents line
up: Doris's internal light `CREATE INDEX` is a synchronous metadata change; the
async `BUILD INDEX` framework (`IndexChangeJob` per partition) is bound to
Doris's tablet / replica / BE-agent model and does not transfer to Lance;
lance-spark's `CREATE INDEX` is synchronous as well.
Semantics: killing the statement before commit leaves nothing persisted
(Lance commits atomically at the end), so interruption is safe. Documentation
will note that build time scales with dataset size.
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
u70b3 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5192741223 Thanks @Gabriel39 — I'd be glad to take ownership of this issue. Please assign it to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] [Feature](lance) Add vector and scalar index lifecycle management [doris]
Gabriel39 commented on issue #66497: URL: https://github.com/apache/doris/issues/66497#issuecomment-5190988080 @u70b3, would you be available to take ownership of this issue and implement it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
