This is an automated email from the ASF dual-hosted git repository.
zhangstar333 pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new c28259d339c branch-4.1: [feature](lance) Add statically validated
Lance index DDL surface (#67201)
c28259d339c is described below
commit c28259d339ca8f57a977d4d6d63a8a513933b545
Author: kid <[email protected]>
AuthorDate: Wed Sep 9 20:29:39 2026 +0800
branch-4.1: [feature](lance) Add statically validated Lance index DDL
surface (#67201)
### What problem does this PR solve?
Issue Number: #66497
Related PR: #66637 (merged), #66671 (open — independent; this PR shares
no files with it)
Problem Summary:
This is the first sub-PR (PR3A) of delivery slice 3 of the v5.1 design
([final 4.2
contract](https://github.com/apache/doris/issues/66497#issuecomment-5301314544),
scope confirmed in [this
review](https://github.com/apache/doris/issues/66497#issuecomment-5301637401)):
the Lance index DDL surface with static validation and target-aware
routing, in reject-all mode. Slice 3's remaining pieces — durable jobs,
same-name fences, unresolved quotas, replay, job SQL, and dispatch with
fake-worker fault tests — land as follow-up PRs; this PR contains no job
admission, no index-metadata reads, and no enablement gate.
What this PR adds:
- Top-level `CREATE [OR REPLACE] INDEX ... USING ANN/BTREE/BITMAP` and
`DROP INDEX [IF EXISTS]` parsing for Lance Directory catalog tables. `OR
REPLACE` is mutually exclusive with `IF NOT EXISTS`; `BTREE` is a new
non-reserved keyword. The `indexDef` rule used by `CREATE TABLE` and
`ALTER TABLE ... ADD INDEX` is unchanged, and `ALTER TABLE ... ADD/DROP
INDEX` remains unsupported per Section 2.1, so previously parseable
internal SQL behaves byte-identically.
- `IndexDefinition` carries `orReplace` and the Lance-only type name
without extending the persisted internal `IndexDef.IndexType` enum
(Section 4.4). New `validate()` guards reject Lance-only syntax on
internal tables and fire only for SQL that was a syntax error before
this PR.
- `LanceIndexMutationValidator` applies the FE static bounds of Section
2.4: the ANN/`IVF_PQ` property matrix (required `index_type=IVF_PQ`,
`metric` in `l2/cosine/dot`, positive required
`num_partitions`/`num_sub_vectors`, fixed `num_bits=8`, unknown and
case-variant duplicate properties rejected), the BTREE/BITMAP
column-type sets (uint64/`LARGEINT` included as integral for both),
exactly one non-null column, and a bounded index name. Arrow-level
revalidation (fixed-size-list, dimension, float16-vs-float32, subvector
divisibility) is deferred to the isolated worker per Sections 2.4/4.2
because `LanceTypeConverter` erases those facts.
- Target-aware routing in `AlterTableCommand.validate` (after the
existing table `ALTER` privilege check and catalog resolution, before
any op validation): top-level CREATE/DROP INDEX on a Lance table is
statically validated and then rejected with a typed per-op message
before any `Env.getNextId()` allocation. Lance REST catalogs receive a
fixed unsupported error mirroring PR1's `SHOW INDEX` stance. Internal
tables never enter this branch.
Explicitly not in this PR: durable job records, fences, quotas, and
replay (PR3B); authoritative `IF` semantics, admission, and job SQL
(Section 2.2, PR3C); dispatch and the isolated worker (PR3D/slice 4);
`FORCE_RELEASE` (PR3E); the mutation enablement configuration (Section
9.7, arrives with admission). Property-value normalization is
validation-local; persisting normalized values belongs to admission.
`SHOW INDEX` behavior from PR1 and the PR2 inspection surface are
untouched.
Known accepted behavior change: the new `BTREE` keyword token makes
stored-procedure bodies that use `btree` as a bare identifier (e.g.
`CLOSE btree`) fail PL parsing, the same hazard class as the
pre-existing `ANN` token; all other identifier positions are covered by
`nonReserved`.
### Release note
Add the `CREATE [OR REPLACE] INDEX ... USING ANN/BTREE/BITMAP` and `DROP
INDEX` SQL surface for Lance Directory catalog tables with static
validation; statements are currently rejected with a typed not-supported
error while the mutation lifecycle is staged.
Co-authored-by: u70b3 <[email protected]>
---
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 1 +
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 5 +-
.../java/org/apache/doris/common/ErrorCode.java | 5 +
.../lance/LanceIndexMutationValidator.java | 230 +++++++++++
.../doris/nereids/parser/LogicalPlanBuilder.java | 10 +-
.../trees/plans/commands/AlterTableCommand.java | 51 ++-
.../trees/plans/commands/info/IndexDefinition.java | 49 ++-
.../lance/LanceIndexMutationValidatorTest.java | 459 +++++++++++++++++++++
.../nereids/parser/CreateIndexParserTest.java | 187 +++++++++
.../commands/AlterTableCommandLanceIndexTest.java | 375 +++++++++++++++++
.../lance/test_lance_index_ddl.groovy | 176 ++++++++
11 files changed, 1541 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
index a56811806ff..f54117a4596 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
@@ -113,6 +113,7 @@ BOTH: 'BOTH';
BRANCH: 'BRANCH';
BRIEF: 'BRIEF';
BROKER: 'BROKER';
+BTREE: 'BTREE';
BUCKETS: 'BUCKETS';
BUILD: 'BUILD';
BUILTIN: 'BUILTIN';
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index cb84945ce76..4d97fdab5b8 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -234,9 +234,9 @@ supportedCreateStatement
name=identifier properties=propertyClause?
#createStoragePolicy
| BUILD INDEX (name=identifier)? ON tableName=multipartIdentifier
partitionSpec?
#buildIndex
- | CREATE INDEX (IF NOT EXISTS)? name=identifier
+ | CREATE (OR REPLACE)? INDEX (IF NOT EXISTS)? name=identifier
ON tableName=multipartIdentifier identifierList
- (USING (NGRAM_BF | INVERTED | ANN))?
+ (USING (NGRAM_BF | INVERTED | ANN | BTREE | BITMAP))?
properties=propertyClause? (COMMENT STRING_LITERAL)?
#createIndex
| CREATE WORKLOAD POLICY (IF NOT EXISTS)? name=identifierOrText
(CONDITIONS LEFT_PAREN workloadPolicyConditions RIGHT_PAREN)?
@@ -2046,6 +2046,7 @@ nonReserved
| BRANCH
| BRIEF
| BROKER
+ | BTREE
| BUCKETS
| BUILD
| BUILTIN
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
index 8f5fe32bb30..92e6cd82a58 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
@@ -1234,6 +1234,11 @@ public enum ErrorCode {
ERR_NO_CLUSTER_ERROR(5099, new byte[]{'4', '2', '0', '0', '0'}, "No
compute group (cloud cluster) selected"),
+ ERR_LANCE_INDEX_INVALID(5100, new byte[]{'4', '2', '0', '0', '0'}, "%s"),
+
+ ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED(5101, new byte[]{'4', '2', '0',
'0', '0'},
+ "%s is not supported for Lance %s"),
+
ERR_NOT_CLOUD_MODE(6000, new byte[]{'4', '2', '0', '0', '0'},
"Command only support in cloud mode.");
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMutationValidator.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMutationValidator.java
new file mode 100644
index 00000000000..bce0ca061a7
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMutationValidator.java
@@ -0,0 +1,230 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lance;
+
+import org.apache.doris.analysis.IndexDef;
+import org.apache.doris.catalog.ArrayType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.nereids.trees.plans.commands.info.IndexDefinition;
+
+import com.google.common.collect.ImmutableSet;
+
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Static FE-side validation for CREATE/DROP INDEX statements targeting Lance
catalog tables.
+ *
+ * <p>These are the FE static bounds of the Lance index lifecycle design
(section 2.4). Exact
+ * Arrow-level revalidation — fixed-size-list-ness, vector dimension, float16
vs float32, and
+ * num_sub_vectors divisibility — is deferred to the isolated index-build
worker per design
+ * sections 2.4/4.2, because {@link LanceTypeConverter} erases those facts
when mapping Arrow
+ * types to Doris types (LanceTypeConverter.java:101-106).
+ */
+public final class LanceIndexMutationValidator {
+ private static final int MAX_INDEX_NAME_BYTES = 64;
+ private static final Set<String> ANN_PROPERTY_KEYS = ImmutableSet.of(
+ "index_type", "metric", "num_partitions", "num_sub_vectors",
"num_bits");
+ private static final Set<String> ANN_METRICS = ImmutableSet.of("l2",
"cosine", "dot");
+
+ private LanceIndexMutationValidator() {
+ }
+
+ /**
+ * Validates a top-level CREATE [OR REPLACE] INDEX statement against a
Lance catalog table.
+ * Returns normally when the statement is statically valid; the caller
then decides whether
+ * the statement is admitted.
+ */
+ public static void validateCreateIndex(LanceExternalCatalog catalog,
LanceExternalTable table,
+ IndexDefinition def) throws AnalysisException {
+ validateCreateIndexCatalog(catalog, def);
+ String lanceType = def.getLanceIndexType();
+ if (lanceType == null) {
+ lanceType = def.getIndexType() == IndexDef.IndexType.ANN ? "ANN" :
null;
+ }
+ if (lanceType == null) {
+ rejectInvalidDefinition("Lance catalog tables only support USING
ANN, BTREE, or BITMAP");
+ }
+ if (def.getCols() == null || def.getCols().size() != 1) {
+ rejectInvalidDefinition("Lance index must be built on exactly one
column");
+ }
+ validateIndexName(def.getIndexName());
+ String columnName = def.getCols().get(0);
+ Column column = table.getColumn(columnName);
+ if (column == null) {
+ rejectInvalidDefinition("Index column '" + columnName + "' does
not exist");
+ }
+ if (column.isAllowNull()) {
+ rejectInvalidDefinition(lanceType + " index must be built on a
column that is not nullable");
+ }
+ switch (lanceType) {
+ case "ANN":
+ validateAnnIndex(column, def.getProperties());
+ break;
+ case "BTREE":
+ validateBtreeIndex(column, def.getProperties());
+ break;
+ case "BITMAP":
+ validateBitmapIndex(column, def.getProperties());
+ break;
+ default:
+ rejectInvalidDefinition("Lance catalog tables only support
USING ANN, BTREE, or BITMAP");
+ }
+ }
+
+ /**
+ * Rejects CREATE INDEX against a Lance REST catalog without resolving its
database or table.
+ */
+ public static void validateCreateIndexCatalog(LanceExternalCatalog
catalog, IndexDefinition def)
+ throws AnalysisException {
+ if (catalog.isRestCatalogConfigured()) {
+ rejectUnsupportedOperation(def.isOrReplace() ? "CREATE OR REPLACE
INDEX" : "CREATE INDEX",
+ "REST catalogs");
+ }
+ }
+
+ /**
+ * Validates a top-level DROP INDEX statement targeting a Lance catalog
table. The REST
+ * rejection keeps failing fast; Directory catalogs then get the same
index-name bounds as
+ * the CREATE path.
+ */
+ public static void validateDropIndex(LanceExternalCatalog catalog, String
indexName)
+ throws AnalysisException {
+ if (catalog.isRestCatalogConfigured()) {
+ rejectUnsupportedOperation("DROP INDEX", "REST catalogs");
+ }
+ validateIndexName(indexName);
+ }
+
+ /**
+ * Shared Lance index-name bounds for the CREATE and DROP paths: the name
becomes the durable
+ * logical identity that an admitted job and its same-name fence key are
built on, so
+ * null/empty names are rejected here instead of being masked as
unsupported operations.
+ */
+ private static void validateIndexName(String indexName) throws
AnalysisException {
+ if (indexName == null || indexName.isEmpty()) {
+ rejectInvalidDefinition("index name cannot be empty");
+ }
+ if (indexName.getBytes(StandardCharsets.UTF_8).length >
MAX_INDEX_NAME_BYTES) {
+ rejectInvalidDefinition("index name too long, the index name
length at most is 64.");
+ }
+ }
+
+ private static void validateAnnIndex(Column column, Map<String, String>
properties)
+ throws AnalysisException {
+ Type columnType = column.getType();
+ if (!(columnType instanceof ArrayType)) {
+ rejectInvalidDefinition("ANN index column must be array type");
+ }
+ Type itemType = ((ArrayType) columnType).getItemType();
+ if (!itemType.isScalarType(PrimitiveType.FLOAT)) {
+ rejectInvalidDefinition("ANN index column item type must be float
type");
+ }
+ // Keys match case-insensitively; the normalized view below is
validation-local only.
+ // Persisting normalized keys/values into the admitted job spec is
owned by admission.
+ Map<String, String> lowerCaseProperties = new HashMap<>();
+ for (Map.Entry<String, String> entry : properties.entrySet()) {
+ String key = entry.getKey().toLowerCase(Locale.ROOT);
+ if (!ANN_PROPERTY_KEYS.contains(key)) {
+ rejectInvalidDefinition("Unknown property '" + entry.getKey()
+ "' for Lance ANN index");
+ }
+ if (lowerCaseProperties.put(key, entry.getValue()) != null) {
+ rejectInvalidDefinition("Duplicate property '" +
entry.getKey() + "' for Lance ANN index");
+ }
+ }
+ String indexType = lowerCaseProperties.get("index_type");
+ if (indexType == null || !indexType.equalsIgnoreCase("IVF_PQ")) {
+ rejectInvalidDefinition("Lance ANN index requires property
\"index_type\" = \"IVF_PQ\"");
+ }
+ String metric = lowerCaseProperties.get("metric");
+ if (metric != null &&
!ANN_METRICS.contains(metric.toLowerCase(Locale.ROOT))) {
+ rejectInvalidDefinition("metric must be one of l2, cosine, dot");
+ }
+ checkRequiredPositiveInt(lowerCaseProperties, "num_partitions");
+ checkRequiredPositiveInt(lowerCaseProperties, "num_sub_vectors");
+ String numBits = lowerCaseProperties.get("num_bits");
+ if (numBits != null && parsePositiveInt(numBits) != 8) {
+ rejectInvalidDefinition("num_bits must be 8");
+ }
+ }
+
+ private static void validateBtreeIndex(Column column, Map<String, String>
properties)
+ throws AnalysisException {
+ if (!properties.isEmpty()) {
+ rejectInvalidDefinition("BTREE indexes do not support properties");
+ }
+ Type columnType = column.getType();
+ // LARGEINT (Arrow uint64) and TIMESTAMPTZ are included deliberately.
+ if (!columnType.isIntegerType() && !columnType.isLargeIntType()
+ && !columnType.isFloatingPointType() &&
!columnType.isDecimalV3()
+ && !columnType.isStringType() && !columnType.isDateV2()
+ && !columnType.isDatetimeV2() && !columnType.isTimeStampTz()) {
+ rejectInvalidDefinition("BTREE index does not support column type
" + columnType);
+ }
+ }
+
+ private static void validateBitmapIndex(Column column, Map<String, String>
properties)
+ throws AnalysisException {
+ if (!properties.isEmpty()) {
+ rejectInvalidDefinition("BITMAP indexes do not support
properties");
+ }
+ Type columnType = column.getType();
+ // LARGEINT (Arrow uint64) is integral, included here exactly as in
the BTREE matrix.
+ if (!columnType.isBoolean() && !columnType.isIntegerType() &&
!columnType.isLargeIntType()
+ && !columnType.isStringType() && !columnType.isDateV2()) {
+ rejectInvalidDefinition("BITMAP index does not support column type
" + columnType);
+ }
+ }
+
+ private static void checkRequiredPositiveInt(Map<String, String>
properties, String key)
+ throws AnalysisException {
+ String value = properties.get(key);
+ if (value == null || parsePositiveInt(value) <= 0) {
+ rejectInvalidDefinition(key + " must be a positive integer");
+ }
+ }
+
+ /**
+ * Rejects a Lance index operation with a stable client-visible error code.
+ */
+ public static void rejectUnsupportedOperation(String operation, String
target)
+ throws AnalysisException {
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ operation, target);
+ }
+
+ private static void rejectInvalidDefinition(String detail) throws
AnalysisException {
+ ErrorReport.reportAnalysisException(ErrorCode.ERR_LANCE_INDEX_INVALID,
detail);
+ }
+
+ private static int parsePositiveInt(String value) {
+ try {
+ return Integer.parseInt(value);
+ } catch (NumberFormatException e) {
+ return -1;
+ }
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 4659bf3b69e..5d485db0353 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -6488,6 +6488,10 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
public Command visitCreateIndex(CreateIndexContext ctx) {
String indexName = ctx.name.getText();
boolean ifNotExists = ctx.EXISTS() != null;
+ boolean orReplace = ctx.REPLACE() != null;
+ if (orReplace && ifNotExists) {
+ throw new AnalysisException("[OR REPLACE] and [IF NOT EXISTS]
cannot used at the same time");
+ }
TableNameInfo tableNameInfo = new
TableNameInfo(visitMultipartIdentifier(ctx.tableName));
List<String> indexCols = visitIdentifierList(ctx.identifierList());
Map<String, String> properties = ctx.properties != null
@@ -6500,10 +6504,14 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
indexType = "INVERTED";
} else if (ctx.ANN() != null) {
indexType = "ANN";
+ } else if (ctx.BTREE() != null) {
+ indexType = "BTREE";
+ } else if (ctx.BITMAP() != null) {
+ indexType = "BITMAP";
}
String comment = ctx.STRING_LITERAL() == null ? "" :
stripQuotes(ctx.STRING_LITERAL().getText());
IndexDefinition indexDefinition = new IndexDefinition(indexName,
ifNotExists, indexCols, indexType,
- properties, comment);
+ properties, comment, orReplace);
List<AlterTableOp> alterTableOps = Lists.newArrayList(new
CreateIndexOp(tableNameInfo,
indexDefinition, false));
return new AlterTableCommand(tableNameInfo, alterTableOps);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java
index 9b7eda21119..05d6a225d2c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java
@@ -37,7 +37,11 @@ import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.UserException;
import org.apache.doris.common.util.InternalDatabaseUtil;
import org.apache.doris.common.util.PropertyAnalyzer;
+import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.iceberg.IcebergExternalTable;
+import org.apache.doris.datasource.lance.LanceExternalCatalog;
+import org.apache.doris.datasource.lance.LanceExternalTable;
+import org.apache.doris.datasource.lance.LanceIndexMutationValidator;
import org.apache.doris.info.TableNameInfo;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.nereids.trees.plans.PlanType;
@@ -47,14 +51,17 @@ import
org.apache.doris.nereids.trees.plans.commands.info.AddPartitionFieldOp;
import org.apache.doris.nereids.trees.plans.commands.info.AddRollupOp;
import org.apache.doris.nereids.trees.plans.commands.info.AlterTableOp;
import org.apache.doris.nereids.trees.plans.commands.info.ColumnDefinition;
+import org.apache.doris.nereids.trees.plans.commands.info.CreateIndexOp;
import
org.apache.doris.nereids.trees.plans.commands.info.CreateOrReplaceBranchOp;
import org.apache.doris.nereids.trees.plans.commands.info.CreateOrReplaceTagOp;
import org.apache.doris.nereids.trees.plans.commands.info.DropBranchOp;
import org.apache.doris.nereids.trees.plans.commands.info.DropColumnOp;
+import org.apache.doris.nereids.trees.plans.commands.info.DropIndexOp;
import org.apache.doris.nereids.trees.plans.commands.info.DropPartitionFieldOp;
import org.apache.doris.nereids.trees.plans.commands.info.DropRollupOp;
import org.apache.doris.nereids.trees.plans.commands.info.DropTagOp;
import org.apache.doris.nereids.trees.plans.commands.info.EnableFeatureOp;
+import org.apache.doris.nereids.trees.plans.commands.info.IndexDefinition;
import
org.apache.doris.nereids.trees.plans.commands.info.ModifyColumnCommentOp;
import org.apache.doris.nereids.trees.plans.commands.info.ModifyColumnOp;
import org.apache.doris.nereids.trees.plans.commands.info.ModifyEngineOp;
@@ -135,14 +142,31 @@ public class AlterTableCommand extends Command implements
ForwardWithSync {
String ctlName = tbl.getCtl();
String dbName = tbl.getDb();
String tableName = tbl.getTbl();
- DatabaseIf dbIf = Env.getCurrentEnv().getCatalogMgr()
- .getCatalogOrException(ctlName, catalog -> new
DdlException("Unknown catalog " + catalog))
- .getDbOrDdlException(dbName);
+ CatalogIf catalog = Env.getCurrentEnv().getCatalogMgr()
+ .getCatalogOrException(ctlName, catalogName -> new
DdlException("Unknown catalog " + catalogName));
+ validateLanceIndexOperationsBeforeResolution(catalog);
+ DatabaseIf dbIf = catalog.getDbOrDdlException(dbName);
TableIf tableIf = dbIf.getTableOrDdlException(tableName);
if (tableIf.isTemporary()) {
throw new AnalysisException("Do not support alter temporary
table[" + tableName + "]");
}
checkColumnOperationsSupported(tableIf, ops);
+ if (tableIf instanceof LanceExternalTable) {
+ // Top-level CREATE/DROP INDEX on Lance catalog tables:
static-validate, then reject
+ // until the Lance index build path lands. ALTER TABLE ADD/DROP
INDEX (alter = true)
+ // falls through to the existing generic external-table rejection
below.
+ for (AlterTableOp op : ops) {
+ if (op instanceof CreateIndexOp && !((CreateIndexOp)
op).isAlter()) {
+ IndexDefinition indexDef = ((CreateIndexOp)
op).getIndexDef();
+
LanceIndexMutationValidator.validateCreateIndex((LanceExternalCatalog) catalog,
+ (LanceExternalTable) tableIf, indexDef);
+ LanceIndexMutationValidator.rejectUnsupportedOperation(
+ indexDef.isOrReplace() ? "CREATE OR REPLACE INDEX"
: "CREATE INDEX", "catalog tables");
+ } else if (op instanceof DropIndexOp && !((DropIndexOp)
op).isAlter()) {
+
LanceIndexMutationValidator.rejectUnsupportedOperation("DROP INDEX", "catalog
tables");
+ }
+ }
+ }
for (AlterTableOp op : ops) {
op.setTableName(tbl);
op.validate(ctx);
@@ -154,6 +178,27 @@ public class AlterTableCommand extends Command implements
ForwardWithSync {
}
}
+ /**
+ * Lance index checks that do not need the resolved table: REST catalogs
fail fast for
+ * top-level CREATE/DROP INDEX before database/table metadata resolution,
and DROP INDEX on
+ * Directory catalogs gets the shared index-name bounds here because the
op-level
+ * DropIndexOp.validate() is never reached on the typed-rejection path
below.
+ */
+ private void validateLanceIndexOperationsBeforeResolution(CatalogIf
catalog) throws AnalysisException {
+ if (!(catalog instanceof LanceExternalCatalog)) {
+ return;
+ }
+ for (AlterTableOp op : ops) {
+ if (op instanceof CreateIndexOp && !((CreateIndexOp)
op).isAlter()) {
+
LanceIndexMutationValidator.validateCreateIndexCatalog((LanceExternalCatalog)
catalog,
+ ((CreateIndexOp) op).getIndexDef());
+ } else if (op instanceof DropIndexOp && !((DropIndexOp)
op).isAlter()) {
+
LanceIndexMutationValidator.validateDropIndex((LanceExternalCatalog) catalog,
+ ((DropIndexOp) op).getIndexName());
+ }
+ }
+ }
+
static void checkColumnOperationsSupported(TableIf table,
List<AlterTableOp> alterTableOps)
throws AnalysisException {
if (table instanceof IcebergExternalTable) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
index 8ab371db208..d725fd7191b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
@@ -52,6 +52,10 @@ public class IndexDefinition {
// instead of the column name which from DorisParser
private List<String> caseSensitivityCols = Lists.newArrayList();
private IndexType indexType;
+ // Lance-only index type name ("BTREE" or "BITMAP"); null for internal
index types.
+ // Deliberately kept out of IndexDef.IndexType so the persisted internal
enum is untouched.
+ private final String lanceIndexType;
+ private final boolean orReplace;
private Map<String, String> properties = new HashMap<>();
private boolean isBuildDeferred = false;
@@ -64,9 +68,19 @@ public class IndexDefinition {
*/
public IndexDefinition(String name, boolean ifNotExists, List<String>
cols, String indexTypeName,
Map<String, String> properties, String comment) {
+ this(name, ifNotExists, cols, indexTypeName, properties, comment,
false);
+ }
+
+ /**
+ * constructor for IndexDefinition
+ */
+ public IndexDefinition(String name, boolean ifNotExists, List<String>
cols, String indexTypeName,
+ Map<String, String> properties, String comment, boolean orReplace)
{
this.name = name;
this.ifNotExists = ifNotExists;
+ this.orReplace = orReplace;
this.cols = Utils.copyRequiredList(cols);
+ String lanceType = null;
this.indexType = IndexType.INVERTED;
if (indexTypeName != null) {
switch (indexTypeName) {
@@ -82,10 +96,19 @@ public class IndexDefinition {
this.indexType = IndexType.ANN;
break;
}
+ case "BTREE":
+ case "BITMAP": {
+ // Lance catalog index types: no IndexDef.IndexType
mapping, validate() rejects
+ // them for internal tables before any internal code path
reads indexType.
+ this.indexType = null;
+ lanceType = indexTypeName;
+ break;
+ }
default:
throw new AnalysisException("unknown index type " +
indexTypeName);
}
}
+ this.lanceIndexType = lanceType;
if (properties != null) {
this.properties.putAll(properties);
@@ -105,6 +128,8 @@ public class IndexDefinition {
public IndexDefinition(String name, PartitionNamesInfo partitionNames,
IndexType indexType) {
this.name = name;
this.indexType = indexType;
+ this.lanceIndexType = null;
+ this.orReplace = false;
this.partitionNames = partitionNames;
this.isBuildDeferred = true;
this.cols = null;
@@ -233,6 +258,14 @@ public class IndexDefinition {
* validate
*/
public void validate() {
+ // Lance-only syntax guards: these fire before any internal validation
so that SQL which
+ // only parses for Lance catalog tables never reaches internal index
code paths.
+ if (lanceIndexType != null) {
+ throw new AnalysisException("USING " + lanceIndexType + " is only
supported for Lance catalog tables");
+ }
+ if (orReplace) {
+ throw new AnalysisException("CREATE OR REPLACE INDEX is only
supported for Lance catalog tables");
+ }
if (partitionNames != null) {
partitionNames.validate();
}
@@ -279,6 +312,10 @@ public class IndexDefinition {
}
}
+ public List<String> getCols() {
+ return cols;
+ }
+
public String getIndexName() {
return name;
}
@@ -287,6 +324,14 @@ public class IndexDefinition {
return indexType;
}
+ public boolean isOrReplace() {
+ return orReplace;
+ }
+
+ public String getLanceIndexType() {
+ return lanceIndexType;
+ }
+
public Index translateToCatalogStyle() {
return new Index(Env.getCurrentEnv().getNextId(), name, cols,
indexType, properties,
comment);
@@ -315,7 +360,7 @@ public class IndexDefinition {
* toSql
*/
public String toSql(String tableName) {
- StringBuilder sb = new StringBuilder("INDEX ");
+ StringBuilder sb = new StringBuilder(orReplace ? "OR REPLACE INDEX " :
"INDEX ");
sb.append(name);
if (tableName != null && !tableName.isEmpty()) {
sb.append(" ON ").append(tableName);
@@ -335,6 +380,8 @@ public class IndexDefinition {
}
if (indexType != null) {
sb.append(" USING ").append(indexType.toString());
+ } else if (lanceIndexType != null) {
+ sb.append(" USING ").append(lanceIndexType);
}
if (properties != null && properties.size() > 0) {
sb.append(" PROPERTIES(");
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexMutationValidatorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexMutationValidatorTest.java
new file mode 100644
index 00000000000..a3a108e575d
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexMutationValidatorTest.java
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lance;
+
+import org.apache.doris.catalog.ArrayType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.nereids.trees.plans.commands.info.IndexDefinition;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Unit coverage for the section 2.4 static validation matrix in
+ * {@link LanceIndexMutationValidator}. Every check is exercised positively
and negatively;
+ * the per-op typed rejections that follow a successful validation live in
+ * AlterTableCommandLanceIndexTest.
+ */
+public class LanceIndexMutationValidatorTest {
+
+ private static LanceExternalCatalog filesystemCatalog() {
+ LanceExternalCatalog catalog =
Mockito.mock(LanceExternalCatalog.class);
+ Mockito.when(catalog.isRestCatalogConfigured()).thenReturn(false);
+ return catalog;
+ }
+
+ private static LanceExternalCatalog restCatalog() {
+ LanceExternalCatalog catalog =
Mockito.mock(LanceExternalCatalog.class);
+ Mockito.when(catalog.isRestCatalogConfigured()).thenReturn(true);
+ return catalog;
+ }
+
+ /**
+ * Builds a Lance table mock whose getColumn lookup is case-insensitive,
mirroring
+ * ExternalTable.getColumn.
+ */
+ private static LanceExternalTable tableWithColumns(Map<String, Column>
columns) {
+ LanceExternalTable table = Mockito.mock(LanceExternalTable.class);
+
Mockito.when(table.getColumn(Mockito.anyString())).thenAnswer(invocation -> {
+ String name = invocation.getArgument(0);
+ for (Map.Entry<String, Column> entry : columns.entrySet()) {
+ if (entry.getKey().equalsIgnoreCase(name)) {
+ return entry.getValue();
+ }
+ }
+ return null;
+ });
+ return table;
+ }
+
+ private static Column notNullColumn(String name, Type type) {
+ return new Column(name, type, false, null, false, null, "");
+ }
+
+ private static Column nullableColumn(String name, Type type) {
+ return new Column(name, type, false, null, true, null, "");
+ }
+
+ private static IndexDefinition indexDef(String name, List<String> cols,
String indexTypeName,
+ Map<String, String> properties) {
+ return new IndexDefinition(name, false, cols, indexTypeName,
properties, "");
+ }
+
+ private static IndexDefinition annDef(Map<String, String> properties) {
+ return indexDef("idx", Collections.singletonList("v"), "ANN",
properties);
+ }
+
+ private static Map<String, String> validAnnProperties() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("index_type", "IVF_PQ");
+ properties.put("metric", "l2");
+ properties.put("num_partitions", "256");
+ properties.put("num_sub_vectors", "16");
+ return properties;
+ }
+
+ private static LanceExternalTable annTable() {
+ return tableWithColumns(Collections.singletonMap("v",
+ notNullColumn("v", new ArrayType(Type.FLOAT))));
+ }
+
+ private static void assertRejected(String expectedMessage, IndexDefinition
def,
+ LanceExternalTable table) {
+ AnalysisException exception =
Assertions.assertThrows(AnalysisException.class,
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), table,
def));
+ Assertions.assertEquals(expectedMessage, exception.getDetailMessage());
+ Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_INVALID,
exception.getMysqlErrorCode());
+ }
+
+ @Test
+ public void testAnnHappyPath() {
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), annTable(),
+ annDef(validAnnProperties())));
+ // num_bits = 8 is the one accepted optional value.
+ Map<String, String> withNumBits = validAnnProperties();
+ withNumBits.put("num_bits", "8");
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), annTable(),
+ annDef(withNumBits)));
+ // metric is optional.
+ Map<String, String> noMetric = validAnnProperties();
+ noMetric.remove("metric");
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), annTable(),
+ annDef(noMetric)));
+ // Column lookup is case-insensitive.
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), annTable(),
+ indexDef("idx", Collections.singletonList("V"), "ANN",
validAnnProperties())));
+ }
+
+ @Test
+ public void testAnnPropertyKeysAndValuesAreCaseInsensitive() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("Index_Type", "ivf_pq");
+ properties.put("METRIC", "COSINE");
+ properties.put("Num_Partitions", "256");
+ properties.put("NUM_SUB_VECTORS", "16");
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), annTable(),
+ annDef(properties)));
+ }
+
+ @Test
+ public void testAnnCaseVariantDuplicatePropertyRejected() {
+ // Case-variant duplicates must fail deterministically regardless of
map iteration order.
+ for (String badValue : new String[] {"garbage", "256"}) {
+ Map<String, String> properties = validAnnProperties();
+ properties.put("NUM_PARTITIONS", badValue);
+ assertRejected("Duplicate property 'NUM_PARTITIONS' for Lance ANN
index",
+ annDef(properties), annTable());
+ }
+ }
+
+ @Test
+ public void testAnnNullableColumnRejected() {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("v",
+ nullableColumn("v", new ArrayType(Type.FLOAT))));
+ assertRejected("ANN index must be built on a column that is not
nullable",
+ annDef(validAnnProperties()), table);
+ }
+
+ @Test
+ public void testAnnNonArrayColumnRejected() {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("v",
+ notNullColumn("v", Type.FLOAT)));
+ assertRejected("ANN index column must be array type",
annDef(validAnnProperties()), table);
+ }
+
+ @Test
+ public void testAnnDoubleItemTypeRejected() {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("v",
+ notNullColumn("v", new ArrayType(Type.DOUBLE))));
+ assertRejected("ANN index column item type must be float type",
+ annDef(validAnnProperties()), table);
+ }
+
+ @Test
+ public void testAnnIndexTypePropertyRequired() {
+ Map<String, String> missing = validAnnProperties();
+ missing.remove("index_type");
+ assertRejected("Lance ANN index requires property \"index_type\" =
\"IVF_PQ\"",
+ annDef(missing), annTable());
+
+ Map<String, String> wrongValue = validAnnProperties();
+ wrongValue.put("index_type", "IVF_FLAT");
+ assertRejected("Lance ANN index requires property \"index_type\" =
\"IVF_PQ\"",
+ annDef(wrongValue), annTable());
+ }
+
+ @Test
+ public void testAnnMetricValidated() {
+ Map<String, String> badMetric = validAnnProperties();
+ badMetric.put("metric", "l1");
+ assertRejected("metric must be one of l2, cosine, dot",
annDef(badMetric), annTable());
+
+ for (String metric : new String[] {"l2", "cosine", "dot", "L2",
"Cosine", "DOT"}) {
+ Map<String, String> properties = validAnnProperties();
+ properties.put("metric", metric);
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), annTable(),
+ annDef(properties)));
+ }
+ }
+
+ @Test
+ public void testAnnPositiveIntegerProperties() {
+ for (String key : new String[] {"num_partitions", "num_sub_vectors"}) {
+ Map<String, String> missing = validAnnProperties();
+ missing.remove(key);
+ assertRejected(key + " must be a positive integer",
annDef(missing), annTable());
+
+ for (String badValue : new String[] {"0", "-1", "abc", "1.5"}) {
+ Map<String, String> bad = validAnnProperties();
+ bad.put(key, badValue);
+ assertRejected(key + " must be a positive integer",
annDef(bad), annTable());
+ }
+ }
+ }
+
+ @Test
+ public void testAnnNumBitsMustBeEight() {
+ Map<String, String> nine = validAnnProperties();
+ nine.put("num_bits", "9");
+ assertRejected("num_bits must be 8", annDef(nine), annTable());
+
+ Map<String, String> notNumeric = validAnnProperties();
+ notNumeric.put("num_bits", "abc");
+ assertRejected("num_bits must be 8", annDef(notNumeric), annTable());
+ }
+
+ @Test
+ public void testAnnUnknownPropertyRejected() {
+ Map<String, String> properties = validAnnProperties();
+ properties.put("Foo", "1");
+ assertRejected("Unknown property 'Foo' for Lance ANN index",
annDef(properties), annTable());
+ }
+
+ @Test
+ public void testBtreeAllowedColumnTypes() {
+ for (Type type : new Type[] {Type.TINYINT, Type.SMALLINT, Type.INT,
Type.BIGINT, Type.LARGEINT,
+ Type.FLOAT, Type.DOUBLE, Type.DEFAULT_DECIMALV3, Type.STRING,
Type.DATEV2,
+ Type.DATETIMEV2, Type.TIMESTAMP_TZ}) {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("c",
+ notNullColumn("c", type)));
+ IndexDefinition def = indexDef("idx",
Collections.singletonList("c"), "BTREE",
+ Collections.emptyMap());
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), table,
def),
+ "BTREE should accept column type " + type);
+ }
+ }
+
+ @Test
+ public void testBtreeRejectedColumnTypes() {
+ for (Type type : new Type[] {Type.BOOLEAN, Type.TIMEV2, Type.VARBINARY,
+ new ArrayType(Type.INT)}) {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("c",
+ notNullColumn("c", type)));
+ IndexDefinition def = indexDef("idx",
Collections.singletonList("c"), "BTREE",
+ Collections.emptyMap());
+ AnalysisException exception =
Assertions.assertThrows(AnalysisException.class,
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), table,
def));
+ Assertions.assertTrue(
+ exception.getDetailMessage().startsWith("BTREE index does
not support column type"),
+ "unexpected message for type " + type + ": " +
exception.getDetailMessage());
+ }
+ }
+
+ @Test
+ public void testBtreePropertiesRejected() {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("c",
+ notNullColumn("c", Type.INT)));
+ IndexDefinition def = indexDef("idx", Collections.singletonList("c"),
"BTREE",
+ Collections.singletonMap("k", "v"));
+ assertRejected("BTREE indexes do not support properties", def, table);
+ }
+
+ @Test
+ public void testBtreeNullableColumnRejected() {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("c",
+ nullableColumn("c", Type.INT)));
+ IndexDefinition def = indexDef("idx", Collections.singletonList("c"),
"BTREE",
+ Collections.emptyMap());
+ assertRejected("BTREE index must be built on a column that is not
nullable", def, table);
+ }
+
+ @Test
+ public void testBitmapAllowedColumnTypes() {
+ for (Type type : new Type[] {Type.BOOLEAN, Type.TINYINT,
Type.SMALLINT, Type.INT, Type.BIGINT,
+ Type.LARGEINT, Type.STRING, Type.DATEV2}) {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("c",
+ notNullColumn("c", type)));
+ IndexDefinition def = indexDef("idx",
Collections.singletonList("c"), "BITMAP",
+ Collections.emptyMap());
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), table,
def),
+ "BITMAP should accept column type " + type);
+ }
+ }
+
+ @Test
+ public void testBitmapRejectedColumnTypes() {
+ // LARGEINT (Arrow uint64) is integral and accepted, exactly as in the
BTREE matrix.
+ for (Type type : new Type[] {Type.FLOAT, Type.DATETIMEV2}) {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("c",
+ notNullColumn("c", type)));
+ IndexDefinition def = indexDef("idx",
Collections.singletonList("c"), "BITMAP",
+ Collections.emptyMap());
+ AnalysisException exception =
Assertions.assertThrows(AnalysisException.class,
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), table,
def));
+ Assertions.assertTrue(
+ exception.getDetailMessage().startsWith("BITMAP index does
not support column type"),
+ "unexpected message for type " + type + ": " +
exception.getDetailMessage());
+ }
+ }
+
+ @Test
+ public void testBitmapPropertiesRejected() {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("c",
+ notNullColumn("c", Type.INT)));
+ IndexDefinition def = indexDef("idx", Collections.singletonList("c"),
"BITMAP",
+ Collections.singletonMap("k", "v"));
+ assertRejected("BITMAP indexes do not support properties", def, table);
+ }
+
+ @Test
+ public void testMultiColumnRejected() {
+ LanceExternalTable table =
tableWithColumns(Collections.singletonMap("v",
+ notNullColumn("v", new ArrayType(Type.FLOAT))));
+ IndexDefinition def = indexDef("idx", Arrays.asList("v", "v"), "ANN",
validAnnProperties());
+ assertRejected("Lance index must be built on exactly one column", def,
table);
+ }
+
+ @Test
+ public void testMissingColumnRejected() {
+ IndexDefinition def = indexDef("idx",
Collections.singletonList("nope"), "ANN",
+ validAnnProperties());
+ assertRejected("Index column 'nope' does not exist", def, annTable());
+ }
+
+ @Test
+ public void testIndexNameLengthBoundIsInUtf8Bytes() {
+ char[] chars = new char[64];
+ Arrays.fill(chars, 'a');
+ IndexDefinition sixtyFour = indexDef(new String(chars),
Collections.singletonList("v"),
+ "ANN", validAnnProperties());
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateCreateIndex(filesystemCatalog(), annTable(),
+ sixtyFour));
+
+ char[] tooLong = new char[65];
+ Arrays.fill(tooLong, 'a');
+ IndexDefinition sixtyFive = indexDef(new String(tooLong),
Collections.singletonList("v"),
+ "ANN", validAnnProperties());
+ assertRejected("index name too long, the index name length at most is
64.", sixtyFive, annTable());
+
+ // 33 two-byte characters are 66 UTF-8 bytes: the bound counts bytes,
not characters.
+ StringBuilder multibyte = new StringBuilder();
+ for (int i = 0; i < 33; i++) {
+ multibyte.append('é');
+ }
+ IndexDefinition multibyteName = indexDef(multibyte.toString(),
Collections.singletonList("v"),
+ "ANN", validAnnProperties());
+ assertRejected("index name too long, the index name length at most is
64.", multibyteName, annTable());
+ }
+
+ @Test
+ public void testUnsupportedIndexTypesRejected() {
+ // A CREATE INDEX without USING defaults to INVERTED, which is not a
Lance index type.
+ IndexDefinition inverted = indexDef("idx",
Collections.singletonList("v"), "INVERTED",
+ Collections.emptyMap());
+ assertRejected("Lance catalog tables only support USING ANN, BTREE, or
BITMAP", inverted, annTable());
+
+ IndexDefinition ngram = indexDef("idx",
Collections.singletonList("v"), "NGRAM_BF",
+ Collections.emptyMap());
+ assertRejected("Lance catalog tables only support USING ANN, BTREE, or
BITMAP", ngram, annTable());
+
+ IndexDefinition noUsing = indexDef("idx",
Collections.singletonList("v"), null,
+ Collections.emptyMap());
+ assertRejected("Lance catalog tables only support USING ANN, BTREE, or
BITMAP", noUsing, annTable());
+ }
+
+ @Test
+ public void testRestCatalogRejectedBeforeAnyOtherCheck() {
+ IndexDefinition createDef = indexDef("idx",
Collections.singletonList("missing"), "BTREE",
+ Collections.singletonMap("k", "v"));
+ AnalysisException createException =
Assertions.assertThrows(AnalysisException.class,
+ () ->
LanceIndexMutationValidator.validateCreateIndex(restCatalog(), annTable(),
createDef));
+ Assertions.assertEquals("CREATE INDEX is not supported for Lance REST
catalogs",
+ createException.getDetailMessage());
+
Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ createException.getMysqlErrorCode());
+
+ IndexDefinition orReplaceDef = new IndexDefinition("idx", false,
Collections.singletonList("v"),
+ "BTREE", Collections.emptyMap(), "", true);
+ AnalysisException orReplaceException =
Assertions.assertThrows(AnalysisException.class,
+ () ->
LanceIndexMutationValidator.validateCreateIndex(restCatalog(), annTable(),
orReplaceDef));
+ Assertions.assertEquals("CREATE OR REPLACE INDEX is not supported for
Lance REST catalogs",
+ orReplaceException.getDetailMessage());
+
Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ orReplaceException.getMysqlErrorCode());
+
+ AnalysisException dropException =
Assertions.assertThrows(AnalysisException.class,
+ () ->
LanceIndexMutationValidator.validateDropIndex(restCatalog(), "idx"));
+ Assertions.assertEquals("DROP INDEX is not supported for Lance REST
catalogs",
+ dropException.getDetailMessage());
+
Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ dropException.getMysqlErrorCode());
+
+ // The REST fail-fast precedes even the index-name bounds on the DROP
path.
+ AnalysisException dropEmptyNameException =
Assertions.assertThrows(AnalysisException.class,
+ () ->
LanceIndexMutationValidator.validateDropIndex(restCatalog(), ""));
+ Assertions.assertEquals("DROP INDEX is not supported for Lance REST
catalogs",
+ dropEmptyNameException.getDetailMessage());
+
Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ dropEmptyNameException.getMysqlErrorCode());
+ }
+
+ @Test
+ public void testBlankIndexNameRejectedOnBothPaths() {
+ // Nereids accepts an empty backquoted identifier (``), so both the
CREATE and the DROP
+ // path must reject the empty name instead of masking it as an
unsupported operation.
+ assertRejected("index name cannot be empty",
+ indexDef("", Collections.singletonList("v"), "ANN",
validAnnProperties()), annTable());
+ assertRejected("index name cannot be empty",
+ indexDef(null, Collections.singletonList("v"), "ANN",
validAnnProperties()), annTable());
+
+ for (String blankName : new String[] {null, ""}) {
+ AnalysisException exception =
Assertions.assertThrows(AnalysisException.class,
+ () ->
LanceIndexMutationValidator.validateDropIndex(filesystemCatalog(), blankName));
+ Assertions.assertEquals("index name cannot be empty",
exception.getDetailMessage());
+ Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_INVALID,
exception.getMysqlErrorCode());
+ }
+ }
+
+ @Test
+ public void testDropIndexNameLengthBound() {
+ char[] tooLong = new char[65];
+ Arrays.fill(tooLong, 'a');
+ AnalysisException exception =
Assertions.assertThrows(AnalysisException.class,
+ () ->
LanceIndexMutationValidator.validateDropIndex(filesystemCatalog(),
+ new String(tooLong)));
+ Assertions.assertEquals("index name too long, the index name length at
most is 64.",
+ exception.getDetailMessage());
+ Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_INVALID,
exception.getMysqlErrorCode());
+ }
+
+ @Test
+ public void testDropIndexOnFilesystemCatalogPasses() {
+ Assertions.assertDoesNotThrow(
+ () ->
LanceIndexMutationValidator.validateDropIndex(filesystemCatalog(), "idx"));
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/CreateIndexParserTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/CreateIndexParserTest.java
new file mode 100644
index 00000000000..c48b63269c2
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/CreateIndexParserTest.java
@@ -0,0 +1,187 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.parser;
+
+import org.apache.doris.analysis.IndexDef;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.exceptions.ParseException;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.commands.AlterTableCommand;
+import org.apache.doris.nereids.trees.plans.commands.info.AlterTableOp;
+import org.apache.doris.nereids.trees.plans.commands.info.CreateIndexOp;
+import org.apache.doris.nereids.trees.plans.commands.info.DropIndexOp;
+import org.apache.doris.nereids.trees.plans.commands.info.IndexDefinition;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Parser coverage for the Lance index DDL grammar: top-level CREATE [OR
REPLACE] INDEX
+ * ... USING ANN/BTREE/BITMAP and top-level DROP INDEX, plus the guarantees
that the
+ * ALTER TABLE ADD INDEX grammar rule is unchanged.
+ */
+public class CreateIndexParserTest extends ParserTestBase {
+ private final NereidsParser parser = new NereidsParser();
+
+ private CreateIndexOp parseCreateIndexOp(String sql) {
+ Plan plan = parser.parseSingle(sql);
+ AlterTableCommand command =
Assertions.assertInstanceOf(AlterTableCommand.class, plan);
+ Assertions.assertEquals(1, command.getNereidsOps().size());
+ AlterTableOp op = command.getNereidsOps().get(0);
+ return Assertions.assertInstanceOf(CreateIndexOp.class, op);
+ }
+
+ private DropIndexOp parseDropIndexOp(String sql) {
+ Plan plan = parser.parseSingle(sql);
+ AlterTableCommand command =
Assertions.assertInstanceOf(AlterTableCommand.class, plan);
+ Assertions.assertEquals(1, command.getNereidsOps().size());
+ AlterTableOp op = command.getNereidsOps().get(0);
+ return Assertions.assertInstanceOf(DropIndexOp.class, op);
+ }
+
+ @Test
+ public void testCreateAnnIndexParses() {
+ CreateIndexOp op = parseCreateIndexOp(
+ "CREATE INDEX idx ON ctl.db.tbl (v) USING ANN "
+ + "PROPERTIES(\"index_type\"=\"IVF_PQ\",
\"metric\"=\"l2\", "
+ + "\"num_partitions\"=\"256\",
\"num_sub_vectors\"=\"16\") COMMENT 'ann index'");
+ Assertions.assertFalse(op.isAlter());
+ IndexDefinition def = op.getIndexDef();
+ Assertions.assertEquals("idx", def.getIndexName());
+ Assertions.assertEquals(IndexDef.IndexType.ANN, def.getIndexType());
+ Assertions.assertNull(def.getLanceIndexType());
+ Assertions.assertFalse(def.isOrReplace());
+ Assertions.assertEquals(Collections.singletonList("v"), def.getCols());
+ Map<String, String> properties = def.getProperties();
+ Assertions.assertEquals(4, properties.size());
+ Assertions.assertEquals("IVF_PQ", properties.get("index_type"));
+ Assertions.assertEquals("l2", properties.get("metric"));
+ Assertions.assertEquals("256", properties.get("num_partitions"));
+ Assertions.assertEquals("16", properties.get("num_sub_vectors"));
+ }
+
+ @Test
+ public void testCreateBtreeIndexParses() {
+ CreateIndexOp op = parseCreateIndexOp("CREATE INDEX idx ON db.tbl (c)
USING BTREE");
+ IndexDefinition def = op.getIndexDef();
+ Assertions.assertNull(def.getIndexType());
+ Assertions.assertEquals("BTREE", def.getLanceIndexType());
+ Assertions.assertFalse(def.isOrReplace());
+ Assertions.assertEquals(Collections.singletonList("c"), def.getCols());
+ Assertions.assertTrue(def.getProperties().isEmpty());
+ }
+
+ @Test
+ public void testCreateBitmapIndexParses() {
+ CreateIndexOp op = parseCreateIndexOp("CREATE INDEX idx ON db.tbl (c)
USING BITMAP");
+ IndexDefinition def = op.getIndexDef();
+ Assertions.assertNull(def.getIndexType());
+ Assertions.assertEquals("BITMAP", def.getLanceIndexType());
+ Assertions.assertFalse(def.isOrReplace());
+ }
+
+ @Test
+ public void testLowercaseUsingClausesParse() {
+ // The lexer stream is case-insensitive; the new tokens must behave
like the old ones.
+ CreateIndexOp btreeOp = parseCreateIndexOp("create index idx on db.tbl
(c) using btree");
+ Assertions.assertEquals("BTREE",
btreeOp.getIndexDef().getLanceIndexType());
+ CreateIndexOp bitmapOp = parseCreateIndexOp("create index idx on
db.tbl (c) using bitmap");
+ Assertions.assertEquals("BITMAP",
bitmapOp.getIndexDef().getLanceIndexType());
+ }
+
+ @Test
+ public void testCreateOrReplaceIndexParses() {
+ CreateIndexOp op = parseCreateIndexOp("CREATE OR REPLACE INDEX idx ON
db.tbl (c) USING BTREE");
+ IndexDefinition def = op.getIndexDef();
+ Assertions.assertTrue(def.isOrReplace());
+ Assertions.assertEquals("BTREE", def.getLanceIndexType());
+ Assertions.assertEquals(
+ "CREATE OR REPLACE INDEX idx ON `db`.`tbl` (`c`) USING BTREE
COMMENT ''", op.toSql());
+ }
+
+ @Test
+ public void testCreateIndexIfNotExistsParses() {
+ CreateIndexOp op = parseCreateIndexOp(
+ "CREATE INDEX IF NOT EXISTS idx ON db.tbl (v) USING ANN "
+ + "PROPERTIES(\"index_type\"=\"IVF_PQ\",
\"num_partitions\"=\"256\", "
+ + "\"num_sub_vectors\"=\"16\")");
+ Assertions.assertFalse(op.getIndexDef().isOrReplace());
+ Assertions.assertEquals(IndexDef.IndexType.ANN,
op.getIndexDef().getIndexType());
+ }
+
+ @Test
+ public void testOrReplaceAndIfNotExistsAreExclusive() {
+ AnalysisException exception =
Assertions.assertThrows(AnalysisException.class,
+ () -> parser.parseSingle("CREATE OR REPLACE INDEX IF NOT
EXISTS idx ON db.tbl (c) USING BTREE"));
+ Assertions.assertEquals("[OR REPLACE] and [IF NOT EXISTS] cannot used
at the same time",
+ exception.getMessage());
+ }
+
+ @Test
+ public void testInternalUsingClausesStillParse() {
+ // Previously-parseable internal index SQL must keep working unchanged.
+ CreateIndexOp noUsing = parseCreateIndexOp("CREATE INDEX idx ON db.tbl
(c)");
+ Assertions.assertEquals(IndexDef.IndexType.INVERTED,
noUsing.getIndexDef().getIndexType());
+ Assertions.assertNull(noUsing.getIndexDef().getLanceIndexType());
+
+ CreateIndexOp inverted = parseCreateIndexOp("CREATE INDEX idx ON
db.tbl (c) USING INVERTED");
+ Assertions.assertEquals(IndexDef.IndexType.INVERTED,
inverted.getIndexDef().getIndexType());
+
+ CreateIndexOp ngram = parseCreateIndexOp(
+ "CREATE INDEX idx ON db.tbl (c) USING NGRAM_BF
PROPERTIES(\"gram_size\"=\"3\", \"bf_size\"=\"10000\")");
+ Assertions.assertEquals(IndexDef.IndexType.NGRAM_BF,
ngram.getIndexDef().getIndexType());
+
+ CreateIndexOp multiColumn = parseCreateIndexOp("CREATE INDEX idx ON
db.tbl (c1, c2) USING INVERTED");
+ Assertions.assertEquals(Arrays.asList("c1", "c2"),
multiColumn.getIndexDef().getCols());
+ }
+
+ @Test
+ public void testAlterTableAddIndexGrammarUnchanged() {
+ // The indexDef rule was deliberately not extended: Lance-only index
types stay
+ // parse errors in ALTER TABLE ADD INDEX.
+ Plan plan = parser.parseSingle("ALTER TABLE db.tbl ADD INDEX idx (c)
USING INVERTED");
+ AlterTableCommand command =
Assertions.assertInstanceOf(AlterTableCommand.class, plan);
+ CreateIndexOp op = Assertions.assertInstanceOf(CreateIndexOp.class,
command.getNereidsOps().get(0));
+ Assertions.assertTrue(op.isAlter());
+
+ Plan annPlan = parser.parseSingle("ALTER TABLE db.tbl ADD INDEX idx
(v) USING ANN");
+ AlterTableOp annOp = ((AlterTableCommand)
annPlan).getNereidsOps().get(0);
+ Assertions.assertTrue(((CreateIndexOp) annOp).isAlter());
+
+ Assertions.assertThrows(ParseException.class,
+ () -> parser.parseSingle("ALTER TABLE db.tbl ADD INDEX idx (c)
USING BTREE"));
+ Assertions.assertThrows(ParseException.class,
+ () -> parser.parseSingle("ALTER TABLE db.tbl ADD INDEX idx (c)
USING BITMAP"));
+ }
+
+ @Test
+ public void testDropIndexParses() {
+ DropIndexOp op = parseDropIndexOp("DROP INDEX idx ON db.tbl");
+ Assertions.assertEquals("idx", op.getIndexName());
+ Assertions.assertFalse(op.isAlter());
+ Assertions.assertFalse(op.isSetIfExists());
+
+ DropIndexOp ifExists = parseDropIndexOp("DROP INDEX IF EXISTS idx ON
ctl.db.tbl");
+ Assertions.assertTrue(ifExists.isSetIfExists());
+ Assertions.assertFalse(ifExists.isAlter());
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandLanceIndexTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandLanceIndexTest.java
new file mode 100644
index 00000000000..eac8627433e
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandLanceIndexTest.java
@@ -0,0 +1,375 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.catalog.ArrayType;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.DatabaseIf;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.CatalogMgr;
+import org.apache.doris.datasource.InternalCatalog;
+import org.apache.doris.datasource.lance.LanceExternalCatalog;
+import org.apache.doris.datasource.lance.LanceExternalDatabase;
+import org.apache.doris.datasource.lance.LanceExternalTable;
+import org.apache.doris.mysql.privilege.AccessControllerManager;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.parser.NereidsParser;
+import org.apache.doris.qe.ConnectContext;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+
+/**
+ * Routes-and-rejection coverage for Lance index DDL: top-level CREATE [OR
REPLACE] INDEX and
+ * DROP INDEX against Lance catalog tables run the static validation matrix
and are then
+ * rejected with typed messages (reject-all mode), ALTER TABLE ADD/DROP INDEX
keeps the generic
+ * external-table rejection, the ALTER privilege check precedes the typed
rejection, and no
+ * Env.getNextId() allocation happens on any rejected path.
+ */
+public class AlterTableCommandLanceIndexTest {
+ private static final String CTL = "lance_ctl";
+ private static final String DB = "db";
+ private static final String TBL = "tbl";
+ private static final String VALID_ANN_PROPERTIES =
+ "PROPERTIES(\"index_type\"=\"IVF_PQ\", \"metric\"=\"l2\", "
+ + "\"num_partitions\"=\"256\",
\"num_sub_vectors\"=\"16\")";
+
+ private final NereidsParser parser = new NereidsParser();
+ private ConnectContext connectContext;
+
+ @BeforeEach
+ public void setUp() {
+ connectContext = new ConnectContext();
+ connectContext.setThreadLocalInfo();
+ }
+
+ @AfterEach
+ public void tearDown() {
+ ConnectContext.remove();
+ }
+
+ /**
+ * Fully mocked catalog resolution chain ending at a LanceExternalTable
with three NOT NULL
+ * columns: v ARRAY<FLOAT>, c INT, s STRING.
+ */
+ private static class LanceFixture implements AutoCloseable {
+ private final MockedStatic<Env> mockedEnv;
+ private final Env env;
+ private final CatalogMgr catalogMgr;
+ private final LanceExternalCatalog catalog;
+ private final LanceExternalDatabase database;
+ private final LanceExternalTable table;
+
+ LanceFixture(boolean restCatalog, boolean alterGranted) throws
DdlException {
+ mockedEnv = Mockito.mockStatic(Env.class);
+ env = Mockito.mock(Env.class);
+ catalogMgr = Mockito.mock(CatalogMgr.class);
+ AccessControllerManager accessManager =
Mockito.mock(AccessControllerManager.class);
+ catalog = Mockito.mock(LanceExternalCatalog.class);
+ database = Mockito.mock(LanceExternalDatabase.class);
+ table = Mockito.mock(LanceExternalTable.class);
+
+ mockedEnv.when(Env::getCurrentEnv).thenReturn(env);
+ Mockito.when(env.getAccessManager()).thenReturn(accessManager);
+
Mockito.when(accessManager.checkTblPriv(Mockito.any(ConnectContext.class),
+ Mockito.eq(CTL), Mockito.eq(DB), Mockito.eq(TBL),
+ Mockito.eq(PrivPredicate.ALTER))).thenReturn(alterGranted);
+ Mockito.when(env.getCatalogMgr()).thenReturn(catalogMgr);
+ Mockito.when(catalogMgr.getCatalogOrException(Mockito.eq(CTL),
Mockito.any()))
+ .thenReturn(catalog);
+ Mockito.doReturn(database).when(catalog).getDbOrDdlException(DB);
+ Mockito.doReturn(table).when(database).getTableOrDdlException(TBL);
+
Mockito.when(catalog.isRestCatalogConfigured()).thenReturn(restCatalog);
+
Mockito.when(table.getColumn(Mockito.anyString())).thenAnswer(invocation -> {
+ String name = invocation.getArgument(0);
+ if ("v".equalsIgnoreCase(name)) {
+ return new Column("v", new ArrayType(Type.FLOAT), false,
null, false, null, "");
+ }
+ if ("c".equalsIgnoreCase(name)) {
+ return new Column("c", Type.INT, false, null, false, null,
"");
+ }
+ if ("s".equalsIgnoreCase(name)) {
+ return new Column("s", Type.STRING, false, null, false,
null, "");
+ }
+ return null;
+ });
+
Mockito.when(table.getType()).thenReturn(TableIf.TableType.LANCE_EXTERNAL_TABLE);
+ Mockito.when(table.getName()).thenReturn(TBL);
+ }
+
+ @Override
+ public void close() {
+ mockedEnv.close();
+ }
+ }
+
+ /**
+ * Mocked catalog resolution chain ending at an internal OlapTable.
+ */
+ private static class InternalFixture implements AutoCloseable {
+ private final MockedStatic<Env> mockedEnv;
+ private final Env env;
+
+ InternalFixture() throws DdlException {
+ mockedEnv = Mockito.mockStatic(Env.class);
+ env = Mockito.mock(Env.class);
+ CatalogMgr catalogMgr = Mockito.mock(CatalogMgr.class);
+ AccessControllerManager accessManager =
Mockito.mock(AccessControllerManager.class);
+ CatalogIf internalCatalog = Mockito.mock(InternalCatalog.class);
+ DatabaseIf database = Mockito.mock(DatabaseIf.class);
+ OlapTable table = Mockito.mock(OlapTable.class);
+
+ mockedEnv.when(Env::getCurrentEnv).thenReturn(env);
+ Mockito.when(env.getAccessManager()).thenReturn(accessManager);
+
Mockito.when(accessManager.checkTblPriv(Mockito.any(ConnectContext.class),
+ Mockito.anyString(), Mockito.anyString(),
Mockito.anyString(),
+ Mockito.eq(PrivPredicate.ALTER))).thenReturn(true);
+ Mockito.when(env.getCatalogMgr()).thenReturn(catalogMgr);
+ Mockito.when(catalogMgr.getCatalogOrException(
+ Mockito.eq(InternalCatalog.INTERNAL_CATALOG_NAME),
Mockito.any()))
+ .thenReturn(internalCatalog);
+
Mockito.doReturn(database).when(internalCatalog).getDbOrDdlException(DB);
+ Mockito.doReturn(table).when(database).getTableOrDdlException(TBL);
+ }
+
+ @Override
+ public void close() {
+ mockedEnv.close();
+ }
+ }
+
+ private String runAndGetMessage(String sql) {
+ AlterTableCommand command = (AlterTableCommand)
parser.parseSingle(sql);
+ try {
+ command.run(connectContext, null);
+ throw new AssertionError("expected an AnalysisException but the
statement succeeded: " + sql);
+ } catch (AnalysisException e) {
+ // Catalog-resolution paths (privilege check, Lance validator,
typed rejections) throw
+ // the legacy AnalysisException; getDetailMessage() strips the
"errCode = 2" prefix.
+ return e.getDetailMessage();
+ } catch (org.apache.doris.nereids.exceptions.AnalysisException e) {
+ // IndexDefinition.validate() guards throw the nereids
AnalysisException instead.
+ return e.getMessage();
+ } catch (Exception e) {
+ throw new AssertionError("unexpected exception " +
e.getClass().getName() + ": " + e.getMessage(), e);
+ }
+ }
+
+ private AnalysisException runAndGetCommonAnalysisException(String sql) {
+ AlterTableCommand command = (AlterTableCommand)
parser.parseSingle(sql);
+ try {
+ command.run(connectContext, null);
+ throw new AssertionError("expected an AnalysisException but the
statement succeeded: " + sql);
+ } catch (AnalysisException e) {
+ return e;
+ } catch (Exception e) {
+ throw new AssertionError("unexpected exception " +
e.getClass().getName() + ": " + e.getMessage(), e);
+ }
+ }
+
+ @Test
+ public void testCreateIndexOnLanceTableIsTypedRejected() throws Exception {
+ try (LanceFixture fixture = new LanceFixture(false, true)) {
+ String message = runAndGetMessage(
+ "CREATE INDEX idx ON " + CTL + "." + DB + "." + TBL + "
(v) USING ANN "
+ + VALID_ANN_PROPERTIES);
+ Assertions.assertEquals("CREATE INDEX is not supported for Lance
catalog tables", message);
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ }
+ }
+
+ @Test
+ public void testCreateOrReplaceIndexOnLanceTableIsTypedRejected() throws
Exception {
+ try (LanceFixture fixture = new LanceFixture(false, true)) {
+ String message = runAndGetMessage(
+ "CREATE OR REPLACE INDEX idx ON " + CTL + "." + DB + "." +
TBL + " (c) USING BTREE");
+ Assertions.assertEquals("CREATE OR REPLACE INDEX is not supported
for Lance catalog tables",
+ message);
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ }
+ }
+
+ @Test
+ public void testDropIndexOnLanceTableIsTypedRejected() throws Exception {
+ try (LanceFixture fixture = new LanceFixture(false, true)) {
+ Assertions.assertEquals("DROP INDEX is not supported for Lance
catalog tables",
+ runAndGetMessage("DROP INDEX idx ON " + CTL + "." + DB +
"." + TBL));
+ // Reject-all mode is uniform: IF EXISTS does not change the
outcome.
+ Assertions.assertEquals("DROP INDEX is not supported for Lance
catalog tables",
+ runAndGetMessage("DROP INDEX IF EXISTS idx ON " + CTL +
"." + DB + "." + TBL));
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ }
+ }
+
+ @Test
+ public void testFilesystemCatalogRejectionsExposeNotSupportedErrorCode()
throws Exception {
+ try (LanceFixture fixture = new LanceFixture(false, true)) {
+ for (String sql : new String[] {
+ "CREATE INDEX idx ON " + CTL + "." + DB + "." + TBL + "
(c) USING BTREE",
+ "CREATE OR REPLACE INDEX idx ON " + CTL + "." + DB + "." +
TBL + " (c) USING BTREE",
+ "DROP INDEX idx ON " + CTL + "." + DB + "." + TBL}) {
+ AnalysisException exception =
runAndGetCommonAnalysisException(sql);
+
Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ exception.getMysqlErrorCode());
+ }
+ }
+ }
+
+ @Test
+ public void testRestCatalogMessagesFireFirst() throws Exception {
+ try (LanceFixture fixture = new LanceFixture(true, true)) {
+ Assertions.assertEquals("CREATE INDEX is not supported for Lance
REST catalogs",
+ runAndGetMessage("CREATE INDEX idx ON " + CTL + "." + DB +
"." + TBL
+ + " (v) USING ANN " + VALID_ANN_PROPERTIES));
+ Assertions.assertEquals("CREATE OR REPLACE INDEX is not supported
for Lance REST catalogs",
+ runAndGetMessage("CREATE OR REPLACE INDEX idx ON " + CTL +
"." + DB + "." + TBL
+ + " (c) USING BTREE"));
+ Assertions.assertEquals("DROP INDEX is not supported for Lance
REST catalogs",
+ runAndGetMessage("DROP INDEX idx ON " + CTL + "." + DB +
"." + TBL));
+ // The REST fail-fast precedes even the index-name bounds.
+ Assertions.assertEquals("CREATE INDEX is not supported for Lance
REST catalogs",
+ runAndGetMessage("CREATE INDEX `` ON " + CTL + "." + DB +
"." + TBL
+ + " (v) USING ANN " + VALID_ANN_PROPERTIES));
+ Assertions.assertEquals("DROP INDEX is not supported for Lance
REST catalogs",
+ runAndGetMessage("DROP INDEX `` ON " + CTL + "." + DB +
"." + TBL));
+ Mockito.verify(fixture.catalog,
Mockito.never()).getDbOrDdlException(Mockito.anyString());
+ Mockito.verify(fixture.database,
Mockito.never()).getTableOrDdlException(Mockito.anyString());
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ }
+ }
+
+ @Test
+ public void testStaticValidationErrorsPrecedeTheTypedRejection() throws
Exception {
+ try (LanceFixture fixture = new LanceFixture(false, true)) {
+ Assertions.assertEquals("metric must be one of l2, cosine, dot",
+ runAndGetMessage("CREATE INDEX idx ON " + CTL + "." + DB +
"." + TBL
+ + " (v) USING ANN
PROPERTIES(\"index_type\"=\"IVF_PQ\", \"metric\"=\"l1\", "
+ + "\"num_partitions\"=\"256\",
\"num_sub_vectors\"=\"16\")"));
+ Assertions.assertEquals("num_partitions must be a positive
integer",
+ runAndGetMessage("CREATE INDEX idx ON " + CTL + "." + DB +
"." + TBL
+ + " (v) USING ANN
PROPERTIES(\"index_type\"=\"IVF_PQ\", "
+ + "\"num_sub_vectors\"=\"16\")"));
+ Assertions.assertEquals("BTREE indexes do not support properties",
+ runAndGetMessage("CREATE INDEX idx ON " + CTL + "." + DB +
"." + TBL
+ + " (c) USING BTREE PROPERTIES(\"k\"=\"v\")"));
+ Assertions.assertEquals("Index column 'nope' does not exist",
+ runAndGetMessage("CREATE INDEX idx ON " + CTL + "." + DB +
"." + TBL
+ + " (nope) USING BTREE"));
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ }
+ }
+
+ @Test
+ public void testBlankQuotedIndexNameRejectedOnBothPaths() throws Exception
{
+ try (LanceFixture fixture = new LanceFixture(false, true)) {
+ // Nereids accepts an empty backquoted identifier, so `CREATE
INDEX `` ` and
+ // `DROP INDEX `` ` reach the command with an empty index name.
Both paths must
+ // surface the name error, not the typed unsupported-operation
rejection.
+ Assertions.assertEquals("index name cannot be empty",
+ runAndGetMessage("CREATE INDEX `` ON " + CTL + "." + DB +
"." + TBL
+ + " (v) USING ANN " + VALID_ANN_PROPERTIES));
+ Assertions.assertEquals("index name cannot be empty",
+ runAndGetMessage("CREATE OR REPLACE INDEX `` ON " + CTL +
"." + DB + "." + TBL
+ + " (c) USING BTREE"));
+ Assertions.assertEquals("index name cannot be empty",
+ runAndGetMessage("DROP INDEX `` ON " + CTL + "." + DB +
"." + TBL));
+ Assertions.assertEquals("index name cannot be empty",
+ runAndGetMessage("DROP INDEX IF EXISTS `` ON " + CTL + "."
+ DB + "." + TBL));
+ AnalysisException exception = runAndGetCommonAnalysisException(
+ "DROP INDEX `` ON " + CTL + "." + DB + "." + TBL);
+ Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_INVALID,
exception.getMysqlErrorCode());
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ }
+ }
+
+ @Test
+ public void testNonLanceIndexTypesRejectedEndToEnd() throws Exception {
+ try (LanceFixture fixture = new LanceFixture(false, true)) {
+ // Index types outside the Lance matrix surface the vocabulary
error through the
+ // command path, before the typed rejection and without any id
allocation.
+ Assertions.assertEquals("Lance catalog tables only support USING
ANN, BTREE, or BITMAP",
+ runAndGetMessage("CREATE INDEX idx ON " + CTL + "." + DB +
"." + TBL
+ + " (c) USING NGRAM_BF"));
+ Assertions.assertEquals("Lance catalog tables only support USING
ANN, BTREE, or BITMAP",
+ runAndGetMessage("CREATE INDEX idx ON " + CTL + "." + DB +
"." + TBL
+ + " (c) USING INVERTED"));
+ Assertions.assertEquals("Lance catalog tables only support USING
ANN, BTREE, or BITMAP",
+ runAndGetMessage("CREATE INDEX idx ON " + CTL + "." + DB +
"." + TBL + " (c)"));
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ }
+ }
+
+ @Test
+ public void testAlterTableAddAndDropIndexKeepTheGenericRejection() throws
Exception {
+ try (LanceFixture fixture = new LanceFixture(true, true)) {
+ // alter = true ops skip both the REST fail-fast branch and the
Lance branch, then
+ // fall through to the existing generic external-table rejection.
+ String addMessage = runAndGetMessage(
+ "ALTER TABLE " + CTL + "." + DB + "." + TBL + " ADD INDEX
idx (c) USING INVERTED");
+ Assertions.assertTrue(addMessage.contains("do not support
SCHEMA_CHANGE clause now"),
+ addMessage);
+
+ String dropMessage = runAndGetMessage(
+ "ALTER TABLE " + CTL + "." + DB + "." + TBL + " DROP INDEX
idx");
+ Assertions.assertTrue(dropMessage.contains("do not support
SCHEMA_CHANGE clause now"),
+ dropMessage);
+ Mockito.verify(fixture.catalog,
Mockito.times(2)).getDbOrDdlException(DB);
+ Mockito.verify(fixture.database,
Mockito.times(2)).getTableOrDdlException(TBL);
+ }
+ }
+
+ @Test
+ public void testPrivilegeDeniedPrecedesTheTypedRejection() throws
Exception {
+ try (LanceFixture fixture = new LanceFixture(true, false)) {
+ String message = runAndGetMessage(
+ "CREATE INDEX idx ON " + CTL + "." + DB + "." + TBL + "
(v) USING ANN "
+ + VALID_ANN_PROPERTIES);
+ Assertions.assertTrue(message.contains("command denied to user"),
message);
+ // The privilege check fires before any catalog resolution.
+ Mockito.verify(fixture.catalogMgr, Mockito.never())
+ .getCatalogOrException(Mockito.anyString(), Mockito.any());
+ Mockito.verify(fixture.catalog,
Mockito.never()).isRestCatalogConfigured();
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ }
+ }
+
+ @Test
+ public void testInternalTableGuardsRejectLanceOnlySyntax() throws
Exception {
+ try (InternalFixture fixture = new InternalFixture()) {
+ Assertions.assertEquals("CREATE OR REPLACE INDEX is only supported
for Lance catalog tables",
+ runAndGetMessage("CREATE OR REPLACE INDEX idx ON
internal." + DB + "." + TBL
+ + " (c) USING INVERTED"));
+ Assertions.assertEquals("USING BTREE is only supported for Lance
catalog tables",
+ runAndGetMessage("CREATE INDEX idx ON internal." + DB +
"." + TBL + " (c) USING BTREE"));
+ Assertions.assertEquals("USING BITMAP is only supported for Lance
catalog tables",
+ runAndGetMessage("CREATE INDEX idx ON internal." + DB +
"." + TBL + " (c) USING BITMAP"));
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ }
+ }
+}
diff --git
a/regression-test/suites/external_table_p0/lance/test_lance_index_ddl.groovy
b/regression-test/suites/external_table_p0/lance/test_lance_index_ddl.groovy
new file mode 100644
index 00000000000..c0bb1550453
--- /dev/null
+++ b/regression-test/suites/external_table_p0/lance/test_lance_index_ddl.groovy
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_lance_index_ddl", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable Lance index DDL test because the Iceberg MinIO
environment is disabled.")
+ return
+ }
+
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ String lanceRestPort = context.config.otherConfigs.get("lance_rest_port")
+ String filesystemCatalog = "test_lance_index_ddl"
+ String restCatalog = "test_lance_index_ddl_rest"
+ String user = "test_lance_index_ddl_user"
+ String password = "C123_567p"
+
+ sql """DROP CATALOG IF EXISTS `${filesystemCatalog}`"""
+ sql """DROP CATALOG IF EXISTS `${restCatalog}`"""
+ try_sql "DROP USER '${user}'@'%'"
+
+ try {
+ sql """
+ CREATE CATALOG `${filesystemCatalog}` PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "s3://warehouse/lance",
+ "s3.endpoint" = "http://${externalEnvIp}:${minioPort}";,
+ "s3.access_key" = "admin",
+ "s3.secret_key" = "password",
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true"
+ )
+ """
+
+ // doris.vs_ivf_pq_f32 schema (all NOT NULL): embedding array<float>,
row_id bigint,
+ // category text, label text. Statically valid index DDL passes the
section 2.4 matrix
+ // and is then uniformly rejected until the Lance index build path
lands.
+ test {
+ sql """CREATE INDEX idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (embedding) USING ANN
+ PROPERTIES("index_type"="IVF_PQ", "metric"="l2",
"num_partitions"="256", "num_sub_vectors"="16")"""
+ exception "CREATE INDEX is not supported for Lance catalog tables"
+ }
+
+ test {
+ sql """CREATE INDEX idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (embedding) USING ANN
+ PROPERTIES("index_type"="IVF_PQ", "metric"="l1",
"num_partitions"="256", "num_sub_vectors"="16")"""
+ exception "metric must be one of l2, cosine, dot"
+ }
+
+ test {
+ sql """CREATE INDEX idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (embedding) USING ANN
+ PROPERTIES("index_type"="IVF_PQ", "num_sub_vectors"="16")"""
+ exception "num_partitions must be a positive integer"
+ }
+
+ test {
+ sql """CREATE INDEX idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (row_id) USING BTREE"""
+ exception "CREATE INDEX is not supported for Lance catalog tables"
+ }
+
+ test {
+ sql """CREATE INDEX idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (row_id) USING BTREE
+ PROPERTIES("k"="v")"""
+ exception "BTREE indexes do not support properties"
+ }
+
+ test {
+ sql """CREATE INDEX idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (category) USING BITMAP"""
+ exception "CREATE INDEX is not supported for Lance catalog tables"
+ }
+
+ test {
+ sql """CREATE OR REPLACE INDEX idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (row_id) USING BTREE"""
+ exception "CREATE OR REPLACE INDEX is not supported for Lance
catalog tables"
+ }
+
+ test {
+ sql """CREATE OR REPLACE INDEX IF NOT EXISTS idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (row_id) USING BTREE"""
+ exception "[OR REPLACE] and [IF NOT EXISTS] cannot used at the
same time"
+ }
+
+ test {
+ sql """DROP INDEX idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32`"""
+ exception "DROP INDEX is not supported for Lance catalog tables"
+ }
+
+ // Reject-all mode is uniform: IF EXISTS does not change the outcome.
+ test {
+ sql """DROP INDEX IF EXISTS idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32`"""
+ exception "DROP INDEX is not supported for Lance catalog tables"
+ }
+
+ // An empty backquoted index name is a blank name, not an unsupported
operation.
+ test {
+ sql """CREATE INDEX `` ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (embedding) USING ANN
+ PROPERTIES("index_type"="IVF_PQ", "num_partitions"="256",
"num_sub_vectors"="16")"""
+ exception "index name cannot be empty"
+ }
+
+ test {
+ sql """DROP INDEX `` ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32`"""
+ exception "index name cannot be empty"
+ }
+
+ // ALTER TABLE ADD/DROP INDEX keeps the generic external-table
rejection.
+ test {
+ sql """ALTER TABLE `${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32`
ADD INDEX idx (category) USING INVERTED"""
+ exception "do not support SCHEMA_CHANGE clause now"
+ }
+
+ sql """
+ CREATE CATALOG `${restCatalog}` PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "http://${externalEnvIp}:${lanceRestPort}";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "doris-lance-rest-test-token",
+ "lance.namespace.root_database" = "default",
+ "s3.endpoint" = "http://${externalEnvIp}:${minioPort}";,
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true",
+ "test_connection" = "true"
+ )
+ """
+
+ test {
+ sql """CREATE INDEX idx ON `${restCatalog}`.`default`.`all_types`
(row_id) USING BTREE"""
+ exception "CREATE INDEX is not supported for Lance REST catalogs"
+ }
+
+ test {
+ sql """CREATE OR REPLACE INDEX idx ON
`${restCatalog}`.`default`.`all_types` (row_id) USING BTREE"""
+ exception "CREATE OR REPLACE INDEX is not supported for Lance REST
catalogs"
+ }
+
+ test {
+ sql """DROP INDEX idx ON `${restCatalog}`.`default`.`all_types`"""
+ exception "DROP INDEX is not supported for Lance REST catalogs"
+ }
+
+ sql """CREATE USER '${user}'@'%' IDENTIFIED BY '${password}'"""
+ sql """GRANT SELECT_PRIV ON regression_test TO '${user}'@'%'"""
+ if (isCloudMode()) {
+ def clusters = sql "SHOW CLUSTERS"
+ assertTrue(!clusters.isEmpty())
+ sql """GRANT USAGE_PRIV ON CLUSTER `${clusters[0][0]}` TO
'${user}'@'%'"""
+ }
+
+ connect(user, password, context.config.jdbcUrl) {
+ // The ALTER privilege check precedes the typed Lance rejection.
+ test {
+ sql """CREATE INDEX idx ON
`${filesystemCatalog}`.`doris`.`vs_ivf_pq_f32` (row_id) USING BTREE"""
+ exception "denied"
+ }
+ }
+ } finally {
+ try_sql "DROP USER '${user}'@'%'"
+ // Keep both catalogs for debugging when the suite fails.
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
