This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 8b5bfb44d4 [#11933] feat(clickhouse): support ngrambf_v1 and
tokenbf_v1 data skipping index types (#12026)
8b5bfb44d4 is described below
commit 8b5bfb44d4c6a5a0b01b48e7cba0159496f6c648
Author: StormSpirit <[email protected]>
AuthorDate: Sat Aug 29 10:02:44 2026 +0800
[#11933] feat(clickhouse): support ngrambf_v1 and tokenbf_v1 data skipping
index types (#12026)
### What changes were proposed in this pull request?
Add support for `ngrambf_v1` and `tokenbf_v1` data skipping index types
in the ClickHouse catalog.
- Add `DATA_SKIPPING_NGRAMBFV1` and `DATA_SKIPPING_TOKENBFV1` to
`Index.IndexType`.
- Add ClickHouse type constants and the documented `Index.properties()`
keys for bloom-filter parameters.
- Read `type_full` from `system.data_skipping_indices` to restore the
required parameters on table load, with a legacy query fallback when the
column is unavailable.
- Generate parameterized DDL for both CREATE TABLE and ALTER TABLE ADD
INDEX, sharing the same validation and granularity handling.
- Add single-node and ON CLUSTER integration coverage for CREATE, LOAD,
ALTER, parameter round-trip, and SHOW CREATE behavior.
### Why are the changes needed?
ClickHouse tables containing `ngrambf_v1` or `tokenbf_v1` indexes
currently load without preserving their index type metadata, while
attempts to create these index types through Gravitino fail because the
catalog does not generate their required parameters. This change
preserves the metadata when `type_full` is available and provides a
defensive legacy-query fallback for older metadata schemas. The current
Gravitino compatibility matrix continues to explicitly verify ClickHouse
24.8.x; the fallback does not declare a new supported server-version
baseline.
Fix: #11933
### Does this PR introduce _any_ user-facing change?
Yes. It adds two `Index.IndexType` values and the following
`Index.properties()` keys: `ngram_size`, `bloom_filter_size`,
`hash_functions`, and `random_seed`; `granularity` remains optional and
defaults to `1`. `ngrambf_v1` requires all four bloom-filter parameters,
while `tokenbf_v1` requires the latter three. On a server whose legacy
`type` metadata is bare and does not include parameters, the catalog can
preserve the index type and fields but cannot reconstruct the required
parameter properties for a full recreate operation; callers must provide
those properties explicitly.
### How was this patch tested?
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:spotlessCheck`
- `./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test -PskipITs
-PskipDockerTests=true` — 61 tests passed.
- `./gradlew compileJava -x test`
- `python3 ~/GitHub/bin/gravitino-pr-precheck.py` — no errors.
- ClickHouse 24.8.14 `CatalogClickHouseIT` — 46 tests passed with no
skips, failures, or errors.
- ClickHouse 24.8.14 `CatalogClickHouseClusterIT` — 18 tests passed with
no skips, failures, or errors, including the new ngrambf/tokenbf cluster
test.
---------
Signed-off-by: jiangxt2 <[email protected]>
---
.../org/apache/gravitino/rel/indexes/Index.java | 16 +-
.../catalog/clickhouse/ClickHouseConstants.java | 18 +
.../operations/ClickHouseTableOperations.java | 393 ++++++++++++---
.../test/CatalogClickHouseClusterIT.java | 90 ++++
.../integration/test/CatalogClickHouseIT.java | 142 ++++++
.../operations/TestClickHouseTableOperations.java | 528 +++++++++++++++++++++
.../TestClickHouseTableOperationsUnit.java | 136 ++++++
docs/jdbc-clickhouse-catalog.md | 8 +-
docs/open-api/indexes.yaml | 2 +
9 files changed, 1265 insertions(+), 68 deletions(-)
diff --git a/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
b/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
index 9f725c19cc..ec3d6cbce7 100644
--- a/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
+++ b/api/src/main/java/org/apache/gravitino/rel/indexes/Index.java
@@ -122,8 +122,7 @@ public interface Index {
/** IVF_HNSW_PQ */
IVF_HNSW_PQ,
- // The following index types are data skipping indexes in ClickHouse,
ngrambf_v1 and tokenbf_v1
- // Will be supported later.
+ // The following index types are data skipping indexes in ClickHouse.
/** minmax data skipping index */
DATA_SKIPPING_MINMAX,
@@ -132,5 +131,18 @@ public interface Index {
/** Set data skipping index */
DATA_SKIPPING_SET,
+
+ /**
+ * Ngram bloom filter data skipping index. Uses n-gram tokenization of
string columns for
+ * bloom-filter-based text search acceleration. Available since ClickHouse
v19.x.
+ */
+ DATA_SKIPPING_NGRAMBFV1,
+
+ /**
+ * Token bloom filter data skipping index. Uses tokenization (by
non-alphanumeric delimiters) of
+ * string columns for bloom-filter-based text search acceleration.
Available since ClickHouse
+ * v19.x.
+ */
+ DATA_SKIPPING_TOKENBFV1,
}
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
index 473ac35d3d..b040ba4bda 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
@@ -70,5 +70,23 @@ public class ClickHouseConstants {
// Key for max unique values (N) in set(N) data-skipping index properties.
public static final String SET_MAX_VALUES = "set_max_values";
+
+ /** The name of the data skipping index type for ngrambf_v1 in ClickHouse.
*/
+ public static final String DATA_SKIPPING_NGRAMBFV1 = "ngrambf_v1";
+
+ /** The name of the data skipping index type for tokenbf_v1 in ClickHouse.
*/
+ public static final String DATA_SKIPPING_TOKENBFV1 = "tokenbf_v1";
+
+ /** Property key for bloom filter size in ngrambf_v1 and tokenbf_v1 index
properties. */
+ public static final String BLOOM_FILTER_SIZE = "bloom_filter_size";
+
+ /** Property key for the number of hash functions in ngrambf_v1 and
tokenbf_v1 properties. */
+ public static final String HASH_FUNCTIONS = "hash_functions";
+
+ /** Property key for the random seed in ngrambf_v1 and tokenbf_v1 index
properties. */
+ public static final String RANDOM_SEED = "random_seed";
+
+ /** Property key for the n-gram size in ngrambf_v1 index properties. */
+ public static final String NGRAM_SIZE = "ngram_size";
}
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
index 8d7afb9ed5..77115b9113 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
@@ -18,10 +18,16 @@
*/
package org.apache.gravitino.catalog.clickhouse.operations;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.BLOOM_FILTER_SIZE;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_BLOOM_FILTER;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_MINMAX_VALUE;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_NGRAMBFV1;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_SET;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.DATA_SKIPPING_TOKENBFV1;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.GRANULARITY;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.HASH_FUNCTIONS;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.NGRAM_SIZE;
+import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.RANDOM_SEED;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.IndexConstants.SET_MAX_VALUES;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseTablePropertiesMetadata.CLICKHOUSE_ENGINE_KEY;
import static
org.apache.gravitino.catalog.clickhouse.ClickHouseTablePropertiesMetadata.ENGINE_PROPERTY_ENTRY;
@@ -129,6 +135,13 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
ORDER BY COLUMN_NAME
""";
+ private static final String SECONDARY_INDEX_QUERY =
+ "SELECT name, type, type_full, expr, granularity FROM
system.data_skipping_indices "
+ + "WHERE database = ? AND table = ? ORDER BY name";
+ private static final String LEGACY_SECONDARY_INDEX_QUERY =
+ "SELECT name, type, expr, granularity FROM system.data_skipping_indices "
+ + "WHERE database = ? AND table = ? ORDER BY name";
+
@Override
public void create(
String databaseName,
@@ -612,37 +625,15 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
sqlBuilder.append(" PRIMARY KEY (").append(fieldStr).append(")");
break;
case DATA_SKIPPING_MINMAX:
- sqlBuilder
- .append(" ")
- .append(
- buildDataSkippingIndexDdl(
- index.name(),
- fieldStr,
- DATA_SKIPPING_MINMAX_VALUE,
- resolveGranularity(index.properties(), 1)));
- break;
case DATA_SKIPPING_BLOOM_FILTER:
- sqlBuilder
- .append(" ")
- .append(
- buildDataSkippingIndexDdl(
- index.name(),
- fieldStr,
- DATA_SKIPPING_BLOOM_FILTER,
- resolveGranularity(index.properties(), 1)));
- break;
case DATA_SKIPPING_SET:
- // SET index: set(N) max unique values default to 0 (unlimited),
configurable via
- // set_max_values property. GRANULARITY defaults to 1, configurable
via granularity
- // property, consistent with minmax and bloom_filter indexes.
+ case DATA_SKIPPING_NGRAMBFV1:
+ case DATA_SKIPPING_TOKENBFV1:
sqlBuilder
.append(" ")
.append(
buildDataSkippingIndexDdl(
- index.name(),
- fieldStr,
- "set(" + resolveSetMaxValues(index.properties()) + ")",
- resolveGranularity(index.properties(), 1)));
+ index.name(), fieldStr, index.type(),
index.properties()));
break;
default:
throw new IllegalArgumentException(
@@ -1119,28 +1110,13 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
Map<String, String> properties = addIndex.getProperties();
switch (addIndex.getType()) {
case DATA_SKIPPING_MINMAX:
- return "ADD "
- + buildDataSkippingIndexDdl(
- addIndex.getName(),
- fieldStr,
- DATA_SKIPPING_MINMAX_VALUE,
- resolveGranularity(properties, 1));
-
case DATA_SKIPPING_BLOOM_FILTER:
- return "ADD "
- + buildDataSkippingIndexDdl(
- addIndex.getName(),
- fieldStr,
- DATA_SKIPPING_BLOOM_FILTER,
- resolveGranularity(properties, 1));
-
case DATA_SKIPPING_SET:
+ case DATA_SKIPPING_NGRAMBFV1:
+ case DATA_SKIPPING_TOKENBFV1:
return "ADD "
+ buildDataSkippingIndexDdl(
- addIndex.getName(),
- fieldStr,
- "set(" + resolveSetMaxValues(properties) + ")",
- resolveGranularity(properties, 1));
+ addIndex.getName(), fieldStr, addIndex.getType(), properties);
case PRIMARY_KEY:
throw new UnsupportedOperationException(
@@ -1613,41 +1589,95 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
private List<Index> getSecondaryIndexes(
Connection connection, String databaseName, String tableName) throws
SQLException {
+ try {
+ return querySecondaryIndexes(
+ connection, databaseName, tableName, SECONDARY_INDEX_QUERY, true);
+ } catch (SQLException e) {
+ if (!isMissingTypeFullColumn(e)) {
+ throw e;
+ }
+ LOG.warn(
+ "ClickHouse server does not expose
system.data_skipping_indices.type_full; "
+ + "falling back to the legacy secondary-index query for {}.{}",
+ databaseName,
+ tableName);
+ return querySecondaryIndexes(
+ connection, databaseName, tableName, LEGACY_SECONDARY_INDEX_QUERY,
false);
+ }
+ }
+
+ private List<Index> querySecondaryIndexes(
+ Connection connection,
+ String databaseName,
+ String tableName,
+ String query,
+ boolean includesTypeFull)
+ throws SQLException {
List<Index> secondaryIndexes = new ArrayList<>();
- try (PreparedStatement preparedStatement =
- connection.prepareStatement(
- "SELECT name, type, expr, granularity FROM
system.data_skipping_indices "
- + "WHERE database = ? AND table = ? ORDER BY name")) {
+ try (PreparedStatement preparedStatement =
connection.prepareStatement(query)) {
preparedStatement.setString(1, databaseName);
preparedStatement.setString(2, tableName);
try (ResultSet resultSet = preparedStatement.executeQuery()) {
while (resultSet.next()) {
String name = resultSet.getString("name");
String type = resultSet.getString("type");
+ String parameterSource = includesTypeFull ?
resultSet.getString("type_full") : type;
+ String parameterSourceName = includesTypeFull ? "type_full" :
"legacy type";
String expression = resultSet.getString("expr");
long granularity = resultSet.getLong("granularity");
+ Index.IndexType indexType;
+ String[][] fields;
try {
- String[][] fields = parseIndexFields(expression);
- if (ArrayUtils.isEmpty(fields)) {
- continue;
- }
- // Only include granularity in properties when it differs from the
default,
- // so that indexes created without explicit granularity have empty
properties
- // and match the original creation state (avoids false
index-change diffs).
- Map<String, String> properties =
- granularity == DEFAULT_INDEX_GRANULARITY
- ? Map.of()
- : Map.of(GRANULARITY, String.valueOf(granularity));
- secondaryIndexes.add(
- Indexes.of(getClickHouseIndexType(type), name, fields,
properties));
+ indexType = getClickHouseIndexType(type);
+ fields = parseIndexFields(expression);
} catch (IllegalArgumentException e) {
LOG.warn(
- "Skip unsupported data skipping index {} for {}.{} with
expression {}",
+ "Skip unsupported data skipping index {} for {}.{} with type
{} "
+ + "(parameter metadata={}) and expression {}",
name,
databaseName,
tableName,
- expression);
+ type,
+ parameterSource,
+ expression,
+ e);
+ continue;
+ }
+ if (ArrayUtils.isEmpty(fields)) {
+ continue;
+ }
+
+ // Only include granularity in properties when it differs from the
default,
+ // so that indexes created without explicit granularity have empty
properties
+ // and match the original creation state (avoids false index-change
diffs).
+ Map<String, String> properties = new HashMap<>();
+ if (granularity != DEFAULT_INDEX_GRANULARITY) {
+ properties.put(GRANULARITY, String.valueOf(granularity));
}
+ Map<String, String> bloomFilterProperties;
+ try {
+ bloomFilterProperties =
+ parseBloomFilterPropertiesForQuery(
+ indexType, parameterSource, name, !includesTypeFull);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(
+ "Failed to load data skipping index '%s' from %s.%s with %s
'%s'"
+ .formatted(name, databaseName, tableName,
parameterSourceName, parameterSource),
+ e);
+ }
+ if (!includesTypeFull
+ && isParameterizedBloomFilterIndex(indexType)
+ && bloomFilterProperties.isEmpty()) {
+ LOG.warn(
+ "Legacy ClickHouse metadata does not expose bloom-filter
parameters for "
+ + "{} index '{}' on {}.{}; loaded Index.properties() is
incomplete",
+ type,
+ name,
+ databaseName,
+ tableName);
+ }
+ properties.putAll(bloomFilterProperties);
+ secondaryIndexes.add(Indexes.of(indexType, name, fields,
properties));
}
}
}
@@ -1655,6 +1685,109 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return secondaryIndexes;
}
+ private static boolean isMissingTypeFullColumn(SQLException exception) {
+ for (SQLException current = exception; current != null; current =
current.getNextException()) {
+ for (Throwable cause = current; cause != null; cause = cause.getCause())
{
+ String message = StringUtils.lowerCase(cause.getMessage());
+ if (message != null
+ && message.contains("type_full")
+ && (message.contains("unknown")
+ || message.contains("missing")
+ || message.contains("column"))) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Parses the positional parameters returned by ClickHouse in {@code
type_full} for the two
+ * parameterized bloom-filter data skipping indexes.
+ *
+ * @param indexType the mapped Gravitino index type
+ * @param typeFull the complete ClickHouse index type expression
+ * @param indexName the index name for validation messages
+ * @return the index properties, or an empty map for non-parameterized index
types
+ * @throws IllegalArgumentException if a supported index has malformed or
invalid parameters
+ */
+ @VisibleForTesting
+ static Map<String, String> parseBloomFilterProperties(
+ Index.IndexType indexType, String typeFull, String indexName) {
+ if (!isParameterizedBloomFilterIndex(indexType)) {
+ return Collections.emptyMap();
+ }
+
+ String expectedType =
+ indexType == Index.IndexType.DATA_SKIPPING_NGRAMBFV1
+ ? DATA_SKIPPING_NGRAMBFV1
+ : DATA_SKIPPING_TOKENBFV1;
+ String normalizedTypeFull = StringUtils.trimToEmpty(typeFull);
+ int paramsStart = normalizedTypeFull.indexOf('(');
+ int paramsEnd = normalizedTypeFull.lastIndexOf(')');
+ Preconditions.checkArgument(
+ paramsStart > 0 && paramsEnd == normalizedTypeFull.length() - 1,
+ "Invalid type_full '%s' for %s index '%s'",
+ typeFull,
+ expectedType,
+ indexName);
+ Preconditions.checkArgument(
+ StringUtils.equalsIgnoreCase(
+ expectedType, normalizedTypeFull.substring(0, paramsStart).trim()),
+ "type_full '%s' does not match %s index '%s'",
+ typeFull,
+ expectedType,
+ indexName);
+
+ String[] params = normalizedTypeFull.substring(paramsStart + 1,
paramsEnd).split(",", -1);
+ int expectedParamCount = indexType ==
Index.IndexType.DATA_SKIPPING_NGRAMBFV1 ? 4 : 3;
+ Preconditions.checkArgument(
+ params.length == expectedParamCount,
+ "Expected %s parameters for %s index '%s', but got %s in '%s'",
+ expectedParamCount,
+ expectedType,
+ indexName,
+ params.length,
+ typeFull);
+
+ Map<String, String> properties = new HashMap<>();
+ int paramIndex = 0;
+ if (indexType == Index.IndexType.DATA_SKIPPING_NGRAMBFV1) {
+ properties.put(
+ NGRAM_SIZE,
+ requireIntWithMin(params[paramIndex++], NGRAM_SIZE, expectedType,
indexName, 1));
+ }
+ properties.put(
+ BLOOM_FILTER_SIZE,
+ requireIntWithMin(params[paramIndex++], BLOOM_FILTER_SIZE,
expectedType, indexName, 1));
+ properties.put(
+ HASH_FUNCTIONS,
+ requireIntWithMin(params[paramIndex++], HASH_FUNCTIONS, expectedType,
indexName, 1));
+ properties.put(
+ RANDOM_SEED,
+ requireIntWithMin(params[paramIndex], RANDOM_SEED, expectedType,
indexName, 0));
+ return Map.copyOf(properties);
+ }
+
+ private static Map<String, String> parseBloomFilterPropertiesForQuery(
+ Index.IndexType indexType,
+ String parameterSource,
+ String indexName,
+ boolean allowBareLegacyType) {
+ if (!isParameterizedBloomFilterIndex(indexType)) {
+ return Collections.emptyMap();
+ }
+ if (allowBareLegacyType && !StringUtils.contains(parameterSource, "(")) {
+ return Collections.emptyMap();
+ }
+ return parseBloomFilterProperties(indexType, parameterSource, indexName);
+ }
+
+ private static boolean isParameterizedBloomFilterIndex(Index.IndexType
indexType) {
+ return indexType == Index.IndexType.DATA_SKIPPING_NGRAMBFV1
+ || indexType == Index.IndexType.DATA_SKIPPING_TOKENBFV1;
+ }
+
/**
* Maps a ClickHouse data skipping index type string to the corresponding
Gravitino {@link
* Index.IndexType}. Returns {@code DATA_SKIPPING_MINMAX} for blank/null
input (ClickHouse
@@ -1679,16 +1812,148 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return Index.IndexType.DATA_SKIPPING_BLOOM_FILTER;
case DATA_SKIPPING_SET:
return Index.IndexType.DATA_SKIPPING_SET;
+ case DATA_SKIPPING_NGRAMBFV1:
+ return Index.IndexType.DATA_SKIPPING_NGRAMBFV1;
+ case DATA_SKIPPING_TOKENBFV1:
+ return Index.IndexType.DATA_SKIPPING_TOKENBFV1;
default:
- // ClickHouse may return "set(N)" with parameter in some versions;
- // match on prefix to handle both "set" and "set(N)" formats.
+ // ClickHouse may return type with parameters in some versions (e.g.
"set(0)",
+ // "ngrambf_v1(3, 512, 3, 0)"). Match on prefix to handle both bare and
+ // parameterized formats.
if (rawType.startsWith(DATA_SKIPPING_SET + "(")) {
return Index.IndexType.DATA_SKIPPING_SET;
}
+ if (rawType.startsWith(DATA_SKIPPING_NGRAMBFV1 + "(")) {
+ return Index.IndexType.DATA_SKIPPING_NGRAMBFV1;
+ }
+ if (rawType.startsWith(DATA_SKIPPING_TOKENBFV1 + "(")) {
+ return Index.IndexType.DATA_SKIPPING_TOKENBFV1;
+ }
throw new IllegalArgumentException("Unsupported data skipping index
type: " + rawType);
}
}
+ /**
+ * Validates that a property value is an integer with a given minimum bound,
returning it as a
+ * string for DDL interpolation. Unlike {@link #resolveIntProperty(Map,
String, int, int)}, this
+ * method treats the value as required — a missing or blank value throws
{@link
+ * IllegalArgumentException} rather than returning a default. Used for
bloom-filter parameters
+ * (e.g. {@code bloom_filter_size}, {@code ngram_size} require ≥ 1;
{@code random_seed}
+ * requires ≥ 0).
+ *
+ * @param value the raw string value from the properties map
+ * @param paramName the parameter name for error messages
+ * @param indexType the index type name (e.g. "ngrambf_v1")
+ * @param indexName the index name for error messages
+ * @param minValue the minimum allowed value (inclusive)
+ * @return the validated value as a string
+ * @throws IllegalArgumentException if the value is null, blank, not an
integer, or below minValue
+ */
+ private static String requireIntWithMin(
+ String value, String paramName, String indexType, String indexName, int
minValue) {
+ Preconditions.checkArgument(
+ value != null && !value.isBlank(),
+ "%s is required for %s index '%s'",
+ paramName,
+ indexType,
+ indexName);
+ try {
+ int intVal = Integer.parseInt(value.strip());
+ Preconditions.checkArgument(
+ intVal >= minValue,
+ "%s must be >= %s for %s index '%s', but got '%s'",
+ paramName,
+ minValue,
+ indexType,
+ indexName,
+ value);
+ return String.valueOf(intVal);
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(
+ String.format(
+ "%s must be a valid integer for %s index '%s', but got '%s'",
+ paramName, indexType, indexName, value),
+ e);
+ }
+ }
+
+ /**
+ * Builds the full type clause for bloom-filter-based data skipping indexes,
combining the type
+ * name (e.g. "ngrambf_v1") with the validated parameter clause. Extracted
to eliminate
+ * duplication between the CREATE TABLE path ({@link #appendIndexesSql}) and
the ALTER TABLE ADD
+ * INDEX path ({@link #addIndexDefinition}).
+ *
+ * @param props the index properties map
+ * @param indexType the index type name (one of the {@code DATA_SKIPPING_*}
constants)
+ * @param indexName the index name, used in error messages
+ * @return the full type clause, e.g. "ngrambf_v1(3, 512, 3, 0)"
+ */
+ private static String buildBloomFilterTypeClause(
+ Map<String, String> props, String indexType, String indexName) {
+ return indexType + resolveBloomFilterParams(props, indexType, indexName);
+ }
+
+ /**
+ * Resolves bloom-filter-based data skipping index parameters from index
properties for {@code
+ * ngrambf_v1} and {@code tokenbf_v1} index types. Used by both CREATE TABLE
(via {@link
+ * Index#properties()}) and ALTER TABLE ADD INDEX (via {@link
+ * TableChange.AddIndex#getProperties()}).
+ *
+ * @param props the index properties map
+ * @param indexType "ngrambf_v1" or "tokenbf_v1" (determines whether
ngram_size is required)
+ * @param indexName the index name, used in error messages
+ * @return the DDL parameter clause, e.g. "(3, 512, 3, 0)" for ngrambf_v1
+ * @throws IllegalArgumentException if any required parameter is missing or
invalid
+ */
+ private static String resolveBloomFilterParams(
+ Map<String, String> props, String indexType, String indexName) {
+ String size =
+ requireIntWithMin(
+ props.get(BLOOM_FILTER_SIZE), "bloom_filter_size", indexType,
indexName, 1);
+ String hashFuncs =
+ requireIntWithMin(props.get(HASH_FUNCTIONS), "hash_functions",
indexType, indexName, 1);
+ String seed = requireIntWithMin(props.get(RANDOM_SEED), "random_seed",
indexType, indexName, 0);
+
+ if (DATA_SKIPPING_NGRAMBFV1.equals(indexType)) {
+ String ngramSize =
+ requireIntWithMin(props.get(NGRAM_SIZE), "ngram_size", indexType,
indexName, 1);
+ return String.format("(%s, %s, %s, %s)", ngramSize, size, hashFuncs,
seed);
+ }
+ return String.format("(%s, %s, %s)", size, hashFuncs, seed);
+ }
+
+ private String buildDataSkippingIndexDdl(
+ String indexName,
+ String fieldStr,
+ Index.IndexType indexType,
+ Map<String, String> properties) {
+ return buildDataSkippingIndexDdl(
+ indexName,
+ fieldStr,
+ resolveDataSkippingIndexTypeClause(indexType, properties, indexName),
+ resolveGranularity(properties, 1));
+ }
+
+ private String resolveDataSkippingIndexTypeClause(
+ Index.IndexType indexType, Map<String, String> properties, String
indexName) {
+ switch (indexType) {
+ case DATA_SKIPPING_MINMAX:
+ return DATA_SKIPPING_MINMAX_VALUE;
+ case DATA_SKIPPING_BLOOM_FILTER:
+ return DATA_SKIPPING_BLOOM_FILTER;
+ case DATA_SKIPPING_SET:
+ // SET index defaults to unlimited distinct values and supports an
optional upper bound.
+ return "set(" + resolveSetMaxValues(properties) + ")";
+ case DATA_SKIPPING_NGRAMBFV1:
+ return buildBloomFilterTypeClause(properties, DATA_SKIPPING_NGRAMBFV1,
indexName);
+ case DATA_SKIPPING_TOKENBFV1:
+ return buildBloomFilterTypeClause(properties, DATA_SKIPPING_TOKENBFV1,
indexName);
+ default:
+ throw new IllegalArgumentException(
+ "Gravitino ClickHouse doesn't support index : " + indexType);
+ }
+ }
+
private String buildDataSkippingIndexDdl(
String indexName, String fieldStr, String typeName, int granularity) {
Preconditions.checkArgument(
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
index 4ac96e8f31..0d2ba7bb4c 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
@@ -545,6 +545,96 @@ public class CatalogClickHouseClusterIT extends BaseIT {
autoIncrementFalseException.getMessage().contains("auto increment is
not supported"));
}
+ @Test
+ public void testNgrambfAndTokenbfIndexesOnCluster() {
+ String tableName = GravitinoITUtils.genRandomName("ck_cluster_skip_idx");
+ NameIdentifier tableIdentifier = NameIdentifier.of(schemaName, tableName);
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+ Map<String, String> ngramProperties =
+ Map.of(
+ "ngram_size", "3",
+ "bloom_filter_size", "512",
+ "hash_functions", "3",
+ "random_seed", "0");
+ Map<String, String> tokenProperties =
+ Map.of(
+ "bloom_filter_size", "256",
+ "hash_functions", "2",
+ "random_seed", "0",
+ "granularity", "4");
+
+ tableCatalog.createTable(
+ tableIdentifier,
+ createColumns(),
+ tableComment,
+ clusterMergeTreeProperties(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ getSortOrders("col_3"),
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_ngram",
+ new String[][] {{"col_3"}},
+ ngramProperties),
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_token",
+ new String[][] {{"col_3"}},
+ tokenProperties)
+ });
+
+ Table loaded = tableCatalog.loadTable(tableIdentifier);
+ assertIndexMetadata(
+ loaded.index(), "idx_ngram", Index.IndexType.DATA_SKIPPING_NGRAMBFV1,
ngramProperties);
+ assertIndexMetadata(
+ loaded.index(), "idx_token", Index.IndexType.DATA_SKIPPING_TOKENBFV1,
tokenProperties);
+
+ tableCatalog.alterTable(
+ tableIdentifier,
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_ngram_alter",
+ new String[][] {{"col_3"}},
+ ngramProperties),
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_token_alter",
+ new String[][] {{"col_3"}},
+ tokenProperties));
+
+ Table altered = tableCatalog.loadTable(tableIdentifier);
+ assertIndexMetadata(
+ altered.index(),
+ "idx_ngram_alter",
+ Index.IndexType.DATA_SKIPPING_NGRAMBFV1,
+ ngramProperties);
+ assertIndexMetadata(
+ altered.index(),
+ "idx_token_alter",
+ Index.IndexType.DATA_SKIPPING_TOKENBFV1,
+ tokenProperties);
+
+ tableCatalog.alterTable(
+ tableIdentifier,
+ TableChange.deleteIndex("idx_ngram", false),
+ TableChange.deleteIndex("idx_token", false),
+ TableChange.deleteIndex("idx_ngram_alter", false),
+ TableChange.deleteIndex("idx_token_alter", false));
+ }
+
+ private void assertIndexMetadata(
+ Index[] indexes, String name, Index.IndexType type, Map<String, String>
properties) {
+ Index index =
+ Arrays.stream(indexes)
+ .filter(candidate -> Objects.equals(name, candidate.name()))
+ .findFirst()
+ .orElseThrow(() -> new AssertionError("Missing index " + name));
+ Assertions.assertEquals(type, index.type());
+ Assertions.assertTrue(Arrays.deepEquals(new String[][] {{"col_3"}},
index.fieldNames()));
+ Assertions.assertEquals(properties, index.properties());
+ }
+
@Test
public void testDropTableOnCluster() {
String dropTableName =
GravitinoITUtils.genRandomName("ck_cluster_drop_tbl");
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
index fcc9c0bf24..72731536f4 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
@@ -3219,4 +3219,146 @@ public class CatalogClickHouseIT extends BaseIT {
normalizedCreateSql.contains("Enum16('low'=100,'medium'=200,'high'=300)"),
"SHOW CREATE TABLE should contain Enum16 definition: " + createSql);
}
+
+ @Test
+ @Tag("gravitino-docker-test")
+ void testCreateAndLoadTableWithNgrambfAndTokenbfIndexes() {
+ String tableName = GravitinoITUtils.genRandomName("ch_ngram_tokenbf_idx_");
+ NameIdentifier tableIdentifier = NameIdentifier.of(schemaName, tableName);
+ Map<String, String> properties = createProperties();
+ Column[] columns =
+ new Column[] {
+ Column.of(
+ CLICKHOUSE_COL_NAME1,
+ Types.IntegerType.get(),
+ "col_1_comment",
+ false,
+ false,
+ DEFAULT_VALUE_NOT_SET),
+ Column.of(
+ CLICKHOUSE_COL_NAME3,
+ Types.StringType.get(),
+ "col_3_comment",
+ false,
+ false,
+ DEFAULT_VALUE_NOT_SET),
+ };
+ SortOrder[] sortOrders = getSortOrders(CLICKHOUSE_COL_NAME1);
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+ Map<String, String> ngramProperties =
+ Map.of(
+ "ngram_size", "3",
+ "bloom_filter_size", "512",
+ "hash_functions", "3",
+ "random_seed", "0");
+ Map<String, String> tokenProperties =
+ Map.of(
+ "bloom_filter_size", "256",
+ "hash_functions", "2",
+ "random_seed", "0",
+ "granularity", "4");
+
+ // Create table with ngrambf_v1 index
+ Index[] indexes =
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_ngram",
+ new String[][] {{CLICKHOUSE_COL_NAME3}},
+ ngramProperties),
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_token",
+ new String[][] {{CLICKHOUSE_COL_NAME3}},
+ tokenProperties),
+ };
+ tableCatalog.createTable(
+ tableIdentifier,
+ columns,
+ table_comment,
+ properties,
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ sortOrders,
+ indexes);
+
+ // Load and verify round-trip
+ Table loaded = tableCatalog.loadTable(tableIdentifier);
+ Index[] loadedIndexes = loaded.index();
+ Assertions.assertNotNull(loadedIndexes);
+
+ Index loadedNgram =
+ Arrays.stream(loadedIndexes)
+ .filter(idx -> Objects.equals(idx.name(), "idx_ngram"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_NGRAMBFV1,
loadedNgram.type());
+ Assertions.assertArrayEquals(new String[][] {{CLICKHOUSE_COL_NAME3}},
loadedNgram.fieldNames());
+ Assertions.assertEquals(
+ Map.of(
+ "ngram_size", "3",
+ "bloom_filter_size", "512",
+ "hash_functions", "3",
+ "random_seed", "0"),
+ loadedNgram.properties());
+
+ Index loadedToken =
+ Arrays.stream(loadedIndexes)
+ .filter(idx -> Objects.equals(idx.name(), "idx_token"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_TOKENBFV1,
loadedToken.type());
+ Assertions.assertArrayEquals(new String[][] {{CLICKHOUSE_COL_NAME3}},
loadedToken.fieldNames());
+ Assertions.assertEquals(tokenProperties, loadedToken.properties());
+
+ // Verify that both new index types also work through ALTER TABLE ADD
INDEX.
+ tableCatalog.alterTable(
+ tableIdentifier,
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_ngram_alter",
+ new String[][] {{CLICKHOUSE_COL_NAME3}},
+ ngramProperties));
+ tableCatalog.alterTable(
+ tableIdentifier,
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_token_alter",
+ new String[][] {{CLICKHOUSE_COL_NAME3}},
+ tokenProperties));
+
+ Table altered = tableCatalog.loadTable(tableIdentifier);
+ Index alteredNgram =
+ Arrays.stream(altered.index())
+ .filter(idx -> Objects.equals(idx.name(), "idx_ngram_alter"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_NGRAMBFV1,
alteredNgram.type());
+ Assertions.assertEquals(
+ Map.of(
+ "ngram_size", "3",
+ "bloom_filter_size", "512",
+ "hash_functions", "3",
+ "random_seed", "0"),
+ alteredNgram.properties());
+
+ Index alteredToken =
+ Arrays.stream(altered.index())
+ .filter(idx -> Objects.equals(idx.name(), "idx_token_alter"))
+ .findFirst()
+ .orElseThrow();
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_TOKENBFV1,
alteredToken.type());
+ Assertions.assertEquals(tokenProperties, alteredToken.properties());
+
+ String createSql =
+ clickhouseService.executeQueryForResult(
+ String.format("SHOW CREATE TABLE `%s`.`%s`", schemaName,
tableName));
+ String normalizedCreateSql = createSql.replaceAll("\\s+", "");
+ Assertions.assertTrue(
+ normalizedCreateSql.contains("ngrambf_v1(3,512,3,0)"),
+ "SHOW CREATE TABLE should retain ngrambf_v1 parameters: " + createSql);
+ Assertions.assertTrue(
+ normalizedCreateSql.contains("tokenbf_v1(256,2,0)"),
+ "SHOW CREATE TABLE should retain tokenbf_v1 parameters: " + createSql);
+ }
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperations.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperations.java
index 5f3fdbd8ea..9ff657b9c3 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperations.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperations.java
@@ -1736,15 +1736,543 @@ public class TestClickHouseTableOperations extends
TestClickHouse {
Assertions.assertEquals(IndexType.DATA_SKIPPING_SET,
ops.getClickHouseIndexType("set(0)"));
Assertions.assertEquals(IndexType.DATA_SKIPPING_SET,
ops.getClickHouseIndexType("set(100)"));
+ // ngrambf_v1 and tokenbf_v1 bloom filter data skipping indexes
+ Assertions.assertEquals(
+ IndexType.DATA_SKIPPING_NGRAMBFV1,
ops.getClickHouseIndexType("ngrambf_v1"));
+ Assertions.assertEquals(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
ops.getClickHouseIndexType("tokenbf_v1"));
+
// Blank/null defaults to MINMAX
Assertions.assertEquals(IndexType.DATA_SKIPPING_MINMAX,
ops.getClickHouseIndexType(""));
Assertions.assertEquals(IndexType.DATA_SKIPPING_MINMAX,
ops.getClickHouseIndexType(null));
+ // Parameterized formats (C2 prefix matching)
+ Assertions.assertEquals(
+ IndexType.DATA_SKIPPING_NGRAMBFV1,
ops.getClickHouseIndexType("ngrambf_v1(3, 512, 3, 0)"));
+ Assertions.assertEquals(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
ops.getClickHouseIndexType("tokenbf_v1(256, 2, 0)"));
+
// Unsupported type
Assertions.assertThrows(
IllegalArgumentException.class, () ->
ops.getClickHouseIndexType("unknown_type"));
}
+ @Test
+ public void testParseBloomFilterProperties() {
+ Assertions.assertEquals(
+ Map.of(
+ "ngram_size", "3",
+ "bloom_filter_size", "512",
+ "hash_functions", "3",
+ "random_seed", "0"),
+ ClickHouseTableOperations.parseBloomFilterProperties(
+ IndexType.DATA_SKIPPING_NGRAMBFV1, "ngrambf_v1(3, 512, 3, 0)",
"idx_ngram"));
+ Assertions.assertEquals(
+ Map.of(
+ "bloom_filter_size", "256",
+ "hash_functions", "2",
+ "random_seed", "1"),
+ ClickHouseTableOperations.parseBloomFilterProperties(
+ IndexType.DATA_SKIPPING_TOKENBFV1, " tokenbf_v1(256,2,1) ",
"idx_token"));
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ClickHouseTableOperations.parseBloomFilterProperties(
+ IndexType.DATA_SKIPPING_NGRAMBFV1, "ngrambf_v1(3,512,3)",
"idx_bad"));
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ClickHouseTableOperations.parseBloomFilterProperties(
+ IndexType.DATA_SKIPPING_TOKENBFV1, "tokenbf_v1(256,2,-1)",
"idx_bad"));
+ }
+
+ @Test
+ public void testCreateTableWithNgrambfAndTokenbfIndexes() {
+ TestableClickHouseTableOperations ops = new
TestableClickHouseTableOperations();
+ ops.initialize(
+ null,
+ new ClickHouseExceptionConverter(),
+ new ClickHouseTypeConverter(),
+ new ClickHouseColumnDefaultValueConverter(),
+ new HashMap<>());
+
+ JdbcColumn[] cols =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("c1")
+ .withType(Types.LongType.get())
+ .withNullable(true)
+ .build(),
+ JdbcColumn.builder()
+ .withName("c2")
+ .withType(Types.StringType.get())
+ .withNullable(true)
+ .build(),
+ };
+
+ // ngrambf_v1 with full parameters
+ Index[] indexes1 =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_ngram",
+ new String[][] {{"c2"}},
+ Map.of(
+ "ngram_size",
+ "3",
+ "bloom_filter_size",
+ "512",
+ "hash_functions",
+ "3",
+ "random_seed",
+ "1")),
+ };
+
+ String sql1 =
+ ops.buildCreateSql(
+ "t_ngram",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ indexes1,
+ ClickHouseUtils.getSortOrders("c1"));
+
+ Assertions.assertTrue(
+ sql1.contains("INDEX `idx_ngram` `c2` TYPE ngrambf_v1(3, 512, 3, 1)
GRANULARITY 1"),
+ "DDL should contain ngrambf_v1 with parameters: " + sql1);
+
+ // tokenbf_v1 with full parameters
+ Index[] indexes2 =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_token",
+ new String[][] {{"c2"}},
+ Map.of("bloom_filter_size", "256", "hash_functions", "2",
"random_seed", "0")),
+ };
+
+ String sql2 =
+ ops.buildCreateSql(
+ "t_token",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ indexes2,
+ ClickHouseUtils.getSortOrders("c1"));
+
+ Assertions.assertTrue(
+ sql2.contains("INDEX `idx_token` `c2` TYPE tokenbf_v1(256, 2, 0)
GRANULARITY 1"),
+ "DDL should contain tokenbf_v1 with parameters: " + sql2);
+
+ // ngrambf_v1 with custom GRANULARITY from properties
+ Index[] indexes3 =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_ngram_g",
+ new String[][] {{"c2"}},
+ Map.of(
+ "ngram_size",
+ "4",
+ "bloom_filter_size",
+ "1024",
+ "hash_functions",
+ "3",
+ "random_seed",
+ "1",
+ "granularity",
+ "8")),
+ };
+
+ String sql3 =
+ ops.buildCreateSql(
+ "t_ngram_g",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ indexes3,
+ ClickHouseUtils.getSortOrders("c1"));
+
+ Assertions.assertTrue(
+ sql3.contains("INDEX `idx_ngram_g` `c2` TYPE ngrambf_v1(4, 1024, 3, 1)
GRANULARITY 8"),
+ "DDL should contain custom GRANULARITY: " + sql3);
+ }
+
+ @Test
+ public void testNgrambfTokenbfMissingParameters() {
+ TestableClickHouseTableOperations ops = new
TestableClickHouseTableOperations();
+ ops.initialize(
+ null,
+ new ClickHouseExceptionConverter(),
+ new ClickHouseTypeConverter(),
+ new ClickHouseColumnDefaultValueConverter(),
+ new HashMap<>());
+
+ JdbcColumn[] cols =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("c1")
+ .withType(Types.LongType.get())
+ .withNullable(true)
+ .build(),
+ JdbcColumn.builder()
+ .withName("c2")
+ .withType(Types.StringType.get())
+ .withNullable(true)
+ .build(),
+ };
+
+ // ngrambf_v1 without ngram_size — should throw
+ Index[] noNgramSize =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_bad",
+ new String[][] {{"c2"}},
+ Map.of("bloom_filter_size", "512", "hash_functions", "3",
"random_seed", "1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ noNgramSize,
+ ClickHouseUtils.getSortOrders("c1")));
+
+ // tokenbf_v1 without bloom_filter_size — should throw
+ Index[] noBloomSize =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_bad2",
+ new String[][] {{"c2"}},
+ Map.of("hash_functions", "3", "random_seed", "1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad2",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ noBloomSize,
+ ClickHouseUtils.getSortOrders("c1")));
+
+ // ngrambf_v1 without hash_functions — should throw
+ Index[] noHashFuncs =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_bad3",
+ new String[][] {{"c2"}},
+ Map.of(
+ "ngram_size", "3",
+ "bloom_filter_size", "512",
+ "random_seed", "1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad3",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ noHashFuncs,
+ ClickHouseUtils.getSortOrders("c1")));
+
+ // tokenbf_v1 without hash_functions — should throw
+ Index[] tokenbfNoHash =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_bad4",
+ new String[][] {{"c2"}},
+ Map.of("bloom_filter_size", "256", "random_seed", "1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad4",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ tokenbfNoHash,
+ ClickHouseUtils.getSortOrders("c1")));
+
+ // tokenbf_v1 without random_seed — should throw
+ Index[] noSeed =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_bad5",
+ new String[][] {{"c2"}},
+ Map.of("bloom_filter_size", "256", "hash_functions", "2")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad5",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ noSeed,
+ ClickHouseUtils.getSortOrders("c1")));
+ }
+
+ @Test
+ public void testNgrambfTokenbfInvalidParameterValues() {
+ TestableClickHouseTableOperations ops = new
TestableClickHouseTableOperations();
+ ops.initialize(
+ null,
+ new ClickHouseExceptionConverter(),
+ new ClickHouseTypeConverter(),
+ new ClickHouseColumnDefaultValueConverter(),
+ new HashMap<>());
+
+ JdbcColumn[] cols =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("c1")
+ .withType(Types.LongType.get())
+ .withNullable(true)
+ .build(),
+ JdbcColumn.builder()
+ .withName("c2")
+ .withType(Types.StringType.get())
+ .withNullable(true)
+ .build(),
+ };
+
+ // Non-numeric bloom_filter_size should throw
+ Index[] nonNumericSize =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_bad",
+ new String[][] {{"c2"}},
+ Map.of(
+ "bloom_filter_size", "abc",
+ "hash_functions", "3",
+ "random_seed", "1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ nonNumericSize,
+ ClickHouseUtils.getSortOrders("c1")));
+
+ // Negative bloom_filter_size should throw
+ Index[] negativeSize =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_bad2",
+ new String[][] {{"c2"}},
+ Map.of(
+ "bloom_filter_size", "-1",
+ "hash_functions", "3",
+ "random_seed", "1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad2",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ negativeSize,
+ ClickHouseUtils.getSortOrders("c1")));
+
+ // Zero bloom_filter_size should throw (>= 1 required)
+ Index[] zeroSize =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_bad3",
+ new String[][] {{"c2"}},
+ Map.of(
+ "bloom_filter_size", "0",
+ "hash_functions", "3",
+ "random_seed", "1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad3",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ zeroSize,
+ ClickHouseUtils.getSortOrders("c1")));
+
+ // Non-numeric hash_functions should throw
+ Index[] nonNumericHash =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_bad4",
+ new String[][] {{"c2"}},
+ Map.of(
+ "ngram_size", "3",
+ "bloom_filter_size", "512",
+ "hash_functions", "abc",
+ "random_seed", "1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad4",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ nonNumericHash,
+ ClickHouseUtils.getSortOrders("c1")));
+
+ // Negative random_seed should throw (>= 0 required)
+ Index[] negativeSeed =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_bad5",
+ new String[][] {{"c2"}},
+ Map.of(
+ "bloom_filter_size", "256",
+ "hash_functions", "3",
+ "random_seed", "-1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad5",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ negativeSeed,
+ ClickHouseUtils.getSortOrders("c1")));
+
+ // Non-numeric ngram_size should throw (ngrambf_v1 specific)
+ Index[] nonNumericNgram =
+ new Index[] {
+ Indexes.of(
+ IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_bad6",
+ new String[][] {{"c2"}},
+ Map.of(
+ "ngram_size", "xyz",
+ "bloom_filter_size", "512",
+ "hash_functions", "3",
+ "random_seed", "1")),
+ };
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "t_bad6",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[0],
+ Distributions.NONE,
+ nonNumericNgram,
+ ClickHouseUtils.getSortOrders("c1")));
+ }
+
+ @Test
+ public void testAlterTableAddIndexWithNgrambfAndTokenbf() {
+ StubClickHouseTableOperations ops = new StubClickHouseTableOperations();
+ ops.initialize(
+ null,
+ new ClickHouseExceptionConverter(),
+ new ClickHouseTypeConverter(),
+ new ClickHouseColumnDefaultValueConverter(),
+ new HashMap<>());
+ ops.setTable(buildStubTable());
+
+ // ALTER TABLE ADD INDEX for ngrambf_v1 with full properties
+ String ngrambfSql =
+ ops.buildAlterSql(
+ "db",
+ "tbl",
+ new TableChange[] {
+ TableChange.addIndex(
+ IndexType.DATA_SKIPPING_NGRAMBFV1,
+ "idx_ngram",
+ new String[][] {{"c2"}},
+ Map.of(
+ "ngram_size", "3",
+ "bloom_filter_size", "512",
+ "hash_functions", "3",
+ "random_seed", "0"))
+ });
+ Assertions.assertTrue(
+ ngrambfSql.contains(
+ "ADD INDEX `idx_ngram` `c2` TYPE ngrambf_v1(3, 512, 3, 0)
GRANULARITY 1"),
+ "DDL should contain ngrambf_v1 with parameters: " + ngrambfSql);
+
+ // ALTER TABLE ADD INDEX for tokenbf_v1 with full properties
+ String tokenbfSql =
+ ops.buildAlterSql(
+ "db",
+ "tbl",
+ new TableChange[] {
+ TableChange.addIndex(
+ IndexType.DATA_SKIPPING_TOKENBFV1,
+ "idx_token",
+ new String[][] {{"c2"}},
+ Map.of(
+ "bloom_filter_size", "256",
+ "hash_functions", "2",
+ "random_seed", "1",
+ "granularity", "4"))
+ });
+ Assertions.assertTrue(
+ tokenbfSql.contains("ADD INDEX `idx_token` `c2` TYPE tokenbf_v1(256,
2, 1) GRANULARITY 4"),
+ "DDL should contain tokenbf_v1 with custom GRANULARITY: " +
tokenbfSql);
+ }
+
@Test
public void testAlterTableNullabilityValidationFails() {
StubClickHouseTableOperations ops = new StubClickHouseTableOperations();
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
index 76d04b4525..8dfe14b204 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsUnit.java
@@ -23,6 +23,7 @@ import static
org.apache.gravitino.catalog.clickhouse.ClickHouseUtils.getSortOrd
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
+import java.sql.SQLException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -32,6 +33,7 @@ import
org.apache.gravitino.catalog.clickhouse.converter.ClickHouseColumnDefault
import
org.apache.gravitino.catalog.clickhouse.converter.ClickHouseExceptionConverter;
import
org.apache.gravitino.catalog.clickhouse.converter.ClickHouseTypeConverter;
import org.apache.gravitino.catalog.jdbc.JdbcColumn;
+import org.apache.gravitino.exceptions.GravitinoRuntimeException;
import org.apache.gravitino.exceptions.NoSuchTableException;
import org.apache.gravitino.rel.expressions.FunctionExpression;
import org.apache.gravitino.rel.expressions.NamedReference;
@@ -474,4 +476,138 @@ public class TestClickHouseTableOperationsUnit {
IllegalArgumentException.class, () ->
newOps().callGenerateCreateTableSql(properties));
Assertions.assertTrue(exception.getMessage().contains("balanced"));
}
+
+ @Test
+ void testGetIndexesFailsOnMalformedParameterizedIndexMetadata() throws
Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet secondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryRs.next()).thenReturn(true, false);
+ Mockito.when(secondaryStmt.executeQuery()).thenReturn(secondaryRs);
+ Mockito.when(secondaryRs.getString("name")).thenReturn("idx_bad");
+ Mockito.when(secondaryRs.getString("type")).thenReturn("ngrambf_v1");
+ Mockito.when(secondaryRs.getString("type_full")).thenReturn(null);
+ Mockito.when(secondaryRs.getString("expr")).thenReturn("col_1");
+ Mockito.when(secondaryRs.getLong("granularity")).thenReturn(1L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ IllegalArgumentException exception =
+ Assertions.assertThrows(
+ IllegalArgumentException.class, () ->
ops.callGetIndexes(connection, "db", "tbl"));
+ Assertions.assertTrue(exception.getMessage().contains("idx_bad"));
+ Assertions.assertTrue(exception.getMessage().contains("type_full"));
+ }
+
+ @Test
+ void testGetIndexesSkipsUnsupportedExpressionForParameterizedIndex() throws
Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet secondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryRs.next()).thenReturn(true, true, false);
+ Mockito.when(secondaryStmt.executeQuery()).thenReturn(secondaryRs);
+ Mockito.when(secondaryRs.getString("name")).thenReturn("idx_bad_expr",
"idx_valid");
+ Mockito.when(secondaryRs.getString("type")).thenReturn("ngrambf_v1",
"tokenbf_v1");
+ Mockito.when(secondaryRs.getString("type_full"))
+ .thenReturn("ngrambf_v1(3, 512, 3, 0)", "tokenbf_v1(256, 2, 0)");
+ Mockito.when(secondaryRs.getString("expr")).thenReturn("cityHash64(col_1)
% 16", "col_2");
+ Mockito.when(secondaryRs.getLong("granularity")).thenReturn(1L, 1L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ List<Index> indexes = ops.callGetIndexes(connection, "db", "tbl");
+
+ Assertions.assertEquals(1, indexes.size());
+ Assertions.assertEquals("idx_valid", indexes.get(0).name());
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_TOKENBFV1,
indexes.get(0).type());
+ Assertions.assertArrayEquals(new String[][] {{"col_2"}},
indexes.get(0).fieldNames());
+ Assertions.assertEquals(
+ Map.of(
+ "bloom_filter_size", "256",
+ "hash_functions", "2",
+ "random_seed", "0"),
+ indexes.get(0).properties());
+ }
+
+ @Test
+ void testGetIndexesFallsBackWhenTypeFullColumnIsMissing() throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement modernSecondaryStmt =
Mockito.mock(PreparedStatement.class);
+ PreparedStatement legacySecondaryStmt =
Mockito.mock(PreparedStatement.class);
+ ResultSet legacySecondaryRs = Mockito.mock(ResultSet.class);
+
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(modernSecondaryStmt.executeQuery())
+ .thenThrow(new SQLException("Unknown identifier 'type_full'"));
+
Mockito.when(legacySecondaryStmt.executeQuery()).thenReturn(legacySecondaryRs);
+ Mockito.when(legacySecondaryRs.next()).thenReturn(true, false);
+ Mockito.when(legacySecondaryRs.getString("name")).thenReturn("idx_legacy");
+
Mockito.when(legacySecondaryRs.getString("type")).thenReturn("ngrambf_v1(3,
512, 3, 0)");
+ Mockito.when(legacySecondaryRs.getString("expr")).thenReturn("col_1");
+ Mockito.when(legacySecondaryRs.getLong("granularity")).thenReturn(1L);
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(modernSecondaryStmt)
+ .thenReturn(legacySecondaryStmt);
+
+ List<Index> indexes = ops.callGetIndexes(connection, "db", "tbl");
+
+ Assertions.assertEquals(1, indexes.size());
+ Assertions.assertEquals(Index.IndexType.DATA_SKIPPING_NGRAMBFV1,
indexes.get(0).type());
+ Assertions.assertEquals(
+ Map.of(
+ "ngram_size", "3",
+ "bloom_filter_size", "512",
+ "hash_functions", "3",
+ "random_seed", "0"),
+ indexes.get(0).properties());
+ }
+
+ @Test
+ void testGetIndexesDoesNotFallbackForOtherSqlErrors() throws Exception {
+ ExposedClickHouseTableOperations ops = newOps();
+
+ PreparedStatement primaryKeyStmt = Mockito.mock(PreparedStatement.class);
+ ResultSet primaryKeyRs = Mockito.mock(ResultSet.class);
+ PreparedStatement secondaryStmt = Mockito.mock(PreparedStatement.class);
+ Mockito.when(primaryKeyRs.next()).thenReturn(false);
+ Mockito.when(primaryKeyStmt.executeQuery()).thenReturn(primaryKeyRs);
+ Mockito.when(secondaryStmt.executeQuery())
+ .thenThrow(new SQLException("Connection reset by peer"));
+
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString()))
+ .thenReturn(primaryKeyStmt)
+ .thenReturn(secondaryStmt);
+
+ GravitinoRuntimeException exception =
+ Assertions.assertThrows(
+ GravitinoRuntimeException.class, () ->
ops.callGetIndexes(connection, "db", "tbl"));
+ Assertions.assertTrue(exception.getCause() instanceof SQLException);
+ Mockito.verify(connection,
Mockito.times(2)).prepareStatement(Mockito.anyString());
+ }
}
diff --git a/docs/jdbc-clickhouse-catalog.md b/docs/jdbc-clickhouse-catalog.md
index 2371ea2bcc..90af2cc5b2 100644
--- a/docs/jdbc-clickhouse-catalog.md
+++ b/docs/jdbc-clickhouse-catalog.md
@@ -172,7 +172,7 @@ See [Manage Catalogs and
Schemas](./manage-catalogs-and-schemas.md#schema-operat
| Mapping | Gravitino table maps to a ClickHouse table
|
| Engines | **MergeTree family** (`MergeTree` default,
`ReplacingMergeTree`, `SummingMergeTree`, `AggregatingMergeTree`,
`CollapsingMergeTree`, `VersionedCollapsingMergeTree`, `GraphiteMergeTree`):
fully supported, data persists across restarts. **Log family** (`TinyLog`,
`StripeLog`, `Log`): supported, data and table definition persist across
restarts. **`Null`**: supported, table persists, data is always discarded by
design. **`Set`**: supported, table definition persists. [...]
| Ordering/Partition | MergeTree-family requires exactly one `ORDER BY`
column; only single-column identity `PARTITION BY` is supported on MergeTree
engines. Other engines reject `ORDER BY`/`PARTITION BY`.
|
-| Indexes | Primary key; data-skipping indexes
`DATA_SKIPPING_MINMAX`, `DATA_SKIPPING_BLOOM_FILTER`, and `DATA_SKIPPING_SET`
(configurable granularity via `Index.properties()`).
[...]
+| Indexes | Primary key; data-skipping indexes
`DATA_SKIPPING_MINMAX`, `DATA_SKIPPING_BLOOM_FILTER`, `DATA_SKIPPING_SET`,
`DATA_SKIPPING_NGRAMBFV1`, and `DATA_SKIPPING_TOKENBFV1` (configurable
granularity via `Index.properties()`).
[...]
| Distribution | Gravitino enforces `Distributions.NONE`; no custom
distribution strategies.
|
| Column defaults | Supported.
|
| Unsupported | Engine change after creation; removing table
properties; auto-increment columns.
|
@@ -243,9 +243,13 @@ If you need Gravitino to manage an existing cluster
database or table, recreate
- `DATA_SKIPPING_MINMAX` (default `GRANULARITY 1`)
- `DATA_SKIPPING_BLOOM_FILTER` (default `GRANULARITY 1`)
- `DATA_SKIPPING_SET` (default `GRANULARITY 1`, plus configurable `set(N)`
max values)
+ - `DATA_SKIPPING_NGRAMBFV1` (`GRANULARITY` customizable via
`Index.properties()`, default 1; requires `ngram_size`, `bloom_filter_size`,
`hash_functions`, `random_seed` in `Index.properties()`)
+ - `DATA_SKIPPING_TOKENBFV1` (`GRANULARITY` customizable via
`Index.properties()`, default 1; requires `bloom_filter_size`,
`hash_functions`, `random_seed` in `Index.properties()`)
Custom `GRANULARITY` can be specified via the `Index.properties()` API (key
`granularity`, value must be a positive integer). For `DATA_SKIPPING_SET`, the
max unique values can be configured via `set_max_values` (non-negative
integer). If not specified, the defaults above apply.
+ On ClickHouse versions without `system.data_skipping_indices.type_full`,
Gravitino falls back to the legacy metadata query. If the legacy `type` value
does not include the bloom-filter parameters, the index type and fields are
preserved but the required parameter properties cannot be reconstructed;
provide the properties explicitly before recreating the table.
+
### Partitioning, Sorting, and Distribution
- `ORDER BY`: required for MergeTree-family engines and only columns identity
are supported;
@@ -350,7 +354,7 @@ Supported:
- Rename column.
- Update column type/comment/default/position/nullability.
- Delete columns (with `IF EXISTS` support).
-- Add data-skipping indexes with custom `GRANULARITY` and `set(N)` via
`Index.properties()`; drop data-skipping indexes. Adding/dropping primary key
is not supported.
+- Add and drop data-skipping indexes; configure custom `GRANULARITY`,
`set(N)`, and `ngrambf_v1`/`tokenbf_v1` Bloom-filter parameters via
`Index.properties()`. Adding/dropping primary key is not supported.
- Update table comment.
Unsupported:
diff --git a/docs/open-api/indexes.yaml b/docs/open-api/indexes.yaml
index c4e1a00cf2..af250b8967 100644
--- a/docs/open-api/indexes.yaml
+++ b/docs/open-api/indexes.yaml
@@ -38,6 +38,8 @@ components:
- "data_skipping_minmax"
- "data_skipping_bloom_filter"
- "data_skipping_set"
+ - "data_skipping_ngrambfv1"
+ - "data_skipping_tokenbfv1"
name:
type: string
description: The name of the index
