This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 23f871f805 [#11802] fix(clickhouse): support user-defined GRANULARITY
and validate shard key type (#11806)
23f871f805 is described below
commit 23f871f8055ac8e93f186a7c7a98e0d5e3c4e0b1
Author: StormSpirit <[email protected]>
AuthorDate: Mon Jul 20 20:15:07 2026 +0800
[#11802] fix(clickhouse): support user-defined GRANULARITY and validate
shard key type (#11806)
### What changes were proposed in this pull request?
#### GRANULARITY round-trip (`ClickHouseTableOperations.java`)
- **Read path**: `getSecondaryIndexes()` queries the `granularity`
column from `system.data_skipping_indices` and stores it in
`Index.properties()`. Default values (1) are omitted so that indexes
created without explicit GRANULARITY have empty properties and match the
original creation state (avoids false index-change diffs).
- **Write path**: `appendIndexesSql()` passes
`resolveGranularity(index.properties(), 1)` to
`buildDataSkippingIndexDdl`, replacing the old hardcoded defaults.
#### Shard key validation (`ClickHouseTableOperations.java`)
- Extracted `validateShardKeyColumns()` shared method covering column
existence, nullable check, and integer type check. Called from both
`create()` (empty columns path) and `handleDistributeTable()` (explicit
columns path).
- Added `isIntegerType()`: accepts Gravitino `IntegralType` (Int8-Int64,
UInt8-UInt64) and ClickHouse wide integers (Int128/256, UInt128/256)
mapped to `ExternalType`, using regex for future-proof coverage.
- Function-wrapped shard keys (e.g. `cityHash64(string_col)`) skip
type/nullability checks — only column existence is verified.
- **Empty columns path**: New `create()` override fetches remote table
columns via `fetchRemoteColumns()` and validates shard keys before
`super.create()`, covering the case where `handleDistributeTable` is
skipped (distributed table using `AS remote_table`).
#### `getColumns()` override (`ClickHouseTableOperations.java`)
- Parent `JdbcTableOperations.getColumns()` uses
`connection.getSchema()` as the schema pattern, which is incorrect for
ClickHouse when the target database differs from the connection's
default database. The override passes `databaseName` directly to
`metaData.getColumns()`.
### Why are the changes needed?
ClickHouse data skipping indexes support a `GRANULARITY` parameter, but
the existing code hardcoded default values. Users who create indexes
with custom GRANULARITY would lose that information on round-trip.
Distributed table shard keys in ClickHouse must be non-nullable integer
columns. Previously this was documented in a TODO comment — invalid
shard keys reached the ClickHouse server and produced opaque error
messages instead of clear client-side validation.
### Does this PR introduce any user-facing change?
1. Custom GRANULARITY for data skipping indexes via `Index.properties()`
— round-trip (create → load → recreate) preserves non-default values.
2. Nullable or non-integer shard keys are rejected at the Gravitino
client level with clear `IllegalArgumentException` messages.
3. Int128/UInt128/Int256/UInt256 columns are now accepted as shard keys.
### How was this patch tested?
**Unit tests** (`-PskipITs`):
`TestClickHouseTableOperations`:
- `testGranularityNormalizedToCanonicalForm` — leading zero / whitespace
trimming
`TestClickHouseTableOperationsCluster`:
- `testIndexNonNumericGranularityRejected`,
`testIndexZeroGranularityRejected`,
`testIndexZeroAfterParsingGranularityRejected`,
`testIndexNegativeGranularityRejected` — invalid GRANULARITY
- `testShardingKeyNullableColumnRejected`,
`testShardingKeyNonIntegerColumnRejected` — shard key rejected
- `testShardingKeyFunctionWithNonIntegerColumnAccepted` —
function-wrapped key
- `testShardingKeyInt128ColumnAccepted`,
`testShardingKeyUInt128ColumnAccepted` — wide integer keys
- `testAlterTableAddIndexDefaultGranularity`,
`testAlterTableAddIndexWithClusterProperties`,
`testAlterTableAddIndexWithNullProperties` — ALTER TABLE ADD INDEX
**Docker integration tests** (`-PskipDockerTests=false`):
`CatalogClickHouseIT` (35 tests, 0 failures):
- `testCreateAndLoadWithCustomGranularity`,
`testCreateAndLoadWithDefaultGranularity`,
`testLoadSqlCreatedTableWithCustomGranularity` — GRANULARITY round-trip
- `testAlterTableAddIndexUsesDefaultGranularity` — ALTER TABLE ADD INDEX
end-to-end
`CatalogClickHouseClusterIT` (17 tests, 0 failures):
- `testDistributedTableNullableShardKeyRejected`,
`testDistributedTableNonIntegerShardKeyRejected` — shard key rejected
- `testDistributedTableIntegerShardKeyAccepted`,
`testDistributedTableWideIntegerShardKeyAccepted`,
`testDistributedTableFunctionShardKeyAccepted` — shard key accepted
Run commands:
```bash
./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test -PskipITs
./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test --tests
"CatalogClickHouseIT" -PskipDockerTests=false
./gradlew :catalogs-contrib:catalog-jdbc-clickhouse:test --tests
"CatalogClickHouseClusterIT" -PskipDockerTests=false
```
Closes #11802.
---------
Signed-off-by: jiangxt2 <[email protected]>
---
.../operations/ClickHouseTableOperations.java | 172 +++++++++-
.../test/CatalogClickHouseClusterIT.java | 252 ++++++++++++++
.../integration/test/CatalogClickHouseIT.java | 169 +++++++++
.../operations/TestClickHouseTableOperations.java | 57 ++++
.../TestClickHouseTableOperationsCluster.java | 379 +++++++++++++++++++++
5 files changed, 1013 insertions(+), 16 deletions(-)
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
index befc2bc761..70b8f06346 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
@@ -42,6 +42,7 @@ import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.regex.Matcher;
@@ -63,6 +64,7 @@ import org.apache.gravitino.catalog.jdbc.JdbcTable;
import org.apache.gravitino.catalog.jdbc.operation.JdbcTableOperations;
import org.apache.gravitino.catalog.jdbc.utils.JdbcConnectorUtils;
import org.apache.gravitino.exceptions.NoSuchTableException;
+import org.apache.gravitino.exceptions.TableAlreadyExistsException;
import org.apache.gravitino.rel.Column;
import org.apache.gravitino.rel.TableChange;
import org.apache.gravitino.rel.expressions.Expression;
@@ -79,11 +81,16 @@ import
org.apache.gravitino.rel.expressions.transforms.Transform;
import org.apache.gravitino.rel.expressions.transforms.Transforms;
import org.apache.gravitino.rel.indexes.Index;
import org.apache.gravitino.rel.indexes.Indexes;
+import org.apache.gravitino.rel.types.Type;
+import org.apache.gravitino.rel.types.Types;
public class ClickHouseTableOperations extends JdbcTableOperations {
private static final String CLICKHOUSE_NOT_SUPPORT_NESTED_COLUMN_MSG =
"Clickhouse does not support nested column names.";
+ /** Default GRANULARITY for data skipping indexes, matching ClickHouse's own
default. */
+ private static final long DEFAULT_INDEX_GRANULARITY = 1;
+
private static final Pattern ORDER_BY_PATTERN =
Pattern.compile(
"(?is)\\bORDER\\s+BY\\s*(.+?)(?=\\bPARTITION\\s+BY\\b|\\bPRIMARY\\s+KEY\\b|\\bSAMPLE\\s+BY\\b|\\bTTL\\b|\\bSETTINGS\\b|\\bCOMMENT\\b|$)");
@@ -95,6 +102,8 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
private static final Pattern DISTRIBUTED_ENGINE_PATTERN =
Pattern.compile(
"(?i)^Distributed\\(([^,]+),\\s*([^,]+),\\s*([^,]+),\\s*(.+)\\)$",
Pattern.DOTALL);
+ /** Matches ClickHouse wide integer type names (Int128/256, UInt128/256, and
future variants). */
+ private static final Pattern WIDE_INTEGER_PATTERN =
Pattern.compile("^U?INT\\d+$");
private static final String QUERY_INDEXES_SQL =
"""
@@ -112,6 +121,110 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
ORDER BY COLUMN_NAME
""";
+ @Override
+ public void create(
+ String databaseName,
+ String tableName,
+ JdbcColumn[] columns,
+ String comment,
+ Map<String, String> properties,
+ Transform[] partitioning,
+ Distribution distribution,
+ Index[] indexes,
+ SortOrder[] sortOrders)
+ throws TableAlreadyExistsException {
+ // When columns are provided, delegate directly to the parent
implementation.
+ if (ArrayUtils.isNotEmpty(columns)) {
+ super.create(
+ databaseName,
+ tableName,
+ columns,
+ comment,
+ properties,
+ partitioning,
+ distribution,
+ indexes,
+ sortOrders);
+ return;
+ }
+
+ // When columns is empty (distributed table using AS remote_table), the
shard key validation
+ // in handleDistributeTable is skipped. Fetch remote table columns and
validate here.
+ Map<String, String> props =
+ MapUtils.isNotEmpty(properties) ? properties : Collections.emptyMap();
+ String engine = props.get(GRAVITINO_ENGINE_KEY);
+ if (StringUtils.isNotEmpty(engine) && ENGINE.DISTRIBUTED ==
ENGINE.fromString(engine)) {
+ String shardingKey = props.get(DistributedTableConstants.SHARDING_KEY);
+ String remoteDb = props.get(DistributedTableConstants.REMOTE_DATABASE);
+ String remoteTbl = props.get(DistributedTableConstants.REMOTE_TABLE);
+ if (StringUtils.isNotBlank(shardingKey)) {
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(remoteDb), "Remote database must be
specified for Distributed");
+ Preconditions.checkArgument(
+ StringUtils.isNotBlank(remoteTbl), "Remote table must be specified
for Distributed");
+ try (Connection conn = getConnection(databaseName)) {
+ JdbcColumn[] remoteCols = fetchRemoteColumns(conn, remoteDb,
remoteTbl);
+ validateShardKeyColumns(
+ remoteCols, shardingKey, "in remote table
%s.%s".formatted(remoteDb, remoteTbl));
+ } catch (SQLException e) {
+ throw exceptionMapper.toGravitinoException(e);
+ }
+ }
+ }
+ super.create(
+ databaseName,
+ tableName,
+ columns,
+ comment,
+ properties,
+ partitioning,
+ distribution,
+ indexes,
+ sortOrders);
+ }
+
+ private JdbcColumn[] fetchRemoteColumns(Connection conn, String db, String
tbl)
+ throws SQLException {
+ List<JdbcColumn> cols = new ArrayList<>();
+ try (ResultSet rs = getColumns(conn, db, tbl)) {
+ while (rs.next()) {
+ JdbcColumn.Builder b = getColumnBuilder(rs, db, tbl);
+ if (b != null) {
+ b.withAutoIncrement(getAutoIncrementInfo(rs));
+ cols.add(b.build());
+ }
+ }
+ }
+ return cols.toArray(new JdbcColumn[0]);
+ }
+
+ /**
+ * Validates that bare-column shard keys exist, are not nullable, and are
integer-typed. Shared by
+ * {@link #create} (empty columns) and {@link #handleDistributeTable}
(explicit columns).
+ */
+ private void validateShardKeyColumns(
+ JdbcColumn[] columns, String shardingKey, String contextMsg) {
+ List<String> shardingColumns =
ClickHouseTableSqlUtils.extractShardingKeyColumns(shardingKey);
+ if (CollectionUtils.isEmpty(shardingColumns)) {
+ return;
+ }
+ boolean isBareColumn =
ClickHouseTableSqlUtils.isSimpleIdentifier(shardingKey.trim());
+ for (String columnName : shardingColumns) {
+ JdbcColumn col = findColumn(columns, columnName);
+ Preconditions.checkArgument(
+ col != null, "Sharding key column %s not found %s", columnName,
contextMsg);
+ if (isBareColumn) {
+ Preconditions.checkArgument(
+ !col.nullable(), "Sharding key column %s must not be nullable",
columnName);
+ Preconditions.checkArgument(
+ isIntegerType(col.dataType()),
+ "Sharding key column %s must be an integer type, but got %s",
+ columnName,
+ col.dataType());
+ }
+ }
+ }
+
@Override
protected List<Index> getIndexes(Connection connection, String databaseName,
String tableName) {
// cause clickhouse not impl getPrimaryKeys yet, ref:
@@ -355,10 +468,6 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
StringUtils.isNotBlank(remoteTable), "Remote table must be specified
for Distributed");
// User must ensure the sharding key is a trusted value.
- // TODO(yuqi) WE need to check the columns in shard keys should be integer
and not nullable,
- // as clickhouse distributed table requires the sharding key to be
integer and not nullable.
- // We can add this validation after we support user defined sharding key
in index, as we can
- // reuse the index field definition for validation.
Preconditions.checkArgument(
StringUtils.isNotBlank(shardingKey), "Sharding key must be specified
for Distributed");
@@ -366,16 +475,7 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
// columns should contain the sharding key, as clickhouse requires the
sharding key must be
// defined in the columns of the distributed table.
if (ArrayUtils.isNotEmpty(columns)) {
- List<String> shardingColumns =
ClickHouseTableSqlUtils.extractShardingKeyColumns(shardingKey);
- if (CollectionUtils.isNotEmpty(shardingColumns)) {
- for (String columnName : shardingColumns) {
- JdbcColumn shardingColumn = findColumn(columns, columnName);
- Preconditions.checkArgument(
- shardingColumn != null,
- "Sharding key column %s must be defined in the table",
- columnName);
- }
- }
+ validateShardKeyColumns(columns, shardingKey, "in the table");
}
if (ArrayUtils.isEmpty(columns)) {
@@ -506,6 +606,9 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
resolveGranularity(index.properties(), 1)));
break;
case DATA_SKIPPING_SET:
+ // SET index: set(N) max unique values default to 0 (unlimited),
configurable via
+ // set_max_values property. GRANULARITY defaults to 1, configurable
via granularity
+ // property, consistent with minmax and bloom_filter indexes.
sqlBuilder
.append(" ")
.append(
@@ -522,6 +625,23 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
}
}
+ /**
+ * Checks whether the given type represents an integer type suitable for
shard keys. Covers both
+ * Gravitino's built-in integral types (Int8-Int64, UInt8-UInt64) and
ClickHouse-specific wide
+ * integers (Int128/256, UInt128/256) that map to {@link
Types.ExternalType}. The regex matches
+ * the ClickHouse naming convention {@code U?INT<width>} to automatically
cover future integer
+ * variants (e.g. Int512) without code changes.
+ */
+ private static boolean isIntegerType(Type type) {
+ if (type instanceof Type.IntegralType) {
+ return true;
+ }
+ if (type instanceof Types.ExternalType ext) {
+ return
WIDE_INTEGER_PATTERN.matcher(ext.catalogString().toUpperCase(Locale.ROOT)).matches();
+ }
+ return false;
+ }
+
@Override
protected boolean getAutoIncrementInfo(ResultSet resultSet) throws
SQLException {
return "YES".equalsIgnoreCase(resultSet.getString("IS_AUTOINCREMENT"));
@@ -681,6 +801,17 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return Transforms.EMPTY_TRANSFORM;
}
+ @Override
+ protected ResultSet getColumns(Connection connection, String databaseName,
String tableName)
+ throws SQLException {
+ // The parent implementation ignores databaseName and uses
connection.getSchema(), which is
+ // incorrect for ClickHouse when the target database differs from the
connection's default
+ // database (e.g., Distributed tables referencing a remote database). Pass
databaseName as
+ // the schema pattern so JDBC metadata is filtered by the intended
database.
+ final DatabaseMetaData metaData = connection.getMetaData();
+ return metaData.getColumns(connection.getCatalog(), databaseName,
tableName, null);
+ }
+
protected ResultSet getTables(Connection connection) throws SQLException {
final DatabaseMetaData metaData = connection.getMetaData();
String catalogName = connection.getCatalog();
@@ -1373,7 +1504,7 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
List<Index> secondaryIndexes = new ArrayList<>();
try (PreparedStatement preparedStatement =
connection.prepareStatement(
- "SELECT name, type, expr FROM system.data_skipping_indices "
+ "SELECT name, type, expr, granularity FROM
system.data_skipping_indices "
+ "WHERE database = ? AND table = ? ORDER BY name")) {
preparedStatement.setString(1, databaseName);
preparedStatement.setString(2, tableName);
@@ -1382,12 +1513,21 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
String name = resultSet.getString("name");
String type = resultSet.getString("type");
String expression = resultSet.getString("expr");
+ long granularity = resultSet.getLong("granularity");
try {
String[][] fields = parseIndexFields(expression);
if (ArrayUtils.isEmpty(fields)) {
continue;
}
- secondaryIndexes.add(Indexes.of(getClickHouseIndexType(type),
name, fields));
+ // Only include granularity in properties when it differs from the
default,
+ // so that indexes created without explicit granularity have empty
properties
+ // and match the original creation state (avoids false
index-change diffs).
+ Map<String, String> properties =
+ granularity == DEFAULT_INDEX_GRANULARITY
+ ? Map.of()
+ : Map.of(GRANULARITY, String.valueOf(granularity));
+ secondaryIndexes.add(
+ Indexes.of(getClickHouseIndexType(type), name, fields,
properties));
} catch (IllegalArgumentException e) {
LOG.warn(
"Skip unsupported data skipping index {} for {}.{} with
expression {}",
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
index 7cfdde2b7b..4ac96e8f31 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseClusterIT.java
@@ -851,4 +851,256 @@ public class CatalogClickHouseClusterIT extends BaseIT {
}
}
}
+
+ //
---------------------------------------------------------------------------
+ // Shard key validation IT tests
+ //
---------------------------------------------------------------------------
+
+ /**
+ * Creating a distributed table with a nullable shard key column should fail
with a clear error
+ * message from Gravitino, rather than an opaque ClickHouse server error.
+ */
+ @Test
+ public void testDistributedTableNullableShardKeyRejected() {
+ String localTbl =
GravitinoITUtils.genRandomName("ck_shard_local_nullable");
+ String distTbl = GravitinoITUtils.genRandomName("ck_shard_dist_nullable");
+ NameIdentifier localIdent = NameIdentifier.of(schemaName, localTbl);
+ NameIdentifier distIdent = NameIdentifier.of(schemaName, distTbl);
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+
+ // Create a local table with a non-nullable sorting key (id) and a
nullable shard column.
+ // The sorting key must be non-nullable because ClickHouse rejects
nullable sorting keys
+ // by default (allow_nullable_key is disabled).
+ Column idCol = Column.of("id", Types.LongType.get(), "pk", false, false,
DEFAULT_VALUE_NOT_SET);
+ Column nullableCol =
+ Column.of(
+ "shard_col", Types.IntegerType.get(), "shard", true, false,
DEFAULT_VALUE_NOT_SET);
+ Column[] cols = new Column[] {idCol, nullableCol};
+
+ tableCatalog.createTable(
+ localIdent,
+ cols,
+ tableComment,
+ clusterMergeTreeProperties(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ getSortOrders("id"),
+ Indexes.EMPTY_INDEXES);
+
+ // Attempt to create a distributed table with the nullable column as shard
key
+ Map<String, String> distProps = distributedProperties(localTbl);
+ distProps.put(SHARDING_KEY, "shard_col");
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ tableCatalog.createTable(
+ distIdent,
+ new Column[] {},
+ tableComment,
+ distProps,
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ null),
+ "Distributed table with nullable shard key column should be rejected");
+ }
+
+ /**
+ * Creating a distributed table with a non-integer shard key column should
fail with a clear error
+ * message from Gravitino.
+ */
+ @Test
+ public void testDistributedTableNonIntegerShardKeyRejected() {
+ String localTbl = GravitinoITUtils.genRandomName("ck_shard_local_str");
+ String distTbl = GravitinoITUtils.genRandomName("ck_shard_dist_str");
+ NameIdentifier localIdent = NameIdentifier.of(schemaName, localTbl);
+ NameIdentifier distIdent = NameIdentifier.of(schemaName, distTbl);
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+
+ // Create a local table with a String column
+ Column stringCol =
+ Column.of(
+ "shard_col", Types.StringType.get(), "shard", false, false,
DEFAULT_VALUE_NOT_SET);
+ Column[] cols = new Column[] {stringCol};
+
+ tableCatalog.createTable(
+ localIdent,
+ cols,
+ tableComment,
+ clusterMergeTreeProperties(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ getSortOrders("shard_col"),
+ Indexes.EMPTY_INDEXES);
+
+ // Attempt to create a distributed table with the String column as shard
key
+ Map<String, String> distProps = distributedProperties(localTbl);
+ distProps.put(SHARDING_KEY, "shard_col");
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ tableCatalog.createTable(
+ distIdent,
+ new Column[] {},
+ tableComment,
+ distProps,
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ null),
+ "Distributed table with non-integer shard key column should be
rejected");
+ }
+
+ /** Creating a distributed table with a valid integer shard key column
should succeed. */
+ @Test
+ public void testDistributedTableIntegerShardKeyAccepted() {
+ String localTbl = GravitinoITUtils.genRandomName("ck_shard_local_int");
+ String distTbl = GravitinoITUtils.genRandomName("ck_shard_dist_int");
+ NameIdentifier localIdent = NameIdentifier.of(schemaName, localTbl);
+ NameIdentifier distIdent = NameIdentifier.of(schemaName, distTbl);
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+
+ // Create a local table with a non-nullable Int32 column
+ Column intCol =
+ Column.of(
+ "shard_col", Types.IntegerType.get(), "shard", false, false,
DEFAULT_VALUE_NOT_SET);
+ Column[] cols = new Column[] {intCol};
+
+ tableCatalog.createTable(
+ localIdent,
+ cols,
+ tableComment,
+ clusterMergeTreeProperties(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ getSortOrders("shard_col"),
+ Indexes.EMPTY_INDEXES);
+
+ // Create a distributed table with the Int32 column as shard key — should
succeed
+ Map<String, String> distProps = distributedProperties(localTbl);
+ distProps.put(SHARDING_KEY, "shard_col");
+
+ Table distTable =
+ tableCatalog.createTable(
+ distIdent,
+ new Column[] {},
+ tableComment,
+ distProps,
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ null);
+
+ Assertions.assertNotNull(distTable);
+ Assertions.assertEquals(distTbl, distTable.name());
+
+ Table loaded = tableCatalog.loadTable(distIdent);
+ Assertions.assertEquals(
+ ENGINE.DISTRIBUTED.getValue(),
loaded.properties().get(GRAVITINO_ENGINE_KEY));
+ }
+
+ /**
+ * Creating a distributed table with an Int128 shard key column should
succeed. Int128 is a valid
+ * ClickHouse integer type that maps to ExternalType in Gravitino's type
system, but should still
+ * be recognized as an integer by the shard key validation logic.
+ */
+ @Test
+ public void testDistributedTableWideIntegerShardKeyAccepted() {
+ String localTbl = GravitinoITUtils.genRandomName("ck_shard_local_i128");
+ String distTbl = GravitinoITUtils.genRandomName("ck_shard_dist_i128");
+ NameIdentifier distIdent = NameIdentifier.of(schemaName, distTbl);
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+
+ // Create a local table with Int128 column via raw SQL (Gravitino API may
not support Int128
+ // directly)
+ clickHouseService.executeQuery(
+ String.format(
+ "CREATE TABLE `%s`.`%s` ON CLUSTER `%s` "
+ + "(shard_col Int128, val String) "
+ + "ENGINE = MergeTree ORDER BY shard_col",
+ schemaName, localTbl, ClickHouseContainer.DEFAULT_CLUSTER_NAME));
+
+ // Create a distributed table with the Int128 column as shard key
+ Map<String, String> distProps = distributedProperties(localTbl);
+ distProps.put(SHARDING_KEY, "shard_col");
+
+ try {
+ Table distTable =
+ tableCatalog.createTable(
+ distIdent,
+ new Column[] {},
+ tableComment,
+ distProps,
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ null);
+
+ Assertions.assertNotNull(distTable);
+ Assertions.assertEquals(distTbl, distTable.name());
+
+ Table loaded = tableCatalog.loadTable(distIdent);
+ Assertions.assertEquals(
+ ENGINE.DISTRIBUTED.getValue(),
loaded.properties().get(GRAVITINO_ENGINE_KEY));
+ } finally {
+ tableCatalog.dropTable(distIdent);
+ clickHouseService.executeQuery(
+ String.format(
+ "DROP TABLE `%s`.`%s` ON CLUSTER `%s` SYNC",
+ schemaName, localTbl, ClickHouseContainer.DEFAULT_CLUSTER_NAME));
+ }
+ }
+
+ /**
+ * Creating a distributed table with a function-wrapped shard key (e.g.
cityHash64(string_col))
+ * should succeed regardless of the inner column type, because the function
returns a valid
+ * integer.
+ */
+ @Test
+ public void testDistributedTableFunctionShardKeyAccepted() {
+ String localTbl = GravitinoITUtils.genRandomName("ck_shard_local_func");
+ String distTbl = GravitinoITUtils.genRandomName("ck_shard_dist_func");
+ NameIdentifier localIdent = NameIdentifier.of(schemaName, localTbl);
+ NameIdentifier distIdent = NameIdentifier.of(schemaName, distTbl);
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+
+ // Create a local table with a String column
+ Column strCol =
+ Column.of("user_name", Types.StringType.get(), "user", false, false,
DEFAULT_VALUE_NOT_SET);
+ Column[] cols = new Column[] {strCol};
+
+ tableCatalog.createTable(
+ localIdent,
+ cols,
+ tableComment,
+ clusterMergeTreeProperties(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ getSortOrders("user_name"),
+ Indexes.EMPTY_INDEXES);
+
+ // Create a distributed table with cityHash64(string_col) as shard key —
should succeed
+ Map<String, String> distProps = distributedProperties(localTbl);
+ distProps.put(SHARDING_KEY, "cityHash64(user_name)");
+
+ try {
+ Table distTable =
+ tableCatalog.createTable(
+ distIdent,
+ new Column[] {},
+ tableComment,
+ distProps,
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ null);
+
+ Assertions.assertNotNull(distTable);
+ Assertions.assertEquals(distTbl, distTable.name());
+
+ Table loaded = tableCatalog.loadTable(distIdent);
+ Assertions.assertEquals(
+ ENGINE.DISTRIBUTED.getValue(),
loaded.properties().get(GRAVITINO_ENGINE_KEY));
+ } finally {
+ tableCatalog.dropTable(distIdent);
+ tableCatalog.dropTable(localIdent);
+ }
+ }
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
index 6fa6f470ca..a433ac058f 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
@@ -35,6 +35,7 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
+import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
import org.apache.commons.lang3.StringUtils;
@@ -566,6 +567,174 @@ public class CatalogClickHouseIT extends BaseIT {
&& Arrays.deepEquals(idx.fieldNames(), new String[][]
{{"user_id"}})));
}
+ @Test
+ void testCreateAndLoadWithCustomGranularity() {
+ String table = GravitinoITUtils.genRandomName("granularity_roundtrip");
+ NameIdentifier ident = NameIdentifier.of(schemaName, table);
+ Column[] cols =
+ new Column[] {
+ Column.of("id", Types.LongType.get(), "id", false, false,
DEFAULT_VALUE_NOT_SET),
+ Column.of("val", Types.StringType.get(), "val", true, false,
DEFAULT_VALUE_NOT_SET),
+ };
+
+ Index[] indexes =
+ new Index[] {
+ Indexes.primary(Indexes.DEFAULT_PRIMARY_KEY_NAME, new String[][]
{{"id"}}),
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_MINMAX,
+ "idx_minmax",
+ new String[][] {{"val"}},
+ Map.of("granularity", "5")),
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_BLOOM_FILTER,
+ "idx_bloom",
+ new String[][] {{"val"}},
+ Map.of("granularity", "10")),
+ };
+
+ SortOrder[] sortOrders =
+ new SortOrder[] {SortOrders.of(NamedReference.field("id"),
SortDirection.ASCENDING)};
+
+ catalog
+ .asTableCatalog()
+ .createTable(
+ ident,
+ cols,
+ "granularity test",
+ createProperties(),
+ null,
+ Distributions.NONE,
+ sortOrders,
+ indexes);
+
+ Table loaded = catalog.asTableCatalog().loadTable(ident);
+ Index[] loadedIndexes = loaded.index();
+
+ // Verify minmax index has custom GRANULARITY
+ Optional<Index> minmaxIdx =
+ Arrays.stream(loadedIndexes).filter(idx ->
"idx_minmax".equals(idx.name())).findFirst();
+ Assertions.assertTrue(minmaxIdx.isPresent(), "idx_minmax should exist");
+ Assertions.assertEquals("5",
minmaxIdx.get().properties().get("granularity"));
+
+ // Verify bloom_filter index has custom GRANULARITY
+ Optional<Index> bloomIdx =
+ Arrays.stream(loadedIndexes).filter(idx ->
"idx_bloom".equals(idx.name())).findFirst();
+ Assertions.assertTrue(bloomIdx.isPresent(), "idx_bloom should exist");
+ Assertions.assertEquals("10",
bloomIdx.get().properties().get("granularity"));
+ }
+
+ @Test
+ void testCreateAndLoadWithDefaultGranularity() {
+ String table = GravitinoITUtils.genRandomName("default_granularity");
+ NameIdentifier ident = NameIdentifier.of(schemaName, table);
+ Column[] cols =
+ new Column[] {
+ Column.of("id", Types.LongType.get(), "id", false, false,
DEFAULT_VALUE_NOT_SET),
+ Column.of("val", Types.StringType.get(), "val", true, false,
DEFAULT_VALUE_NOT_SET),
+ };
+
+ // Create index without specifying GRANULARITY — should default to 1
+ Index[] indexes =
+ new Index[] {
+ Indexes.primary(Indexes.DEFAULT_PRIMARY_KEY_NAME, new String[][]
{{"id"}}),
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_MINMAX, "idx_minmax_default", new
String[][] {{"val"}}),
+ };
+
+ SortOrder[] sortOrders =
+ new SortOrder[] {SortOrders.of(NamedReference.field("id"),
SortDirection.ASCENDING)};
+
+ catalog
+ .asTableCatalog()
+ .createTable(
+ ident,
+ cols,
+ "default granularity test",
+ createProperties(),
+ null,
+ Distributions.NONE,
+ sortOrders,
+ indexes);
+
+ Table loaded = catalog.asTableCatalog().loadTable(ident);
+ Index[] loadedIndexes = loaded.index();
+
+ // Default GRANULARITY (1) should NOT be persisted in properties
+ Optional<Index> minmaxIdx =
+ Arrays.stream(loadedIndexes)
+ .filter(idx -> "idx_minmax_default".equals(idx.name()))
+ .findFirst();
+ Assertions.assertTrue(minmaxIdx.isPresent(), "idx_minmax_default should
exist");
+ Assertions.assertFalse(
+ minmaxIdx.get().properties().containsKey("granularity"),
+ "Default GRANULARITY (1) should not be stored in properties");
+ }
+
+ @Test
+ void testLoadSqlCreatedTableWithCustomGranularity() {
+ String tableName = GravitinoITUtils.genRandomName("sql_granularity");
+ clickhouseService.executeQuery(
+ String.format(
+ "CREATE TABLE `%s`.`%s` ("
+ + " id UInt64,"
+ + " val String,"
+ + " INDEX idx_sql_gran val TYPE minmax GRANULARITY 7"
+ + ") ENGINE = MergeTree ORDER BY id",
+ schemaName, tableName));
+
+ Table loaded =
catalog.asTableCatalog().loadTable(NameIdentifier.of(schemaName, tableName));
+ Index[] loadedIndexes = loaded.index();
+
+ Optional<Index> idx =
+ Arrays.stream(loadedIndexes).filter(i ->
"idx_sql_gran".equals(i.name())).findFirst();
+ Assertions.assertTrue(idx.isPresent(), "idx_sql_gran should exist");
+ Assertions.assertEquals(
+ "7",
+ idx.get().properties().get("granularity"),
+ "SQL-created index with GRANULARITY 7 should be readable");
+ }
+
+ @Test
+ void testAlterTableAddIndexUsesDefaultGranularity() {
+ String table = GravitinoITUtils.genRandomName("alter_idx_gran");
+ NameIdentifier ident = NameIdentifier.of(schemaName, table);
+ Column[] cols =
+ new Column[] {
+ Column.of("id", Types.LongType.get(), "id", false, false,
DEFAULT_VALUE_NOT_SET),
+ Column.of("val", Types.StringType.get(), "val", true, false,
DEFAULT_VALUE_NOT_SET),
+ };
+
+ catalog
+ .asTableCatalog()
+ .createTable(
+ ident,
+ cols,
+ "alter index granularity test",
+ createProperties(),
+ null,
+ Distributions.NONE,
+ new SortOrder[] {SortOrders.of(NamedReference.field("id"),
SortDirection.ASCENDING)},
+ Indexes.EMPTY_INDEXES);
+
+ // Add index via ALTER TABLE — GRANULARITY should default to 1
+ catalog
+ .asTableCatalog()
+ .alterTable(
+ ident,
+ TableChange.addIndex(
+ Index.IndexType.DATA_SKIPPING_MINMAX,
+ "idx_alter_minmax",
+ new String[][] {{"val"}}));
+
+ Table loaded = catalog.asTableCatalog().loadTable(ident);
+ Optional<Index> idx =
+ Arrays.stream(loaded.index()).filter(i ->
"idx_alter_minmax".equals(i.name())).findFirst();
+ Assertions.assertTrue(idx.isPresent(), "idx_alter_minmax should exist");
+ Assertions.assertFalse(
+ idx.get().properties().containsKey("granularity"),
+ "ALTER TABLE ADD INDEX should use default GRANULARITY (1), not
persisted in properties");
+ }
+
@Test
void testCreateAndLoadWithPartitionTransforms() {
assertPartitionRoundTrip("identity_part",
Transforms.identity("event_time"));
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperations.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperations.java
index 635c41f021..5cea6b2801 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperations.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperations.java
@@ -1183,6 +1183,63 @@ public class TestClickHouseTableOperations extends
TestClickHouse {
Assertions.assertTrue(sql.contains("INDEX `idx_set` `c2` TYPE set(100)
GRANULARITY 3"));
}
+ @Test
+ void testGranularityNormalizedToCanonicalForm() {
+ TestableClickHouseTableOperations ops = new
TestableClickHouseTableOperations();
+ ops.initialize(
+ null,
+ new ClickHouseExceptionConverter(),
+ new ClickHouseTypeConverter(),
+ new ClickHouseColumnDefaultValueConverter(),
+ new HashMap<>());
+
+ JdbcColumn[] cols =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("c1")
+ .withType(Types.IntegerType.get())
+ .withNullable(false)
+ .build(),
+ };
+
+ Index[] indexes =
+ new Index[] {
+ Indexes.primary(Indexes.DEFAULT_PRIMARY_KEY_NAME, new String[][]
{{"c1"}}),
+ Indexes.of(
+ IndexType.DATA_SKIPPING_MINMAX,
+ "idx_c1",
+ new String[][] {{"c1"}},
+ Map.of("granularity", "005")),
+ Indexes.of(
+ IndexType.DATA_SKIPPING_BLOOM_FILTER,
+ "idx_c2",
+ new String[][] {{"c1"}},
+ Map.of("granularity", " 3 ")),
+ };
+
+ String sql =
+ ops.buildCreateSql(
+ "t_norm",
+ cols,
+ "comment",
+ new HashMap<>(),
+ new Transform[] {Transforms.identity("c1")},
+ Distributions.NONE,
+ indexes,
+ ClickHouseUtils.getSortOrders("c1"));
+
+ // "005" should be normalized to "5" after parsing
+ Assertions.assertTrue(
+ sql.contains("GRANULARITY 5"),
+ "Leading-zero granularity should be normalized, actual SQL: " + sql);
+ Assertions.assertFalse(
+ sql.contains("GRANULARITY 005"), "Raw leading-zero granularity must
not appear in DDL");
+ // Whitespace around the granularity value should be tolerated.
+ Assertions.assertTrue(
+ sql.contains("GRANULARITY 3"),
+ "Whitespace around granularity should be trimmed, actual SQL: " + sql);
+ }
+
@Test
void testGenerateCreateTableSqlWithAutoIncrementColumnUnsupported() {
TestableClickHouseTableOperations ops = new
TestableClickHouseTableOperations();
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsCluster.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsCluster.java
index e20e476a8f..5978fe212c 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsCluster.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsCluster.java
@@ -40,6 +40,8 @@ import org.apache.gravitino.rel.expressions.sorts.SortOrder;
import org.apache.gravitino.rel.expressions.sorts.SortOrders;
import org.apache.gravitino.rel.expressions.transforms.Transform;
import org.apache.gravitino.rel.indexes.Index;
+import org.apache.gravitino.rel.indexes.Index.IndexType;
+import org.apache.gravitino.rel.indexes.Indexes;
import org.apache.gravitino.rel.types.Types;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
@@ -433,6 +435,381 @@ class TestClickHouseTableOperationsCluster {
Assertions.assertTrue(sql.contains("ADD COLUMN"), "ALTER TABLE should
contain ADD COLUMN");
}
+ /** Shard key with nullable column should be rejected. */
+ @Test
+ void testShardingKeyNullableColumnRejected() {
+ Map<String, String> props = new HashMap<>();
+ props.put(ClusterConstants.CLUSTER_NAME, "ck_cluster");
+ props.put(ClusterConstants.ON_CLUSTER, "true");
+ props.put(GRAVITINO_ENGINE_KEY, "Distributed");
+ props.put(DistributedTableConstants.REMOTE_DATABASE, "remote_db");
+ props.put(DistributedTableConstants.REMOTE_TABLE, "remote_table");
+ props.put(DistributedTableConstants.SHARDING_KEY, "user_id");
+
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("user_id")
+ .withType(Types.IntegerType.get())
+ .withNullable(true)
+ .build()
+ };
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "tbl", columns, "comment", props, null, Distributions.NONE,
new Index[0], null));
+ }
+
+ /** Shard key with non-integer column type should be rejected. */
+ @Test
+ void testShardingKeyNonIntegerColumnRejected() {
+ Map<String, String> props = new HashMap<>();
+ props.put(ClusterConstants.CLUSTER_NAME, "ck_cluster");
+ props.put(ClusterConstants.ON_CLUSTER, "true");
+ props.put(GRAVITINO_ENGINE_KEY, "Distributed");
+ props.put(DistributedTableConstants.REMOTE_DATABASE, "remote_db");
+ props.put(DistributedTableConstants.REMOTE_TABLE, "remote_table");
+ props.put(DistributedTableConstants.SHARDING_KEY, "user_id");
+
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("user_id")
+ .withType(Types.StringType.get())
+ .withNullable(false)
+ .build()
+ };
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "tbl", columns, "comment", props, null, Distributions.NONE,
new Index[0], null));
+ }
+
+ /** Function-wrapped shard key with non-integer inner column should be
accepted. */
+ @Test
+ void testShardingKeyFunctionWithNonIntegerColumnAccepted() {
+ Map<String, String> props = new HashMap<>();
+ props.put(ClusterConstants.CLUSTER_NAME, "ck_cluster");
+ props.put(ClusterConstants.ON_CLUSTER, "true");
+ props.put(GRAVITINO_ENGINE_KEY, "Distributed");
+ props.put(DistributedTableConstants.REMOTE_DATABASE, "remote_db");
+ props.put(DistributedTableConstants.REMOTE_TABLE, "remote_table");
+ props.put(DistributedTableConstants.SHARDING_KEY, "cityHash64(user_name)");
+
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("user_name")
+ .withType(Types.StringType.get())
+ .withNullable(false)
+ .build()
+ };
+
+ // cityHash64(string_col) returns UInt64, so it should be accepted
+ String sql =
+ ops.buildCreateSql(
+ "tbl", columns, "comment", props, null, Distributions.NONE, new
Index[0], null);
+ Assertions.assertTrue(sql.contains("cityHash64"));
+ }
+
+ /**
+ * Bare Int128 column as shard key should be accepted because it is a valid
integer type in
+ * ClickHouse, even though it maps to ExternalType in Gravitino's type
system.
+ */
+ @Test
+ void testShardingKeyInt128ColumnAccepted() {
+ Map<String, String> props = new HashMap<>();
+ props.put(ClusterConstants.CLUSTER_NAME, "ck_cluster");
+ props.put(ClusterConstants.ON_CLUSTER, "true");
+ props.put(GRAVITINO_ENGINE_KEY, "Distributed");
+ props.put(DistributedTableConstants.REMOTE_DATABASE, "remote_db");
+ props.put(DistributedTableConstants.REMOTE_TABLE, "remote_table");
+ props.put(DistributedTableConstants.SHARDING_KEY, "id");
+
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("id")
+ .withType(Types.ExternalType.of("Int128"))
+ .withNullable(false)
+ .build()
+ };
+
+ // Int128 is a valid integer type in ClickHouse. The isIntegerType() check
recognizes
+ // ExternalType with integer catalog strings (Int128/256, UInt128/256).
+ String sql =
+ ops.buildCreateSql(
+ "tbl", columns, "comment", props, null, Distributions.NONE, new
Index[0], null);
+ Assertions.assertTrue(sql.contains("Distributed"));
+ }
+
+ /**
+ * Bare UInt128 column as shard key should be accepted, same as Int128 —
both are wide integer
+ * types recognized by isIntegerType().
+ */
+ @Test
+ void testShardingKeyUInt128ColumnAccepted() {
+ Map<String, String> props = new HashMap<>();
+ props.put(ClusterConstants.CLUSTER_NAME, "ck_cluster");
+ props.put(ClusterConstants.ON_CLUSTER, "true");
+ props.put(GRAVITINO_ENGINE_KEY, "Distributed");
+ props.put(DistributedTableConstants.REMOTE_DATABASE, "remote_db");
+ props.put(DistributedTableConstants.REMOTE_TABLE, "remote_table");
+ props.put(DistributedTableConstants.SHARDING_KEY, "id");
+
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("id")
+ .withType(Types.ExternalType.of("UInt128"))
+ .withNullable(false)
+ .build()
+ };
+
+ String sql =
+ ops.buildCreateSql(
+ "tbl", columns, "comment", props, null, Distributions.NONE, new
Index[0], null);
+ Assertions.assertTrue(sql.contains("Distributed"));
+ }
+
+ /** A non-numeric index granularity should be rejected to avoid malformed
DDL. */
+ @Test
+ void testIndexNonNumericGranularityRejected() {
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("c1")
+ .withType(Types.IntegerType.get())
+ .withNullable(false)
+ .build()
+ };
+
+ Index[] indexes =
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_MINMAX,
+ "idx_c1",
+ new String[][] {{"c1"}},
+ Map.of("granularity", "abc"))
+ };
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "tbl",
+ columns,
+ "comment",
+ new HashMap<>(),
+ null,
+ Distributions.NONE,
+ indexes,
+ null));
+ }
+
+ /** Index with GRANULARITY 0 should be rejected since ClickHouse requires
GRANULARITY >= 1. */
+ @Test
+ void testIndexZeroGranularityRejected() {
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("c1")
+ .withType(Types.IntegerType.get())
+ .withNullable(false)
+ .build()
+ };
+
+ Index[] indexes =
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_MINMAX,
+ "idx_c1",
+ new String[][] {{"c1"}},
+ Map.of("granularity", "0"))
+ };
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "tbl",
+ columns,
+ "comment",
+ new HashMap<>(),
+ null,
+ Distributions.NONE,
+ indexes,
+ null));
+ }
+
+ /**
+ * Index with GRANULARITY that parses to 0 (e.g. "00") should be rejected,
regardless of
+ * formatting.
+ */
+ @Test
+ void testIndexZeroAfterParsingGranularityRejected() {
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("c1")
+ .withType(Types.IntegerType.get())
+ .withNullable(false)
+ .build()
+ };
+
+ Index[] indexes =
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_MINMAX,
+ "idx_c1",
+ new String[][] {{"c1"}},
+ Map.of("granularity", "00"))
+ };
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "tbl",
+ columns,
+ "comment",
+ new HashMap<>(),
+ null,
+ Distributions.NONE,
+ indexes,
+ null));
+ }
+
+ /** Index with negative GRANULARITY (e.g. "-1") should be rejected. */
+ @Test
+ void testIndexNegativeGranularityRejected() {
+ JdbcColumn[] columns =
+ new JdbcColumn[] {
+ JdbcColumn.builder()
+ .withName("c1")
+ .withType(Types.IntegerType.get())
+ .withNullable(false)
+ .build()
+ };
+
+ Index[] indexes =
+ new Index[] {
+ Indexes.of(
+ Index.IndexType.DATA_SKIPPING_MINMAX,
+ "idx_c1",
+ new String[][] {{"c1"}},
+ Map.of("granularity", "-1"))
+ };
+
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ ops.buildCreateSql(
+ "tbl",
+ columns,
+ "comment",
+ new HashMap<>(),
+ null,
+ Distributions.NONE,
+ indexes,
+ null));
+ }
+
+ /** ALTER TABLE ADD INDEX should use the default GRANULARITY value. */
+ @Test
+ void testAlterTableAddIndexDefaultGranularity() {
+ StubClickHouseTableOperations stubOps = new
StubClickHouseTableOperations();
+ stubOps.initialize(
+ null,
+ new ClickHouseExceptionConverter(),
+ new ClickHouseTypeConverter(),
+ new ClickHouseColumnDefaultValueConverter(),
+ new HashMap<>());
+ stubOps.setTable(buildStubTableWithCluster("ck_cluster", true));
+
+ String minmaxSql =
+ stubOps.buildAlterSql(
+ "default",
+ "orders",
+ new TableChange[] {
+ TableChange.addIndex(
+ IndexType.DATA_SKIPPING_MINMAX, "idx_mm", new String[][]
{{"id"}})
+ });
+ Assertions.assertTrue(
+ minmaxSql.contains("ADD INDEX `idx_mm` `id` TYPE minmax GRANULARITY
1"),
+ "minmax ADD INDEX should use default GRANULARITY 1, actual: " +
minmaxSql);
+
+ String bloomSql =
+ stubOps.buildAlterSql(
+ "default",
+ "orders",
+ new TableChange[] {
+ TableChange.addIndex(
+ IndexType.DATA_SKIPPING_BLOOM_FILTER, "idx_bf", new
String[][] {{"id"}})
+ });
+ Assertions.assertTrue(
+ bloomSql.contains("ADD INDEX `idx_bf` `id` TYPE bloom_filter
GRANULARITY 1"),
+ "bloom_filter ADD INDEX should use default GRANULARITY 1, actual: " +
bloomSql);
+ }
+
+ /** ALTER TABLE ADD INDEX on a cluster table should include ON CLUSTER in
SQL. */
+ @Test
+ void testAlterTableAddIndexWithClusterProperties() {
+ StubClickHouseTableOperations stubOps = new
StubClickHouseTableOperations();
+ stubOps.initialize(
+ null,
+ new ClickHouseExceptionConverter(),
+ new ClickHouseTypeConverter(),
+ new ClickHouseColumnDefaultValueConverter(),
+ new HashMap<>());
+ stubOps.setTable(buildStubTableWithCluster("ck_cluster", true));
+
+ String sql =
+ stubOps.buildAlterSql(
+ "default",
+ "orders",
+ new TableChange[] {
+ TableChange.addIndex(
+ IndexType.DATA_SKIPPING_MINMAX, "idx_mm", new String[][]
{{"id"}})
+ });
+
+ Assertions.assertTrue(
+ sql.contains("ON CLUSTER"), "ALTER TABLE ADD INDEX should include ON
CLUSTER");
+ Assertions.assertTrue(sql.contains("`ck_cluster`"), "ALTER TABLE should
include cluster name");
+ Assertions.assertTrue(
+ sql.contains("ADD INDEX `idx_mm` `id` TYPE minmax GRANULARITY 1"),
+ "Should contain ADD INDEX with default GRANULARITY");
+ }
+
+ /** ALTER TABLE ADD INDEX with null properties should not throw NPE. */
+ @Test
+ void testAlterTableAddIndexWithNullProperties() {
+ StubClickHouseTableOperations stubOps = new
StubClickHouseTableOperations();
+ stubOps.initialize(
+ null,
+ new ClickHouseExceptionConverter(),
+ new ClickHouseTypeConverter(),
+ new ClickHouseColumnDefaultValueConverter(),
+ new HashMap<>());
+ stubOps.setTable(buildStubTableWithNullProperties());
+
+ String sql =
+ stubOps.buildAlterSql(
+ "default",
+ "orders",
+ new TableChange[] {
+ TableChange.addIndex(
+ IndexType.DATA_SKIPPING_MINMAX, "idx_mm", new String[][]
{{"id"}})
+ });
+
+ Assertions.assertFalse(sql.contains("ON CLUSTER"), "ALTER TABLE should NOT
include ON CLUSTER");
+ Assertions.assertTrue(sql.contains("ADD INDEX"), "ALTER TABLE should
contain ADD INDEX");
+ }
+
private static class TestableClickHouseTableOperations extends
ClickHouseTableOperations {
String buildCreateSql(
String tableName,
@@ -483,6 +860,7 @@ class TestClickHouseTableOperationsCluster {
return JdbcTable.builder()
.withName("orders")
.withColumns(new JdbcColumn[] {c1})
+ .withIndexes(new Index[0])
.withProperties(props)
.withTableOperation(null)
.build();
@@ -498,6 +876,7 @@ class TestClickHouseTableOperationsCluster {
return JdbcTable.builder()
.withName("orders")
.withColumns(new JdbcColumn[] {c1})
+ .withIndexes(new Index[0])
.withTableOperation(null)
.build();
}
