This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new ed5694877f [#12843] fix(clickhouse): Expose native partition
expression via partition-key property (#13080)
ed5694877f is described below
commit ed5694877faa3f4a8a9b39c7096c33012cb4b64f
Author: cwq222 <[email protected]>
AuthorDate: Thu Sep 17 21:00:22 2026 +0800
[#12843] fix(clickhouse): Expose native partition expression via
partition-key property (#13080)
### What changes were proposed in this pull request?
Remove the partition-specific SHOW CREATE regex parsing path and treat
`system.tables.partition_key` as the single authoritative source of the
native partition expression.
Introduce a visible, read-only `partition-key` table property that
carries the canonical native expression for every loaded table:
- identity / `toYear` / `toDate` / `toYYYYMM` mappings keep producing
their existing structured `Transform`s, and now also expose the
canonical native expression through `partition-key`.
- A native expression that cannot be structured (e.g.
`cityHash64(toString(sm4_cipher_msg)) % 7`) no longer throws;
`partitioning()` stays empty and the raw expression is preserved
verbatim in `partition-key`.
- An unpartitioned table exposes `partition-key` as an empty string so
the property key is always present and callers can rely on
`Map.get(...)` without a `containsKey` check.
Supporting changes:
- Make structural conversion best-effort: `parsePartitionExpression`
returns `null` for unsupported expressions instead of throwing, and
`parsePartitioning` returns an empty array when any element of the
partition key cannot be structured.
- Require a strict (plain-column) identifier for the existing
`toYear`/`toDate`/`toYYYYMM` mappings so nested expressions such as
`toYear(f(x))` are no longer misrepresented as a single-field transform.
- Fix `convertFromJdbcProperties`/`transformToJdbcProperties` to pass
through all properties (keeping the existing `ENGINE` key rename) so
that `partition-key` and other properties survive the conversion; filter
`partition-key` out of `transformToJdbcProperties` so it is never
written back to ClickHouse.
- Register `partition-key` as a reserved, immutable property, which
makes the framework reject it on create, set, and remove.
-
### Why are the changes needed?
Loading a valid MergeTree table with `PARTITION BY
cityHash64(toString(sm4_cipher_msg)) % 7` currently throws
`UnsupportedOperationException` from the strict SHOW CREATE parser,
blocking ordinary table metadata access. Returning only an empty
`Transform` array would also be ambiguous, since callers could not
distinguish an unpartitioned table from a native expression that
Gravitino cannot structure.
Fix: #12843
### Does this PR introduce _any_ user-facing change?
Yes. Loaded ClickHouse tables now expose a visible, read-only
`partition-key` property:
- Non-empty value: ClickHouse's canonical native partition expression as
returned by `system.tables.partition_key`.
- Empty string: the table is unpartitioned (the key is always present).
`Table.partitioning()` is unchanged for structured expressions and
returns an empty array for native expressions that have no equivalent
structured `Transform`. The property is read-only: callers cannot supply
it on create, set it on alter, or remove it.
Known limitation:
- A partition key is structured on an all-or-nothing basis: if any
element of a tuple partition key cannot be structured, `partitioning()`
is empty and the full expression is preserved verbatim in
`partition-key`; callers cannot tell which tuple elements were
individually structurable.
### How was this patch tested?
- Unit test `TestClickHouseTableOperationsPartitioning`: structured
expressions (identity/year/month/day/tuple) and unsupported
native/nested expressions.
- New IT
`CatalogClickHouseIT.testLoadTableWithNativePartitionExpression`: native
expression loads successfully and is preserved in `partition-key`.
- New IT `CatalogClickHouseIT.testLoadTableWithoutPartition`:
unpartitioned table exposes `partition-key` as an empty string.
- Manual end-to-end: real ClickHouse + Gravitino server + REST,
verifying all four scenarios (native expression, unpartitioned,
`toDate`, tuple) return the expected `partitioning()` and
`partition-key` values.
---
.../catalog/clickhouse/ClickHouseConstants.java | 7 ++
.../ClickHouseTablePropertiesMetadata.java | 53 ++++++++------
.../operations/ClickHouseTableOperations.java | 80 +++++----------------
.../operations/ClickHouseTableSqlUtils.java | 66 +++++++++--------
.../TestClickHouseTablePropertiesMetadata.java | 83 ++++++++++++++++++++++
.../integration/test/CatalogClickHouseIT.java | 45 ++++++++++++
.../TestClickHouseTableOperationsPartitioning.java | 27 ++++++-
docs/jdbc-clickhouse-catalog.md | 3 +
docs/partitions.md | 7 +-
9 files changed, 251 insertions(+), 120 deletions(-)
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
index b040ba4bda..94420487d7 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseConstants.java
@@ -51,6 +51,13 @@ public class ClickHouseConstants {
/** Parameters for supported parameterized MergeTree engines, without
outer parentheses. */
public static final String ENGINE_PARAMETERS = "engine_parameters";
+
+ /**
+ * Read-only property that exposes ClickHouse's canonical native partition
expression as
+ * returned by system.tables.partition_key. It carries expressions that
cannot be mapped to a
+ * structured Transform (identity, year, month, or day).
+ */
+ public static final String PARTITION_KEY = "partition-key";
}
public static final class IndexConstants {
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseTablePropertiesMetadata.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseTablePropertiesMetadata.java
index 50ab7af396..cdcd83b051 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseTablePropertiesMetadata.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/ClickHouseTablePropertiesMetadata.java
@@ -20,6 +20,7 @@ package org.apache.gravitino.catalog.clickhouse;
import static
org.apache.gravitino.connector.PropertyEntry.enumImmutablePropertyEntry;
import static
org.apache.gravitino.connector.PropertyEntry.stringOptionalPropertyEntry;
+import static org.apache.gravitino.connector.PropertyEntry.stringPropertyEntry;
import static
org.apache.gravitino.connector.PropertyEntry.stringReservedPropertyEntry;
import java.util.Collections;
@@ -110,6 +111,21 @@ public class ClickHouseTablePropertiesMetadata extends
JdbcTablePropertiesMetada
"",
false);
+ /**
+ * ClickHouse's canonical native partition expression as returned by
system.tables.partition_key.
+ * Read-only: populated when loading a table, exposing an empty string for
unpartitioned tables so
+ * that the property key is always present.
+ */
+ public static final PropertyEntry<String> PARTITION_KEY_PROPERTY_ENTRY =
+ stringPropertyEntry(
+ TableConstants.PARTITION_KEY,
+ "The canonical native partition expression of a ClickHouse table",
+ false,
+ true,
+ "",
+ false,
+ true);
+
private static final Map<String, PropertyEntry<?>> PROPERTIES_METADATA =
createPropertiesMetadata();
@@ -137,6 +153,7 @@ public class ClickHouseTablePropertiesMetadata extends
JdbcTablePropertiesMetada
map.put(CLUSTER_REMOTE_TABLE_PROPERTY_ENTRY.getName(),
CLUSTER_REMOTE_TABLE_PROPERTY_ENTRY);
map.put(CLUSTER_SHARDING_KEY_PROPERTY_ENTRY.getName(),
CLUSTER_SHARDING_KEY_PROPERTY_ENTRY);
map.put(ENGINE_PARAMETERS_PROPERTY_ENTRY.getName(),
ENGINE_PARAMETERS_PROPERTY_ENTRY);
+ map.put(PARTITION_KEY_PROPERTY_ENTRY.getName(),
PARTITION_KEY_PROPERTY_ENTRY);
return Collections.unmodifiableMap(map);
}
@@ -237,33 +254,25 @@ public class ClickHouseTablePropertiesMetadata extends
JdbcTablePropertiesMetada
@Override
public Map<String, String> transformToJdbcProperties(Map<String, String>
properties) {
- return Collections.unmodifiableMap(
- new HashMap<String, String>() {
- {
- properties.forEach(
- (key, value) -> {
- if (GRAVITINO_CONFIG_TO_CLICKHOUSE.containsKey(key)) {
- put(GRAVITINO_CONFIG_TO_CLICKHOUSE.get(key), value);
- }
- });
- }
- });
+ // Reuse the parent implementation to filter out the internal
gravitino.identifier property.
+ Map<String, String> transformed = new
HashMap<>(super.transformToJdbcProperties(properties));
+ // partition-key is read-only metadata; never write it back to ClickHouse.
+ transformed.remove(TableConstants.PARTITION_KEY);
+ // Rename the engine key to the ClickHouse-specific form.
+ String engine = transformed.remove(GRAVITINO_ENGINE_KEY);
+ if (engine != null) {
+ transformed.put(CLICKHOUSE_ENGINE_KEY, engine);
+ }
+ return Collections.unmodifiableMap(transformed);
}
@Override
public Map<String, String> convertFromJdbcProperties(Map<String, String>
properties) {
BidiMap<String, String> clickhouseConfigToGravitino =
GRAVITINO_CONFIG_TO_CLICKHOUSE.inverseBidiMap();
- return Collections.unmodifiableMap(
- new HashMap<String, String>() {
- {
- properties.forEach(
- (key, value) -> {
- if (clickhouseConfigToGravitino.containsKey(key)) {
- put(clickhouseConfigToGravitino.get(key), value);
- }
- });
- }
- });
+ Map<String, String> converted = new HashMap<>();
+ properties.forEach(
+ (key, value) ->
converted.put(clickhouseConfigToGravitino.getOrDefault(key, key), value));
+ return Collections.unmodifiableMap(converted);
}
}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
index e21109f84d..b6f566a4e0 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableOperations.java
@@ -42,7 +42,6 @@ import java.sql.DatabaseMetaData;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
-import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
@@ -92,7 +91,6 @@ import
org.apache.gravitino.rel.expressions.sorts.SortDirection;
import org.apache.gravitino.rel.expressions.sorts.SortOrder;
import org.apache.gravitino.rel.expressions.sorts.SortOrders;
import org.apache.gravitino.rel.expressions.transforms.Transform;
-import org.apache.gravitino.rel.expressions.transforms.Transforms;
import org.apache.gravitino.rel.indexes.Index;
import org.apache.gravitino.rel.indexes.Indexes;
import org.apache.gravitino.rel.types.Type;
@@ -112,9 +110,6 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
ENGINE.SUMMINGMERGETREE,
ENGINE.COLLAPSINGMERGETREE,
ENGINE.VERSIONEDCOLLAPSINGMERGETREE));
- private static final Pattern PARTITION_BY_PATTERN =
- Pattern.compile(
-
"(?is)\\bPARTITION\\s+BY\\s*(.+?)(?=\\bORDER\\s+BY\\b|\\bPRIMARY\\s+KEY\\b|\\bSAMPLE\\s+BY\\b|\\bTTL\\b|\\bSETTINGS\\b|\\bCOMMENT\\b|$)");
private static final Pattern SETTINGS_PATTERN =
Pattern.compile("(?is)\\bSETTINGS\\s+(.+?)(?=\\bCOMMENT\\b|$)");
private static final Pattern DISTRIBUTED_ENGINE_PATTERN =
@@ -878,12 +873,8 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
SystemTableMetadata systemTableMetadata =
getSystemTableMetadata(connection, databaseName, tableName);
- ShowCreateTableMetadata showCreateMetadata =
parseShowCreateTable(connection, tableName);
- Transform[] partitioning = showCreateMetadata.partitioning;
- if (ArrayUtils.isEmpty(partitioning)) {
- partitioning = getTablePartitioning(connection, databaseName,
tableName);
- }
- jdbcTableBuilder.withPartitioning(partitioning);
+ String partitionKey = getPartitionKey(connection, databaseName,
tableName);
+ jdbcTableBuilder.withPartitioning(parsePartitioning(partitionKey));
jdbcTableBuilder.withSortOrders(systemTableMetadata.sortOrders());
Distribution distribution = getDistributionInfo(connection,
databaseName, tableName);
@@ -895,12 +886,14 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
// mistaken for table SETTINGS. These values take precedence
// over any settings.* keys that might exist in system.tables (though
getTableProperties()
// currently does not read SETTINGS from system.tables, so no overlap
occurs in practice).
+ Map<String, String> merged = new HashMap<>(tableProperties);
if (!systemTableMetadata.settings().isEmpty()) {
- Map<String, String> merged = new HashMap<>(tableProperties);
merged.putAll(systemTableMetadata.settings());
- tableProperties = Collections.unmodifiableMap(merged);
}
- jdbcTableBuilder.withProperties(tableProperties);
+ // Expose ClickHouse's canonical native partition expression.
Unpartitioned tables expose an
+ // empty string so that the property key is always present.
+ merged.put(TableConstants.PARTITION_KEY,
StringUtils.defaultString(partitionKey));
+ jdbcTableBuilder.withProperties(Collections.unmodifiableMap(merged));
correctJdbcTableFields(connection, databaseName, tableName,
jdbcTableBuilder);
@@ -950,6 +943,13 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
@Override
protected Transform[] getTablePartitioning(
Connection connection, String databaseName, String tableName) throws
SQLException {
+ return parsePartitioning(getPartitionKey(connection, databaseName,
tableName));
+ }
+
+ @VisibleForTesting
+ @Nullable
+ String getPartitionKey(Connection connection, String databaseName, String
tableName)
+ throws SQLException {
try (PreparedStatement statement =
connection.prepareStatement(
"SELECT partition_key FROM system.tables WHERE database = ? AND
name = ?")) {
@@ -957,22 +957,12 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
statement.setString(2, tableName);
try (ResultSet resultSet = statement.executeQuery()) {
if (resultSet.next()) {
- String partitionKey = resultSet.getString("partition_key");
- try {
- return parsePartitioning(partitionKey);
- } catch (IllegalArgumentException | UnsupportedOperationException e)
{
- LOG.warn(
- "Skip unsupported partition expression {} for {}.{}",
- partitionKey,
- databaseName,
- tableName);
- return Transforms.EMPTY_TRANSFORM;
- }
+ return resultSet.getString("partition_key");
}
}
}
- return Transforms.EMPTY_TRANSFORM;
+ return null;
}
@Override
@@ -1474,24 +1464,10 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
}
@VisibleForTesting
- Transform[] parsePartitioning(String partitionKey) {
+ Transform[] parsePartitioning(@Nullable String partitionKey) {
return ClickHouseTableSqlUtils.parsePartitioning(partitionKey);
}
- private ShowCreateTableMetadata parseCreateStatement(String createSql) {
- ShowCreateTableMetadata metadata = new ShowCreateTableMetadata();
- if (StringUtils.isBlank(createSql)) {
- return metadata;
- }
-
- Matcher partitionMatcher = PARTITION_BY_PATTERN.matcher(createSql);
- if (partitionMatcher.find()) {
- metadata.partitioning = parsePartitioning(partitionMatcher.group(1));
- }
-
- return metadata;
- }
-
// Parses "key1 = val1, key2 = val2" from a SETTINGS clause.
// Keys are prefixed with "settings." to match the write path convention in
// appendTableProperties(). ClickHouse SETTINGS values are scalar (UInt64,
Bool,
@@ -1524,24 +1500,6 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
return Collections.emptyMap();
}
- private ShowCreateTableMetadata parseShowCreateTable(Connection connection,
String tableName)
- throws SQLException {
- String createSql = parseShowCreateTableSql(connection, tableName);
- return parseCreateStatement(createSql);
- }
-
- private String parseShowCreateTableSql(Connection connection, String
tableName)
- throws SQLException {
- String sql = "SHOW CREATE TABLE " + quoteIdentifier(tableName);
- try (Statement statement = connection.createStatement();
- ResultSet resultSet = statement.executeQuery(sql)) {
- if (resultSet.next()) {
- return resultSet.getString(1);
- }
- throw new SQLException("SHOW CREATE TABLE returned no rows for " +
tableName);
- }
- }
-
private String unquote(String value) {
String trimmed = StringUtils.trimToEmpty(value);
if (StringUtils.length(trimmed) >= 2) {
@@ -1659,10 +1617,6 @@ public class ClickHouseTableOperations extends
JdbcTableOperations {
}
}
- private static final class ShowCreateTableMetadata {
- private Transform[] partitioning = Transforms.EMPTY_TRANSFORM;
- }
-
private static final class TablePropertiesWithClusterMetadata {
private final Map<String, String> properties;
private final boolean hasClusterMetadata;
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java
index 9681c0d301..bcf1dbeb0b 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/operations/ClickHouseTableSqlUtils.java
@@ -26,6 +26,7 @@ import java.util.List;
import java.util.Locale;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.gravitino.rel.expressions.NamedReference;
import org.apache.gravitino.rel.expressions.transforms.Transform;
@@ -44,7 +45,7 @@ final class ClickHouseTableSqlUtils {
private ClickHouseTableSqlUtils() {}
- static Transform[] parsePartitioning(String partitionKey) {
+ static Transform[] parsePartitioning(@Nullable String partitionKey) {
if (StringUtils.isBlank(partitionKey)) {
return Transforms.EMPTY_TRANSFORM;
}
@@ -61,7 +62,13 @@ final class ClickHouseTableSqlUtils {
if (StringUtils.isBlank(expression)) {
continue;
}
- transforms.add(parsePartitionExpression(expression, partitionKey));
+ Transform transform = parsePartitionExpression(expression);
+ if (transform == null) {
+ // A single unsupported native expression means the whole partition
key cannot be
+ // represented as structured transforms.
+ return Transforms.EMPTY_TRANSFORM;
+ }
+ transforms.add(transform);
}
return transforms.toArray(new Transform[0]);
@@ -178,49 +185,48 @@ final class ClickHouseTableSqlUtils {
return normalizeIndexExpression(current);
}
- private static Transform parsePartitionExpression(
- String expression, String originalPartitionKey) {
+ @Nullable
+ private static Transform parsePartitionExpression(String expression) {
String trimmedExpression = StringUtils.trim(expression);
Matcher toYearMatcher = TO_YEAR_PATTERN.matcher(trimmedExpression);
if (toYearMatcher.matches()) {
- String identifier = normalizeIdentifier(toYearMatcher.group(1));
- Preconditions.checkArgument(
- StringUtils.isNotBlank(identifier),
- "Unsupported partition expression: " + originalPartitionKey);
- return Transforms.year(identifier);
+ String identifier = extractPartitionIdentifier(toYearMatcher.group(1));
+ return identifier == null ? null : Transforms.year(identifier);
}
Matcher toYYYYMMMatcher = TO_MONTH_PATTERN.matcher(trimmedExpression);
if (toYYYYMMMatcher.matches()) {
- String identifier = normalizeIdentifier(toYYYYMMMatcher.group(1));
- Preconditions.checkArgument(
- StringUtils.isNotBlank(identifier),
- "Unsupported partition expression: " + originalPartitionKey);
- return Transforms.month(identifier);
+ String identifier = extractPartitionIdentifier(toYYYYMMMatcher.group(1));
+ return identifier == null ? null : Transforms.month(identifier);
}
Matcher toDateMatcher = TO_DATE_PATTERN.matcher(trimmedExpression);
if (toDateMatcher.matches()) {
- String identifier = normalizeIdentifier(toDateMatcher.group(1));
- Preconditions.checkArgument(
- StringUtils.isNotBlank(identifier),
- "Unsupported partition expression: " + originalPartitionKey);
- return Transforms.day(identifier);
+ String identifier = extractPartitionIdentifier(toDateMatcher.group(1));
+ return identifier == null ? null : Transforms.day(identifier);
}
- if (trimmedExpression.contains("(") && trimmedExpression.contains(")")) {
- throw new UnsupportedOperationException(
- "Currently Gravitino only supports toYear, toYYYYMM, toDate
partition expressions, but got: "
- + trimmedExpression);
- }
+ String identifier = extractPartitionIdentifier(trimmedExpression);
+ return identifier == null ? null : Transforms.identity(identifier);
+ }
- String identifier = normalizeIdentifier(trimmedExpression);
- Preconditions.checkArgument(
- isStrictIdentifier(identifier),
- "Only simple identifier is supported for partition expression, but
got: "
- + originalPartitionKey);
- return Transforms.identity(identifier);
+ /**
+ * Extracts a partition column name from an expression. A backtick-quoted
identifier (which may
+ * contain special characters such as {@code -}) is always treated as a
column name. Otherwise the
+ * expression must match the strict column-name pattern. Returns {@code
null} for arbitrary
+ * expressions such as {@code f(x)} that cannot be represented as a single
column reference.
+ */
+ @Nullable
+ private static String extractPartitionIdentifier(String expression) {
+ String trimmed = StringUtils.trim(expression);
+ if (StringUtils.startsWith(trimmed, "`")
+ && StringUtils.endsWith(trimmed, "`")
+ && trimmed.length() >= 2) {
+ String inner = trimmed.substring(1, trimmed.length() - 1);
+ return StringUtils.isNotBlank(inner) ? inner : null;
+ }
+ return isStrictIdentifier(trimmed) ? trimmed : null;
}
private static String normalizePartitionKey(String partitionKey) {
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/TestClickHouseTablePropertiesMetadata.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/TestClickHouseTablePropertiesMetadata.java
new file mode 100644
index 0000000000..81b6fbba79
--- /dev/null
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/TestClickHouseTablePropertiesMetadata.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.clickhouse;
+
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.gravitino.StringIdentifier;
+import
org.apache.gravitino.catalog.clickhouse.ClickHouseConstants.TableConstants;
+import org.apache.gravitino.connector.PropertyEntry;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+public class TestClickHouseTablePropertiesMetadata {
+
+ private ClickHouseTablePropertiesMetadata metadata;
+
+ @BeforeEach
+ void setUp() {
+ metadata = new ClickHouseTablePropertiesMetadata();
+ }
+
+ @Test
+ void testTransformToJdbcProperties() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put(StringIdentifier.ID_KEY, "gravitino-id-123");
+ properties.put(TableConstants.PARTITION_KEY, "cityHash64(x) % 7");
+ properties.put(TableConstants.ENGINE, "MergeTree");
+ properties.put(TableConstants.SETTINGS_PREFIX + "index_granularity",
"8192");
+
+ Map<String, String> jdbcProperties =
metadata.transformToJdbcProperties(properties);
+
+ // gravitino.identifier and partition-key must not be written to
ClickHouse.
+
Assertions.assertFalse(jdbcProperties.containsKey(StringIdentifier.ID_KEY));
+
Assertions.assertFalse(jdbcProperties.containsKey(TableConstants.PARTITION_KEY));
+ // engine is renamed to the ClickHouse-specific key.
+ Assertions.assertEquals("MergeTree",
jdbcProperties.get(TableConstants.ENGINE_UPPER));
+ Assertions.assertFalse(jdbcProperties.containsKey(TableConstants.ENGINE));
+ // Other properties pass through unchanged.
+ Assertions.assertEquals(
+ "8192", jdbcProperties.get(TableConstants.SETTINGS_PREFIX +
"index_granularity"));
+ }
+
+ @Test
+ void testConvertFromJdbcProperties() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put(TableConstants.ENGINE_UPPER, "MergeTree");
+ properties.put(TableConstants.PARTITION_KEY, "cityHash64(x) % 7");
+
+ Map<String, String> gravitinoProperties =
metadata.convertFromJdbcProperties(properties);
+
+ // engine is restored to the Gravitino key, and partition-key is preserved.
+ Assertions.assertEquals("MergeTree",
gravitinoProperties.get(TableConstants.ENGINE));
+ Assertions.assertEquals(
+ "cityHash64(x) % 7",
gravitinoProperties.get(TableConstants.PARTITION_KEY));
+ }
+
+ @Test
+ void testPartitionKeyPropertyEntry() {
+ PropertyEntry<?> entry =
ClickHouseTablePropertiesMetadata.PARTITION_KEY_PROPERTY_ENTRY;
+ Assertions.assertEquals(TableConstants.PARTITION_KEY, entry.getName());
+ Assertions.assertTrue(entry.isReserved());
+ Assertions.assertTrue(entry.isImmutable());
+ Assertions.assertFalse(entry.isHidden());
+ Assertions.assertEquals("", entry.getDefaultValue());
+ }
+}
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
index a9ce6f64ea..a6619bd95d 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
@@ -515,6 +515,8 @@ public class CatalogClickHouseIT extends BaseIT {
Assertions.assertEquals(Transforms.NAME_OF_MONTH, partitioning[0].name());
Assertions.assertArrayEquals(
new String[] {"event_time"}, ((NamedReference)
partitioning[0].arguments()[0]).fieldName());
+ Assertions.assertEquals(
+ "toYYYYMM(event_time)",
loaded.properties().get(TableConstants.PARTITION_KEY));
Index[] indexes = loaded.index();
Assertions.assertTrue(
@@ -590,6 +592,49 @@ public class CatalogClickHouseIT extends BaseIT {
"Recreated table must not contain a fabricated replacement index: " +
recreatedCreateSql);
}
+ @Test
+ void testLoadTableWithNativePartitionExpression() {
+ // A valid MergeTree table whose PARTITION BY uses a native expression
outside the structured
+ // identity/year/month/day subset must still be loadable. partitioning()
stays empty, and the
+ // canonical native expression is exposed through the read-only
partition-key property.
+ String name = GravitinoITUtils.genRandomName("native_partition_expr");
+ clickhouseService.executeQuery(
+ String.format(
+ "CREATE TABLE `%s`.`%s` (\n"
+ + " `id` UInt64,\n"
+ + " `sm4_cipher_msg` String\n"
+ + ")\n"
+ + "ENGINE = MergeTree\n"
+ + "PARTITION BY cityHash64(toString(sm4_cipher_msg)) %% 7\n"
+ + "ORDER BY id",
+ schemaName, name));
+
+ Table loaded =
catalog.asTableCatalog().loadTable(NameIdentifier.of(schemaName, name));
+ Assertions.assertEquals(0, loaded.partitioning().length);
+ Assertions.assertEquals(
+ "cityHash64(toString(sm4_cipher_msg)) % 7",
+ loaded.properties().get(TableConstants.PARTITION_KEY));
+ }
+
+ @Test
+ void testLoadTableWithoutPartition() {
+ // An unpartitioned table exposes an empty partition-key property so the
key is always present,
+ // and partitioning() stays empty.
+ String name = GravitinoITUtils.genRandomName("no_partition");
+ clickhouseService.executeQuery(
+ String.format(
+ "CREATE TABLE `%s`.`%s` (\n"
+ + " `id` UInt64\n"
+ + ")\n"
+ + "ENGINE = MergeTree\n"
+ + "ORDER BY id",
+ schemaName, name));
+
+ Table loaded =
catalog.asTableCatalog().loadTable(NameIdentifier.of(schemaName, name));
+ Assertions.assertEquals(0, loaded.partitioning().length);
+ Assertions.assertEquals("",
loaded.properties().get(TableConstants.PARTITION_KEY));
+ }
+
@Test
void testCreateAndLoadCompositePrimaryKey() {
String table = GravitinoITUtils.genRandomName("composite_primary_key");
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsPartitioning.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsPartitioning.java
index 24cf0c574a..00c0895a08 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsPartitioning.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/operations/TestClickHouseTableOperationsPartitioning.java
@@ -40,9 +40,9 @@ public class TestClickHouseTableOperationsPartitioning {
Assertions.assertEquals(1, yearPartitions.length);
assertSingleFieldTransform(yearPartitions[0], Transforms.NAME_OF_YEAR,
"event_time");
- Assertions.assertThrows(
- UnsupportedOperationException.class,
- () -> operations.parsePartitioning("cityHash64(user_id) % 16"));
+ // A native expression that cannot be structured returns an empty
transform array. The raw
+ // expression is instead exposed through the read-only partition-key
property during load.
+ Assertions.assertEquals(0,
operations.parsePartitioning("cityHash64(user_id) % 16").length);
Transform[] identityPartitions =
operations.parsePartitioning("metric_type");
Assertions.assertEquals(1, identityPartitions.length);
@@ -57,6 +57,27 @@ public class TestClickHouseTableOperationsPartitioning {
Assertions.assertEquals(0, operations.parsePartitioning(" ").length);
}
+ @Test
+ public void testNestedExpressionInsideKnownTransformIsNotMisrepresented() {
+ // A nested expression inside toYear cannot be mapped to a single-field
year transform, so the
+ // whole partition key is treated as unsupported and returns an empty
transform array rather
+ // than misrepresenting it as year("f(x)").
+ Assertions.assertEquals(0,
operations.parsePartitioning("toYear(toString(event_time))").length);
+ }
+
+ @Test
+ public void testBacktickQuotedColumnNameIsStructured() {
+ // A backtick-quoted column name (which may contain special characters
such as "-") is a real
+ // column reference and must be structured rather than rejected.
+ Transform[] monthPartitions =
operations.parsePartitioning("toYYYYMM(`event-time`)");
+ Assertions.assertEquals(1, monthPartitions.length);
+ assertSingleFieldTransform(monthPartitions[0], Transforms.NAME_OF_MONTH,
"event-time");
+
+ Transform[] identityPartitions =
operations.parsePartitioning("`event-time`");
+ Assertions.assertEquals(1, identityPartitions.length);
+ assertSingleFieldTransform(identityPartitions[0],
Transforms.NAME_OF_IDENTITY, "event-time");
+ }
+
private void assertSingleFieldTransform(
Transform transform, String expectedName, String expectedColumn) {
Assertions.assertEquals(expectedName, transform.name());
diff --git a/docs/jdbc-clickhouse-catalog.md b/docs/jdbc-clickhouse-catalog.md
index 468d80909d..845964b021 100644
--- a/docs/jdbc-clickhouse-catalog.md
+++ b/docs/jdbc-clickhouse-catalog.md
@@ -234,6 +234,7 @@ If you need Gravitino to manage an existing cluster
database or table, recreate
| `cluster-remote-table` | Remote table for `Distributed` engine
| (none) |
No\*\* | No | No |
| `cluster-sharding-key` | Sharding key for `Distributed` engine
(expression allowed; referenced columns must be non-null integral) | (none)
| No\*\* | No | No |
| `settings.<name>` | ClickHouse engine setting forwarded as `SETTINGS
<name>=<value>` | (none) | No
| No | No |
+| `partition-key` | ClickHouse's canonical native partition
expression (from `system.tables.partition_key`). Read-only; always present on
load, empty string means unpartitioned. | `""` | No | Yes
| Yes |
\* Required when `on-cluster=true` or `engine=Distributed`.
\*\* Required when `engine=Distributed`.
@@ -269,6 +270,8 @@ The `engine_parameters` property applies to
`ReplacingMergeTree`, `SummingMergeT
- Functions: `PARTITION BY toDate(column_name)`, `PARTITION BY
toYear(column_name)`, `PARTITION BY toYYYYMM(column_name)`. Other functions are
not supported.
- Not support: `PARTITION BY (column_name + 1)`, `PARTITION BY
(toYear(column_name) + 1)`, etc. (Note: ClickHouse itself does support
arbitrary partitioning expressions, but Gravitino supports only the above
patterns for partitioning).
+ The patterns above apply when creating a table. When loading a table,
Gravitino preserves ClickHouse's canonical native partition expression (as
returned by `system.tables.partition_key`) in the read-only `partition-key`
property. An arbitrary native expression is therefore retained on load even
when it cannot be mapped to one of the supported `Transform`s; in that case
`Table.partitioning()` is empty and the full expression is exposed through
`partition-key`.
+
- Distribution: fixed to `Distributions.NONE`. For a `Distributed` engine
table, you can specify the sharding key and remote database/table through table
properties to fulfill the same use cases. We will later consider adding more
flexible distribution strategies if there is demand.
### Create a Table
diff --git a/docs/partitions.md b/docs/partitions.md
index 2a3c25106d..1195b9226d 100644
--- a/docs/partitions.md
+++ b/docs/partitions.md
@@ -94,8 +94,11 @@ Individual partitions are not listed, and adding or dropping
one goes through th
The display comes from the table itself, so it works for every relational
catalog including Iceberg.
An Iceberg table partitioned by `day(event_time)` shows that transform like
any other table, because
-the partition spec is read back and presented in the same form. A count of
zero means the table
-declares no partitioning rather than that partitions could not be read.
+the partition spec is read back and presented in the same form. A count of
zero usually means the
+table declares no partitioning rather than that partitions could not be read.
One exception is
+ClickHouse: a table whose `PARTITION BY` uses a native expression that
Gravitino cannot structure
+shows zero partition fields, with the canonical expression preserved in the
read-only `partition-key`
+property instead.
## Permissions
