This is an automated email from the ASF dual-hosted git repository.
mchades pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new d7b5fa0588 [#10561] fix(hive): Tighten and document non-Hive table
filter in listTables (#11193)
d7b5fa0588 is described below
commit d7b5fa0588b590c993d097207a0c846521946043
Author: mchades <[email protected]>
AuthorDate: Fri May 22 20:44:33 2026 +0800
[#10561] fix(hive): Tighten and document non-Hive table filter in
listTables (#11193)
### What changes were proposed in this pull request?
- Fix a `startsWith` over-removal bug in
`HiveCatalogOperations#listTables` that
incorrectly dropped tables whose names happened to start with a Hudi
base table
name (e.g. `<base>_root`).
- Extract the non-Hive filtering logic into a focused private method and
consolidate filter-string construction.
- Document the Hive Metastore `listTableNamesByFilter` limitation
(dot-free
parameter keys only) and provide a user-side workaround on the
`list-all-tables` catalog property and in the Hive catalog docs.
### Why are the changes needed?
`HiveCatalogOperations#listTables` filters out non-Hive tables (Iceberg,
Paimon,
Hudi) via `IMetaStoreClient#listTableNamesByFilter`. Two issues:
1. Spark-managed Hudi tables registered via `saveAsTable` only set
`spark.sql.sources.provider=hudi` — a dotted key the HMS filter grammar
cannot match — so they leak into the listing.
2. The `_ro` / `_rt` cleanup used `startsWith`, removing unrelated
tables that
merely shared a prefix with a Hudi base table.
(1) cannot be fully fixed server-side (HMS limitation, confirmed against
Hive
`ExpressionTree` and Trino's `HiveUtil`); (2) is fixed here, and the gap
from
(1) is now clearly documented with a one-line user-side workaround.
Fix: #10561
### Does this PR introduce _any_ user-facing change?
- Updated description of the `list-all-tables` catalog property.
- New "Known limitation" + "Workaround" section in
`docs/apache-hive-catalog.md`.
- No API or behavior change for callers using the default
`list-all-tables=false`.
### How was this patch tested?
- `./gradlew :catalogs:catalog-hive:test -PskipITs --tests
"org.apache.gravitino.catalog.hive.TestHiveCatalogOperations"` passes.
- Manual review of generated filter strings against Hive
`ExpressionTree` JDOQL.
---
.../catalog/hive/HiveCatalogOperations.java | 76 ++++++++++++++--------
.../hive/HiveCatalogPropertiesMetadata.java | 5 +-
.../catalog/hive/TestHiveCatalogOperations.java | 60 +++++++++++++++++
docs/apache-hive-catalog.md | 32 +++++++--
4 files changed, 139 insertions(+), 34 deletions(-)
diff --git
a/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java
b/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java
index 3f8c37d7b2..5ff4f27ddc 100644
---
a/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java
+++
b/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java
@@ -399,26 +399,7 @@ public class HiveCatalogOperations
allTables.removeAll(views);
if (!listAllTables) {
- // The reason for using the listTableNamesByFilter function is that the
- // getTableObjectiesByName function has poor performance. Currently,
we focus on the
- // Iceberg, Paimon and Hudi table. In the future, if necessary, we
will need to filter out
- // other tables.
- String icebergAndPaimonFilter = getIcebergAndPaimonFilter();
- List<String> icebergAndPaimonTables =
- clientPool.run(
- c ->
- c.listTableNamesByFilter(
- catalogName, schemaIdent.name(),
icebergAndPaimonFilter, MAX_TABLES));
- allTables.removeAll(icebergAndPaimonTables);
-
- // filter out the Hudi tables
- String hudiFilter = String.format("%sprovider like \"hudi\"",
HIVE_FILTER_FIELD_PARAMS);
- List<String> hudiTables =
- clientPool.run(
- c ->
- c.listTableNamesByFilter(
- catalogName, schemaIdent.name(), hudiFilter,
MAX_TABLES));
- removeHudiTables(allTables, hudiTables);
+ filterOutNonHiveTables(schemaIdent.name(), allTables);
}
return allTables.stream()
@@ -430,20 +411,59 @@ public class HiveCatalogOperations
}
}
- private static String getIcebergAndPaimonFilter() {
+ /**
+ * Best-effort removal of non-Hive tables (Iceberg, Paimon, Hudi) from
{@code allTables} via HMS
+ * server-side {@code listTableNamesByFilter}. HMS only supports exact
lookups on dot-free
+ * property keys, so Spark-managed Hudi tables that only expose {@code
+ * spark.sql.sources.provider=hudi} cannot be filtered here. We keep this
strategy because {@code
+ * getTableObjectsByName} materializes every table and is slow on large
databases.
+ *
+ * @param database the database name
+ * @param allTables all table names fetched from HMS before non-Hive
filtering
+ * @throws InterruptedException if the HMS client call is interrupted
+ */
+ private void filterOutNonHiveTables(String database, List<String> allTables)
+ throws InterruptedException {
+ List<String> icebergAndPaimonTables =
+ clientPool.run(
+ c ->
+ c.listTableNamesByFilter(
+ catalogName, database, buildIcebergAndPaimonFilter(),
MAX_TABLES));
+ allTables.removeAll(icebergAndPaimonTables);
+
+ // HoodieHiveSyncTool sets `provider=hudi` only on the base table; derived
`_ro` / `_rt`
+ // tables carry only dotted keys, so we strip them by exact name match
against the base list.
+ List<String> hudiBaseTables =
+ clientPool.run(
+ c ->
+ c.listTableNamesByFilter(
+ catalogName, database, buildHudiBaseTableFilter(),
MAX_TABLES));
+ removeHudiDerivedTables(allTables, hudiBaseTables);
+ }
+
+ private static String buildIcebergAndPaimonFilter() {
String icebergFilter = String.format("%stable_type like \"ICEBERG\"",
HIVE_FILTER_FIELD_PARAMS);
String paimonFilter = String.format("%stable_type like \"PAIMON\"",
HIVE_FILTER_FIELD_PARAMS);
return String.format("%s or %s", icebergFilter, paimonFilter);
}
- private void removeHudiTables(List<String> allTables, List<String>
hudiTables) {
- for (String hudiTable : hudiTables) {
- allTables.removeIf(
- t ->
- t.equals(hudiTable)
- || t.startsWith(hudiTable + "_ro")
- || t.startsWith(hudiTable + "_rt"));
+ private static String buildHudiBaseTableFilter() {
+ return String.format("%sprovider like \"hudi\"", HIVE_FILTER_FIELD_PARAMS);
+ }
+
+ /**
+ * Removes Hudi base tables together with their derived read-optimized
({@code _ro}) and real-time
+ * ({@code _rt}) tables. Exact name match is used because {@code startsWith}
would incorrectly
+ * drop unrelated tables such as {@code <base>_root}.
+ */
+ private void removeHudiDerivedTables(List<String> allTables, List<String>
hudiBaseTables) {
+ Set<String> hudiTablesToRemove = new HashSet<>();
+ for (String hudiBase : hudiBaseTables) {
+ hudiTablesToRemove.add(hudiBase);
+ hudiTablesToRemove.add(hudiBase + "_ro");
+ hudiTablesToRemove.add(hudiBase + "_rt");
}
+ allTables.removeIf(hudiTablesToRemove::contains);
}
/**
diff --git
a/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
b/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
index c316b1b72e..2577bdf944 100644
---
a/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
+++
b/catalogs/catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogPropertiesMetadata.java
@@ -114,7 +114,10 @@ public class HiveCatalogPropertiesMetadata extends
BaseCatalogPropertiesMetadata
LIST_ALL_TABLES,
PropertyEntry.booleanPropertyEntry(
LIST_ALL_TABLES,
- "Lists all tables in a database, including non-Hive tables,
such as Iceberg, etc.",
+ "Whether to list all tables in a database, including
non-Hive tables such as "
+ + "Iceberg, Paimon and Hudi. When false, non-Hive tables
are filtered out "
+ + "on a best-effort basis; see the Hive catalog
documentation for known "
+ + "limitations.",
false /* required */,
false /* immutable */,
DEFAULT_LIST_ALL_TABLES,
diff --git
a/catalogs/catalog-hive/src/test/java/org/apache/gravitino/catalog/hive/TestHiveCatalogOperations.java
b/catalogs/catalog-hive/src/test/java/org/apache/gravitino/catalog/hive/TestHiveCatalogOperations.java
index c6c4b313a2..56154cbf41 100644
---
a/catalogs/catalog-hive/src/test/java/org/apache/gravitino/catalog/hive/TestHiveCatalogOperations.java
+++
b/catalogs/catalog-hive/src/test/java/org/apache/gravitino/catalog/hive/TestHiveCatalogOperations.java
@@ -869,6 +869,66 @@ class TestHiveCatalogOperations {
.listTableNamesByFilter(anyString(), anyString(), anyString(),
anyShort());
}
+ @Test
+ void testListTablesRemovesOnlyHudiBaseAndDerivedTables() throws Exception {
+ HiveCatalogOperations op = new HiveCatalogOperations();
+ op.initialize(ImmutableMap.of(LIST_ALL_TABLES, "false"), null,
HIVE_PROPERTIES_METADATA);
+
+ CachedClientPool clientPool = mock(CachedClientPool.class);
+ HiveClient hiveClient = mock(HiveClient.class);
+ when(hiveClient.getAllDatabases(anyString())).thenReturn(List.of("db"));
+ when(hiveClient.getAllTables(anyString(), anyString()))
+ .thenReturn(
+ new ArrayList<>(
+ List.of(
+ "v1",
+ "hive_tbl",
+ "ice_tbl",
+ "hudi_base",
+ "hudi_base_ro",
+ "hudi_base_rt",
+ "hudi_base_root")));
+ when(hiveClient.listTablesByType(anyString(), anyString(), anyString(),
anyString()))
+ .thenReturn(List.of("v1"));
+
+ String icebergAndPaimonFilter =
+ String.format(
+ "%stable_type like \"ICEBERG\" or %stable_type like \"PAIMON\"",
+ HiveConstants.HIVE_FILTER_FIELD_PARAMS,
HiveConstants.HIVE_FILTER_FIELD_PARAMS);
+ String hudiBaseFilter =
+ String.format("%sprovider like \"hudi\"",
HiveConstants.HIVE_FILTER_FIELD_PARAMS);
+ when(hiveClient.listTableNamesByFilter(anyString(), anyString(),
anyString(), anyShort()))
+ .thenAnswer(
+ invocation -> {
+ String filter = invocation.getArgument(2);
+ if (icebergAndPaimonFilter.equals(filter)) {
+ return List.of("ice_tbl");
+ }
+
+ if (hudiBaseFilter.equals(filter)) {
+ return List.of("hudi_base");
+ }
+
+ return List.of();
+ });
+ when(clientPool.run(any()))
+ .thenAnswer(
+ invocation -> {
+ ClientPool.Action<?, HiveClient, ?> action =
invocation.getArgument(0);
+ return action.run(hiveClient);
+ });
+ op.clientPool = clientPool;
+
+ NameIdentifier[] tables = op.listTables(Namespace.of("metalake", "hive",
"db"));
+
+ Assertions.assertEquals(2, tables.length);
+ Assertions.assertEquals("hive_tbl", tables[0].name());
+ Assertions.assertEquals("hudi_base_root", tables[1].name());
+ verify(hiveClient)
+ .listTableNamesByFilter(eq("hive"), eq("db"),
eq(icebergAndPaimonFilter), anyShort());
+ verify(hiveClient).listTableNamesByFilter(eq("hive"), eq("db"),
eq(hudiBaseFilter), anyShort());
+ }
+
@Test
void testListViewsUsesTypeListing() throws Exception {
HiveCatalogOperations op = new HiveCatalogOperations();
diff --git a/docs/apache-hive-catalog.md b/docs/apache-hive-catalog.md
index 89ed09c1a3..f27fec46ab 100644
--- a/docs/apache-hive-catalog.md
+++ b/docs/apache-hive-catalog.md
@@ -40,14 +40,36 @@ Besides the [common catalog
properties](./gravitino-server-config.md#apache-grav
| `kerberos.keytab-uri` | The uri of key tab for the
catalog. Now supported protocols are `https`, `http`, `ftp`, `file`.
| (none) |
required if you use kerberos | 0.4.0 |
| `kerberos.check-interval-sec` | The interval to check validness
of the principal
| 60 | No
| 0.4.0 |
| `kerberos.keytab-fetch-timeout-sec` | The timeout to fetch key tab
| 60 | No
| 0.4.0 |
-| `list-all-tables` | Lists all tables in a database,
including non-Hive tables, such as Iceberg, Hudi, etc.
| false | No
| 0.5.1 |
+| `list-all-tables` | Whether to list all tables in a
database, including non-Hive tables such as Iceberg, Paimon, and Hudi. When
false, non-Hive tables are filtered out on a best-effort basis; see the note
below for known limitations. | false | No
| 0.5.1 |
| `default.catalog` | The default catalog name for the
Hive3 metastore backend; this configuration is ignored when using a Hive2
metastore.
| hive | No
| 1.1.0 |
:::note
-For `list-all-tables=false`, the Hive catalog will filter out:
-- Iceberg tables by table property `table_type=ICEBERG`
-- Paimon tables by table property `table_type=PAIMON`
-- Hudi tables by table property `provider=hudi`
+When `list-all-tables=false`, the Hive catalog removes the following on a
best-effort basis:
+- Iceberg tables (table property `table_type=ICEBERG`)
+- Paimon tables (table property `table_type=PAIMON`)
+- Hudi tables (table property `provider=hudi`), together with their `_ro` and
`_rt` siblings
+
+**Known limitation.** Filtering is performed server-side via the Hive
Metastore, which only
+supports exact-key lookups on dot-free property keys. Hudi tables registered
directly by Spark
+(e.g. via `saveAsTable`) typically only set `spark.sql.sources.provider=hudi`
without also
+setting `provider=hudi`, so they cannot be filtered out and will appear in the
listing.
+
+**Workaround.** Add a dot-free `provider=hudi` property to such tables so the
server-side
+filter can match them. Either after creation:
+
+```sql
+ALTER TABLE <db>.<table> SET TBLPROPERTIES ('provider'='hudi');
+```
+
+or at write time via Hudi's Hive sync option:
+
+```scala
+df.write.format("hudi")
+ .option("hoodie.datasource.hive_sync.table_properties", "provider=hudi")
+ .saveAsTable("<db>.<table>")
+```
+
+The corresponding `_ro` / `_rt` siblings are removed automatically based on
the base table name.
:::
When you use the Gravitino with Trino. You can pass the Trino Hive connector
configuration using prefix `trino.bypass.`. For example, using
`trino.bypass.hive.config.resources` to pass the `hive.config.resources` to the
Gravitino Hive catalog in Trino runtime.
