diqiu50 commented on code in PR #11186:
URL: https://github.com/apache/gravitino/pull/11186#discussion_r3315130407
##########
spark-connector/spark-common/src/test/java/org/apache/gravitino/spark/connector/integration/test/util/SparkUtilIT.java:
##########
@@ -134,12 +135,45 @@ protected List<String> getTableMetadata(String
getTableMetadataSql) {
}
// Create SparkTableInfo from SparkBaseTable retrieved from LogicalPlan.
+ // In Spark 3.3/3.5: DESC TABLE EXTENDED returns DescribeRelation.
+ // In Spark 3.4: DESC TABLE EXTENDED returns DescribeTableCommand (different
class hierarchy).
+ // Use the v2 Catalog API (CatalogManager + TableCatalog.loadTable) for
cross-version
+ // compatibility.
protected SparkTableInfo getTableInfo(String tableName) {
- Dataset ds = getSparkSession().sql("DESC TABLE EXTENDED " + tableName);
- CommandResult result = (CommandResult) ds.logicalPlan();
- DescribeRelation relation = (DescribeRelation) result.commandLogicalPlan();
- ResolvedTable table = (ResolvedTable) relation.child();
- return SparkTableInfo.create(table.table());
+ CatalogManager catalogManager =
getSparkSession().sessionState().catalogManager();
+
+ // Parse tableName: could be short (tbl), partially-qualified (db.tbl),
+ // or fully-qualified (cat.db.tbl).
+ String[] parts = tableName.split("\\.");
+ Identifier identifier;
+ TableCatalog tableCatalog;
+ if (parts.length == 1) {
+ // Short table name: use current catalog + current V2 namespace.
+ // catalog().currentDatabase() returns the V1 Hive session catalog
database and is NOT
+ // updated when USE <db> is issued against a V2 catalog (e.g. Glue) in
Spark 3.3.
+ // catalogManager.currentNamespace() reflects the V2 namespace correctly.
+ CatalogPlugin currentCatalog = catalogManager.currentCatalog();
+ String[] currentNamespace = catalogManager.currentNamespace();
+ identifier = Identifier.of(currentNamespace, parts[0]);
+ tableCatalog = (TableCatalog) currentCatalog;
+ } else if (parts.length == 2) {
+ // Partially qualified: db.table
+ identifier = Identifier.of(new String[] {parts[0]}, parts[1]);
+ CatalogPlugin currentCatalog = catalogManager.currentCatalog();
+ tableCatalog = (TableCatalog) currentCatalog;
+ } else if (parts.length == 3) {
+ // Fully qualified: cat.db.table — namespace is only the db part, not
the catalog
+ identifier = Identifier.of(new String[] {parts[1]}, parts[2]);
+ CatalogPlugin catalog = catalogManager.catalog(parts[0]);
+ tableCatalog = (TableCatalog) catalog;
+ } else {
+ throw new IllegalArgumentException("Invalid table name format: " +
tableName);
+ }
+ try {
+ return SparkTableInfo.create(tableCatalog.loadTable(identifier));
+ } catch (NoSuchTableException e) {
+ throw new RuntimeException(e);
Review Comment:
TableCatalog.loadTable() declares throws NoSuchTableException in Spark's
API, It' not a subclass of Runtime Exception ,Without it the code won't
compile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
