yuqi1129 commented on code in PR #11087:
URL: https://github.com/apache/gravitino/pull/11087#discussion_r3257160955
##########
catalogs/catalog-glue/build.gradle.kts:
##########
@@ -36,9 +36,14 @@ dependencies {
}
implementation(libs.aws.glue)
+ implementation(libs.aws.kms)
+ implementation(libs.aws.s3)
implementation(libs.aws.sts)
implementation(libs.commons.lang3)
implementation(libs.guava)
+ implementation(libs.iceberg.core)
+ implementation(libs.iceberg.api)
Review Comment:
sort this part.
##########
catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogOperations.java:
##########
@@ -173,6 +173,13 @@ public void close() throws IOException {
glueClient.close();
glueClient = null;
}
+ if (icebergGlueCatalog instanceof AutoCloseable) {
Review Comment:
You can use `icebergGlueCatalog instanceof AutoCloseable closeable `
##########
catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogOperations.java:
##########
@@ -148,12 +147,13 @@ public void initialize(
.map(s -> s.toLowerCase(Locale.ROOT))
.collect(Collectors.toSet());
}
+ this.icebergGlueCatalog = GlueIcebergTableHelper.createGlueCatalog(config);
}
@Override
public void testConnection(
NameIdentifier catalogIdent,
- Catalog.Type type,
+ org.apache.gravitino.Catalog.Type type,
Review Comment:
Why do we change to use the full qualified name here?
##########
catalogs/catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueCatalogOperations.java:
##########
@@ -351,8 +371,26 @@ public GlueTable loadTable(NameIdentifier ident) throws
NoSuchTableException {
GetTableRequest.Builder req =
GetTableRequest.builder().databaseName(dbName).name(ident.name());
applyCatalogId(catalogId, req::catalogId);
try {
- GlueTable table =
- GlueTable.fromGlueTable(glueClient.getTable(req.build()).table(),
typeConverter);
+ software.amazon.awssdk.services.glue.model.Table rawGlueTable =
+ glueClient.getTable(req.build()).table();
+ GlueTable table = GlueTable.fromGlueTable(rawGlueTable, typeConverter);
+
+ // Recover Iceberg-specific partitioning and sort orders from the
Iceberg metadata.
+ // AWS Glue Table.partitionKeys() is empty for Iceberg tables, so we
load the Iceberg
+ // Table to obtain the accurate partition spec and sort order.
+ if (GlueIcebergTableHelper.isIcebergTable(rawGlueTable) &&
icebergGlueCatalog != null) {
+ try {
+ GlueIcebergTableHelper.loadTable(icebergGlueCatalog, dbName,
ident.name(), table);
+ } catch (Exception e) {
+ LOG.error(
Review Comment:
If it's generally okay that we can't load information via
`GlueIcebergTableHelper.loadTable`, warnning level may be more proper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
