singhpk234 commented on code in PR #8931:
URL: https://github.com/apache/iceberg/pull/8931#discussion_r1381933633
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/MigrateTableSparkAction.java:
##########
@@ -108,6 +109,23 @@ public MigrateTableSparkAction backupTableName(String
tableName) {
return this;
}
+ @Override
+ public MigrateTableSparkAction destCatalogName(String catalogName) {
+ CatalogManager catalogManager = spark().sessionState().catalogManager();
+
+ CatalogPlugin catalogPlugin;
+ if (catalogManager.isCatalogRegistered(catalogName)) {
+ catalogPlugin = catalogManager.catalog(catalogName);
+ } else {
+ LOG.warn(
+ "{} doesn't exist in SparkSession. " + "Fallback to current
SparkSession catalog.",
+ catalogName);
+ catalogPlugin = catalogManager.currentCatalog();
+ }
+ this.destCatalog = checkDestinationCatalog(catalogPlugin);
Review Comment:
> In this part, the renameTable checks if the source table is Iceberg or
not. However, for the migrate procedure, the source table must be spark_catalog
(for example, if the glue_catalog is specified, it fails as described above).
I see the source table being non-iceberg table is making the check of rename
fail and it defers to V2SessionCatalog.rename, and this step can't be skipped
as we want to rename the source to backup table to halt the writes. wondering
then adding the support for rename in GlueMetaStore client would be best, but
it comes with it's own challenges frankly, how we have implemented rename in
GlueCatalog of iceberg is a bit tricky we read the source table, create a
destination table and then delete the source table (As glue doesn't have rename
API). I am not sure if something goes south will we be able to correctly
restore the source table state from the backup.
for ex, i am assuming non-iceberg table would track partition info also in
glue which i don't think is correctly being propagated in iceberg GlueCatalog
impl, thoughts :
https://github.com/apache/iceberg/blob/2c890109c17bfb971490cab007be23029b81bad8/aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java#L383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
