yihua commented on code in PR #19426:
URL: https://github.com/apache/hudi/pull/19426#discussion_r3687997595
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -239,9 +240,9 @@ protected void syncHoodieTable(String tableName, boolean
useRealtimeInputFormat,
log.info("Trying to sync hoodie table {} with base path {} of type {}",
tableName, syncClient.getBasePath(), syncClient.getTableType());
final boolean tableExists = syncClient.tableExists(tableName);
- // if table exists and location of the metastore table doesn't match the
hoodie base path, recreate the table
- if (tableExists &&
!FSUtils.comparePathsWithoutScheme(syncClient.getBasePath(),
syncClient.getTableLocation(tableName))) {
- log.info("basepath is updated for the table {}", tableName);
+ // recreate the table if it exists and either its metastore location no
longer matches the hoodie base path,
+ // or a full recreate was explicitly requested
+ if (tableExists && shouldRecreateTable(tableName)) {
Review Comment:
This is a pre-existing issue. Will fix.
##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/HoodieSyncConfig.java:
##########
@@ -182,6 +182,16 @@ public class HoodieSyncConfig extends HoodieConfig {
.withDocumentation("If true, TOUCH partition events will be emitted
during meta sync. "
+ "TOUCH events indicate partitions that exist in both storage and
metastore, no schema or location change, but the partition has received data.");
+ public static final ConfigProperty<Boolean> META_SYNC_FORCE_RECREATE_TABLE =
ConfigProperty
+ .key("hoodie.meta.sync.force.recreate.table")
Review Comment:
Given that the force recreate table can be applied to any catalog sync, it
is intentional to make the config generic, rather than having catalog-specific
config. There will be follow-up PR to make other catalog follow the same.
This PR intends to support Hive and Glue catalogs only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
