nsivabalan commented on code in PR #18197:
URL: https://github.com/apache/hudi/pull/18197#discussion_r2897753630
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1682,4 +1684,25 @@ private InternalSchema
getInternalSchema(TableSchemaResolver schemaUtil) {
}
});
}
+
+ private void runPreWriteCleanerPolicy(HoodieTableMetaClient metaClient) {
+ if (!config.areTableServicesEnabled()) {
+ log.info("Skipping pre-write cleaner policy since table services are
disabled");
+ return;
+ }
+ HoodiePreWriteCleanerPolicy policy = config.getPreWriteCleanerPolicy();
+ if (policy.isNone()) {
+ return;
+ }
+ switch (policy) {
+ case CLEAN:
+ tableServiceClient.clean(Option.empty(), true);
Review Comment:
why can't we call the `clean()` in BaseHoodieWriteClient only?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1682,4 +1684,25 @@ private InternalSchema
getInternalSchema(TableSchemaResolver schemaUtil) {
}
});
}
+
+ private void runPreWriteCleanerPolicy(HoodieTableMetaClient metaClient) {
+ if (!config.areTableServicesEnabled()) {
+ log.info("Skipping pre-write cleaner policy since table services are
disabled");
+ return;
+ }
+ HoodiePreWriteCleanerPolicy policy = config.getPreWriteCleanerPolicy();
+ if (policy.isNone()) {
+ return;
+ }
+ switch (policy) {
+ case CLEAN:
+ tableServiceClient.clean(Option.empty(), true);
+ break;
+ case ROLLBACK_FAILED_WRITES:
+ tableServiceClient.rollbackFailedWrites(metaClient);
Review Comment:
same here. can we call `rollbackFailedWrites(metaClient)` method in
BaseHoodieWriteClient
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCleanConfig.java:
##########
@@ -212,6 +213,12 @@ public class HoodieCleanConfig extends HoodieConfig {
+ "By using local engine context, file listing is performed on the
driver, allowing targeted memory scaling. "
+ "When enabled, both non-partitioned datasets and metadata tables
use the driver for scheduling cleans.");
+ public static final ConfigProperty<String> PREWRITE_CLEANER_POLICY =
ConfigProperty
+ .key("hoodie.cleaner.prewrite.cleaner.policy")
Review Comment:
`hoodie.prewrite.cleaner.policy`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
