nsivabalan commented on code in PR #17935:
URL: https://github.com/apache/hudi/pull/17935#discussion_r2720313593
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -1011,6 +1024,11 @@ public Builder withLogCompactBlocksThreshold(int
logCompactBlocksThreshold) {
return this;
}
+ public HoodieMetadataConfig.Builder withCleanerPolicy(HoodieCleaningPolicy
policy) {
Review Comment:
We need to add tests.
check out `TestHoodieMetadataConfig`
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -151,6 +153,13 @@ public final class HoodieMetadataConfig extends
HoodieConfig {
.sinceVersion("0.14.0")
.withDocumentation("Controls the criteria to log compacted files groups
in metadata table.");
+ public static final ConfigProperty<String> CLEANER_POLICY = ConfigProperty
+ .key(METADATA_PREFIX + ".clean.policy")
+ .defaultValue(KEEP_LATEST_FILE_VERSIONS.name())
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("This config determines the cleaner policy for
metadata table.");
Review Comment:
oh, here we are only adding the policy override. the values required for
these policies are still derived from data table.
but are we not making a assumption here that, both data table and metadata
will be using same clean policy.
what incase users configure different policy for each data table and mdt
table.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -151,6 +153,13 @@ public final class HoodieMetadataConfig extends
HoodieConfig {
.sinceVersion("0.14.0")
.withDocumentation("Controls the criteria to log compacted files groups
in metadata table.");
+ public static final ConfigProperty<String> CLEANER_POLICY = ConfigProperty
+ .key(METADATA_PREFIX + ".clean.policy")
+ .defaultValue(KEEP_LATEST_FILE_VERSIONS.name())
+ .markAdvanced()
+ .sinceVersion("1.2.0")
+ .withDocumentation("This config determines the cleaner policy for
metadata table.");
Review Comment:
may be, if the policy does not align between data table and mdt, we could
throw exceptions.
and also add a configuration for the multiplier. leave the default as 1.2 x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
