nsivabalan commented on code in PR #19045:
URL: https://github.com/apache/hudi/pull/19045#discussion_r3611572286
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -383,6 +383,29 @@ public static final String getDefaultPayloadClassName() {
.sinceVersion("1.1.0")
.withDocumentation("This property when set, will define how two versions
of the record will be merged together when records are partially formed");
+ public static final ConfigProperty<String> METADATA_LAYOUT_CLASS =
ConfigProperty
+ .key("hoodie.metadata.layout.class")
+ .noDefaultValue()
+ .sinceVersion("1.3.0")
+ .withDocumentation("Fully-qualified class name of the
HoodieMetadataTableLayout implementation that organizes "
+ + "MDT file groups on disk. When unset, MDT uses the flat layout
(file groups directly under each metadata "
+ + "partition). Set once at MDT initialization; immutable
thereafter.");
+
+ public static final ConfigProperty<Integer> METADATA_LAYOUT_BUCKET_SIZE =
ConfigProperty
+ .key("hoodie.metadata.layout.bucket.size")
+ .defaultValue(1000)
+ .sinceVersion("1.3.0")
+ .withDocumentation("Layout-specific: maximum number of file groups per
bucket sub-directory when the layout is "
+ + "SubDirBucketedMDTLayout. Ignored otherwise.");
+
+ public static final ConfigProperty<String>
METADATA_LAYOUT_PARTITION_FILE_GROUP_COUNTS = ConfigProperty
+ .key("hoodie.metadata.layout.partition.file.group.counts")
Review Comment:
Great question — worth doing but I would like to keep it out of this patch.
Today `hoodie.metadata.layout.bucketed.file.group.per.bucket` is a single
global value that applies to every MDT partition under the bucketed layout.
Making it configurable per-partition (e.g.
`hoodie.metadata.layout.bucketed.file.group.per.bucket.record_index=…`,
`.files=…`, etc.) is a legitimate follow-up — different MDT partitions have
very different file-group counts in practice (RLI can be O(10k), files
partition is O(10), col_stats depends on schema width), and one bucket size
probably does not fit all.
Filing this as a follow-up in the RFC that will cover the broader layout SPI
(partitioned RLI bucketing lands there too). Happy to loop you in when the RFC
is up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
