[ https://issues.apache.org/jira/browse/HUDI-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinoth Chandar updated HUDI-6117: --------------------------------- Fix Version/s: (was: 1.0.0) > Parallelize creation of initial file groups for MDT partitions > -------------------------------------------------------------- > > Key: HUDI-6117 > URL: https://issues.apache.org/jira/browse/HUDI-6117 > Project: Apache Hudi > Issue Type: Improvement > Reporter: Prashant Wason > Assignee: Prashant Wason > Priority: Major > Labels: pull-request-available > Fix For: 0.14.0 > > > When there are large number of file groups in a MDT partition (record index > having billions of records can have 10k+ partitions, creating the initial log > files in a for-loop can take a long time (100ms per create = 1000 seconds = > 16mins) but routinely this latency is as high as 500msec / create. > The initial file group creation can be optimized to be done parallelly > speeding up MDT partition initialization. -- This message was sent by Atlassian Jira (v8.20.10#820010)