[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1227127259 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java: ## @@ -0,0 +1,111 @@ +/* + * Licensed to the A

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1227126602 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java: ## @@ -0,0 +1,111 @@ +/* + * Licensed to the A

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1219045032 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -626,8 +657,7 @@ private void initializeFileGrou

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1219043175 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java: ## @@ -338,14 +338,20 @@ protected void initMetadataTable(Option ins

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1219042633 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java: ## @@ -338,14 +338,20 @@ protected void initMetadataTable(Option ins

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1203532474 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -1052,51 +1091,81 @@ protected HoodieData prepR

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1203530662 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -760,15 +815,14 @@ private void initializeFileGr

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1203530124 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -564,53 +532,147 @@ private boolean isCommitRe

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1203525353 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -375,105 +357,91 @@ public List getEnabledParti

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1203521583 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -489,7 +457,7 @@ private boolean metadataTable

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1203518705 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java: ## @@ -337,15 +337,33 @@ protected void initMetadataTable(Option ins

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1203503606 ## hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java: ## @@ -694,17 +695,75 @@ private Long getTableChecksum() { return getLong(TABL

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1194268603 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -1097,87 +1165,76 @@ protected void cleanIfNeces

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1194268190 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -1097,87 +1165,76 @@ protected void cleanIfNeces

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1194267719 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -1097,87 +1165,76 @@ protected void cleanIfNeces

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1194226408 ## hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java: ## @@ -2482,6 +2483,14 @@ public void testMetadataMet

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1194213033 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java: ## @@ -0,0 +1,111 @@ +/* + * Licensed to the A

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1194209832 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -824,25 +863,22 @@ private interface ConvertMeta

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1194208983 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -562,53 +532,144 @@ private boolean isCommitRe

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1194204960 ## hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java: ## @@ -670,17 +670,75 @@ private Long getTableChecksum() { return getLong(TABL

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192180200 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -1378,6 +1339,206 @@ public static Set getInflightAndCompletedMetadataPart

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192169835 ## hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java: ## @@ -1378,6 +1339,206 @@ public static Set getInflightAndCompletedMetadataPart

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192166826 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -873,17 +908,7 @@ public void buildMetadataParti

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192164647 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java: ## @@ -118,46 +123,32 @@ protected void initRegist

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192162585 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -824,25 +863,22 @@ private interface ConvertMeta

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192161869 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -824,25 +863,22 @@ private interface ConvertMeta

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192161017 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -619,41 +680,34 @@ private boolean anyPendingDa

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192160408 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -373,105 +356,92 @@ public List getEnabledParti

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192159380 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -562,53 +532,144 @@ private boolean isCommitRe

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192156057 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java: ## @@ -562,53 +532,144 @@ private boolean isCommitRe

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192140304 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/ScheduleIndexActionExecutor.java: ## @@ -100,15 +99,6 @@ public Option execute() {

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192139220 ## hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java: ## @@ -33,10 +33,6 @@ public enum MetadataPartitionType { private final String p

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192138716 ## hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java: ## @@ -670,17 +670,75 @@ private Long getTableChecksum() { return getLong(TABL

[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.

prashantwason commented on code in PR #8684: URL: https://github.com/apache/hudi/pull/8684#discussion_r1192137497 ## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java: ## @@ -0,0 +1,109 @@ +/* + * Licensed to the A