lokeshj1703 commented on code in PR #14138:
URL: https://github.com/apache/hudi/pull/14138#discussion_r2455295108
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java:
##########
@@ -294,12 +295,24 @@ public static void
dropNonV1IndexPartitions(HoodieWriteConfig config, HoodieEngi
HoodieTable table,
SupportsUpgradeDowngrade upgradeDowngradeHelper, String operationType) {
HoodieTableMetaClient metaClient = table.getMetaClient();
try (BaseHoodieWriteClient writeClient =
upgradeDowngradeHelper.getWriteClient(config, context)) {
- List<String> mdtPartitions =
metaClient.getTableConfig().getMetadataPartitions()
+ Set<String> metadataPartitions =
metaClient.getTableConfig().getMetadataPartitions();
+ List<String> mdtPartitions = metadataPartitions
.stream()
.filter(partition ->
metaClient.getIndexForMetadataPartition(partition)
.map(indexDef ->
HoodieIndexVersion.V1.lowerThan(indexDef.getVersion()))
.orElse(false))
.collect(Collectors.toList());
+
+ // If col stats V2 is being deleted and partition stats exists, delete
partition stats as well
+ // This handles the case where partition stats might not have an index
definition in index.json
+ String colStatsPartition =
MetadataPartitionType.COLUMN_STATS.getPartitionPath();
+ String partitionStatsPartition =
MetadataPartitionType.PARTITION_STATS.getPartitionPath();
Review Comment:
Is there a possibility that column stats is V1 but partition stats is v2 and
without index definition?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1112,9 +1112,14 @@ public void dropIndex(List<String> metadataPartitions) {
HoodieTableMetaClient metaClient = table.getMetaClient();
// For secondary index and expression index with wrong parameters, index
definition for the MDT partition is
// removed so that such indices are not recreated while initializing the
writer.
+ // Also remove index definitions for col stats and partition stats when
they are dropped (e.g., during downgrade).
metadataPartitions.forEach(partition -> {
- if (MetadataPartitionType.isExpressionOrSecondaryIndex(partition)) {
- metaClient.deleteIndexDefinition(partition);
+ if (MetadataPartitionType.isExpressionOrSecondaryIndex(partition)
+ ||
partition.equals(MetadataPartitionType.COLUMN_STATS.getPartitionPath())
+ ||
partition.equals(MetadataPartitionType.PARTITION_STATS.getPartitionPath())) {
Review Comment:
I think we can remove these conditions and allow index definition deletion
to proceed if the index is getting dropped.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java:
##########
@@ -294,12 +295,24 @@ public static void
dropNonV1IndexPartitions(HoodieWriteConfig config, HoodieEngi
HoodieTable table,
SupportsUpgradeDowngrade upgradeDowngradeHelper, String operationType) {
HoodieTableMetaClient metaClient = table.getMetaClient();
try (BaseHoodieWriteClient writeClient =
upgradeDowngradeHelper.getWriteClient(config, context)) {
- List<String> mdtPartitions =
metaClient.getTableConfig().getMetadataPartitions()
+ Set<String> metadataPartitions =
metaClient.getTableConfig().getMetadataPartitions();
+ List<String> mdtPartitions = metadataPartitions
.stream()
.filter(partition ->
metaClient.getIndexForMetadataPartition(partition)
.map(indexDef ->
HoodieIndexVersion.V1.lowerThan(indexDef.getVersion()))
.orElse(false))
.collect(Collectors.toList());
+
+ // If col stats V2 is being deleted and partition stats exists, delete
partition stats as well
+ // This handles the case where partition stats might not have an index
definition in index.json
Review Comment:
Can we create another GH issue to track the partition stats index definition
not getting created issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
