Davis-Zhang-Onehouse commented on code in PR #19717:
URL: https://github.com/apache/hudi/pull/19717#discussion_r3845659591
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -435,8 +453,16 @@ private void initializeMetadataPartition(
Indexer indexer,
String dataTableInstantTime,
Map<String, List<FileInfo>> partitionToAllFilesMap,
Review Comment:
Done, reworded to 'is treated as a partially applied earlier commit'.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -820,6 +861,10 @@ public void buildMetadataPartitions(HoodieEngineContext
engineContext, List<Hood
String indexUptoInstantTime =
indexPartitionInfos.get(0).getIndexUptoInstant();
List<String> partitionPaths = new ArrayList<>();
List<MetadataPartitionType> partitionTypes = new ArrayList<>();
+ // The plan names partition paths; for the index types that cover many
partitions that is the only thing
Review Comment:
Done, split into two sentences keyed to path-vs-type.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -435,8 +453,16 @@ private void initializeMetadataPartition(
Indexer indexer,
String dataTableInstantTime,
Map<String, List<FileInfo>> partitionToAllFilesMap,
- Lazy<List<FileSliceAndPartition>> lazyMergedFileSlices) throws
IOException {
- String instantTimeForPartition =
generateUniqueInstantTime(dataTableInstantTime);
+ Lazy<List<FileSliceAndPartition>> lazyMergedFileSlices,
+ Option<String> requestedIndexPartition) throws IOException {
+ // A requested partition initializes under a fresh solo-family instant,
never the indexing
+ // action's own instant. The action's completion applies its data commit
to the metadata table
+ // too, and finding that instant already completed there reads as a
partial earlier application:
+ // it is rolled back and re-applied, destroying the initialization records
while leaving the
+ // file groups. The solo family is the established shape for
metadata-table-only bootstrap
+ // commits and survives that reconciliation.
+ String instantTimeForPartition = requestedIndexPartition.isPresent()
Review Comment:
Intended, with one nuance: the solo instant applies only when the requested
partition's index definition already exists at initialization time.
HoodieIndexer's first-time, config-driven index has no definition yet at that
point, so it takes the definition-lookup branch and keeps the previous instant
behavior; CREATE INDEX (HoodieSparkIndexClient registers the definition before
scheduling) and a rebuild of an existing definition take the solo instant. The
switch is the point of the second commit: committing under the action's own
instant collides with the action's completion applying that same data commit to
the metadata table, and whichever lands second rolls the other back, leaving
file groups with no records. Coverage: TestHoodieIndexer (18, including
testIndexerForSecondaryIndex and testIndexerForExpressionIndex via
HoodieIndexer) and TestSecondaryIndexPruning (36, CREATE INDEX flows including
two-index sequences that read the index contents back) both pass with the solo
instan
t; the new
testIndexerBuildsTheRequestedSecondaryIndexWhenSeveralAreUninitialized asserts
the built partition has file slices on storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
