yihua commented on code in PR #19717:
URL: https://github.com/apache/hudi/pull/19717#discussion_r4018987339
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/index/BaseIndexer.java:
##########
@@ -51,6 +56,54 @@ protected BaseIndexer(
this.dataTableMetaClient = dataTableMetaClient;
}
+ /**
+ * Resolves which partition of a definition-driven index type to initialize.
+ * <p>
+ * An indexing action names the partition in the context, and that partition
is initialized
+ * whenever its index definition exists, regardless of how many other
definitions of the type
+ * are still uninitialized. A regular write names nothing, and the partition
is inferred from
+ * the uninitialized definitions: exactly one means that one, any other
count means nothing
+ * is initialized. A requested partition without a definition goes through
the same inference
+ * (that is where a first-time index mints its definition from the write
config), but when the
+ * inference cannot resolve to exactly one partition the action fails rather
than completing
+ * with nothing built and the requested partition marked complete.
+ *
+ * @param context the initialization context
+ * @param uninitializedPartitions the uninitialized partitions of this
type, as the definition
+ * lookup reports them
+ * @param indexType the index type, for the messages
+ * @return the partitions to initialize: exactly one, or none
+ */
+ protected Set<String> resolvePartitionsToInit(IndexInitializationContext
context,
+ Set<String>
uninitializedPartitions,
+ MetadataPartitionType
indexType) {
+ Option<String> requested = context.requestedIndexPartition();
+ if (requested.isPresent()
+ &&
dataTableMetaClient.getTableConfig().getMetadataPartitions().contains(requested.get()))
{
+ // Already initialized, typically by the write that committed between
scheduling and running the
+ // indexing action. Re-initializing would commit at an instant the
metadata table already holds
+ // completed, and the rollback-and-recommit inside that commit destroys
the earlier commit's records.
+ log.info("Metadata partition {} is already initialized, skipping",
requested.get());
+ return Collections.emptySet();
+ }
+ if (requested.isPresent() &&
dataTableMetaClient.getIndexForMetadataPartition(requested.get()).isPresent()) {
+ return Collections.singleton(requested.get());
+ }
+ if (uninitializedPartitions.size() == 1) {
Review Comment:
Good catch, this was real and I have pushed a fix. The single-candidate
branch ran before the requested-but-undefined check, so a HoodieIndexer run
naming an index with no definition yet built whichever other index happened to
be the one uninitialized definition, and RunIndexActionExecutor then marked the
requested partition complete with nothing in it. A single candidate now answers
a request only when it is the requested partition itself, which is what a
first-time index looks like once getIndexPartitionsToInit has minted its
definition; anything else falls through to the throw. TestSecondaryIndexer
gains testFailsWhenTheOnlyUninitializedPartitionIsNotTheRequestedOne for the
bad case and testInitializesTheRequestedPartitionWhoseDefinitionTheLookupMinted
to pin the first-time case that must keep working. Verified the new test fails
without the guard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
