danny0405 commented on code in PR #19304:
URL: https://github.com/apache/hudi/pull/19304#discussion_r4043490458
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1544,6 +1546,38 @@ protected boolean loadActiveTimelineOnTableInit() {
return true;
}
+ /**
+ * Resolves the record key encoding of a single-field {@code
ComplexKeyGenerator} table onto the write config,
+ * so that every key generator built from it keys records the way the
table's data is keyed. A table that does
+ * not carry {@link HoodieTableConfig#COMPLEX_KEYGEN_ENCODING} yet gets it
deduced from its data and backfilled
+ * under the transaction lock, or the write fails when the encoding cannot
be determined and
+ * {@code hoodie.write.complex.keygen.validation.enable} is on.
+ * Public because the streamer keys its records before {@link #initTable}
runs and has to call this itself.
+ */
+ public void resolveComplexKeygenEncoding(HoodieTableMetaClient metaClient) {
+ if
(!KeyGenUtils.isComplexKeyGenEncodingTracked(metaClient.getTableConfig())) {
+ return;
+ }
+ if (!metaClient.getTableConfig().getComplexKeyGenEncoding().isPresent()) {
+ executeUsingTxnManager(Option.empty(), () ->
backfillComplexKeygenEncoding(metaClient));
+ }
+ config.setValue(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING,
metaClient.getTableConfig().getComplexKeyGenEncoding().get().name());
+ }
+
+ private void backfillComplexKeygenEncoding(HoodieTableMetaClient metaClient)
{
Review Comment:
didn't get why we set up the table config inside the write config, they
should be different configurations, write config can be updated freely during
execution while table config is kind of immutable.
The new option should only be applied to the `ComplexAvroKeyGenerator` which
already has a `TypedProperties` in its constructor, so why not set up the props
before instantiating the `ComplexAvroKeyGenerator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
