yihua commented on code in PR #19304:
URL: https://github.com/apache/hudi/pull/19304#discussion_r4102160327
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -419,9 +446,7 @@ public static boolean
isAutoGeneratedRecordKeysEnabled(TypedProperties props) {
}
public static boolean
isComplexKeyGeneratorWithSingleRecordKeyField(HoodieTableConfig tableConfig) {
- Option<String[]> recordKeyFields = tableConfig.getRecordKeyFields();
- return KeyGeneratorType.isComplexKeyGenerator(tableConfig)
- && recordKeyFields.isPresent() && recordKeyFields.get().length == 1;
+ return tableConfig.isComplexKeyGenWithSingleRecordKeyField();
}
Review Comment:
Addressed: the wrapper is gone, callers use
`tableConfig.isComplexKeyGenWithSingleRecordKeyField()` directly.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -436,15 +461,213 @@ public static String getComplexKeygenErrorMessage(String
operation) {
+ "`hoodie.write.complex.keygen.validation.enable=false` to skip this
validation.";
}
+ /**
+ * Whether a complex key generator with a single record key field prepends
the field name to the key.
+ *
+ * <p>{@link HoodieTableConfig#COMPLEX_KEYGEN_ENCODING} wins whenever it is
present in the properties, because
+ * it describes what the table's data carries. Otherwise the write table
version decides: 9 and above always
+ * prefix, 8 and below follow {@code
hoodie.write.complex.keygen.new.encoding}.
+ */
public static boolean
encodeSingleKeyFieldNameForComplexKeyGen(TypedProperties props) {
+ String tableEncoding =
props.getProperty(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING.key());
Review Comment:
Addressed: reads through `ConfigUtils.getStringWithAltKeys` now.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -436,15 +461,213 @@ public static String getComplexKeygenErrorMessage(String
operation) {
+ "`hoodie.write.complex.keygen.validation.enable=false` to skip this
validation.";
}
+ /**
+ * Whether a complex key generator with a single record key field prepends
the field name to the key.
+ *
+ * <p>{@link HoodieTableConfig#COMPLEX_KEYGEN_ENCODING} wins whenever it is
present in the properties, because
+ * it describes what the table's data carries. Otherwise the write table
version decides: 9 and above always
+ * prefix, 8 and below follow {@code
hoodie.write.complex.keygen.new.encoding}.
+ */
public static boolean
encodeSingleKeyFieldNameForComplexKeyGen(TypedProperties props) {
+ String tableEncoding =
props.getProperty(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING.key());
+ if (!StringUtils.isNullOrEmpty(tableEncoding)) {
+ return
ComplexKeyGenEncoding.fromString(tableEncoding).encodesFieldName();
+ }
int tableVersionCode = ConfigUtils.getIntWithAltKeys(props,
WRITE_TABLE_VERSION);
HoodieTableVersion tableVersion =
HoodieTableVersion.fromVersionCode(tableVersionCode);
return tableVersion.greaterThanOrEquals(HoodieTableVersion.NINE)
|| !ConfigUtils.getBooleanWithAltKeys(props,
COMPLEX_KEYGEN_NEW_ENCODING);
}
- public static boolean mayUseNewEncodingForComplexKeyGen(HoodieTableConfig
tableConfig) {
- return tableConfig.getTableVersion().lesserThan(HoodieTableVersion.NINE)
- && isComplexKeyGeneratorWithSingleRecordKeyField(tableConfig);
+ /**
+ * Whether the table's record key encoding is tracked by {@link
HoodieTableConfig#COMPLEX_KEYGEN_ENCODING}:
+ * a complex key generator with a single record key field and a populated
{@code _hoodie_record_key}.
+ * Without the meta field there is no stored key whose encoding could
diverge from the key generator's.
+ */
+ public static boolean isComplexKeyGenEncodingTracked(HoodieTableConfig
tableConfig) {
Review Comment:
Addressed: renamed to `requireComplexKeyGenEncodingTracked`.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -436,15 +461,213 @@ public static String getComplexKeygenErrorMessage(String
operation) {
+ "`hoodie.write.complex.keygen.validation.enable=false` to skip this
validation.";
}
+ /**
+ * Whether a complex key generator with a single record key field prepends
the field name to the key.
+ *
+ * <p>{@link HoodieTableConfig#COMPLEX_KEYGEN_ENCODING} wins whenever it is
present in the properties, because
+ * it describes what the table's data carries. Otherwise the write table
version decides: 9 and above always
+ * prefix, 8 and below follow {@code
hoodie.write.complex.keygen.new.encoding}.
+ */
public static boolean
encodeSingleKeyFieldNameForComplexKeyGen(TypedProperties props) {
+ String tableEncoding =
props.getProperty(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING.key());
+ if (!StringUtils.isNullOrEmpty(tableEncoding)) {
+ return
ComplexKeyGenEncoding.fromString(tableEncoding).encodesFieldName();
+ }
int tableVersionCode = ConfigUtils.getIntWithAltKeys(props,
WRITE_TABLE_VERSION);
HoodieTableVersion tableVersion =
HoodieTableVersion.fromVersionCode(tableVersionCode);
return tableVersion.greaterThanOrEquals(HoodieTableVersion.NINE)
|| !ConfigUtils.getBooleanWithAltKeys(props,
COMPLEX_KEYGEN_NEW_ENCODING);
}
- public static boolean mayUseNewEncodingForComplexKeyGen(HoodieTableConfig
tableConfig) {
- return tableConfig.getTableVersion().lesserThan(HoodieTableVersion.NINE)
- && isComplexKeyGeneratorWithSingleRecordKeyField(tableConfig);
+ /**
+ * Whether the table's record key encoding is tracked by {@link
HoodieTableConfig#COMPLEX_KEYGEN_ENCODING}:
+ * a complex key generator with a single record key field and a populated
{@code _hoodie_record_key}.
+ * Without the meta field there is no stored key whose encoding could
diverge from the key generator's.
+ */
+ public static boolean isComplexKeyGenEncodingTracked(HoodieTableConfig
tableConfig) {
+ return tableConfig.isComplexKeyGenWithSingleRecordKeyField() &&
tableConfig.isRecordKeyPopulated();
+ }
+
+ /**
+ * Records a missing complex key generator encoding before ingestion starts.
Engine entry points call this
+ * once during setup, before creating record keys. Existing data determines
the encoding under the table lock.
+ */
+ public static void
recordComplexKeygenEncodingIfMissing(HoodieTableMetaClient metaClient,
HoodieWriteConfig config) {
+ if (!isComplexKeyGenEncodingTracked(metaClient.getTableConfig())
+ || metaClient.getTableConfig().getComplexKeyGenEncoding().isPresent())
{
+ return;
+ }
+ try (TransactionManager transactionManager = new
TransactionManager(config, metaClient.getStorage())) {
+ transactionManager.beginStateChange(Option.empty(), Option.empty());
+ try {
+ metaClient.reloadTableConfig();
+ if
(metaClient.getTableConfig().getComplexKeyGenEncoding().isPresent()) {
+ return;
+ }
+ ComplexKeyGenEncoding encoding =
resolveComplexKeyGenEncodingForWrite(metaClient, config)
+ .orElseThrow(() -> new
HoodieException(getComplexKeygenErrorMessage("ingestion")));
+ Properties props = new Properties();
+ props.setProperty(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING.key(),
encoding.name());
+ HoodieTableConfig.update(metaClient.getStorage(),
metaClient.getMetaPath(), props);
+ metaClient.reloadTableConfig();
+ LOG.info("Recorded complex keygen record key encoding {} on table {}",
encoding, metaClient.getBasePath());
+ } finally {
+ transactionManager.endStateChange(Option.empty());
+ }
+ }
+ }
+
+ /**
+ * Resolves the record key encoding of a single-field complex key generator
table for readers, without
+ * modifying the table: the persisted property, otherwise the encoding
deduced from the table's data.
+ *
+ * @return empty when the table does not use a single-field complex key
generator, or when its encoding
+ * cannot be determined, in which case record-key based file pruning must be
skipped
+ */
+ public static Option<ComplexKeyGenEncoding>
resolveComplexKeyGenEncoding(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (!tableConfig.isComplexKeyGenWithSingleRecordKeyField()) {
+ return Option.empty();
+ }
+ if (!tableConfig.isRecordKeyPopulated()) {
+ // the keys are regenerated by the key generator, which follows the
table version
+ return
tableConfig.getTableVersion().greaterThanOrEquals(HoodieTableVersion.NINE)
+ ? Option.of(ComplexKeyGenEncoding.FIELD_PREFIXED) : Option.empty();
+ }
+ Option<ComplexKeyGenEncoding> persisted =
tableConfig.getComplexKeyGenEncoding();
+ return persisted.isPresent() ? persisted :
deduceComplexKeyGenEncodingFromData(metaClient);
+ }
+
+ /**
+ * Resolves the record key encoding a writer must use and persist on a table
whose encoding is tracked
+ * ({@link #isComplexKeyGenEncodingTracked}) but not yet recorded: the
encoding deduced from the data,
+ * otherwise the configured {@code hoodie.write.complex.keygen.new.encoding}
when
+ * {@code hoodie.write.complex.keygen.validation.enable} is false.
+ *
+ * @return empty when the encoding cannot be determined and the validation
is enabled; the caller fails
+ * the operation with {@link #getComplexKeygenErrorMessage}
+ */
+ public static Option<ComplexKeyGenEncoding>
resolveComplexKeyGenEncodingForWrite(HoodieTableMetaClient metaClient,
+
HoodieWriteConfig config) {
+ Option<ComplexKeyGenEncoding> deduced =
deduceComplexKeyGenEncodingFromData(metaClient);
+ if (deduced.isPresent() || config.enableComplexKeygenValidation()) {
+ return deduced;
+ }
+ ComplexKeyGenEncoding configured =
+
ComplexKeyGenEncoding.fromUseNewEncoding(config.getBooleanOrDefault(COMPLEX_KEYGEN_NEW_ENCODING));
+ LOG.warn("Could not determine the record key encoding of table {} from its
data; recording the configured {} "
+ + "because {} is disabled. If the existing keys are not {}, fix {}
and rerun, or the records written "
+ + "from now on will not match the existing ones.",
+ metaClient.getBasePath(), configured,
ENABLE_COMPLEX_KEYGEN_VALIDATION.key(), configured,
+ COMPLEX_KEYGEN_NEW_ENCODING.key());
+ return Option.of(configured);
+ }
+
+ /**
+ * Deduces the record key encoding of a single-field complex key generator
table from the
+ * {@code _hoodie_record_key} stored in its most recently written data file.
+ *
+ * @return {@link ComplexKeyGenEncoding#FIELD_PREFIXED} for a table that
never had data files, the encoding
+ * read from the first readable base or log file otherwise, or empty when
the table has (or had) data files
+ * but none of them yields a record key
+ */
+ public static Option<ComplexKeyGenEncoding>
deduceComplexKeyGenEncodingFromData(HoodieTableMetaClient metaClient) {
+ String expectedPrefix =
metaClient.getTableConfig().getRecordKeyFields().get()[0] +
DEFAULT_COLUMN_VALUE_SEPARATOR;
+ HoodieTimeline completedTimeline =
metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants();
+ boolean hasDataFiles = false;
+ for (HoodieInstant instant :
completedTimeline.getReverseOrderedInstants().collect(Collectors.toList())) {
+ for (HoodieWriteStat writeStat : getWriteStats(instant,
completedTimeline)) {
+ if (StringUtils.isNullOrEmpty(writeStat.getPath())) {
+ continue;
+ }
+ hasDataFiles = true;
+ StoragePath path = new StoragePath(metaClient.getBasePath(),
writeStat.getPath());
+ Option<String> recordKey = readFirstRecordKey(metaClient, path);
+ if (recordKey.isPresent()) {
+ ComplexKeyGenEncoding encoding =
+
ComplexKeyGenEncoding.fromUseNewEncoding(!recordKey.get().startsWith(expectedPrefix));
+ LOG.info("Deduced complex keygen record key encoding {} of table {}
from {}",
+ encoding, metaClient.getBasePath(), path);
+ return Option.of(encoding);
+ }
+ }
+ }
+ if (hasDataFiles || hasArchivedCommits(metaClient)) {
+ LOG.warn("No data file with a readable record key found in table {}; the
complex keygen record key "
+ + "encoding cannot be deduced", metaClient.getBasePath());
+ return Option.empty();
+ }
+ return Option.of(ComplexKeyGenEncoding.FIELD_PREFIXED);
+ }
+
+ /**
+ * Data files may have been written by commits that have since been
archived; only a table without any is new.
+ * A listing of the archive folder answers that without loading the archived
timeline.
+ */
+ private static boolean hasArchivedCommits(HoodieTableMetaClient metaClient) {
Review Comment:
Addressed: `hasArchivedCommits` is gone, no archive listing.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -436,15 +461,213 @@ public static String getComplexKeygenErrorMessage(String
operation) {
+ "`hoodie.write.complex.keygen.validation.enable=false` to skip this
validation.";
}
+ /**
+ * Whether a complex key generator with a single record key field prepends
the field name to the key.
+ *
+ * <p>{@link HoodieTableConfig#COMPLEX_KEYGEN_ENCODING} wins whenever it is
present in the properties, because
+ * it describes what the table's data carries. Otherwise the write table
version decides: 9 and above always
+ * prefix, 8 and below follow {@code
hoodie.write.complex.keygen.new.encoding}.
+ */
public static boolean
encodeSingleKeyFieldNameForComplexKeyGen(TypedProperties props) {
+ String tableEncoding =
props.getProperty(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING.key());
+ if (!StringUtils.isNullOrEmpty(tableEncoding)) {
+ return
ComplexKeyGenEncoding.fromString(tableEncoding).encodesFieldName();
+ }
int tableVersionCode = ConfigUtils.getIntWithAltKeys(props,
WRITE_TABLE_VERSION);
HoodieTableVersion tableVersion =
HoodieTableVersion.fromVersionCode(tableVersionCode);
return tableVersion.greaterThanOrEquals(HoodieTableVersion.NINE)
|| !ConfigUtils.getBooleanWithAltKeys(props,
COMPLEX_KEYGEN_NEW_ENCODING);
}
- public static boolean mayUseNewEncodingForComplexKeyGen(HoodieTableConfig
tableConfig) {
- return tableConfig.getTableVersion().lesserThan(HoodieTableVersion.NINE)
- && isComplexKeyGeneratorWithSingleRecordKeyField(tableConfig);
+ /**
+ * Whether the table's record key encoding is tracked by {@link
HoodieTableConfig#COMPLEX_KEYGEN_ENCODING}:
+ * a complex key generator with a single record key field and a populated
{@code _hoodie_record_key}.
+ * Without the meta field there is no stored key whose encoding could
diverge from the key generator's.
+ */
+ public static boolean isComplexKeyGenEncodingTracked(HoodieTableConfig
tableConfig) {
+ return tableConfig.isComplexKeyGenWithSingleRecordKeyField() &&
tableConfig.isRecordKeyPopulated();
+ }
+
+ /**
+ * Records a missing complex key generator encoding before ingestion starts.
Engine entry points call this
+ * once during setup, before creating record keys. Existing data determines
the encoding under the table lock.
+ */
+ public static void
recordComplexKeygenEncodingIfMissing(HoodieTableMetaClient metaClient,
HoodieWriteConfig config) {
+ if (!isComplexKeyGenEncodingTracked(metaClient.getTableConfig())
+ || metaClient.getTableConfig().getComplexKeyGenEncoding().isPresent())
{
+ return;
+ }
+ try (TransactionManager transactionManager = new
TransactionManager(config, metaClient.getStorage())) {
+ transactionManager.beginStateChange(Option.empty(), Option.empty());
+ try {
+ metaClient.reloadTableConfig();
+ if
(metaClient.getTableConfig().getComplexKeyGenEncoding().isPresent()) {
+ return;
+ }
+ ComplexKeyGenEncoding encoding =
resolveComplexKeyGenEncodingForWrite(metaClient, config)
+ .orElseThrow(() -> new
HoodieException(getComplexKeygenErrorMessage("ingestion")));
+ Properties props = new Properties();
+ props.setProperty(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING.key(),
encoding.name());
+ HoodieTableConfig.update(metaClient.getStorage(),
metaClient.getMetaPath(), props);
+ metaClient.reloadTableConfig();
+ LOG.info("Recorded complex keygen record key encoding {} on table {}",
encoding, metaClient.getBasePath());
+ } finally {
+ transactionManager.endStateChange(Option.empty());
+ }
+ }
+ }
+
+ /**
+ * Resolves the record key encoding of a single-field complex key generator
table for readers, without
+ * modifying the table: the persisted property, otherwise the encoding
deduced from the table's data.
+ *
+ * @return empty when the table does not use a single-field complex key
generator, or when its encoding
+ * cannot be determined, in which case record-key based file pruning must be
skipped
+ */
+ public static Option<ComplexKeyGenEncoding>
resolveComplexKeyGenEncoding(HoodieTableMetaClient metaClient) {
+ HoodieTableConfig tableConfig = metaClient.getTableConfig();
+ if (!tableConfig.isComplexKeyGenWithSingleRecordKeyField()) {
+ return Option.empty();
+ }
+ if (!tableConfig.isRecordKeyPopulated()) {
+ // the keys are regenerated by the key generator, which follows the
table version
+ return
tableConfig.getTableVersion().greaterThanOrEquals(HoodieTableVersion.NINE)
+ ? Option.of(ComplexKeyGenEncoding.FIELD_PREFIXED) : Option.empty();
+ }
+ Option<ComplexKeyGenEncoding> persisted =
tableConfig.getComplexKeyGenEncoding();
+ return persisted.isPresent() ? persisted :
deduceComplexKeyGenEncodingFromData(metaClient);
+ }
+
+ /**
+ * Resolves the record key encoding a writer must use and persist on a table
whose encoding is tracked
+ * ({@link #isComplexKeyGenEncodingTracked}) but not yet recorded: the
encoding deduced from the data,
+ * otherwise the configured {@code hoodie.write.complex.keygen.new.encoding}
when
+ * {@code hoodie.write.complex.keygen.validation.enable} is false.
+ *
+ * @return empty when the encoding cannot be determined and the validation
is enabled; the caller fails
+ * the operation with {@link #getComplexKeygenErrorMessage}
+ */
+ public static Option<ComplexKeyGenEncoding>
resolveComplexKeyGenEncodingForWrite(HoodieTableMetaClient metaClient,
+
HoodieWriteConfig config) {
+ Option<ComplexKeyGenEncoding> deduced =
deduceComplexKeyGenEncodingFromData(metaClient);
+ if (deduced.isPresent() || config.enableComplexKeygenValidation()) {
+ return deduced;
+ }
+ ComplexKeyGenEncoding configured =
+
ComplexKeyGenEncoding.fromUseNewEncoding(config.getBooleanOrDefault(COMPLEX_KEYGEN_NEW_ENCODING));
+ LOG.warn("Could not determine the record key encoding of table {} from its
data; recording the configured {} "
+ + "because {} is disabled. If the existing keys are not {}, fix {}
and rerun, or the records written "
+ + "from now on will not match the existing ones.",
+ metaClient.getBasePath(), configured,
ENABLE_COMPLEX_KEYGEN_VALIDATION.key(), configured,
+ COMPLEX_KEYGEN_NEW_ENCODING.key());
+ return Option.of(configured);
+ }
+
+ /**
+ * Deduces the record key encoding of a single-field complex key generator
table from the
+ * {@code _hoodie_record_key} stored in its most recently written data file.
+ *
+ * @return {@link ComplexKeyGenEncoding#FIELD_PREFIXED} for a table that
never had data files, the encoding
+ * read from the first readable base or log file otherwise, or empty when
the table has (or had) data files
+ * but none of them yields a record key
+ */
+ public static Option<ComplexKeyGenEncoding>
deduceComplexKeyGenEncodingFromData(HoodieTableMetaClient metaClient) {
+ String expectedPrefix =
metaClient.getTableConfig().getRecordKeyFields().get()[0] +
DEFAULT_COLUMN_VALUE_SEPARATOR;
+ HoodieTimeline completedTimeline =
metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants();
+ boolean hasDataFiles = false;
+ for (HoodieInstant instant :
completedTimeline.getReverseOrderedInstants().collect(Collectors.toList())) {
+ for (HoodieWriteStat writeStat : getWriteStats(instant,
completedTimeline)) {
+ if (StringUtils.isNullOrEmpty(writeStat.getPath())) {
+ continue;
+ }
+ hasDataFiles = true;
+ StoragePath path = new StoragePath(metaClient.getBasePath(),
writeStat.getPath());
+ Option<String> recordKey = readFirstRecordKey(metaClient, path);
+ if (recordKey.isPresent()) {
+ ComplexKeyGenEncoding encoding =
+
ComplexKeyGenEncoding.fromUseNewEncoding(!recordKey.get().startsWith(expectedPrefix));
+ LOG.info("Deduced complex keygen record key encoding {} of table {}
from {}",
+ encoding, metaClient.getBasePath(), path);
+ return Option.of(encoding);
+ }
+ }
+ }
+ if (hasDataFiles || hasArchivedCommits(metaClient)) {
Review Comment:
Addressed: the scan is capped at the 20 most recent completed commits and
falls through to `Option.empty()` with a warning naming the count.
`hasDataFiles` is gone -- an empty commits timeline is what now identifies a
table that was never written to, since archival always leaves the most recent
commits behind.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -436,15 +461,213 @@ public static String getComplexKeygenErrorMessage(String
operation) {
+ "`hoodie.write.complex.keygen.validation.enable=false` to skip this
validation.";
}
+ /**
+ * Whether a complex key generator with a single record key field prepends
the field name to the key.
+ *
+ * <p>{@link HoodieTableConfig#COMPLEX_KEYGEN_ENCODING} wins whenever it is
present in the properties, because
+ * it describes what the table's data carries. Otherwise the write table
version decides: 9 and above always
+ * prefix, 8 and below follow {@code
hoodie.write.complex.keygen.new.encoding}.
+ */
public static boolean
encodeSingleKeyFieldNameForComplexKeyGen(TypedProperties props) {
+ String tableEncoding =
props.getProperty(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING.key());
Review Comment:
Not moved, unfortunately: `BaseKeyGenerator` lives in hudi-common, while
this method reads `HoodieWriteConfig.WRITE_TABLE_VERSION` and
`COMPLEX_KEYGEN_NEW_ENCODING` from hudi-client-common, so moving it there would
invert the module dependency. If the table config becomes authoritative and the
table-version fallback goes away, the method collapses to a one-line read and
this stops mattering.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java:
##########
@@ -436,15 +461,213 @@ public static String getComplexKeygenErrorMessage(String
operation) {
+ "`hoodie.write.complex.keygen.validation.enable=false` to skip this
validation.";
}
+ /**
+ * Whether a complex key generator with a single record key field prepends
the field name to the key.
+ *
+ * <p>{@link HoodieTableConfig#COMPLEX_KEYGEN_ENCODING} wins whenever it is
present in the properties, because
+ * it describes what the table's data carries. Otherwise the write table
version decides: 9 and above always
+ * prefix, 8 and below follow {@code
hoodie.write.complex.keygen.new.encoding}.
+ */
public static boolean
encodeSingleKeyFieldNameForComplexKeyGen(TypedProperties props) {
+ String tableEncoding =
props.getProperty(HoodieTableConfig.COMPLEX_KEYGEN_ENCODING.key());
+ if (!StringUtils.isNullOrEmpty(tableEncoding)) {
+ return
ComplexKeyGenEncoding.fromString(tableEncoding).encodesFieldName();
+ }
int tableVersionCode = ConfigUtils.getIntWithAltKeys(props,
WRITE_TABLE_VERSION);
HoodieTableVersion tableVersion =
HoodieTableVersion.fromVersionCode(tableVersionCode);
return tableVersion.greaterThanOrEquals(HoodieTableVersion.NINE)
|| !ConfigUtils.getBooleanWithAltKeys(props,
COMPLEX_KEYGEN_NEW_ENCODING);
}
- public static boolean mayUseNewEncodingForComplexKeyGen(HoodieTableConfig
tableConfig) {
- return tableConfig.getTableVersion().lesserThan(HoodieTableVersion.NINE)
- && isComplexKeyGeneratorWithSingleRecordKeyField(tableConfig);
+ /**
+ * Whether the table's record key encoding is tracked by {@link
HoodieTableConfig#COMPLEX_KEYGEN_ENCODING}:
+ * a complex key generator with a single record key field and a populated
{@code _hoodie_record_key}.
+ * Without the meta field there is no stored key whose encoding could
diverge from the key generator's.
+ */
+ public static boolean isComplexKeyGenEncodingTracked(HoodieTableConfig
tableConfig) {
+ return tableConfig.isComplexKeyGenWithSingleRecordKeyField() &&
tableConfig.isRecordKeyPopulated();
+ }
+
+ /**
+ * Records a missing complex key generator encoding before ingestion starts.
Engine entry points call this
+ * once during setup, before creating record keys. Existing data determines
the encoding under the table lock.
+ */
+ public static void
recordComplexKeygenEncodingIfMissing(HoodieTableMetaClient metaClient,
HoodieWriteConfig config) {
+ if (!isComplexKeyGenEncodingTracked(metaClient.getTableConfig())
+ || metaClient.getTableConfig().getComplexKeyGenEncoding().isPresent())
{
+ return;
+ }
+ try (TransactionManager transactionManager = new
TransactionManager(config, metaClient.getStorage())) {
Review Comment:
All three call sites are outside the write transaction today:
`DataSourceUtils` and `StreamSync` run before the write client exists, and the
Java client path runs after `doInitTable`'s `executeUsingTxnManager` has closed
and after `startCommit` has released its lock in the `finally`. I would keep
the lock though -- it is not guarding the encoding value (concurrent writers
deduce the same thing from the same data) but `hoodie.properties` itself, since
`HoodieTableConfig.modify` is a non-atomic read-modify-write of the whole file
and can race an upgrade writing the table version.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java:
##########
@@ -1595,6 +1605,16 @@ public Properties build() {
KeyGeneratorType type = KeyGeneratorType.valueOf(keyGeneratorType);
tableConfig.setValue(HoodieTableConfig.KEY_GENERATOR_TYPE,
type.name());
}
+ if (tableConfig.isComplexKeyGenWithSingleRecordKeyField() &&
metaFieldsMode.isRecordKeyPopulated()) {
+ // New tables store `<field>:<value>` keys. Only releases up to 1.0.2
(table version 8 and below) ever
+ // wrote bare values, so that encoding can only be declared for a
table created at such a version.
+ ComplexKeyGenEncoding encoding = complexKeyGenEncoding != null ?
complexKeyGenEncoding : ComplexKeyGenEncoding.FIELD_PREFIXED;
+ checkArgument(encoding == ComplexKeyGenEncoding.FIELD_PREFIXED ||
tableVersion.lesserThan(HoodieTableVersion.NINE),
+ String.format("%s=%s is only valid for a table created at version
%s or below; tables at version %s and above "
Review Comment:
Yes -- a pre-v9 VALUE_ONLY table upgrades to v9/v10 and keeps VALUE_ONLY.
`TestComplexKeyGenExistingTableUpgrade#testUpgradeToCurrentVersionWithDefaultsNoDuplicates`
asserts exactly that, and the property is in
`CONFIGS_REQUIRED_FOR_OLDER_VERSIONED_TABLES` so it survives version filtering.
The v9 restriction only bites at table *creation*, and I think you are right
that it is too strict there: it is what is failing CI, since
`TestHoodieTableSource#testRecordLevelIndexPruningHonorsRecordedEncoding[2]`
creates a current-version table with VALUE_ONLY and hits
`IllegalArgumentException:
hoodie.table.complex.keygenerator.encoding=VALUE_ONLY is only valid for a table
created at version 8 or below`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
