wombatu-kun commented on code in PR #20346:
URL: https://github.com/apache/hudi/pull/20346#discussion_r4281303592
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -701,6 +701,11 @@ public static void create(HoodieStorage storage,
StoragePath metadataFolder, Pro
hoodieConfig.setValue(POPULATE_META_FIELDS,
Boolean.toString(metaFieldsMode.toLegacyPopulateMetaFields()));
}
+ String orderingFields =
hoodieConfig.getProps().getProperty(ORDERING_FIELDS.key());
+ if (orderingFields != null &&
tableVersion.lesserThan(HoodieTableVersion.NINE)) {
+ // readers of tables below version 9 only know the deprecated
precombine key, which the 9 to 8 downgrade writes too
+ hoodieConfig.setValue(PRECOMBINE_FIELD, orderingFields);
Review Comment:
Done e2f12b57a041
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -701,6 +701,12 @@ public static void create(HoodieStorage storage,
StoragePath metadataFolder, Pro
hoodieConfig.setValue(POPULATE_META_FIELDS,
Boolean.toString(metaFieldsMode.toLegacyPopulateMetaFields()));
}
+ List<String> orderingFields =
StringUtils.split(hoodieConfig.getProps().getProperty(ORDERING_FIELDS.key()),
",");
+ if (orderingFields.size() == 1 &&
tableVersion.lesserThan(HoodieTableVersion.NINE)) {
+ // readers of tables below version 9 only know the deprecated
precombine key, which the 9 to 8 downgrade writes too,
+ // and take its value as one field name
+ hoodieConfig.setValue(PRECOMBINE_FIELD, orderingFields.get(0));
Review Comment:
Done 8a10f739ff42: below version 9 a `MySqlDebeziumAvroPayload` table now
gets `hoodie.table.precombine.field=_event_seq`, the same as after the
downgrade. The payload is detected through `hoodie.compaction.payload.class`,
since `hoodie.table.legacy.payload.class` is not set below version 9.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -701,6 +701,12 @@ public static void create(HoodieStorage storage,
StoragePath metadataFolder, Pro
hoodieConfig.setValue(POPULATE_META_FIELDS,
Boolean.toString(metaFieldsMode.toLegacyPopulateMetaFields()));
}
+ List<String> orderingFields =
StringUtils.split(hoodieConfig.getProps().getProperty(ORDERING_FIELDS.key()),
",");
+ if (orderingFields.size() == 1 &&
tableVersion.lesserThan(HoodieTableVersion.NINE)) {
+ // readers of tables below version 9 only know the deprecated
precombine key, which the 9 to 8 downgrade writes too,
+ // and take its value as one field name
+ hoodieConfig.setValue(PRECOMBINE_FIELD, orderingFields.get(0));
Review Comment:
Reverted in 42514b97ede2 after [danny0405's
comment](https://github.com/apache/hudi/pull/20346#discussion_r4300817866): a
table declared with `_event_seq` still gets the same precombine key as a
downgraded one, and one declared with `ts` now keeps `ts`. With `_event_seq`
next to `hoodie.table.ordering.fields=ts`, a 0.x or 1.0.x Spark writer would
dedup on `_event_seq` while 1.x dedups on `ts`, or fail with `Config conflict`
when passed `ts`.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -701,6 +701,17 @@ public static void create(HoodieStorage storage,
StoragePath metadataFolder, Pro
hoodieConfig.setValue(POPULATE_META_FIELDS,
Boolean.toString(metaFieldsMode.toLegacyPopulateMetaFields()));
}
+ List<String> orderingFields =
StringUtils.split(hoodieConfig.getProps().getProperty(ORDERING_FIELDS.key()),
",");
Review Comment:
With 42514b97ede2 the block is back to one `if`, the size of the
`POPULATE_META_FIELDS` step above it, so it stays inline.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -701,6 +701,17 @@ public static void create(HoodieStorage storage,
StoragePath metadataFolder, Pro
hoodieConfig.setValue(POPULATE_META_FIELDS,
Boolean.toString(metaFieldsMode.toLegacyPopulateMetaFields()));
}
+ List<String> orderingFields =
StringUtils.split(hoodieConfig.getProps().getProperty(ORDERING_FIELDS.key()),
",");
+ if (!orderingFields.isEmpty() &&
tableVersion.lesserThan(HoodieTableVersion.NINE)) {
+ // readers of tables below version 9 only know the deprecated
precombine key, which the 9 to 8 downgrade writes too,
+ // and take its value as one field name
+ if
(MySqlDebeziumAvroPayload.class.getName().equals(hoodieConfig.getString(PAYLOAD_CLASS_NAME)))
{
+ // below version 9 this payload orders on its sequence field, the
value the downgrade writes for it
+ hoodieConfig.setValue(PRECOMBINE_FIELD,
DebeziumConstants.ADDED_SEQ_COL_NAME);
Review Comment:
Done 42514b97ede2: the legacy key now repeats the declared ordering field
whatever the payload, and stays unset when none is declared. There is no such
validation: this payload's ordering fields are only rewritten at version 9
creation, by the 8 to 9 and 9 to 8 handlers, and by `StreamSync` on version 9
tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
