cshuo commented on code in PR #19205:
URL: https://github.com/apache/hudi/pull/19205#discussion_r3643625867
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:
##########
@@ -478,6 +478,8 @@ HoodieTableMetaClient
initializeEmptyTable(HoodieTableMetaClient.TableBuilder ta
.setRecordKeyFields(props.getProperty(DataSourceWriteOptions.RECORDKEY_FIELD().key()))
.setPopulateMetaFields(props.getBoolean(HoodieTableConfig.POPULATE_META_FIELDS.key(),
HoodieTableConfig.POPULATE_META_FIELDS.defaultValue()))
+
.setMetaFieldsModeFromString(props.getString(HoodieTableConfig.META_FIELDS_MODE.key(),
Review Comment:
HIGH: This persists the mode during table creation, but existing-table
writes do not merge the stored mode back into the Streamer write config. If a
COMMIT_TIME_ONLY table is restarted with hoodie.populate.meta.fields=false but
without repeating hoodie.meta.fields.mode, the write config resolves to NONE
while the table still advertises COMMIT_TIME_ONLY, so incremental queries can
silently omit the new rows. Please inherit or validate the effective on-disk
mode and add a two-run restart test.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowCreateHandle.java:
##########
@@ -158,10 +159,45 @@ public HoodieRowCreateHandle(HoodieTable table,
* @throws IOException
*/
public void write(InternalRow row) throws IOException {
- if (populateMetaFields) {
- writeRow(row);
- } else {
- writeRowNoMetaFields(row);
+ switch (metaFieldsMode) {
+ case ALL:
+ writeRow(row);
+ break;
+ case NONE:
+ writeRowNoMetaFields(row);
+ break;
+ default:
+ writeRowSelectiveMetaFields(row);
+ break;
+ }
+ }
+
+ /**
+ * Selective meta-field write path: populate only the meta columns opted in
via
+ * {@code hoodie.meta.fields.mode} — {@code _hoodie_commit_time} and/or
{@code _hoodie_file_name}.
+ * The other meta columns stay null on disk. Record key is never populated
in this path, so the
+ * record key is not registered with the write support (bloom filter / RLI
hooks are meaningless
+ * without the record-key column).
+ */
+ private void writeRowSelectiveMetaFields(InternalRow row) {
+ try {
+ UTF8String[] metaFields = new UTF8String[5];
+ if (metaFieldsMode.isCommitTimePopulated()) {
+ metaFields[HoodieRecord.COMMIT_TIME_METADATA_FIELD_ORD] =
shouldPreserveHoodieMetadata
+ ? row.getUTF8String(HoodieRecord.COMMIT_TIME_METADATA_FIELD_ORD) :
commitTime;
+ }
+ if (metaFieldsMode.isFileNamePopulated()) {
+ metaFields[HoodieRecord.FILENAME_META_FIELD_ORD] =
shouldPreserveHoodieMetadata
Review Comment:
HIGH: Clustering defaults shouldPreserveHoodieMetadata to true, so this
copies _hoodie_file_name from the source row into the newly created clustered
file. That value must identify the containing output file; the ALL path
correctly always uses the new fileName. Please preserve the original commit
time but always assign the new file name, and strengthen the clustering test to
compare _hoodie_file_name with the actual input file name rather than checking
only non-nullness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
