nsivabalan commented on code in PR #19205:
URL: https://github.com/apache/hudi/pull/19205#discussion_r3651620570
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -3883,6 +3919,33 @@ private void validate() {
checkArgument(ttlStatsMaxParallelism > 0,
String.format("%s must be positive, but was %d",
HoodieTTLConfig.STATS_MAX_PARALLELISM.key(),
ttlStatsMaxParallelism));
+
+ // hoodie.meta.fields.mode is an additive opt-in on top of
populate.meta.fields=false. Setting
+ // populate.meta.fields=true together with a non-ALL mode is ambiguous
(the mode has no effect
+ // when all meta fields are already populated) so reject it explicitly
rather than silently
+ // ignore. MetaFieldsMode.fromConfig also throws on unrecognized on-disk
values.
+ MetaFieldsMode metaFieldsMode = writeConfig.getMetaFieldsMode();
+ boolean populateMetaFields = writeConfig.populateMetaFields();
+ String rawMode =
writeConfig.getStringOrDefault(HoodieTableConfig.META_FIELDS_MODE);
+ checkArgument(!(populateMetaFields && rawMode != null &&
!rawMode.isEmpty()),
+ String.format("%s must be empty when %s=true. Disable
populate.meta.fields or clear the mode.",
+ HoodieTableConfig.META_FIELDS_MODE.key(),
+ HoodieTableConfig.POPULATE_META_FIELDS.key()));
+ // Selective meta-field modes are CoW-only in this release. MoR
log-write path does not yet
+ // respect the mode, which would silently produce log records with null
meta columns.
+ boolean isSelective = metaFieldsMode != MetaFieldsMode.ALL &&
metaFieldsMode != MetaFieldsMode.NONE;
+ checkArgument(!(writeConfig.getTableType() ==
HoodieTableType.MERGE_ON_READ && isSelective),
+ String.format("%s=%s is currently supported for COPY_ON_WRITE tables
only. MoR support is a follow-up. "
+ + "For MoR either keep %s=true or use NONE mode.",
+ HoodieTableConfig.META_FIELDS_MODE.key(), metaFieldsMode,
+ HoodieTableConfig.POPULATE_META_FIELDS.key()));
+ // Selective meta-field modes are wired only for the Spark writer path
in this release. Flink
+ // RowData / Java-client writers ignore the mode and would silently
produce NONE-mode output.
+ checkArgument(!(engineType != EngineType.SPARK && isSelective),
Review Comment:
Good catch, and it's broader than ORC — `newHFileFileWriter` has the same
gap. It passes the raw `POPULATE_META_FIELDS` boolean straight through
(`HoodieAvroFileWriterFactory.java:129`), so under a selective mode it silently
behaves as `NONE`; ORC (`:141`) receives no meta-field signal at all.
Punting both to a separate PR rather than half-wiring them here. That patch
will either thread `MetaFieldsMode` through both writers or add a
base-file-format guard to `validate()` so a selective mode on a non-Parquet
table is rejected up-front instead of producing a table whose config disagrees
with its data.
Since selective modes are already gated to Spark + CoW at writer init, the
exposure until then is a CoW+Spark table that explicitly sets
`hoodie.base.file.format=ORC/HFILE`. Leaving this thread open to track it.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:
##########
@@ -478,6 +478,8 @@ HoodieTableMetaClient
initializeEmptyTable(HoodieTableMetaClient.TableBuilder ta
.setRecordKeyFields(props.getProperty(DataSourceWriteOptions.RECORDKEY_FIELD().key()))
.setPopulateMetaFields(props.getBoolean(HoodieTableConfig.POPULATE_META_FIELDS.key(),
HoodieTableConfig.POPULATE_META_FIELDS.defaultValue()))
+
.setMetaFieldsModeFromString(props.getString(HoodieTableConfig.META_FIELDS_MODE.key(),
Review Comment:
Thanks — this pushed me to re-check the resolution order, and
[`1656733`](https://github.com/apache/hudi/pull/19205/commits/16567335f5d1)
closes the case you describe.
`hoodie.meta.fields.mode` is now the source of truth and is persisted
verbatim (including `ALL` / `NONE`). An explicit mode on disk wins over the
deprecated boolean, so an existing `COMMIT_TIME_ONLY` table restarted with only
`hoodie.populate.meta.fields=false` resolves to `COMMIT_TIME_ONLY`, not `NONE`.
The boolean can no longer silently downgrade a mode recorded on the table, so
new records don't get null commit times.
One correction to something I nearly claimed here: `HoodieStreamer` *does*
already run the immutability guard — `HoodieStreamer.java:760` calls
`HoodieWriterUtils.validateTableConfig` with the full property map whenever the
target table already exists. So the streamer path was not unguarded.
The residual gap is narrower: tables created *before* this PR have no mode
property on disk, so they fall back to the boolean, and the explicit `null ->
non-empty` check in `validateTableConfig` only fires when the mode is being
introduced via write options. I'll take a separate patch to tighten that
transition and add streamer-restart test coverage, but it's a
pre-existing-table concern rather than a live divergence on the write path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
