cshuo commented on code in PR #19205:
URL: https://github.com/apache/hudi/pull/19205#discussion_r3643621674
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -3883,6 +3919,33 @@ private void validate() {
checkArgument(ttlStatsMaxParallelism > 0,
String.format("%s must be positive, but was %d",
HoodieTTLConfig.STATS_MAX_PARALLELISM.key(),
ttlStatsMaxParallelism));
+
+ // hoodie.meta.fields.mode is an additive opt-in on top of
populate.meta.fields=false. Setting
+ // populate.meta.fields=true together with a non-ALL mode is ambiguous
(the mode has no effect
+ // when all meta fields are already populated) so reject it explicitly
rather than silently
+ // ignore. MetaFieldsMode.fromConfig also throws on unrecognized on-disk
values.
+ MetaFieldsMode metaFieldsMode = writeConfig.getMetaFieldsMode();
+ boolean populateMetaFields = writeConfig.populateMetaFields();
+ String rawMode =
writeConfig.getStringOrDefault(HoodieTableConfig.META_FIELDS_MODE);
+ checkArgument(!(populateMetaFields && rawMode != null &&
!rawMode.isEmpty()),
+ String.format("%s must be empty when %s=true. Disable
populate.meta.fields or clear the mode.",
+ HoodieTableConfig.META_FIELDS_MODE.key(),
+ HoodieTableConfig.POPULATE_META_FIELDS.key()));
+ // Selective meta-field modes are CoW-only in this release. MoR
log-write path does not yet
+ // respect the mode, which would silently produce log records with null
meta columns.
+ boolean isSelective = metaFieldsMode != MetaFieldsMode.ALL &&
metaFieldsMode != MetaFieldsMode.NONE;
+ checkArgument(!(writeConfig.getTableType() ==
HoodieTableType.MERGE_ON_READ && isSelective),
+ String.format("%s=%s is currently supported for COPY_ON_WRITE tables
only. MoR support is a follow-up. "
+ + "For MoR either keep %s=true or use NONE mode.",
+ HoodieTableConfig.META_FIELDS_MODE.key(), metaFieldsMode,
+ HoodieTableConfig.POPULATE_META_FIELDS.key()));
+ // Selective meta-field modes are wired only for the Spark writer path
in this release. Flink
+ // RowData / Java-client writers ignore the mode and would silently
produce NONE-mode output.
+ checkArgument(!(engineType != EngineType.SPARK && isSelective),
Review Comment:
HIGH: The Spark-engine check is insufficient: a CoW ORC table passes this
validation, but only the Parquet writer receives MetaFieldsMode. Other
base-file writer paths can therefore produce physical metadata that disagrees
with hoodie.meta.fields.mode. Please either restrict selective modes to fully
wired base-file formats or propagate the mode through ORC, Lance, and Vortex
paths, with format-specific tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
