nsivabalan commented on code in PR #19205:
URL: https://github.com/apache/hudi/pull/19205#discussion_r3651627953
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -3883,6 +3919,33 @@ private void validate() {
checkArgument(ttlStatsMaxParallelism > 0,
String.format("%s must be positive, but was %d",
HoodieTTLConfig.STATS_MAX_PARALLELISM.key(),
ttlStatsMaxParallelism));
+
+ // hoodie.meta.fields.mode is an additive opt-in on top of
populate.meta.fields=false. Setting
+ // populate.meta.fields=true together with a non-ALL mode is ambiguous
(the mode has no effect
+ // when all meta fields are already populated) so reject it explicitly
rather than silently
+ // ignore. MetaFieldsMode.fromConfig also throws on unrecognized on-disk
values.
+ MetaFieldsMode metaFieldsMode = writeConfig.getMetaFieldsMode();
+ boolean populateMetaFields = writeConfig.populateMetaFields();
+ String rawMode =
writeConfig.getStringOrDefault(HoodieTableConfig.META_FIELDS_MODE);
+ checkArgument(!(populateMetaFields && rawMode != null &&
!rawMode.isEmpty()),
+ String.format("%s must be empty when %s=true. Disable
populate.meta.fields or clear the mode.",
+ HoodieTableConfig.META_FIELDS_MODE.key(),
+ HoodieTableConfig.POPULATE_META_FIELDS.key()));
+ // Selective meta-field modes are CoW-only in this release. MoR
log-write path does not yet
+ // respect the mode, which would silently produce log records with null
meta columns.
+ boolean isSelective = metaFieldsMode != MetaFieldsMode.ALL &&
metaFieldsMode != MetaFieldsMode.NONE;
+ checkArgument(!(writeConfig.getTableType() ==
HoodieTableType.MERGE_ON_READ && isSelective),
+ String.format("%s=%s is currently supported for COPY_ON_WRITE tables
only. MoR support is a follow-up. "
+ + "For MoR either keep %s=true or use NONE mode.",
+ HoodieTableConfig.META_FIELDS_MODE.key(), metaFieldsMode,
+ HoodieTableConfig.POPULATE_META_FIELDS.key()));
+ // Selective meta-field modes are wired only for the Spark writer path
in this release. Flink
+ // RowData / Java-client writers ignore the mode and would silently
produce NONE-mode output.
+ checkArgument(!(engineType != EngineType.SPARK && isSelective),
Review Comment:
Correction to my previous reply — I was wrong to call this a gap, and
there's no follow-up needed here.
**ORC never supported virtual keys in the first place.**
`HoodieAvroOrcWriter#writeAvroWithMetadata` calls `prepRecordWithMetadata(...)`
unconditionally — there is no `populateMetaFields` branch anywhere in that
class, on this branch or on master. ORC has always written all five meta
columns regardless of `hoodie.populate.meta.fields`. So selective modes don't
introduce a config-vs-data divergence for ORC; the pre-existing behavior is
that ORC ignores meta-field configuration entirely, and that is unchanged by
this PR.
**HFile is not a data-table format.** It's used only by the metadata table,
which is always `ALL`. Passing it the derived boolean at
`HoodieAvroFileWriterFactory.java:129` is therefore never exercised with a
selective mode.
So the config/data disagreement I described isn't reachable: ORC is
all-or-nothing today by construction, and HFile never sees a selective mode.
Wiring `MetaFieldsMode` into ORC would be part of adding virtual-key support to
ORC generally, which is well outside the scope of this PR and isn't a
regression from it.
Thanks for the push on this one — I should have checked the ORC writer
before claiming a follow-up. Resolving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
