danny0405 opened a new pull request, #19845:
URL: https://github.com/apache/hudi/pull/19845

   ### Describe the issue this Pull Request addresses
   
   Tables that omit `_hoodie_record_key` currently cannot retain 
`_hoodie_commit_seqno` and `_hoodie_partition_path`: those columns are 
populated only by `ALL`. Add `ALL_EXCEPT_RECORD_KEY` so a table can retain 
commit time, commit sequence number, partition path, and file name without 
storing the record-key meta column.
   
   ### Summary and Changelog
   
   - Add `ALL_EXCEPT_RECORD_KEY` to `MetaFieldsMode`, with explicit 
`commitSeqnoPopulated` and `partitionPathPopulated` members and getters.
   - Include both flags in `isWiderThan`, so transitions that add either column 
are treated as widening.
   - Populate the selected columns in the Avro and Spark Parquet writers and 
Spark row-create handle.
   - Support the new mode in Flink's Parquet writer, Lance writer, and 
row-create handle, including generated and preserved commit metadata. Write the 
destination file name when that column is enabled.
   - Update the configuration description and CLI help. Update Spark concat, 
Flink COW insert/update/concat, bulk-insert, and preservation assertions to use 
each column's population flag instead of assuming sequence number and partition 
path are ALL-only.
   - Use named column access in the Spark metadata-preservation test.
   
   No code was copied from external sources.
   
   ### Impact
   
   Users can set `hoodie.meta.fields.mode=ALL_EXCEPT_RECORD_KEY` on supported 
Spark and Flink COPY_ON_WRITE tables. This adds a public enum value and two 
public population predicates, and changes physical column population only for 
tables selecting the new mode. Existing modes, defaults, and file schemas are 
unchanged.
   
   The compatibility boundary is opt-in: older builds cannot parse the new enum 
value. The new mode inherits the existing selective-mode restrictions, 
including the Java-writer and MERGE_ON_READ restrictions, and maps to `false` 
for the deprecated `hoodie.populate.meta.fields` boolean. Record-key-dependent 
features remain subject to existing validation. Storage savings were not 
benchmarked.
   
   ### Risk Level
   
   medium
   
   The change touches Avro, Spark, and Flink metadata population and table-mode 
transitions. Validation on the branch rebased onto master `eccfee6dfe7d` passed 
218 cases:
   
   - Flink 2.2 reactor: 143 cases across `TestMetaFieldsMode` (75), 
`TestHoodieWriteConfigMetaFieldsMode` (24), `TestHoodieRowDataCreateHandle` 
(14), `TestHoodieRowDataLanceWriter#testMetaFieldsMode` (6), 
`TestBulkInsertWriteHelper#testMetaFieldsMode` (12), and the COW 
metadata/concat methods of `TestFlinkWriteClientFunctional` (12). The mode 
matrices exercise all six modes, including insert, update, delete, concat, bulk 
insert, and metadata preservation.
   - Spark 3.5: the full `TestMetaFieldsModeE2E` class (62 cases) and 
`TestHoodieRowCreateHandle#testAllExceptRecordKeyPreservesMetadata` (2 cases). 
This includes mode persistence across table versions, row and non-row writes, 
concat, clustering, incremental reads, and multi-partition upserts.
   - Common/Flink dependency configuration: `TestHoodieAvroParquetWriter` (2 
cases) and `TestHoodieTableConfigMetaFieldsMode` (9 cases).
   
   Compilation, applicable Checkstyle checks, and `git diff --check` passed. 
The standalone Avro test initially encountered a missing Hadoop runtime 
dependency under the Spark profile; it passed under the common/Flink 
configuration. Streamer assertions were updated, but its suite and the full 
repository suite were not run.
   
   ### Documentation Update
   
   Updated `HoodieTableConfig.META_FIELDS_MODE`, `MetaFieldsMode` Javadoc, and 
CLI help to describe the new mode and preserve master's Spark/Flink 
COPY_ON_WRITE support documentation. A Hudi website documentation follow-up is 
still needed to list `ALL_EXCEPT_RECORD_KEY`, its populated columns, and the 
selective-mode restrictions; website changes are not included in this code PR.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to