danny0405 opened a new pull request, #19247: URL: https://github.com/apache/hudi/pull/19247
### Describe the issue this Pull Request addresses This closes #19246 Native log format metadata is currently serialized as a JSON object under the single footer key `hudi.log.format.metadata`. This makes individual metadata entries opaque to native file-format tooling and requires Hudi-specific JSON serialization and parsing. This PR changes the native log storage contract so each header value is stored as an independent, namespaced footer entry such as `hudi.log.format.VERSION` and `hudi.log.format.SCHEMA`. This is an on-disk format change: readers no longer fall back to the previous single-JSON-entry representation. ### Summary and Changelog Native log metadata can now be discovered by prefix and consumed directly from native file footers without an intermediate JSON envelope. - Replaced `hudi.log.format.metadata` JSON serialization with separate `hudi.log.format.<HeaderMetadataType>` entries. - Added `FileFormatUtils.readFooterWithPrefix` implementations for Parquet, ORC, HFile, and Lance. - Extended `HFileReader` to expose file-info entries needed for prefix filtering. - Updated `HoodieNativeLogFileReader` and native-log schema discovery to read the `hudi.log.format.` namespace. - Improved missing-schema diagnostics to report the exact `hudi.log.format.SCHEMA` key. - Preserved format-version validation and forward compatibility for unknown header types. - Updated RFC-103 and tests for the new footer representation. ### Impact This changes the on-disk metadata representation for native log files. Files written with the previous interim `hudi.log.format.metadata` JSON representation are not readable through the new metadata path. The change also adds methods to the `FileFormatUtils` and `HFileReader` abstractions. Parquet, ORC, HFile, and Lance support prefix-based reads; Vortex remains unsupported, consistent with its existing footer-read behavior. There are no configuration changes. Footer metadata is still loaded through the native format reader and filtered in memory, so no significant performance impact is expected. ### Risk Level medium The change affects the native log storage contract and shared footer APIs. Risk is limited to native log metadata handling and is mitigated by focused validation: - `TestNativeLogFooterMetadata` - `TestHoodieNativeLogFileReader` - `TestTableSchemaResolver#testReadSchemaFromNativeDataLogFile` - `TestHoodieHFileReaderWriter#testReadFooterMetadata` - `TestHoodieOrcReaderWriter#testAddFooterMetadata` ### Documentation Update RFC-103 was updated to document the individual `hudi.log.format.*` footer entries and clarify storage of the physical and logical schemas. No additional user-facing documentation is required. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
