Davis-Zhang-Onehouse opened a new pull request, #20692: URL: https://github.com/apache/hudi/pull/20692
### Describe the issue this Pull Request addresses issue: #20691 Part of #20683, RFC-118 (#20684). Stacked on #20690: review only the last commit. Until the PRs below it merge, this PR also shows their commits. First RFC-118 stage: the cloud-event source contract and BLOB ingestion in Hudi Streamer. ### Summary and Changelog Adds `UnstructuredDataSource` in blob mode: one row per object version from the S3/GCS events table, bytes inline up to a size threshold. - `UnstructuredDataSource`, its typed settings (`hoodie.unstructured.source.*`: mode, cloud) and `UnstructuredObjectMaterializer` (a `CloudObjectMaterializer` from the previous PR). - Source contract (`SourceContract`, `RequiredSourceSettings`, `RequiredTransformerChain`): a source declares the transformer chain, record key, Parquet injector and marker settings it requires; Hudi Streamer fills them in and refuses conflicting ones. Inert for existing sources. - `InstantAwareTransformer`: `ChainedTransformer` and `ErrorTableAwareChainedTransformer` pass the round's instant to transformers that ask for it. Existing transformers are unaffected; master's corrupt-record validation and restoration in the chain are kept. - Blob table contract (`BlobTableSchema`, `BlobTableConfig`, `hoodie.blob.table.*`), `BlobRowBuilder`, and `BlobCopyTransformer` / `BlobObjectCopier`, which copy objects above the inline threshold into managed files under the table's `.hoodie/.blob`. - `StreamSync`, `HoodieStreamer`, `HoodieStreamerConfig`, `HoodieTableMetaClient`: wire in the contract and the round instant; master's async service switching and key generator recording are kept, and the early heartbeat restarts after the old write client closes. - Path partition names are trimmed and blank segments dropped, matching the source and embedding schemas. - 25 new test files; `HoodieDeltaStreamerTestBase` and `UtilitiesTestBase` get the helpers they need. Known limitations, both resolved later in this chain: - A failed round can leave completed, unreferenced copies under `.hoodie/.blob`; the blob-rollback PR adds marker-based cleanup for them. - Copying large objects into the table is replaced by follow-up sub-issue 37 of the plan with an in-place, unmanaged reference to the source object (RFC-118 ยง2). No third-party code was copied. ### Impact New source and settings; opt-in. Existing sources and transformers keep their current path; the source contract and instant-aware transformer are inert unless a source or transformer declares them. Managed blob payload files live under the table's `.hoodie` folder. ### Risk Level high: this stage touches `StreamSync` and transformer dispatch. Verified on JDK 11 / Spark 3.5 / Scala 2.12 with 191 tests (including master's error-table cases), module compilation, checkstyle and RAT. The partition-name fix has a regression test that fails without it. Tests read cloud events from local files; no live S3 or GCS I/O was exercised, and the local runs log a GCS connector service-provider warning that does not fail any test. ### Documentation Update New settings carry their config documentation. RFC-118 describes the source; a website page follows once the chain lands. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
