Davis-Zhang-Onehouse opened a new pull request, #20692:
URL: https://github.com/apache/hudi/pull/20692

   ### Describe the issue this Pull Request addresses
   
   issue: #20691
   Part of #20683, RFC-118 (#20684).
   Stacked on #20690: review only the last commit. Until the PRs below it 
merge, this PR also shows their commits.
   
   First RFC-118 stage: the cloud-event source contract and BLOB ingestion in 
Hudi Streamer.
   
   ### Summary and Changelog
   
   Adds `UnstructuredDataSource` in blob mode: one row per object version from 
the S3/GCS events table, bytes inline up to a size threshold.
   
   - `UnstructuredDataSource`, its typed settings 
(`hoodie.unstructured.source.*`: mode, cloud) and 
`UnstructuredObjectMaterializer` (a `CloudObjectMaterializer` from the previous 
PR).
   - Source contract (`SourceContract`, `RequiredSourceSettings`, 
`RequiredTransformerChain`): a source declares the transformer chain, record 
key, Parquet injector and marker settings it requires; Hudi Streamer fills them 
in and refuses conflicting ones. Inert for existing sources.
   - `InstantAwareTransformer`: `ChainedTransformer` and 
`ErrorTableAwareChainedTransformer` pass the round's instant to transformers 
that ask for it. Existing transformers are unaffected; master's corrupt-record 
validation and restoration in the chain are kept.
   - Blob table contract (`BlobTableSchema`, `BlobTableConfig`, 
`hoodie.blob.table.*`), `BlobRowBuilder`, and `BlobCopyTransformer` / 
`BlobObjectCopier`, which copy objects above the inline threshold into managed 
files under the table's `.hoodie/.blob`.
   - `StreamSync`, `HoodieStreamer`, `HoodieStreamerConfig`, 
`HoodieTableMetaClient`: wire in the contract and the round instant; master's 
async service switching and key generator recording are kept, and the early 
heartbeat restarts after the old write client closes.
   - Path partition names are trimmed and blank segments dropped, matching the 
source and embedding schemas.
   - 25 new test files; `HoodieDeltaStreamerTestBase` and `UtilitiesTestBase` 
get the helpers they need.
   
   Known limitations, both resolved later in this chain:
   - A failed round can leave completed, unreferenced copies under 
`.hoodie/.blob`; the blob-rollback PR adds marker-based cleanup for them.
   - Copying large objects into the table is replaced by follow-up sub-issue 37 
of the plan with an in-place, unmanaged reference to the source object (RFC-118 
ยง2).
   
   No third-party code was copied.
   
   ### Impact
   
   New source and settings; opt-in. Existing sources and transformers keep 
their current path; the source contract and instant-aware transformer are inert 
unless a source or transformer declares them. Managed blob payload files live 
under the table's `.hoodie` folder.
   
   ### Risk Level
   
   high: this stage touches `StreamSync` and transformer dispatch. Verified on 
JDK 11 / Spark 3.5 / Scala 2.12 with 191 tests (including master's error-table 
cases), module compilation, checkstyle and RAT. The partition-name fix has a 
regression test that fails without it. Tests read cloud events from local 
files; no live S3 or GCS I/O was exercised, and the local runs log a GCS 
connector service-provider warning that does not fail any test.
   
   ### Documentation Update
   
   New settings carry their config documentation. RFC-118 describes the source; 
a website page follows once the chain lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(ingest): RFC-118: ... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to