Davis-Zhang-Onehouse opened a new issue, #20683:
URL: https://github.com/apache/hudi/issues/20683

   ### Feature Description
   
   **What the feature achieves:**
   
   Tracking issue for RFC-118. Adds `UnstructuredDataSource` to Hudi Streamer 
with two modes over the S3/GCS events table the stock events pipeline already 
builds:
   
   - **blob**: one row per object version (modification time, size). Bytes are 
stored inline up to `hoodie.blob.table.inline.max.bytes`; a larger object is an 
unmanaged out-of-line reference (`managed=false`) to the source object in 
place. Nothing is copied into the table. The trade-offs of referencing in place 
are listed in RFC-118 ยง2.
   - **embedding**: prepares files into text chunks, images, video frames and 
speech transcripts (`prep_only`), and optionally computes embeddings with 
configured models into one VECTOR column per model (`prep_and_compute`). The 
only files it writes beside data files are extracted video frames, inside the 
round's instant, so a failed round rolls them back.
   
   Builds on RFC-100 (BLOB type, read_blob) and runs alongside the existing 
unstructured ingestion (#19278); produces tables usable with RFC-102/RFC-109 
vector search.
   
   **Why this feature is needed:**
   
   To keep blob and embedding tables up to date incrementally from 
object-storage events, once per file version, with bounded decoding of PDF, 
image and video inputs and bad files quarantined in the error table instead of 
failing the pipeline.
   
   ### User Experience
   
   **How users will use this feature:**
   
   - Configuration changes needed: `--source-class 
org.apache.hudi.utilities.sources.UnstructuredDataSource`, 
`hoodie.unstructured.source.mode` (`blob` | `embedding`), 
`hoodie.unstructured.source.cloud` (`s3` | `gcs`); for embedding, 
`hoodie.unstructured.embedding.mode` and the model keys. The source contract 
fills in the required transformer, record key, Parquet injector and marker 
settings, and refuses conflicting ones.
   - API changes: two generic Hudi Streamer additions, inert for existing 
sources and transformers: a source contract and an instant-aware transformer.
   - Usage examples: in RFC-118.
   
   The implementation lands as a chain of PRs, each compiling and tested on its 
own; each code PR opens its own sub-issue of this issue. PRs after the first 
segment open as drafts and become ready as their predecessor merges. Sub-issues 
4, 10, 27, 28, 29, 31, 32, 34 and 36 of the plan chiefly carry a blob copy path 
(large objects copied under `.hoodie/.blob`) that follow-up 37 replaces with 
the in-place reference RFC-118 describes; the end state is the chain plus 37. 
Sub-issue 38 is a generic memory fix, independent of the chain.
   
   ### Hudi RFC Requirements
   
   **RFC PR link:** RFC-118 (the RFC PR references this issue)
   
   **Why RFC is/isn't needed:**
   - Does this change public interfaces/APIs? Yes (a new Streamer source and 
two Streamer extension points)
   - Does this change storage format? No (BLOB and VECTOR columns as 
RFC-100/RFC-102 define them; table-config properties of the new tables)
   - Justification: a new Hudi Streamer source with generic streamer additions 
needs an RFC.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to