Davis-Zhang-Onehouse opened a new pull request, #20707:
URL: https://github.com/apache/hudi/pull/20707

   ### Describe the issue this Pull Request addresses
   
   issue: #20706
   Part of #20683, RFC-118 (#20684).
   Stacked on #20700: review only the last commit. Until the PRs below it 
merge, this PR also shows their commits.
   
   The `prep_and_compute` embedding round: one Hudi Streamer round prepares the 
new files into chunks and computes their embeddings with the configured models, 
writing one VECTOR column per model, with the PDF path working end to end. It 
covers in-step and task retries of the compute step without re-running prep, 
provider outages, error-table rows for failed files, one commit per round, and 
blob and embedding streams running concurrently.
   
   ### Summary and Changelog
   
   - The `prep_and_compute` mode of `EmbeddingTableTransformer`: prep and 
compute in one round, one commit, one VECTOR column per configured model.
   - Compute-step behaviour: in-step retries, task retries that do not re-run 
prep, shuffle balance, provider outages surfacing as round failures, and failed 
files routed to the error table.
   - `UnstructuredDataSource` end-to-end coverage: S3 and GCS events, target 
schema, ordering field, customer transformers, embedding from S3 events without 
a blob table, overwrite convergence, and blob and embedding streams side by 
side.
   - 27 test classes (23 new).
   
   Test-only adaptations to master: cases for a delete option master does not 
have are removed, compression is read from master's writer configuration, and 
the two-stream concurrency test turns off the column-stats index of its 
source-events fixture. With that index on, two streams in one Spark session can 
deadlock inside Spark's cache manager (`CacheManager.uncacheQuery`), reached 
through data skipping. That is outside this feature; production code is 
unchanged and the fixture still checks both streams' data, checkpoints and 
provider failures.
   
   All files are in hudi-utilities and were added earlier in this chain or 
here. No code was copied from other projects.
   
   Known limitation, closed by the next PR (blob rollback): a failed blob round 
can still leave unreferenced copies under `.hoodie/.blob`.
   
   ### Impact
   
   `UnstructuredDataSource` in embedding mode can now compute embeddings, not 
only prepare inputs. Existing sources are unaffected.
   
   ### Risk Level
   
   medium: first end-to-end compute round of the new source. Verified on JDK 11 
/ Spark 3.5 / Scala 2.12 with 155 tests (none skipped), module compilation, 
checkstyle and RAT.
   
   ### Documentation Update
   
   The embedding mode settings carry their config documentation. RFC-118 
describes the round; a website page follows once the chain lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(ingest): RFC-118: ... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to