Davis-Zhang-Onehouse opened a new pull request, #20729:
URL: https://github.com/apache/hudi/pull/20729

   ### Describe the issue this Pull Request addresses
   
   issue: #20728
   Part of #20683, RFC-118 (#20684).
   Stacked on #20727: review only the last commit. Until the PRs below it 
merge, this PR also shows their commits.
   
   Prepared inputs (image and video frame payloads) can be written to an input 
store instead of inline, and prep work is balanced by size and type: a 
balancing strategy plans task groups from the declared object sizes and kinds, 
with its parallelism taken from the num-partitions setting, local mode, Spark 
dynamic allocation or the executor count.
   
   ### Summary and Changelog
   
   - Input store: prepared image and video-frame payloads can be written out of 
line (with markers, so a failed round rolls them back) instead of inline.
   - `SizeTypeAwareBalancingStrategy`: plans prep task groups from the declared 
object sizes and kinds. Its parallelism comes from the num-partitions setting, 
local mode, `spark.dynamicAllocation.maxExecutors` or 
`spark.executor.instances`.
   - The video embedding round and opt-in memory cells of the video round 
(`-Dvideo.memory.cells`, not run by default CI), with a large-frames fixture 
and its generator script.
   - 31 test classes (21 new).
   
   No code was copied from other projects.
   
   ### Impact
   
   Large media rounds spread their work by size and type and can keep payloads 
out of line. No change outside the new source.
   
   ### Risk Level
   
   medium: changes how prep work is distributed for the new source. Verified on 
JDK 11 / Spark 3.5 / Scala 2.12 with 245 tests (7 skipped: external model input 
or opt-in cells), module compilation, checkstyle and RAT.
   
   ### Documentation Update
   
   New balancing and input-store settings carry their config documentation; a 
website page follows once the chain lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(ingest): RFC-118: ... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to