Davis-Zhang-Onehouse opened a new pull request, #20719:
URL: https://github.com/apache/hudi/pull/20719

   ### Describe the issue this Pull Request addresses
   
   issue: #20718
   Part of #20683, RFC-118 (#20684).
   Stacked on #20717: review only the last commit. Until the PRs below it 
merge, this PR also shows their commits.
   
   Video frame preprocessing for embedding tables: keyframe sampling, frame 
dedup and JPEG encoding behind a frame extraction strategy, a seekable BLOB 
input stream and byte channel, and the FFmpeg keyframe frame source. FFmpeg 
(LGPL) stays an optional, user-supplied dependency: it is never bundled, and 
the build bans it at compile and runtime scope.
   
   ### Summary and Changelog
   
   - Video frame preprocessing: keyframe sampling, frame dedup and JPEG 
encoding behind a `FrameExtractionStrategy`, a seekable BLOB input stream and 
byte channel, and the FFmpeg keyframe frame source.
   - FFmpeg (org.bytedeco javacv/javacpp/ffmpeg, LGPL) is optional and never 
shipped: provided scope, test-scope natives, and a root enforcer ban on 
`org.bytedeco` at compile and runtime scope. A job that prepares video with 
FFmpeg adds the jars at launch, for example through `spark.jars`.
   - `scripts/ci/check_no_ffmpeg_in_bundles.sh` checks a built bundle jar for 
`org/bytedeco` entries and FFmpeg natives (exit 0 clean, 1 with the offending 
entries). It is not wired into CI yet; it can become a step after the utilities 
bundle build in `bot.yml` if reviewers want that.
   - Adds the JCodec dependency (BSD-2), which the seekable channel uses, with 
its LICENSE entry.
   - Test clips: `video/README.md` names every clip's source, and 
`video/generate-av-fixtures.sh` regenerates the derived clips from the base 
clip. 21 test classes (18 new).
   
   Upstream files touched: root `pom.xml`, `hudi-utilities/pom.xml`, both 
utilities bundle poms, `LICENSE`. No code was copied from other projects.
   
   ### Impact
   
   The embedding stream can prepare video when the user supplies FFmpeg. 
Bundles gain JCodec; FFmpeg is never bundled.
   
   ### Risk Level
   
   medium: new dependency and bundle changes. Verified on JDK 11 / Spark 3.5 / 
Scala 2.12 with 213 tests (none skipped), module compilation, checkstyle and 
RAT. Both utilities bundles were rebuilt and checked for the absence of FFmpeg 
classes and natives.
   
   ### Documentation Update
   
   New video settings carry their config documentation, including how to add 
FFmpeg at launch; a website page follows once the chain lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(ingest): RFC-118: ... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to