Davis-Zhang-Onehouse opened a new pull request, #20719: URL: https://github.com/apache/hudi/pull/20719
### Describe the issue this Pull Request addresses issue: #20718 Part of #20683, RFC-118 (#20684). Stacked on #20717: review only the last commit. Until the PRs below it merge, this PR also shows their commits. Video frame preprocessing for embedding tables: keyframe sampling, frame dedup and JPEG encoding behind a frame extraction strategy, a seekable BLOB input stream and byte channel, and the FFmpeg keyframe frame source. FFmpeg (LGPL) stays an optional, user-supplied dependency: it is never bundled, and the build bans it at compile and runtime scope. ### Summary and Changelog - Video frame preprocessing: keyframe sampling, frame dedup and JPEG encoding behind a `FrameExtractionStrategy`, a seekable BLOB input stream and byte channel, and the FFmpeg keyframe frame source. - FFmpeg (org.bytedeco javacv/javacpp/ffmpeg, LGPL) is optional and never shipped: provided scope, test-scope natives, and a root enforcer ban on `org.bytedeco` at compile and runtime scope. A job that prepares video with FFmpeg adds the jars at launch, for example through `spark.jars`. - `scripts/ci/check_no_ffmpeg_in_bundles.sh` checks a built bundle jar for `org/bytedeco` entries and FFmpeg natives (exit 0 clean, 1 with the offending entries). It is not wired into CI yet; it can become a step after the utilities bundle build in `bot.yml` if reviewers want that. - Adds the JCodec dependency (BSD-2), which the seekable channel uses, with its LICENSE entry. - Test clips: `video/README.md` names every clip's source, and `video/generate-av-fixtures.sh` regenerates the derived clips from the base clip. 21 test classes (18 new). Upstream files touched: root `pom.xml`, `hudi-utilities/pom.xml`, both utilities bundle poms, `LICENSE`. No code was copied from other projects. ### Impact The embedding stream can prepare video when the user supplies FFmpeg. Bundles gain JCodec; FFmpeg is never bundled. ### Risk Level medium: new dependency and bundle changes. Verified on JDK 11 / Spark 3.5 / Scala 2.12 with 213 tests (none skipped), module compilation, checkstyle and RAT. Both utilities bundles were rebuilt and checked for the absence of FFmpeg classes and natives. ### Documentation Update New video settings carry their config documentation, including how to add FFmpeg at launch; a website page follows once the chain lands. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
