Davis-Zhang-Onehouse opened a new pull request, #20733: URL: https://github.com/apache/hudi/pull/20733
### Describe the issue this Pull Request addresses issue: #20732 Part of #20683, RFC-118 (#20684). Stacked on #20731: review only the last commit. Until the PRs below it merge, this PR also shows their commits. Embedding clients for Amazon Bedrock (Titan and Cohere text models, signed with the AWS SDK default credential chain), Google Vertex AI (text and multimodal, with token refresh and its retry loop) and Jina, each with its compute writer. Also closes correctness and resource-limit gaps found in review of the earlier PRs of this chain. ### Summary and Changelog - Embedding clients with compute writers: Amazon Bedrock (Titan and Cohere text models), Google Vertex AI (text and multimodal, token refresh, retry loop) and Jina. - Adds `software.amazon.awssdk:bedrockruntime` to hudi-utilities. Like the other AWS SDK clients there (sqs, kinesis, sts), it is not shaded into the utilities bundles: a job that uses the Bedrock provider adds the AWS SDK jars (bedrockruntime, its SDK core dependencies and apache-client) at launch. Requests are signed with the AWS SDK default credential chain; there is no credential key in the embedding config. This PR also closes correctness and resource-limit gaps in the earlier PRs of this chain, each with a regression test, a passing control and a revert-to-red run: - Compute rounds keep customer columns whose names match the internal outcome column, including case variants. - A denied source read fails the round and keeps its checkpoint, instead of being treated as a missing file. - PDF extraction charges repeated font materializations, scans the decoder bytes the parser actually consumes, and records a truncated empty page as a durable loss. The PDF record implementation version advances to v3. - Older embedding tables gain the nullable image BLOB column before prep or compute. - Image preparation admits source-width PNG/BMP/GIF row buffers before subsampling, and reads PNG row layout without copying unrelated metadata. - 12 provider test classes plus the regression tests; recorded responses under `unstructured/vertex/`; no live endpoint is called. No code was copied from other projects. ### Impact The embedding stream can call Bedrock, Vertex AI and Jina. hudi-utilities gains the bedrockruntime dependency (not bundled). Fixes listed above change behaviour only inside the new source. ### Risk Level medium: new provider dependency and fixes across the earlier stages. Verified on JDK 11 / Spark 3.5 / Scala 2.12 with 339 tests (none skipped), module compilation, checkstyle and RAT. The integrated chain up to this PR passed 1987 tests with 0 failures or errors and 10 accepted, gated skips; all 24 selected memory cells passed; both bundles were checked. ### Documentation Update New provider settings carry their config documentation; a website page follows once the chain lands. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
