vinothchandar opened a new pull request, #19278: URL: https://github.com/apache/hudi/pull/19278
### Describe the issue this Pull Request addresses Landing unstructured data (documents, images, videos) in a lakehouse today means gluing Spark's `binaryFile` reader to hand-rolled parsing UDFs, with no incremental ingestion, no upsert semantics, no blob management, and no embedding lifecycle. With the BLOB and VECTOR types available since 1.2, HoodieStreamer can offer this as one command: point at a folder on cloud storage, get a queryable Hudi table with parsed text, chunks, and embedding vectors that stay current as files arrive or change. ### Summary and Changelog Two additive pieces in `hudi-utilities`, no changes to existing behavior: **`UnstructuredFileDFSSource`** (RowSource) - Reuses `DFSPathSelector` for modification-time-checkpointed incremental discovery: each sync ingests only new/changed files; record key = `path` with `modification_time` ordering makes re-ingested files upsert in place. - Per-record blob placement by configurable size threshold (default 1 MiB): small files store bytes INLINE in the BLOB column; larger files store an OUT_OF_LINE reference to the original file in place. Large-file bytes never enter Spark rows, so memory and shuffle stay bounded regardless of file sizes (`--source-limit` already caps total inline bytes per batch). - Embedded text extraction behind a pluggable `DocumentParser`; default `TikaDocumentParser` (Apache Tika, in-process, no services). Parsing never fails the job: per-row `parse_status`/`parse_error` record SUCCESS/TRUNCATED/EMPTY/SKIPPED/FAILED. Extracted text is chunked (size/overlap configurable) into a nested `chunks` column. - Only `tika-core` (~800 KB) ships in the utilities bundles; format parser modules (PDF, Office, ...) are supplied at runtime (e.g. spark-submit `--packages org.apache.tika:tika-parsers-standard-package`), degrading gracefully to EMPTY parses when absent. **`EmbeddingTransformer`** (chained via `--transformer-class`) - Appends a `VECTOR(dimension)` column by calling an embeddings API, batching at the record level within each partition (up to `batch.size`, default 1024, records per request) — executors stay busy, large request batches keep request rates low, and retry with `Retry-After`-honoring backoff is the only flow control. - Pluggable `EmbeddingProvider`; default targets any OpenAI-compatible `/v1/embeddings` endpoint (Ollama, TEI, vLLM, OpenAI, Voyage). API keys come from an environment variable named in config, never config values. Errors after bounded retries fail the batch loudly; rows without text (images, videos, failed parses) get a null vector and are never sent to the API. - Since only new/changed records flow through each sync, embeddings stay current with the data incrementally. Also: `tika.version` property in the root pom; `tika-core` added to both utilities bundle include lists. No code copied from other projects. ### Impact New optional source + transformer in hudi-utilities; no existing Hudi APIs, configs, storage format, or bundles change behavior. New user-facing configs under `hoodie.streamer.source.unstructured.*` and `hoodie.streamer.transformer.embedding.*` (documented via `@ConfigClassProperty`). Utilities bundles grow by ~800 KB (tika-core). ### Risk Level low — additive code paths only. Verified by 10 offline tests (chunker boundaries, Tika status mapping on generated fixtures, source-level inline/out-of-line + checkpoint semantics, BLOB logical type surviving Row-to-Avro schema conversion, embedding batching/429-retry/fail-fast against a stub HTTP server) plus a streamer E2E writing a Parquet COW table across two syncs (blob struct round-trip for both placements, upsert refresh, vector logical type present in the committed table schema). Additionally validated end to end locally with real PDFs/DOCX/HTML/images, live Ollama embeddings, and `hudi_vector_search` queries across insert/update/delete cycles. ### Documentation Update Website docs for the new source/transformer configs and a quickstart example are planned as a follow-up PR (config reference tables generate from the config classes). Two notes to include: `hudi_vector_search` requires `spark-mllib` on the classpath (present in standard Spark distributions), and Office-format parsing needs a commons-compress version aligned with Tika's POI. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
