rahil-c opened a new pull request, #19677: URL: https://github.com/apache/hudi/pull/19677
### Describe the issue this Pull Request addresses Closes #19675 ### Summary and Changelog Fixes the defects that stop `UnstructuredFileDFSSource` working on real folders, plus several smaller ones in the same code paths. Discovery moves off `DFSPathSelector` onto a selector belonging to this source, which fixes three problems at once. Selected files are returned as a list rather than a comma-joined string, so a filename containing a comma no longer tears apart and fails the batch. Batches are bounded by file count through a new `hoodie.streamer.source.unstructured.max.files.per.batch`, since `--source-limit` bounds only bytes while file count is what drives driver memory. The checkpoint carries `(modificationTime, lastPath)` rather than a bare timestamp, which lets a group of files sharing one modification time be split across syncs without the remainder being stranded by the next sync's `mtime > checkpoint` filter. A legacy timestamp-only checkpoint keeps its previous meaning, so upgrading re-ingests nothing. `DFSPathSelector` itself is untouched, so other DFS-based sources are unaffected. Each selected file now carries the size and modification time the driver learned while listing, removing a second round of metadata requests from the executors. Also in this PR: the chunk-boundary search is bounded to its acceptable window instead of scanning to the start of the document; `TikaDocumentParser` catches `Exception` so an `Error` propagates rather than becoming a `FAILED` row; the embedding worker pool is released on task completion however the task ends; embedding HTTP clients are shared per JVM instead of created per partition; `blob.inline.max.bytes` is validated against `Integer.MAX_VALUE`; and out-of-line blob references record the ingested length in the field already present in the schema. ### Impact One new config, `hoodie.streamer.source.unstructured.max.files.per.batch`, defaulting to 10000. Existing tables are unaffected: the checkpoint format change is backward compatible, and out-of-line references gain a value in a field that was previously written as null. No public API, storage format or bundle change. ### Risk Level low Contained to `hudi-utilities` and to a source and transformer that are opt-in. Every fix has a test that fails without it, verified by reverting each one. Beyond unit tests, each of the three selector problems was reproduced and then re-run end to end through HoodieStreamer on Spark 3.5.5: a corpus containing a comma in a filename, a 2 GB corpus of 64,487 files at the default source limit that previously exhausted an 8 GB driver, and a 25,167-file corpus sharing one modification time under a 64 MiB limit that previously ingested 800 MiB. ### Documentation Update The new config carries `@ConfigClassProperty` documentation and generates into the config reference. No website change is needed; the source and transformer are not yet documented on the site. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
