rahil-c opened a new issue, #19675: URL: https://github.com/apache/hudi/issues/19675
`UnstructuredFileDFSSource` and `EmbeddingTransformer` landed in #19278. Running them against real folders surfaces several defects, most of which stop ingestion outright. **A filename containing a comma fails the whole sync.** Selected paths are joined into one comma-delimited string and split back apart, so `Q3,2024 report.pdf` is torn into fragments and the batch dies with `FileNotFoundException`. One badly named file among a million blocks all ingestion, and the only remedy available to the user is renaming their source data. Commas are ordinary in document corpora. **The batch is unbounded by file count.** `--source-limit` bounds bytes but never how many files, and file count is what drives driver memory and task count. At the shipped default of `Long.MAX_VALUE`, a 2 GB folder of 64,487 small files exhausted an 8 GB driver heap. **A group of files sharing one modification time ignores the limit entirely.** Selection breaks only when the byte budget is exhausted *and* the current file's mtime is strictly newer than the last accepted one, so a same-mtime group can never be split. A 64 MiB limit ingested 800 MiB. This is common rather than exotic: bulk uploads share a timestamp, and object stores report modification times at second granularity. **Smaller defects in the same code paths:** - Every file is re-`getFileStatus`ed on the executor although the driver already stat'd it during listing, doubling metadata requests against object storage. - `TextChunker.findBreak` calls `lastIndexOf` with no lower bound, scanning back to index 0 whenever a boundary is absent. Chunking is quadratic in document length; a document at the `parse.max.text.chars` default of 1,000,000 spends over a second of CPU in chunking alone. - `TikaDocumentParser` catches `Throwable`, so an `OutOfMemoryError` becomes an ordinary `FAILED` row and ingestion continues on a JVM in an undefined state. - `EmbeddingIterator` shuts its worker pool down from `end()`, which `LazyIterableIterator` only invokes when the input drains normally. A task killed by an embeddings failure strands `max.inflight.requests` threads on an executor JVM that Spark keeps reusing. - `OpenAICompatibleEmbeddingProvider` builds one `HttpClient` per Spark partition. It is not `Closeable` before Java 21 and owns a selector thread plus a connection pool, so there is no release path. - `blob.inline.max.bytes` is not validated against `Integer.MAX_VALUE`, so a threshold above 2 GiB surfaces as a `NegativeArraySizeException` inside an executor rather than a clear error. - An out-of-line blob reference records only a path, with `length` left null although the field exists. A copy that preserves the modification time changes content without the source ever re-ingesting it, and a historical row's reference always resolves to the file's current bytes, both with nothing to detect the mismatch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
