rahil-c opened a new issue, #19697: URL: https://github.com/apache/hudi/issues/19697
### Problem `UnstructuredFileDFSSource` (added in #19278) can only discover files by listing a prefix. Every sync walks the whole prefix whether or not anything arrived, and the walk is sequential per prefix on the driver with no parallelism, because an object store has no directories and each prefix is its own LIST round trip. Cost therefore scales with the size of the bucket rather than with what is new, and never shrinks. Hudi already avoids this for data files. `S3EventsSource` lands notifications in a metadata table and `S3EventsHoodieIncrSource` reads that table incrementally; `GcsEventsSource` and `GcsEventsHoodieIncrSource` are the GCS equivalents. Unstructured files cannot use any of it, because the step that turns the selected objects into rows is hardwired to a Spark datasource read. Measured on a 4,032-object corpus across 1,008 prefixes on S3, a sync with nothing new to ingest takes 121.8s by listing versus 12.3s by notification, a 9.9x difference, and the listing path issues roughly 1,363 sequential LIST calls that the notification path does not issue at all. ### Blockers behind it - The object-key predicate is derived from the configured data file format, which defaults to `parquet`, so a document corpus silently matches nothing. - Partition count is derived from the bytes referenced. For documents that is a poor proxy for cost, since a file above `parse.max.bytes` is referenced without being read. - `CloudObjectMetadata` carries no timestamp, so repeated writes to the same path cannot be ordered. ### Expected Unstructured files should be ingestable from S3 and GCS notifications, producing the same table the folder-based source produces, with the existing columnar behaviour unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
