Davis-Zhang-Onehouse opened a new issue, #20689: URL: https://github.com/apache/hudi/issues/20689
Part of #20683 (RFC-118, #20684). The cloud incremental sources (S3/GCS events) select objects and read them in `CloudDataFetcher`, and the read is fixed to a columnar Spark datasource scan. A source that parses documents instead (the RFC-118 unstructured source) needs to supply its own key predicate, partition sizing and read while reusing the selection. Add a `CloudObjectMaterializer` interface with `ColumnarFileMaterializer` as the default (today's behaviour), a `CloudDataFetcher` constructor that takes a materializer, a `generateFilter` overload that appends the materializer's key predicate, and an optional ordering token on `CloudObjectMetadata`. Existing sources and constructors keep working unchanged. PR: (linked once opened) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
