Davis-Zhang-Onehouse opened a new issue, #20689:
URL: https://github.com/apache/hudi/issues/20689

   Part of #20683 (RFC-118, #20684).
   
   The cloud incremental sources (S3/GCS events) select objects and read them 
in `CloudDataFetcher`, and the read is fixed to a columnar Spark datasource 
scan. A source that parses documents instead (the RFC-118 unstructured source) 
needs to supply its own key predicate, partition sizing and read while reusing 
the selection.
   
   Add a `CloudObjectMaterializer` interface with `ColumnarFileMaterializer` as 
the default (today's behaviour), a `CloudDataFetcher` constructor that takes a 
materializer, a `generateFilter` overload that appends the materializer's key 
predicate, and an optional ordering token on `CloudObjectMetadata`. Existing 
sources and constructors keep working unchanged.
   
   PR: (linked once opened)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to