houqp commented on a change in pull request #1665:
URL: https://github.com/apache/arrow-datafusion/pull/1665#discussion_r792297397
##########
File path: datafusion/src/datasource/object_store/mod.rs
##########
@@ -141,7 +141,7 @@ pub trait ObjectStore: Sync + Send + Debug {
let suffix = suffix.to_owned();
Ok(Box::pin(file_stream.filter(move |fr| {
let has_suffix = match fr {
- Ok(f) => f.path().ends_with(&suffix),
+ Ok(f) => do_path_filter_with_suffix(f.path(), &suffix),
Review comment:
Forcing the filter here will restrict the usefullness of the object
store abstraction. for example, I won't be able to use `list_file_with_suffix`
in delta-rs to manage delta lake metadata. Object store should be a very
generic module that is agnostic of this type of application specific file
naming conventions. I think a better place to host this particular logic is in
the ListingTable struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
