houqp commented on pull request #950: URL: https://github.com/apache/arrow-datafusion/pull/950#issuecomment-907525589
@rdettai on the topics of prefix semantic and filter pushdown, in my mind, these are higher level logic/abstractions that are not natively supported by the object store. For example, I am under the assumption that `TableProvider` is responsible for parsing the partition parameters from the URI `bucket/prefix?partition=year&partition=month` and constructing the list object API calls with appropriate prefixes. Similar for URIs like `bucket/prefix?versionAsOf=v2`, the delta table `TableProvider` should be responsible to parse out `versionAsOf=v2` and issue corresponding object list calls to gather the right list of commit/snapshot objects to fetch from the object store. I am also having a hard time imaging how will filter push down expressions be used to help the object store build a better or more efficient API call. It seems like these kind of higher level logic should be handled at layers above ObjectStore? They seem to be object store agnostics. As a result, it would be better to just implement them once in abstraction layers like `TableProvider` instead of adding separate implementation in each type of object store. Does this make sense to you? Or am I missing specific features from some object stores that can support native filter push down optimizations at the API level? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
