houqp commented on issue #944:
URL:
https://github.com/apache/arrow-datafusion/issues/944#issuecomment-907571328
Here is my understanding based on @yjshen 's PRs so far:
* CatalogProvider is a mapping from schema name to SchemaProvider (schema
here refers to a collection of tables, not table schema)
* SchemaProvider is a mapping from table name to TableProvider
* TableProvider provides table partitions, field schemas and a scan method
to perform table scan using ObjectStore to drive the IO.
* TableProvider::scan takes push down filter expressions as argument,
then issue corresponding ObjectStore list and read calls to perform minimal IO
needed to fetch the data from object store
I am guessing a hive metastore connector will need to touch both
SchemaProvider and TableProvider? I do think that ObjectStore is too low level
to implement hive metastore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
