matthewmturner commented on issue #1836: URL: https://github.com/apache/arrow-datafusion/issues/1836#issuecomment-1042422114
@returnString thanks for your work to enable this! it's going to be very helpful on these next steps. One question, what would be an example of an implementer of `CatalogProvider` or `SchemaProvider` traits that doesnt end up essentially being a `MemoryCatalogProvider` or `MemorySchemaProvider`? I would have thought these could have been a struct but I'm probably missing something. As some additional context - I'm working on a s3 extension for datafusion (https://github.com/datafusion-contrib/datafusion-objectstore-s3) and im trying to figure out if I should be making something like a `S3Catalog` and / or `S3Schema` that implement the aforementioned traits or if I should just use the Memory ones complemented with a builder pattern like @alamb mentioned or just add methods to create a schema from an `ObjectStore` instance. The way I'm thinking about it now is that the majority of the heavy lifting / logic will be on creating many tables in a schema - for example by looking at a directory and reading all sub directories or files within as tables. So I think I will focus my efforts on playing with an API for that. I have the below in mind right now but I guess that's why I was interested in learning more about how the `CatalogProvider` and `SchemaProvider` traits were intended to be used outside of their memory implementations. ``` let s3 = S3FileSystem::default(); let active_schema = s3.create_schema(uri: "s3://bucket/active"); let hist_schema = s3.create_schema(uri: "s3://bucket/hist"); let catalog = MemoryCatalogProvider::new().register_schema(name: "active", schema: active_shema).register_schema(name: "hist", schema: hist_schema); ``` Thanks again for your input! Hope I'm not over complicating this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
