tustvold commented on issue #4533: URL: https://github.com/apache/arrow-datafusion/issues/4533#issuecomment-1348482218
> Unfortunately not, FileOpener requires ObjectStore as an argument to open fn. But we could change that? All the `FileOpener` are constructed in a context that could resolve the object store if it wanted to. All that would change is the logic would move out of `FileStream::new` > DataFusion could have Its own trait for ObjectStore read(get) operations, that exposes low-level interface and simplifies integration. No objection on principle, but I'm sceptical that introducing more indirection is necessary nor desirable > Could you tell me what are best practices for interacting with SessionContext ? I don't honestly know, I believe @alamb is currently working on making the state/config slightly less impenetrable. > What do you think about moving schema inference into scan and removing it from TableProvider trait ? I don't think this is possible, as planning needs to know the schema. In general though performing schema inference per query is very expensive, especially for non-parquet data. I strongly recommend investing in some sort of catalog to store this data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
