Re: [DISCUSS] Supporting hive on DataSourceV2

2020-03-24 Thread JackyLee
Hi Blue, I have created a jira for supporting hive on DataSourceV2,we can associate specific modules on this jira. https://issues.apache.org/jira/browse/SPARK-31241 Could you provide a google doc for current design, so that we can discuss and improve it in detail here? -- Sent from:

Re: [DISCUSS] Supporting hive on DataSourceV2

2020-03-24 Thread JackyLee
Glad to hear that you have already supported it, that is just the thing we are doing. And these exceptions you said doesn't conflict with hive support, we can easily make it compatible. >Do you have an idea about where the connector should be developed? I don’t think it makes sense for it to be

Re: [DISCUSS] Supporting hive on DataSourceV2

2020-03-23 Thread Ryan Blue
Hi Jacky, We’ve internally released support for Hive tables (and Spark FileFormat tables) using DataSourceV2 so that we can switch between catalogs; sounds like that’s what you are planning to build as well. It would be great to work with the broader community on a Hive connector. I will get a

[DISCUSS] Supporting hive on DataSourceV2

2020-03-23 Thread JackyLee
Hi devs, I’d like to start a discussion about Supporting Hive on DatasourceV2. We’re now working on a project using DataSourceV2 to provide multiple source support and it works with the data lake solution very well, yet it does not yet support HiveTable. There are 3 reasons why we need to support