Re: Public v2 interface location

2018-11-30 Thread JackyLee
Hi, Ryan Blue. I don't think it would be a good idea to add the sql-api module. I prefer to add sql-api to sql/core. The sql is just another representation of dataset, thus there is no need to add new module to do this. Besides, it would be easier to add sql-api in core. By the way, I don't

Public v2 interface location

2018-11-30 Thread Ryan Blue
Hi everyone, In the DSv2 sync this week, we discussed adding a new SQL module, sql-api, that would contain the interfaces for authors to plug in external sources. The rationale for adding this package is that the common logical plans and rules to validate those plans should live in Catalyst, but

Re: [SS] No reponse on a PR: Report numOutputRows in SinkProgress

2018-11-30 Thread Gabor Somogyi
Hi Vaclav, As I see it's conflicting at the moment. BR, G On Fri, Nov 30, 2018 at 11:49 AM Vaclav Kosar wrote: > Fellow Structured Streamers, > > I am having trouble getting any feedback on my PR for reporting number > of written rows in Structured Streaming: >

[SS] No reponse on a PR: Report numOutputRows in SinkProgress

2018-11-30 Thread Vaclav Kosar
Fellow Structured Streamers, I am having trouble getting any feedback on my PR for reporting number of written rows in Structured Streaming: https://github.com/apache/spark/pull/21919 Please help me either close or suggest a change to the PR. Thanks, Vaclav