Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path

+1 (non-binding) > On Sep 6, 2017, at 7:29 PM, Wenchen Fan wrote: > > Hi all, > > In the previous discussion, we decided to split the read and write path of > data source v2 into 2 SPIPs, and I'm sending this email to call a vote for > Data Source V2 read path only. >

sessionState could not be accessed in spark-shell command line

Hi, when I use spark-shell to get the logical plan of sql, an error occurs scala> spark.sessionState :30: error: lazy value sessionState in class SparkSession cannot be accessed in org.apache.spark.sql.SparkSession spark.sessionState ^ But if I use spark-submit to access

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path

+1 On Wed, Sep 6, 2017 at 8:53 PM, Xiao Li wrote: > +1 > > Xiao > > 2017-09-06 19:37 GMT-07:00 Wenchen Fan : > >> adding my own +1 (binding) >> >> On Thu, Sep 7, 2017 at 10:29 AM, Wenchen Fan wrote: >> >>> Hi all, >>> >>> In the

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path

+1 Xiao 2017-09-06 19:37 GMT-07:00 Wenchen Fan : > adding my own +1 (binding) > > On Thu, Sep 7, 2017 at 10:29 AM, Wenchen Fan wrote: > >> Hi all, >> >> In the previous discussion, we decided to split the read and write path >> of data source v2 into 2

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path

adding my own +1 (binding) On Thu, Sep 7, 2017 at 10:29 AM, Wenchen Fan wrote: > Hi all, > > In the previous discussion, we decided to split the read and write path of > data source v2 into 2 SPIPs, and I'm sending this email to call a vote for > Data Source V2 read path

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Hi Ryan, Yea I agree with you that we should discuss some substantial details during the vote, and I addressed your comments about schema inference API in my new PR, please take a look. I've also called a new vote for the read path, please vote there, thanks! On Thu, Sep 7, 2017 at 7:55 AM,

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

I'm all for keeping this moving and not getting too far into the details (like naming), but I think the substantial details should be clarified first since they are in the proposal that's being voted on. I would prefer moving the write side to a separate SPIP, too, since there isn't much detail

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Hi all, I've submitted a PR for a basic data source v2, i.e., only contains features we already have in data source v1. We can discuss API details like naming in that PR: https://github.com/apache/spark/pull/19136 In the meanwhile, let's keep this vote open and collecting more feedbacks. Thanks

Re: Putting Kafka 0.8 behind an (opt-in) profile

I kind of doubt the kafka 0.10 integration is going to change much at all before the upgrade to 0.11 On Wed, Sep 6, 2017 at 8:57 AM, Sean Owen wrote: > Thanks, I can do that. We're then in the funny position of having one > deprecated Kafka API, and one experimental one. > >

Re: Putting Kafka 0.8 behind an (opt-in) profile

Thanks, I can do that. We're then in the funny position of having one deprecated Kafka API, and one experimental one. Is the Kafka 0.10 integration as stable as it is going to be, and worth marking as such for 2.3.0? On Tue, Sep 5, 2017 at 4:12 PM Cody Koeninger wrote: > +1

Re: [VOTE][SPIP] SPARK-21190: Vectorized UDFs in Python

Hi all, Thank you for voting and suggestions. As Wenchen mentioned and also we're discussing at JIRA, we need to discuss the size hint for the 0-parameter UDF. But I believe we got a consensus about the basic APIs except for the size hint, I'd like to submit a pr based on the current proposal

Re: descriptions not updated in "Useful Developer Tools" of the website

ok, I will. thanks! On Wed, Sep 6, 2017 at 4:19 PM, Sean Owen wrote: > True, can you make a pull request vs github.com/apache/spark-website? I > think users probably have to add this to spark.{executor|driver}. > extraJavaOptions > > On Wed, Sep 6, 2017 at 8:08 AM Takeshi

Re: descriptions not updated in "Useful Developer Tools" of the website

True, can you make a pull request vs github.com/apache/spark-website? I think users probably have to add this to spark.{executor|driver}.extraJavaOptions On Wed, Sep 6, 2017 at 8:08 AM Takeshi Yamamuro wrote: > hi, devs, > > I found some descriptions were not updated in >

descriptions not updated in "Useful Developer Tools" of the website

hi, devs, I found some descriptions were not updated in https://spark.apache.org/developer-tools.html#profiling # I think `SPARK_JAVA_OPTS` not used anymore. not sure who makes the website updated though, this is just a head-up. Thanks, -- --- Takeshi Yamamuro