Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

> Ideally also getting sort orders _after_ getting filters. Yea we should have a deterministic order when applying various push downs, and I think filter should definitely go before sort. This is one of the details we can discuss during PR review :) On Fri, Sep 1, 2017 at 9:19 AM, James Baker

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

I think that makes sense. I didn't understand backcompat was the primary driver. I actually don't care right now about aggregations on the datasource I'm integrating with - I just care about receiving all the filters (and ideally also the desired sort order) at the same time. I am mostly fine

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

I think that makes sense. I didn't understand backcompat was the primary driver. I actually don't care right now about aggregations on the datasource I'm integrating with - I just care about receiving all the filters (and ideally also the desired sort order) at the same time. I am mostly fine

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

I think that makes sense. I didn't understand backcompat was the primary driver. I actually don't care right now about aggregations on the datasource I'm integrating with - I just care about receiving all the filters (and ideally also the desired sort order) at the same time. I am mostly fine

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations! Xiao 2017-08-31 9:38 GMT-07:00 Imran Rashid : > Congrats Jerry! > > On Mon, Aug 28, 2017 at 8:28 PM, Matei Zaharia > wrote: > >> Hi everyone, >> >> The PMC recently voted to add Saisai (Jerry) Shao as a committer. Saisai >> has

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Hi Ryan, I think for a SPIP, we should not worry too much about details, as we can discuss them during PR review after the vote pass. I think we should focus more on the overall design, like James did. The interface mix-in vs plan push down discussion was great, hope we can get a consensus on

Re: Updates on migration guides

+1 think we do migration guide changes for ML and R in separate JIRA/PR/commit but we definition should have it updated before the release. From: linguin@gmail.com Sent: Wednesday, August 30, 2017 8:27:17 AM To: Dongjoon Hyun Cc: Xiao

Re: Welcoming Saisai (Jerry) Shao as a committer

Congrats Jerry! On Mon, Aug 28, 2017 at 8:28 PM, Matei Zaharia wrote: > Hi everyone, > > The PMC recently voted to add Saisai (Jerry) Shao as a committer. Saisai > has been contributing to many areas of the project for a long time, so it’s > great to see him join. Join

Re: SPIP: Spark on Kubernetes

The proposal is in the process of being updated to include the details on testing that we have, that Imran pointed out. Please expect an update on the SPARK-18278 . Mridul had a couple of points as well, about exposing an SPI and we've been

Re: Moving Scala 2.12 forward one step

I don't think there's a target. The changes aren't all that hard (see the SPARK-14220 umbrella) but there are some changes that are hard or impossible without changing key APIs, as far as we can see. That would suggest 3.0. One motivation I have here for getting it as far as possible otherwise is

Re: Moving Scala 2.12 forward one step

Hi Sean, Do we have a planned target version for Scala 2.12 support? Several other projects like Zeppelin, Livy which rely on Spark repl also require changes to support this Scala 2.12. Thanks Jerry On Thu, Aug 31, 2017 at 5:55 PM, Sean Owen wrote: > No, this doesn't let

Re: Moving Scala 2.12 forward one step

No, this doesn't let Spark build and run on 2.12. It makes changes that will be required though, the ones that are really no loss to the current 2.11 build. On Thu, Aug 31, 2017, 10:48 Denis Bolshakov wrote: > Hello, > > Sounds amazing. Is there any improvements in

Re: Moving Scala 2.12 forward one step

Hello, Sounds amazing. Is there any improvements in benchmarks? On 31 August 2017 at 12:25, Sean Owen wrote: > Calling attention to the question of Scala 2.12 again for moment. I'd like > to make a modest step towards support. Have a look again, if you would, at >

Moving Scala 2.12 forward one step

Calling attention to the question of Scala 2.12 again for moment. I'd like to make a modest step towards support. Have a look again, if you would, at SPARK-14280: https://github.com/apache/spark/pull/18645 This is a lot of the change for 2.12 that doesn't break 2.11, and really doesn't add any