Re: [VOTE][SPIP] SPARK-22026 data source v2 write path

2017-10-09 Thread Reynold Xin
+1 One thing with MetadataSupport - It's a bad idea to call it that unless adding new functions in that trait wouldn't break source/binary compatibility in the future. On Mon, Oct 9, 2017 at 6:07 PM, Wenchen Fan wrote: > I'm adding my own +1 (binding). > > On Tue, Oct 10,

[SPIP] SPARK-22229: RDMA Accelerated Shuffle Engine

2017-10-09 Thread Yuval Degani
Dear Spark community, I would like to call for the review of SPARK-9: "RDMA Accelerated Shuffle Engine". The request purpose is to embed an RDMA accelerated Shuffle Manager into mainstream Spark. Such implementation is available as an external plugin as part of the "SparkRDMA" project:

Re: [VOTE][SPIP] SPARK-22026 data source v2 write path

2017-10-09 Thread Wenchen Fan
I'm adding my own +1 (binding). On Tue, Oct 10, 2017 at 9:07 AM, Wenchen Fan wrote: > I'm going to update the proposal: for the last point, although the > user-facing API (`df.write.format(...).option(...).mode(...).save()`) > mixes data and metadata operations, we are

Re: [VOTE][SPIP] SPARK-22026 data source v2 write path

2017-10-09 Thread Wenchen Fan
I'm going to update the proposal: for the last point, although the user-facing API (`df.write.format(...).option(...).mode(...).save()`) mixes data and metadata operations, we are still able to separate them in the data source write API. We can have a mix-in trait `MetadataSupport` which has a

Re: Raise Jenkins timeout?

2017-10-09 Thread Josh Rosen
I bumped the timeouts up to 255 minutes (to exceed https://github.com/apache/spark/blame/master/dev/run-tests-jenkins.py#L185). Let's see if this resolves the problem. On Mon, Oct 9, 2017 at 9:30 AM shane knapp wrote: > ++joshrosen > > On Mon, Oct 9, 2017 at 1:48 AM, Sean

Re: Raise Jenkins timeout?

2017-10-09 Thread shane knapp
++joshrosen On Mon, Oct 9, 2017 at 1:48 AM, Sean Owen wrote: > I'm seeing jobs killed regularly, presumably because the time out (210 > minutes?) > > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA% >

Raise Jenkins timeout?

2017-10-09 Thread Sean Owen
I'm seeing jobs killed regularly, presumably because the time out (210 minutes?) https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.6/3907/console Possibly related: this master-SBT-2.7 build hasn't passed in weeks: