[RESULT] [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-12 Thread Patrick Wendell
This vote passes with 14 +1 (7 binding) votes and no 0 or -1 votes. +1 (14): Patrick Wendell Reynold Xin Sean Owen Burak Yavuz Mark Hamstra Michael Armbrust Andrew Or York, Brennon Krishna Sankar Luciano Resende Holden Karau Tom Graves Denny Lee Sean McNamara - Patrick On Wed, Jul 8, 2015 at 10:

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-12 Thread Patrick Wendell
I think we can close this vote soon. Any addition votes/testing would be much appreciated! On Fri, Jul 10, 2015 at 11:30 AM, Sean McNamara wrote: > +1 > > Sean > >> On Jul 8, 2015, at 11:55 PM, Patrick Wendell wrote: >> >> Please vote on releasing the following candidate as Apache Spark version

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-10 Thread Sean McNamara
+1 Sean > On Jul 8, 2015, at 11:55 PM, Patrick Wendell wrote: > > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://s.apache.org/spark-1.4.1 > > The tag to be voted on is v1

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-10 Thread Denny Lee
+1 (non binding) On Fri, Jul 10, 2015 at 9:12 PM Tom Graves wrote: > +1 > > Tom > > > > On Thursday, July 9, 2015 12:55 AM, Patrick Wendell > wrote: > > > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spar

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-10 Thread Tom Graves
+1 Tom On Thursday, July 9, 2015 12:55 AM, Patrick Wendell wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Holden Karau
+1 - compiled on ubuntu & centos, spark-perf run against yarn in client mode on a small cluster comparing 1.4.0 & 1.4.1 (for core) doesn't have any huge jumps (albeit with a small scaling factor). On Wed, Jul 8, 2015 at 11:58 PM, Patrick Wendell wrote: > +1 > > On Wed, Jul 8, 2015 at 10:55 PM, P

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Luciano Resende
+1 (non-binding) mostly looking in the legal aspects of the release. On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Krishna Sankar
+1 1. Compiled OSX 10.10 (Yosemite) OK Total time: 38:11 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK 2.3. Decision Tree, Naive Bayes OK 2.4. KMeans OK Center And S

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread York, Brennon
+1 (non-binding) * ran spark-on-YARN MLLib ALS recommendation pipeline (success) * no regression / performance issues * ran spark-on-YARN GraphX pipeline (success) * no regression / performance issues On 7/8/15, 10:55 PM, "Patrick Wendell" wrote: >Please vote on releasing the following ca

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Andrew Or
+1 2015-07-09 10:26 GMT-07:00 Michael Armbrust : > +1 > > On Thu, Jul 9, 2015 at 10:07 AM, Mark Hamstra > wrote: > >> +1 >> >> On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell >> wrote: >> >>> Please vote on releasing the following candidate as Apache Spark version >>> 1.4.1! >>> >>> This relea

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Michael Armbrust
+1 On Thu, Jul 9, 2015 at 10:07 AM, Mark Hamstra wrote: > +1 > > On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell > wrote: > >> Please vote on releasing the following candidate as Apache Spark version >> 1.4.1! >> >> This release fixes a handful of known issues in Spark 1.4.0, listed here: >> h

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Mark Hamstra
+1 On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://s.apache.org/spark-1.4.1 > > The tag to be voted on is v1.4.1-rc4

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Burak Yavuz
+1 nonbinding. On Thu, Jul 9, 2015 at 7:38 AM, Sean Owen wrote: > +1 nonbinding. All previous RC issues appear resolved. All tests pass > with the "-Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver" invocation. > Signatures et al are OK. > > On Thu, Jul 9, 2015 at 6:55 AM, Patrick Wendell > wrote:

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Sean Owen
+1 nonbinding. All previous RC issues appear resolved. All tests pass with the "-Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver" invocation. Signatures et al are OK. On Thu, Jul 9, 2015 at 6:55 AM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version >

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-09 Thread Reynold Xin
+1 On Wed, Jul 8, 2015 at 11:58 PM, Patrick Wendell wrote: > +1 > > On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell > wrote: > > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > > > This release fixes a handful of known issues in Spark 1.4.0, listed here: >

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-08 Thread Patrick Wendell
+1 On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://s.apache.org/spark-1.4.1 > > The tag to be voted on is v1.4.1-rc4

[VOTE] Release Apache Spark 1.4.1 (RC4)

2015-07-08 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=co

[RESULT] [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Patrick Wendell
This vote is cancelled in favor of RC4. - Patrick On Tue, Jul 7, 2015 at 12:06 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://s.apache.org/spark

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Patrick Wendell
Hey All, The issue that Josh pointed out is not just a test failure, it's an issue with an important bug fix that was not correctly back-ported into the 1.4 branch. Unfortunately the overall state of the 1.4 branch tests on Jenkins was not in great shape so this was missed earlier on. Given that

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Josh Rosen
I've filed https://issues.apache.org/jira/browse/SPARK-8903 to fix the DataFrameStatSuite test failure. The problem turned out to be caused by a mistake made while resolving a merge-conflict when backporting that patch to branch-1.4. I've submitted https://github.com/apache/spark/pull/7295 to fix

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Sean Owen
I see, but shouldn't this test not be run when Hive isn't in the build? On Wed, Jul 8, 2015 at 7:13 PM, Andrew Or wrote: > @Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and > `-Phive-thriftserver`. The MissingRequirementsError is just complaining that > it can't find the right

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Andrew Or
@Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and `-Phive-thriftserver`. The MissingRequirementsError is just complaining that it can't find the right classes. The other one (DataFrameStatSuite) is a little more concerning. 2015-07-08 10:43 GMT-07:00 Pradeep Bashyal : > Hi Shi

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Pradeep Bashyal
Hi Shivaram, I created a Jira Issue for the documentation error. https://issues.apache.org/jira/browse/SPARK-8901 Thanks Pradeep On Wed, Jul 8, 2015 at 11:40 AM, Shivaram Venkataraman < shiva...@eecs.berkeley.edu> wrote: > Hi Pradeep > > Thanks for the catch -- Lets open a JIRA and PR for it.

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Patrick Wendell
Yeah - we can fix the docs separately from the release. - Patrick On Wed, Jul 8, 2015 at 10:03 AM, Mark Hamstra wrote: > HiveSparkSubmitSuite is fine for me, but I do see the same issue with > DataFrameStatSuite -- OSX 10.10.4, java > > 1.7.0_75, -Phive -Phive-thriftserver -Phadoop-2.4 -Pyarn >

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Mark Hamstra
HiveSparkSubmitSuite is fine for me, but I do see the same issue with DataFrameStatSuite -- OSX 10.10.4, java 1.7.0_75, -Phive -Phive-thriftserver -Phadoop-2.4 -Pyarn On Wed, Jul 8, 2015 at 4:18 AM, Sean Owen wrote: > The POM issue is resolved and the build succeeds. The license and sigs > stil

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Shivaram Venkataraman
Hi Pradeep Thanks for the catch -- Lets open a JIRA and PR for it. I don't think documentation changes affect the release though Patrick can confirm that. Thanks Shivaram On Wed, Jul 8, 2015 at 9:35 AM, Pradeep Bashyal wrote: > Here's one thing I ran into: > > The SparkR documentation example

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Sean Owen
Although that should be fixed if it's incorrect, it's not something that would nearly block a release. The question here is whether this artifact can be released as 1.4.1, or whether it has a blocking regression from 1.4.0. On Wed, Jul 8, 2015 at 5:35 PM, Pradeep Bashyal wrote: > Here's one thing

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Pradeep Bashyal
Here's one thing I ran into: The SparkR documentation example in http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is incorrect. sc <- sparkR.init(packages="com.databricks:spark-csv_2.11:1.0.3") should be sc <- sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-08 Thread Sean Owen
The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with "-Pyarn -Phadoop-2.6", with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (.

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-07 Thread Krishna Sankar
+1 (non-binding, of course) 1. Compiled OSX 10.10 (Yosemite) OK Total time: 27:24 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK 2.3. Decision Tree, Naive Bayes OK 2.4. KMea

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-07 Thread Andrew Or
+1 Verified that the previous blockers SPARK-8781 and SPARK-8819 are now resolved. 2015-07-07 12:06 GMT-07:00 Patrick Wendell : > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http

[VOTE] Release Apache Spark 1.4.1 (RC3)

2015-07-07 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=co

[RESULT] [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-07 Thread Patrick Wendell
Hey All, This vote is cancelled in favor of RC3. - Patrick On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://s.apache.

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Patrick Wendell
Let's continue the disucssion on the other thread relating to the master build. On Fri, Jul 3, 2015 at 4:13 PM, Patrick Wendell wrote: > Thanks - it appears this is just a legitimate issue with the build, > affecting all versions of Maven. > > On Fri, Jul 3, 2015 at 4:02 PM, Krishna Sankar wrote

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Patrick Wendell
Thanks - it appears this is just a legitimate issue with the build, affecting all versions of Maven. On Fri, Jul 3, 2015 at 4:02 PM, Krishna Sankar wrote: > I have 3.3.3 > USS-Defiant:NW ksankar$ mvn -version > Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; > 2015-04-22T04:57:37-07

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Krishna Sankar
I have 3.3.3 USS-Defiant:NW ksankar$ mvn -version Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T04:57:37-07:00) Maven home: /usr/local/apache-maven-3.3.3 Java version: 1.7.0_60, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Conten

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Patrick Wendell
Hm - what if you do a fresh git checkout (just to make sure you don't have an older maven version downloaded). It also might be that this really is an issue even with Maven 3.3.3. I just am not sure why it's not reflected in our continuous integration or the build of the release packages themselves

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Robin East
I used the following build command: build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package this also gave the ‘Dependency-reduced POM’ loop Robin > On 3 Jul 2015, at 23:41, Patrick Wendell wrote: > > What if you use the built-in maven (i.e. build/mvn). It might be that

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Sean Owen
Sorry to say same happens on 3.3.3. I tried Shade 2.4 too. It is indeed MSHADE-148 that Andrew was trying to fix in the first place. I'm also trying to think of workarounds here. On Fri, Jul 3, 2015 at 11:41 PM, Patrick Wendell wrote: > What if you use the built-in maven (i.e. build/mvn). It migh

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Patrick Wendell
What if you use the built-in maven (i.e. build/mvn). It might be that we require a newer version of maven than you have. The release itself is built with maven 3.3.3: https://github.com/apache/spark/blob/master/build/mvn#L72 - Patrick On Fri, Jul 3, 2015 at 3:19 PM, Krishna Sankar wrote: > Yep,

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Krishna Sankar
Yep, happens to me as well. Build loops. Cheers On Fri, Jul 3, 2015 at 2:40 PM, Ted Yu wrote: > Patrick: > I used the following command: > ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean > package > > The build doesn't seem to stop. > Here is tail of build output: > >

Re: [VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Ted Yu
Patrick: I used the following command: ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package The build doesn't seem to stop. Here is tail of build output: [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml [INFO] Dependen

[VOTE] Release Apache Spark 1.4.1 (RC2)

2015-07-03 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=co

[RESULT] [VOTE] Release Apache Spark 1.4.1

2015-07-03 Thread Patrick Wendell
This vote is cancelled in favor of RC2. Thanks very much to Sean Owen for triaging an important bug associated with RC1. I took a look at the branch-1.4 contents and I think its safe to cut RC2 from the head of that branch (i.e no very high risk patches that I could see). JIRA management around th

Re: [VOTE] Release Apache Spark 1.4.1

2015-07-03 Thread Sean Owen
Great, thanks for the fix. Anything marked as fixed for 1.4.2 should now be marked as fixed for 1.4.1 right? I saw you were already updating many of those; OK to finish that? >From skimming them, it looks like mostly bug fixes and docs, which are pretty safe. A few things are kind of minor behavi

Re: [VOTE] Release Apache Spark 1.4.1

2015-07-02 Thread Andrew Or
@Sean I believe that is a real issue. I have submitted a patch to fix it: https://github.com/apache/spark/pull/7193. Unfortunately this would mean we need to cut a new RC to include it. When we do so we should also do another careful pass over the commits that are merged since the first RC. -1 20

Re: [VOTE] Release Apache Spark 1.4.1

2015-07-02 Thread Shivaram Venkataraman
+1 Tested the EC2 launch scripts and the Spark version and EC2 branch etc. look good. Shivaram On Thu, Jul 2, 2015 at 8:22 AM, Patrick Wendell wrote: > Hey Sean - yes I think that is an issue. Our published poms need to > have the dependency versions inlined. > > We probably need to revert that

Re: [VOTE] Release Apache Spark 1.4.1

2015-07-02 Thread Patrick Wendell
Hey Sean - yes I think that is an issue. Our published poms need to have the dependency versions inlined. We probably need to revert that bit of the build patch. - Patrick On Thu, Jul 2, 2015 at 7:21 AM, vaquar khan wrote: > +1 > > On 2 Jul 2015 18:03, "shenyan zhen" wrote: >> >> +1 >> >> On J

Re: [VOTE] Release Apache Spark 1.4.1

2015-07-02 Thread vaquar khan
+1 On 2 Jul 2015 18:03, "shenyan zhen" wrote: > +1 > On Jun 30, 2015 8:28 PM, "Reynold Xin" wrote: > >> +1 >> >> On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell >> wrote: >> >>> Please vote on releasing the following candidate as Apache Spark version >>> 1.4.1! >>> >>> This release fixes a ha

Re: [VOTE] Release Apache Spark 1.4.1

2015-07-02 Thread shenyan zhen
+1 On Jun 30, 2015 8:28 PM, "Reynold Xin" wrote: > +1 > > On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell > wrote: > >> Please vote on releasing the following candidate as Apache Spark version >> 1.4.1! >> >> This release fixes a handful of known issues in Spark 1.4.0, listed here: >> http://s

Re: [VOTE] Release Apache Spark 1.4.1

2015-07-02 Thread Sean Owen
I wanted to flag a potential blocker here, but pardon me if this is still after all this time just my misunderstanding of the POM/build theory -- So this is the final candiate release POM right? https://repository.apache.org/content/repositories/orgapachespark-1118/org/apache/spark/spark-core_2.10

Re: [VOTE] Release Apache Spark 1.4.1

2015-07-01 Thread Tathagata Das
+1 On Tue, Jun 30, 2015 at 8:12 PM, Bobby Chowdary wrote: > +1 Tested on CentOS 7 > On Jun 30, 2015 19:38, "Joseph Bradley" wrote: > >> +1 >> >> On Tue, Jun 30, 2015 at 5:27 PM, Reynold Xin wrote: >> >>> +1 >>> >>> On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell >>> wrote: >>> Please v

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-30 Thread Bobby Chowdary
+1 Tested on CentOS 7 On Jun 30, 2015 19:38, "Joseph Bradley" wrote: > +1 > > On Tue, Jun 30, 2015 at 5:27 PM, Reynold Xin wrote: > >> +1 >> >> On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell >> wrote: >> >>> Please vote on releasing the following candidate as Apache Spark version >>> 1.4.1!

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-30 Thread Joseph Bradley
+1 On Tue, Jun 30, 2015 at 5:27 PM, Reynold Xin wrote: > +1 > > On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell > wrote: > >> Please vote on releasing the following candidate as Apache Spark version >> 1.4.1! >> >> This release fixes a handful of known issues in Spark 1.4.0, listed here: >> h

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-30 Thread Reynold Xin
+1 On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http://s.apache.org/spark-1.4.1 > > The tag to be voted on is v1.4.1-rc1

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-30 Thread York, Brennon
+1 (non-binding) * built Spark on OSX10.10 w/java version 1.8.0_45 via `dev/run-tests` * all tests ran successfully * ran spark-on-YARN core ETL pipeline (success) * no regression issues / performance issues * ran spark-on-YARN MLLib ALS recommendation pipeline (success) * no regression issu

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Ted Yu
Andrew: I agree with your assessment. Cheers On Mon, Jun 29, 2015 at 3:33 PM, Andrew Or wrote: > Hi Ted, > > We haven't observed a StreamingContextSuite failure on our test > infrastructure recently. Given that we cannot reproduce it even locally it > is unlikely that this uncovers a real bug.

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Andrew Or
Hi Ted, We haven't observed a StreamingContextSuite failure on our test infrastructure recently. Given that we cannot reproduce it even locally it is unlikely that this uncovers a real bug. Even if it does I would not block the release on it because many in the community are waiting for a few impo

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Ted Yu
The test passes when run alone on my machine as well. Please run test suite. Thanks On Mon, Jun 29, 2015 at 2:01 PM, Tathagata Das wrote: > @Ted, I ran the following two commands. > > mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive -DskipTests clean > package > mvn -Phadoop-2.4 -Dhadoop.

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Tathagata Das
@Ted, I ran the following two commands. mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive -DskipTests clean package mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive -DwildcardSuites=org.apache.spark.streaming.StreamingContextSuite test Using Java version "1.7.0_51", the tests passed norm

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Krishna Sankar
+1 (non-binding, of course) 1. Compiled OSX 10.10 (Yosemite) OK Total time: 13:26 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK 2.3. Decision Tree, Naive Bayes OK 2.4. KMea

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Ted Yu
Here is the command I used: mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive package Java: 1.8.0_45 OS: Linux x.com 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Cheers On Mon, Jun 29, 2015 at 12:04 AM, Tathagata Das wrote: > @Ted, could you ela

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Yin Huai
+1. I tested those SQL blocker bugs in my laptop and they have been fixed. On Mon, Jun 29, 2015 at 6:51 AM, Sean Owen wrote: > +1 sigs, license, etc check out. > > All tests pass for me in the Hadoop 2.6 + Hive configuration on Ubuntu. > (I still get those pesky cosmetic UDF test failures in Jav

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Sean Owen
+1 sigs, license, etc check out. All tests pass for me in the Hadoop 2.6 + Hive configuration on Ubuntu. (I still get those pesky cosmetic UDF test failures in Java 8, but they are clearly just test issues.) I'll follow up on retargeting 1.4.1 issues afterwards as needed, but again feel free to m

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Tom Graves
+1. Tested on yarn on hadoop 2.6 cluster Tom On Monday, June 29, 2015 2:04 AM, Tathagata Das wrote: @Ted, could you elaborate more on what was the test command that you ran? What profiles, using SBT or Maven?  TD On Sun, Jun 28, 2015 at 12:21 PM, Patrick Wendell wrote: Hey Krishn

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-29 Thread Tathagata Das
@Ted, could you elaborate more on what was the test command that you ran? What profiles, using SBT or Maven? TD On Sun, Jun 28, 2015 at 12:21 PM, Patrick Wendell wrote: > Hey Krishna - this is still the current release candidate. > > - Patrick > > On Sun, Jun 28, 2015 at 12:14 PM, Krishna Sanka

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-28 Thread Patrick Wendell
Hey Krishna - this is still the current release candidate. - Patrick On Sun, Jun 28, 2015 at 12:14 PM, Krishna Sankar wrote: > Patrick, >Haven't seen any replies on test results. I will byte ;o) - Should I test > this version or is another one in the wings ? > Cheers > > > On Tue, Jun 23, 2

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-28 Thread Krishna Sankar
Patrick, Haven't seen any replies on test results. I will byte ;o) - Should I test this version or is another one in the wings ? Cheers On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This releas

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-26 Thread Ted Yu
Pardon. During earlier test run, I got: ^[[32mStreamingContextSuite:^[[0m ^[[32m- from no conf constructor^[[0m ^[[32m- from no conf + spark home^[[0m ^[[32m- from no conf + spark home + env^[[0m ^[[32m- from conf with settings^[[0m ^[[32m- from existing SparkContext^[[0m ^[[32m- from existing Spa

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-26 Thread Ted Yu
I got the following when running test suite: [INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null) ^[[0m[^[[0minfo^[[0m] ^[[0mCompiling 2 Scala sources and 1 Java source to /home/hbase/spark-1.4.1/streaming/target/scala-2.10/test-classes...^[[0m ^[[0m[^[[31merror^[[0m]

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-26 Thread Patrick Wendell
Hey Tom - no one voted on this yet, so I need to keep it open until people vote. But I'm not aware of specific things we are waiting for. Anyone else? - Patrick On Fri, Jun 26, 2015 at 7:10 AM, Tom Graves wrote: > So is this open for vote then or are we waiting on other things? > > Tom > > > > O

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-26 Thread Tom Graves
So is this open for vote then or are we waiting on other things? Tom On Thursday, June 25, 2015 10:32 AM, Andrew Ash wrote: I would guess that many tickets targeted at 1.4.1 were set that way during the tail end of the 1.4.0 voting process as people realized they wouldn't make the

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-25 Thread Andrew Ash
I would guess that many tickets targeted at 1.4.1 were set that way during the tail end of the 1.4.0 voting process as people realized they wouldn't make the .0 release in time. In that case, they were likely aiming for a 1.4.x release, not necessarily 1.4.1 specifically. Maybe creating a "1.4.x"

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-25 Thread Sean Owen
That makes sense to me -- there's an urgent fix to get out. I missed that part. Not that it really matters but was that expressed elsewhere? I know we tend to start the RC process even when a few more changes are still in progress, to get a first wave or two of testing done early, knowing that the

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-24 Thread Josh Rosen
At least a couple of those issues are mistargeted; some of the flaky test JIRAs + test improvement tasks should probably be targeted for 1.5.0 instead. On Wed, Jun 24, 2015 at 8:56 AM, Patrick Wendell wrote: > Hey Sean, > > This is being shipped now because there is a severe bug in 1.4.0 that >

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-24 Thread Patrick Wendell
Hey Sean, This is being shipped now because there is a severe bug in 1.4.0 that can cause data corruption for Parquet users. There are no blockers targeted for 1.4.1 - so I don't see that JIRA is inconsistent with shipping a release now. The goal of having every single targeted JIRA cleared by th

Re: [VOTE] Release Apache Spark 1.4.1

2015-06-24 Thread Sean Owen
There are 44 issues still targeted for 1.4.1. None are Blockers; 12 are Critical. ~80% were opened and/or set by committers. Compare with 90 issues resolved for 1.4.1. I'm concerned that committers are targeting lots more for a release even in the short term than realistically can go in. On its fa

[VOTE] Release Apache Spark 1.4.1

2015-06-23 Thread Patrick Wendell
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=co