[RESULT] [VOTE] Release Apache Spark 1.4.1 (RC4)
This vote passes with 14 +1 (7 binding) votes and no 0 or -1 votes. +1 (14): Patrick Wendell Reynold Xin Sean Owen Burak Yavuz Mark Hamstra Michael Armbrust Andrew Or York, Brennon Krishna Sankar Luciano Resende Holden Karau Tom Graves Denny Lee Sean McNamara - Patrick On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
I think we can close this vote soon. Any addition votes/testing would be much appreciated! On Fri, Jul 10, 2015 at 11:30 AM, Sean McNamara sean.mcnam...@webtrends.com wrote: +1 Sean On Jul 8, 2015, at 11:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 Sean On Jul 8, 2015, at 11:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 Tom On Thursday, July 9, 2015 12:55 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 nonbinding. All previous RC issues appear resolved. All tests pass with the -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver invocation. Signatures et al are OK. On Thu, Jul 9, 2015 at 6:55 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 On Thu, Jul 9, 2015 at 10:07 AM, Mark Hamstra m...@clearstorydata.com wrote: +1 On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 2015-07-09 10:26 GMT-07:00 Michael Armbrust mich...@databricks.com: +1 On Thu, Jul 9, 2015 at 10:07 AM, Mark Hamstra m...@clearstorydata.com wrote: +1 On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 (non-binding) * ran spark-on-YARN MLLib ALS recommendation pipeline (success) * no regression / performance issues * ran spark-on-YARN GraphX pipeline (success) * no regression / performance issues On 7/8/15, 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 On Wed, Jul 8, 2015 at 11:58 PM, Patrick Wendell pwend...@gmail.com wrote: +1 On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 (non-binding) mostly looking in the legal aspects of the release. On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 - compiled on ubuntu centos, spark-perf run against yarn in client mode on a small cluster comparing 1.4.0 1.4.1 (for core) doesn't have any huge jumps (albeit with a small scaling factor). On Wed, Jul 8, 2015 at 11:58 PM, Patrick Wendell pwend...@gmail.com wrote: +1 On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau Linked In: https://www.linkedin.com/in/holdenkarau
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 1. Compiled OSX 10.10 (Yosemite) OK Total time: 38:11 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK 2.3. Decision Tree, Naive Bayes OK 2.4. KMeans OK Center And Scale OK 2.5. RDD operations OK State of the Union Texts - MapReduce, Filter,sortByKey (word count) 2.6. Recommendation (Movielens medium dataset ~1 M ratings) OK Model evaluation/optimization (rank, numIter, lambda) with itertools OK 3. Scala - MLlib 3.1. statistics (min,max,mean,Pearson,Spearman) OK 3.2. LinearRegressionWithSGD OK 3.3. Decision Tree OK 3.4. KMeans OK 3.5. Recommendation (Movielens medium dataset ~1 M ratings) OK 3.6. saveAsParquetFile OK 3.7. Read and verify the 4.3 save(above) - sqlContext.parquetFile, registerTempTable, sql OK 3.8. result = sqlContext.sql(SELECT OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders INNER JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID) OK 4.0. Spark SQL from Python OK 4.1. result = sqlContext.sql(SELECT * from people WHERE State = 'WA') OK 5.0. Packages 5.1. com.databricks.spark.csv - read/write OK 6.0. DataFrames 6.1. cast,dtypes OK 6.2. groupBy,avg,crosstab,corr,isNull,na.drop OK 6.3. joins,sql,set operations,udf OK Cheers k/ On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC4)
+1 nonbinding. On Thu, Jul 9, 2015 at 7:38 AM, Sean Owen so...@cloudera.com wrote: +1 nonbinding. All previous RC issues appear resolved. All tests pass with the -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver invocation. Signatures et al are OK. On Thu, Jul 9, 2015 at 6:55 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with -Pyarn -Phadoop-2.6, with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Here's one thing I ran into: The SparkR documentation example in http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is incorrect. sc - sparkR.init(packages=com.databricks:spark-csv_2.11:1.0.3) should be sc - sparkR.init(sparkPackages=com.databricks:spark-csv_2.11:1.0.3) Thanks Pradeep On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen so...@cloudera.com wrote: The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with -Pyarn -Phadoop-2.6, with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
I see, but shouldn't this test not be run when Hive isn't in the build? On Wed, Jul 8, 2015 at 7:13 PM, Andrew Or and...@databricks.com wrote: @Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and `-Phive-thriftserver`. The MissingRequirementsError is just complaining that it can't find the right classes. The other one (DataFrameStatSuite) is a little more concerning. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Hi Pradeep Thanks for the catch -- Lets open a JIRA and PR for it. I don't think documentation changes affect the release though Patrick can confirm that. Thanks Shivaram On Wed, Jul 8, 2015 at 9:35 AM, Pradeep Bashyal prad...@bashyal.com wrote: Here's one thing I ran into: The SparkR documentation example in http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is incorrect. sc - sparkR.init(packages=com.databricks:spark-csv_2.11:1.0.3) should be sc - sparkR.init(sparkPackages=com.databricks:spark-csv_2.11:1.0.3) Thanks Pradeep On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen so...@cloudera.com wrote: The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with -Pyarn -Phadoop-2.6, with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
HiveSparkSubmitSuite is fine for me, but I do see the same issue with DataFrameStatSuite -- OSX 10.10.4, java 1.7.0_75, -Phive -Phive-thriftserver -Phadoop-2.4 -Pyarn On Wed, Jul 8, 2015 at 4:18 AM, Sean Owen so...@cloudera.com wrote: The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with -Pyarn -Phadoop-2.6, with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Although that should be fixed if it's incorrect, it's not something that would nearly block a release. The question here is whether this artifact can be released as 1.4.1, or whether it has a blocking regression from 1.4.0. On Wed, Jul 8, 2015 at 5:35 PM, Pradeep Bashyal prad...@bashyal.com wrote: Here's one thing I ran into: The SparkR documentation example in http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is incorrect. sc - sparkR.init(packages=com.databricks:spark-csv_2.11:1.0.3) should be sc - sparkR.init(sparkPackages=com.databricks:spark-csv_2.11:1.0.3) Thanks Pradeep On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen so...@cloudera.com wrote: The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with -Pyarn -Phadoop-2.6, with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Yeah - we can fix the docs separately from the release. - Patrick On Wed, Jul 8, 2015 at 10:03 AM, Mark Hamstra m...@clearstorydata.com wrote: HiveSparkSubmitSuite is fine for me, but I do see the same issue with DataFrameStatSuite -- OSX 10.10.4, java 1.7.0_75, -Phive -Phive-thriftserver -Phadoop-2.4 -Pyarn On Wed, Jul 8, 2015 at 4:18 AM, Sean Owen so...@cloudera.com wrote: The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with -Pyarn -Phadoop-2.6, with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Hi Shivaram, I created a Jira Issue for the documentation error. https://issues.apache.org/jira/browse/SPARK-8901 Thanks Pradeep On Wed, Jul 8, 2015 at 11:40 AM, Shivaram Venkataraman shiva...@eecs.berkeley.edu wrote: Hi Pradeep Thanks for the catch -- Lets open a JIRA and PR for it. I don't think documentation changes affect the release though Patrick can confirm that. Thanks Shivaram On Wed, Jul 8, 2015 at 9:35 AM, Pradeep Bashyal prad...@bashyal.com wrote: Here's one thing I ran into: The SparkR documentation example in http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is incorrect. sc - sparkR.init(packages=com.databricks:spark-csv_2.11:1.0.3) should be sc - sparkR.init(sparkPackages=com.databricks:spark-csv_2.11:1.0.3) Thanks Pradeep On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen so...@cloudera.com wrote: The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with -Pyarn -Phadoop-2.6, with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
@Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and `-Phive-thriftserver`. The MissingRequirementsError is just complaining that it can't find the right classes. The other one (DataFrameStatSuite) is a little more concerning. 2015-07-08 10:43 GMT-07:00 Pradeep Bashyal prad...@bashyal.com: Hi Shivaram, I created a Jira Issue for the documentation error. https://issues.apache.org/jira/browse/SPARK-8901 Thanks Pradeep On Wed, Jul 8, 2015 at 11:40 AM, Shivaram Venkataraman shiva...@eecs.berkeley.edu wrote: Hi Pradeep Thanks for the catch -- Lets open a JIRA and PR for it. I don't think documentation changes affect the release though Patrick can confirm that. Thanks Shivaram On Wed, Jul 8, 2015 at 9:35 AM, Pradeep Bashyal prad...@bashyal.com wrote: Here's one thing I ran into: The SparkR documentation example in http://people.apache.org/~pwendell/spark-releases/latest/sparkr.html is incorrect. sc - sparkR.init(packages=com.databricks:spark-csv_2.11:1.0.3) should be sc - sparkR.init(sparkPackages=com.databricks:spark-csv_2.11:1.0.3) Thanks Pradeep On Wed, Jul 8, 2015 at 6:18 AM, Sean Owen so...@cloudera.com wrote: The POM issue is resolved and the build succeeds. The license and sigs still work. The tests pass for me with -Pyarn -Phadoop-2.6, with the following two exceptions. Is anyone else seeing these? this is consistent on Ubuntu 14 with Java 7/8: DataFrameStatSuite: ... - special crosstab elements (., '', null, ``) *** FAILED *** java.lang.NullPointerException: at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:131) at org.apache.spark.sql.execution.stat.StatFunctions$$anonfun$4.apply(StatFunctions.scala:121) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Map$Map4.foreach(Map.scala:181) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.execution.stat.StatFunctions$.crossTabulate(StatFunctions.scala:121) at org.apache.spark.sql.DataFrameStatFunctions.crosstab(DataFrameStatFunctions.scala:94) at org.apache.spark.sql.DataFrameStatSuite$$anonfun$5.apply$mcV$sp(DataFrameStatSuite.scala:97) ... HiveSparkSubmitSuite: - SPARK-8368: includes jars passed in through --jars *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8020: set sql conf in spark conf *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) - SPARK-8489: MissingRequirementError during reflection *** FAILED *** Process returned with exit code 1. See the log4j logs for more detail. (HiveSparkSubmitSuite.scala:92) On Tue, Jul 7, 2015 at 8:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
Hey All, The issue that Josh pointed out is not just a test failure, it's an issue with an important bug fix that was not correctly back-ported into the 1.4 branch. Unfortunately the overall state of the 1.4 branch tests on Jenkins was not in great shape so this was missed earlier on. Given that this is fixed now, I have prepared another RC and am leaning towards restarting the vote. If anyone feels strongly one way or the other let me know, otherwise I'll restart it in a few hours. I figured since this will likely finalize over the weekend anyways, it's not so bad to wait 1 additional day in order to get that fix. - Patrick On Wed, Jul 8, 2015 at 12:00 PM, Josh Rosen rosenvi...@gmail.com wrote: I've filed https://issues.apache.org/jira/browse/SPARK-8903 to fix the DataFrameStatSuite test failure. The problem turned out to be caused by a mistake made while resolving a merge-conflict when backporting that patch to branch-1.4. I've submitted https://github.com/apache/spark/pull/7295 to fix this issue. On Wed, Jul 8, 2015 at 11:30 AM, Sean Owen so...@cloudera.com wrote: I see, but shouldn't this test not be run when Hive isn't in the build? On Wed, Jul 8, 2015 at 7:13 PM, Andrew Or and...@databricks.com wrote: @Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and `-Phive-thriftserver`. The MissingRequirementsError is just complaining that it can't find the right classes. The other one (DataFrameStatSuite) is a little more concerning. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
[VOTE] Release Apache Spark 1.4.1 (RC4)
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc4 (commit dbaa5c2): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= dbaa5c294eb565f84d7032e387e4b8c1a56e4cd2 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1125/ [published as version: 1.4.1-rc4] https://repository.apache.org/content/repositories/orgapachespark-1126/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc4-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Sunday, July 12, at 06:55 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
[RESULT] [VOTE] Release Apache Spark 1.4.1 (RC3)
This vote is cancelled in favor of RC4. - Patrick On Tue, Jul 7, 2015 at 12:06 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
I've filed https://issues.apache.org/jira/browse/SPARK-8903 to fix the DataFrameStatSuite test failure. The problem turned out to be caused by a mistake made while resolving a merge-conflict when backporting that patch to branch-1.4. I've submitted https://github.com/apache/spark/pull/7295 to fix this issue. On Wed, Jul 8, 2015 at 11:30 AM, Sean Owen so...@cloudera.com wrote: I see, but shouldn't this test not be run when Hive isn't in the build? On Wed, Jul 8, 2015 at 7:13 PM, Andrew Or and...@databricks.com wrote: @Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and `-Phive-thriftserver`. The MissingRequirementsError is just complaining that it can't find the right classes. The other one (DataFrameStatSuite) is a little more concerning. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC3)
+1 Verified that the previous blockers SPARK-8781 and SPARK-8819 are now resolved. 2015-07-07 12:06 GMT-07:00 Patrick Wendell pwend...@gmail.com: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc3 (commit 3e8ae38): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 3e8ae38944f13895daf328555c1ad22cd590b089 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1123/ [published as version: 1.4.1-rc3] https://repository.apache.org/content/repositories/orgapachespark-1124/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc3-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Friday, July 10, at 20:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
[RESULT] [VOTE] Release Apache Spark 1.4.1 (RC2)
Hey All, This vote is cancelled in favor of RC3. - Patrick On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC2)
Hm - what if you do a fresh git checkout (just to make sure you don't have an older maven version downloaded). It also might be that this really is an issue even with Maven 3.3.3. I just am not sure why it's not reflected in our continuous integration or the build of the release packages themselves: https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/ It could be that it's dependent on which modules are enabled. On Fri, Jul 3, 2015 at 3:46 PM, Robin East robin.e...@xense.co.uk wrote: which got me thinking: build/mvn -version Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c; 2015-03-13T20:10:27+00:00) Maven home: /usr/local/Cellar/maven/3.3.1/libexec Java version: 1.8.0_40, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: mac os x, version: 10.10.2, arch: x86_64, family: “mac Seems to be using 3.3.1 On 3 Jul 2015, at 23:44, Robin East robin.e...@xense.co.uk wrote: I used the following build command: build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package this also gave the ‘Dependency-reduced POM’ loop Robin On 3 Jul 2015, at 23:41, Patrick Wendell pwend...@gmail.com wrote: What if you use the built-in maven (i.e. build/mvn). It might be that we require a newer version of maven than you have. The release itself is built with maven 3.3.3: https://github.com/apache/spark/blob/master/build/mvn#L72 - Patrick On Fri, Jul 3, 2015 at 3:19 PM, Krishna Sankar ksanka...@gmail.com wrote: Yep, happens to me as well. Build loops. Cheers k/ On Fri, Jul 3, 2015 at 2:40 PM, Ted Yu yuzhih...@gmail.com wrote: Patrick: I used the following command: ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package The build doesn't seem to stop. Here is tail of build output: [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml Here is part of the stack trace for the build process: http://pastebin.com/xL2Y0QMU FYI On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC2)
I have 3.3.3 USS-Defiant:NW ksankar$ mvn -version Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T04:57:37-07:00) Maven home: /usr/local/apache-maven-3.3.3 Java version: 1.7.0_60, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: mac os x, version: 10.10.3, arch: x86_64, family: mac Let me nuke it and reinstall maven. Cheers k/ On Fri, Jul 3, 2015 at 3:41 PM, Patrick Wendell pwend...@gmail.com wrote: What if you use the built-in maven (i.e. build/mvn). It might be that we require a newer version of maven than you have. The release itself is built with maven 3.3.3: https://github.com/apache/spark/blob/master/build/mvn#L72 - Patrick On Fri, Jul 3, 2015 at 3:19 PM, Krishna Sankar ksanka...@gmail.com wrote: Yep, happens to me as well. Build loops. Cheers k/ On Fri, Jul 3, 2015 at 2:40 PM, Ted Yu yuzhih...@gmail.com wrote: Patrick: I used the following command: ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package The build doesn't seem to stop. Here is tail of build output: [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml Here is part of the stack trace for the build process: http://pastebin.com/xL2Y0QMU FYI On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC2)
Let's continue the disucssion on the other thread relating to the master build. On Fri, Jul 3, 2015 at 4:13 PM, Patrick Wendell pwend...@gmail.com wrote: Thanks - it appears this is just a legitimate issue with the build, affecting all versions of Maven. On Fri, Jul 3, 2015 at 4:02 PM, Krishna Sankar ksanka...@gmail.com wrote: I have 3.3.3 USS-Defiant:NW ksankar$ mvn -version Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T04:57:37-07:00) Maven home: /usr/local/apache-maven-3.3.3 Java version: 1.7.0_60, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: mac os x, version: 10.10.3, arch: x86_64, family: mac Let me nuke it and reinstall maven. Cheers k/ On Fri, Jul 3, 2015 at 3:41 PM, Patrick Wendell pwend...@gmail.com wrote: What if you use the built-in maven (i.e. build/mvn). It might be that we require a newer version of maven than you have. The release itself is built with maven 3.3.3: https://github.com/apache/spark/blob/master/build/mvn#L72 - Patrick On Fri, Jul 3, 2015 at 3:19 PM, Krishna Sankar ksanka...@gmail.com wrote: Yep, happens to me as well. Build loops. Cheers k/ On Fri, Jul 3, 2015 at 2:40 PM, Ted Yu yuzhih...@gmail.com wrote: Patrick: I used the following command: ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package The build doesn't seem to stop. Here is tail of build output: [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml Here is part of the stack trace for the build process: http://pastebin.com/xL2Y0QMU FYI On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC2)
Sorry to say same happens on 3.3.3. I tried Shade 2.4 too. It is indeed MSHADE-148 that Andrew was trying to fix in the first place. I'm also trying to think of workarounds here. On Fri, Jul 3, 2015 at 11:41 PM, Patrick Wendell pwend...@gmail.com wrote: What if you use the built-in maven (i.e. build/mvn). It might be that we require a newer version of maven than you have. The release itself is built with maven 3.3.3: https://github.com/apache/spark/blob/master/build/mvn#L72 - Patrick On Fri, Jul 3, 2015 at 3:19 PM, Krishna Sankar ksanka...@gmail.com wrote: Yep, happens to me as well. Build loops. Cheers k/ On Fri, Jul 3, 2015 at 2:40 PM, Ted Yu yuzhih...@gmail.com wrote: Patrick: I used the following command: ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package The build doesn't seem to stop. Here is tail of build output: [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml Here is part of the stack trace for the build process: http://pastebin.com/xL2Y0QMU FYI On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC2)
I used the following build command: build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package this also gave the ‘Dependency-reduced POM’ loop Robin On 3 Jul 2015, at 23:41, Patrick Wendell pwend...@gmail.com wrote: What if you use the built-in maven (i.e. build/mvn). It might be that we require a newer version of maven than you have. The release itself is built with maven 3.3.3: https://github.com/apache/spark/blob/master/build/mvn#L72 - Patrick On Fri, Jul 3, 2015 at 3:19 PM, Krishna Sankar ksanka...@gmail.com wrote: Yep, happens to me as well. Build loops. Cheers k/ On Fri, Jul 3, 2015 at 2:40 PM, Ted Yu yuzhih...@gmail.com wrote: Patrick: I used the following command: ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package The build doesn't seem to stop. Here is tail of build output: [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml Here is part of the stack trace for the build process: http://pastebin.com/xL2Y0QMU FYI On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC2)
Yep, happens to me as well. Build loops. Cheers k/ On Fri, Jul 3, 2015 at 2:40 PM, Ted Yu yuzhih...@gmail.com wrote: Patrick: I used the following command: ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package The build doesn't seem to stop. Here is tail of build output: [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml Here is part of the stack trace for the build process: http://pastebin.com/xL2Y0QMU FYI On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC2)
Thanks - it appears this is just a legitimate issue with the build, affecting all versions of Maven. On Fri, Jul 3, 2015 at 4:02 PM, Krishna Sankar ksanka...@gmail.com wrote: I have 3.3.3 USS-Defiant:NW ksankar$ mvn -version Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T04:57:37-07:00) Maven home: /usr/local/apache-maven-3.3.3 Java version: 1.7.0_60, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: mac os x, version: 10.10.3, arch: x86_64, family: mac Let me nuke it and reinstall maven. Cheers k/ On Fri, Jul 3, 2015 at 3:41 PM, Patrick Wendell pwend...@gmail.com wrote: What if you use the built-in maven (i.e. build/mvn). It might be that we require a newer version of maven than you have. The release itself is built with maven 3.3.3: https://github.com/apache/spark/blob/master/build/mvn#L72 - Patrick On Fri, Jul 3, 2015 at 3:19 PM, Krishna Sankar ksanka...@gmail.com wrote: Yep, happens to me as well. Build loops. Cheers k/ On Fri, Jul 3, 2015 at 2:40 PM, Ted Yu yuzhih...@gmail.com wrote: Patrick: I used the following command: ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package The build doesn't seem to stop. Here is tail of build output: [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml Here is part of the stack trace for the build process: http://pastebin.com/xL2Y0QMU FYI On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
[RESULT] [VOTE] Release Apache Spark 1.4.1
This vote is cancelled in favor of RC2. Thanks very much to Sean Owen for triaging an important bug associated with RC1. I took a look at the branch-1.4 contents and I think its safe to cut RC2 from the head of that branch (i.e no very high risk patches that I could see). JIRA management around the time of the RC voting is an interesting topic, Sean I like your most recent proposal. Maybe we can put that on the wiki or start a DISCUSS thread to cover that topic. On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
[VOTE] Release Apache Spark 1.4.1 (RC2)
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1 (RC2)
Patrick: I used the following command: ~/apache-maven-3.3.1/bin/mvn -DskipTests -Phadoop-2.4 -Pyarn -Phive clean package The build doesn't seem to stop. Here is tail of build output: [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml [INFO] Dependency-reduced POM written at: /home/hbase/spark-1.4.1/bagel/dependency-reduced-pom.xml Here is part of the stack trace for the build process: http://pastebin.com/xL2Y0QMU FYI On Fri, Jul 3, 2015 at 1:15 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc2 (commit 07b95c7): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 07b95c7adf88f0662b7ab1c47e302ff5e6859606 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1120/ [published as version: 1.4.1-rc2] https://repository.apache.org/content/repositories/orgapachespark-1121/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc2-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Monday, July 06, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
Great, thanks for the fix. Anything marked as fixed for 1.4.2 should now be marked as fixed for 1.4.1 right? I saw you were already updating many of those; OK to finish that? From skimming them, it looks like mostly bug fixes and docs, which are pretty safe. A few things are kind of minor behavior changes like https://issues.apache.org/jira/browse/SPARK-8630 Still probably not wrong to include. Ideally these would not be merged in branch 1.4 while the RC process is in progress, but then, that bottlenecks things. Once the RC for 1.4.1 is cut, should be mark everything else merged into branch 1.4 as fixed for 1.4.2? and then if a new RC is cut, mark them as fixed for 1.4.1 instead? that's a nice easy convention. Anything that might be mergeable for a later 1.4.x release but shouldn't go into 1.4.1 could be left out of branch 1.4 and marked as backport-needed. That should be a rare occasion for late in the RC process. Obviously, the faster the RC process goes the smaller this issue is - the smaller and more disciplined the list of target issues is when the process is, the better. On Thu, Jul 2, 2015 at 7:11 PM, Andrew Or and...@databricks.com wrote: @Sean I believe that is a real issue. I have submitted a patch to fix it: https://github.com/apache/spark/pull/7193. Unfortunately this would mean we need to cut a new RC to include it. When we do so we should also do another careful pass over the commits that are merged since the first RC. -1 2015-07-02 9:10 GMT-07:00 Shivaram Venkataraman shiva...@eecs.berkeley.edu: +1 Tested the EC2 launch scripts and the Spark version and EC2 branch etc. look good. Shivaram On Thu, Jul 2, 2015 at 8:22 AM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean - yes I think that is an issue. Our published poms need to have the dependency versions inlined. We probably need to revert that bit of the build patch. - Patrick On Thu, Jul 2, 2015 at 7:21 AM, vaquar khan vaquar.k...@gmail.com wrote: +1 On 2 Jul 2015 18:03, shenyan zhen shenya...@gmail.com wrote: +1 On Jun 30, 2015 8:28 PM, Reynold Xin r...@databricks.com wrote: +1 On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
I wanted to flag a potential blocker here, but pardon me if this is
still after all this time just my misunderstanding of the POM/build
theory --
So this is the final candiate release POM right?
https://repository.apache.org/content/repositories/orgapachespark-1118/org/apache/spark/spark-core_2.10/1.4.1/spark-core_2.10-1.4.1.pom
Compare to for example:
https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/1.4.0/spark-core_2.10-1.4.0.pom
and see:
https://issues.apache.org/jira/browse/SPARK-8781
For instance, in 1.4.0 it had
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-launcher_2.10/artifactId
version1.4.0/version
scopecompile/scope
/dependency
but now that's:
dependency
groupIdorg.apache.spark/groupId
artifactIdspark-launcher_${scala.binary.version}/artifactId
version${project.version}/version
/dependency
JIRA suggests it had to do with adding:
createDependencyReducedPomfalse/createDependencyReducedPom
Am I missing something or is that indeed not going to work as a release POM?
On Wed, Jun 24, 2015 at 6:37 AM, Patrick Wendell pwend...@gmail.com wrote:
Please vote on releasing the following candidate as Apache Spark version
1.4.1!
This release fixes a handful of known issues in Spark 1.4.0, listed here:
http://s.apache.org/spark-1.4.1
The tag to be voted on is v1.4.1-rc1 (commit 60e08e5):
https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=
60e08e50751fe3929156de956d62faea79f5b801
The release files, including signatures, digests, etc. can be found at:
http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/
Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/pwendell.asc
The staging repository for this release can be found at:
[published as version: 1.4.1]
https://repository.apache.org/content/repositories/orgapachespark-1118/
[published as version: 1.4.1-rc1]
https://repository.apache.org/content/repositories/orgapachespark-1119/
The documentation corresponding to this release can be found at:
http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/
Please vote on releasing this package as Apache Spark 1.4.1!
The vote is open until Saturday, June 27, at 06:32 UTC and passes
if a majority of at least 3 +1 PMC votes are cast.
[ ] +1 Release this package as Apache Spark 1.4.1
[ ] -1 Do not release this package because ...
To learn more about Apache Spark, please see
http://spark.apache.org/
-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
+1 On Jun 30, 2015 8:28 PM, Reynold Xin r...@databricks.com wrote: +1 On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
+1 On 2 Jul 2015 18:03, shenyan zhen shenya...@gmail.com wrote: +1 On Jun 30, 2015 8:28 PM, Reynold Xin r...@databricks.com wrote: +1 On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
+1 Tested the EC2 launch scripts and the Spark version and EC2 branch etc. look good. Shivaram On Thu, Jul 2, 2015 at 8:22 AM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean - yes I think that is an issue. Our published poms need to have the dependency versions inlined. We probably need to revert that bit of the build patch. - Patrick On Thu, Jul 2, 2015 at 7:21 AM, vaquar khan vaquar.k...@gmail.com wrote: +1 On 2 Jul 2015 18:03, shenyan zhen shenya...@gmail.com wrote: +1 On Jun 30, 2015 8:28 PM, Reynold Xin r...@databricks.com wrote: +1 On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
@Sean I believe that is a real issue. I have submitted a patch to fix it: https://github.com/apache/spark/pull/7193. Unfortunately this would mean we need to cut a new RC to include it. When we do so we should also do another careful pass over the commits that are merged since the first RC. -1 2015-07-02 9:10 GMT-07:00 Shivaram Venkataraman shiva...@eecs.berkeley.edu : +1 Tested the EC2 launch scripts and the Spark version and EC2 branch etc. look good. Shivaram On Thu, Jul 2, 2015 at 8:22 AM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean - yes I think that is an issue. Our published poms need to have the dependency versions inlined. We probably need to revert that bit of the build patch. - Patrick On Thu, Jul 2, 2015 at 7:21 AM, vaquar khan vaquar.k...@gmail.com wrote: +1 On 2 Jul 2015 18:03, shenyan zhen shenya...@gmail.com wrote: +1 On Jun 30, 2015 8:28 PM, Reynold Xin r...@databricks.com wrote: +1 On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
+1 On Tue, Jun 30, 2015 at 5:27 PM, Reynold Xin r...@databricks.com wrote: +1 On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
+1 On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
+1 sigs, license, etc check out. All tests pass for me in the Hadoop 2.6 + Hive configuration on Ubuntu. (I still get those pesky cosmetic UDF test failures in Java 8, but they are clearly just test issues.) I'll follow up on retargeting 1.4.1 issues afterwards as needed, but again feel free to move those you're sure won't be in this release. On Wed, Jun 24, 2015 at 6:37 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
+1. Tested on yarn on hadoop 2.6 cluster Tom On Monday, June 29, 2015 2:04 AM, Tathagata Das tathagata.das1...@gmail.com wrote: @Ted, could you elaborate more on what was the test command that you ran? What profiles, using SBT or Maven? TD On Sun, Jun 28, 2015 at 12:21 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Krishna - this is still the current release candidate. - Patrick On Sun, Jun 28, 2015 at 12:14 PM, Krishna Sankar ksanka...@gmail.com wrote: Patrick, Haven't seen any replies on test results. I will byte ;o) - Should I test this version or is another one in the wings ? Cheers k/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
+1. I tested those SQL blocker bugs in my laptop and they have been fixed. On Mon, Jun 29, 2015 at 6:51 AM, Sean Owen so...@cloudera.com wrote: +1 sigs, license, etc check out. All tests pass for me in the Hadoop 2.6 + Hive configuration on Ubuntu. (I still get those pesky cosmetic UDF test failures in Java 8, but they are clearly just test issues.) I'll follow up on retargeting 1.4.1 issues afterwards as needed, but again feel free to move those you're sure won't be in this release. On Wed, Jun 24, 2015 at 6:37 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
Here is the command I used: mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive package Java: 1.8.0_45 OS: Linux x.com 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Cheers On Mon, Jun 29, 2015 at 12:04 AM, Tathagata Das tathagata.das1...@gmail.com wrote: @Ted, could you elaborate more on what was the test command that you ran? What profiles, using SBT or Maven? TD On Sun, Jun 28, 2015 at 12:21 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Krishna - this is still the current release candidate. - Patrick On Sun, Jun 28, 2015 at 12:14 PM, Krishna Sankar ksanka...@gmail.com wrote: Patrick, Haven't seen any replies on test results. I will byte ;o) - Should I test this version or is another one in the wings ? Cheers k/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
The test passes when run alone on my machine as well. Please run test suite. Thanks On Mon, Jun 29, 2015 at 2:01 PM, Tathagata Das tathagata.das1...@gmail.com wrote: @Ted, I ran the following two commands. mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive -DskipTests clean package mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive -DwildcardSuites=org.apache.spark.streaming.StreamingContextSuite test Using Java version 1.7.0_51, the tests passed normally. On Mon, Jun 29, 2015 at 1:05 PM, Krishna Sankar ksanka...@gmail.com wrote: +1 (non-binding, of course) 1. Compiled OSX 10.10 (Yosemite) OK Total time: 13:26 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK 2.3. Decision Tree, Naive Bayes OK 2.4. KMeans OK Center And Scale OK 2.5. RDD operations OK State of the Union Texts - MapReduce, Filter,sortByKey (word count) 2.6. Recommendation (Movielens medium dataset ~1 M ratings) OK Model evaluation/optimization (rank, numIter, lambda) with itertools OK 3. Scala - MLlib 3.1. statistics (min,max,mean,Pearson,Spearman) OK 3.2. LinearRegressionWithSGD OK 3.3. Decision Tree OK 3.4. KMeans OK 3.5. Recommendation (Movielens medium dataset ~1 M ratings) OK 3.6. saveAsParquetFile OK 3.7. Read and verify the 4.3 save(above) - sqlContext.parquetFile, registerTempTable, sql OK 3.8. result = sqlContext.sql(SELECT OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders INNER JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID) OK 4.0. Spark SQL from Python OK 4.1. result = sqlContext.sql(SELECT * from people WHERE State = 'WA') OK 5.0. Packages 5.1. com.databricks.spark.csv - read/write OK Cheers k/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
Hi Ted, We haven't observed a StreamingContextSuite failure on our test infrastructure recently. Given that we cannot reproduce it even locally it is unlikely that this uncovers a real bug. Even if it does I would not block the release on it because many in the community are waiting for a few important fixes. In general, there will always be outstanding issues in Spark that we cannot address in every release. -Andrew 2015-06-29 14:29 GMT-07:00 Ted Yu yuzhih...@gmail.com: The test passes when run alone on my machine as well. Please run test suite. Thanks On Mon, Jun 29, 2015 at 2:01 PM, Tathagata Das tathagata.das1...@gmail.com wrote: @Ted, I ran the following two commands. mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive -DskipTests clean package mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive -DwildcardSuites=org.apache.spark.streaming.StreamingContextSuite test Using Java version 1.7.0_51, the tests passed normally. On Mon, Jun 29, 2015 at 1:05 PM, Krishna Sankar ksanka...@gmail.com wrote: +1 (non-binding, of course) 1. Compiled OSX 10.10 (Yosemite) OK Total time: 13:26 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK 2.3. Decision Tree, Naive Bayes OK 2.4. KMeans OK Center And Scale OK 2.5. RDD operations OK State of the Union Texts - MapReduce, Filter,sortByKey (word count) 2.6. Recommendation (Movielens medium dataset ~1 M ratings) OK Model evaluation/optimization (rank, numIter, lambda) with itertools OK 3. Scala - MLlib 3.1. statistics (min,max,mean,Pearson,Spearman) OK 3.2. LinearRegressionWithSGD OK 3.3. Decision Tree OK 3.4. KMeans OK 3.5. Recommendation (Movielens medium dataset ~1 M ratings) OK 3.6. saveAsParquetFile OK 3.7. Read and verify the 4.3 save(above) - sqlContext.parquetFile, registerTempTable, sql OK 3.8. result = sqlContext.sql(SELECT OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders INNER JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID) OK 4.0. Spark SQL from Python OK 4.1. result = sqlContext.sql(SELECT * from people WHERE State = 'WA') OK 5.0. Packages 5.1. com.databricks.spark.csv - read/write OK Cheers k/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
+1 (non-binding, of course) 1. Compiled OSX 10.10 (Yosemite) OK Total time: 13:26 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK 2.3. Decision Tree, Naive Bayes OK 2.4. KMeans OK Center And Scale OK 2.5. RDD operations OK State of the Union Texts - MapReduce, Filter,sortByKey (word count) 2.6. Recommendation (Movielens medium dataset ~1 M ratings) OK Model evaluation/optimization (rank, numIter, lambda) with itertools OK 3. Scala - MLlib 3.1. statistics (min,max,mean,Pearson,Spearman) OK 3.2. LinearRegressionWithSGD OK 3.3. Decision Tree OK 3.4. KMeans OK 3.5. Recommendation (Movielens medium dataset ~1 M ratings) OK 3.6. saveAsParquetFile OK 3.7. Read and verify the 4.3 save(above) - sqlContext.parquetFile, registerTempTable, sql OK 3.8. result = sqlContext.sql(SELECT OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders INNER JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID) OK 4.0. Spark SQL from Python OK 4.1. result = sqlContext.sql(SELECT * from people WHERE State = 'WA') OK 5.0. Packages 5.1. com.databricks.spark.csv - read/write OK Cheers k/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
Andrew: I agree with your assessment. Cheers On Mon, Jun 29, 2015 at 3:33 PM, Andrew Or and...@databricks.com wrote: Hi Ted, We haven't observed a StreamingContextSuite failure on our test infrastructure recently. Given that we cannot reproduce it even locally it is unlikely that this uncovers a real bug. Even if it does I would not block the release on it because many in the community are waiting for a few important fixes. In general, there will always be outstanding issues in Spark that we cannot address in every release. -Andrew 2015-06-29 14:29 GMT-07:00 Ted Yu yuzhih...@gmail.com: The test passes when run alone on my machine as well. Please run test suite. Thanks On Mon, Jun 29, 2015 at 2:01 PM, Tathagata Das tathagata.das1...@gmail.com wrote: @Ted, I ran the following two commands. mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive -DskipTests clean package mvn -Phadoop-2.4 -Dhadoop.version=2.7.0 -Pyarn -Phive -DwildcardSuites=org.apache.spark.streaming.StreamingContextSuite test Using Java version 1.7.0_51, the tests passed normally. On Mon, Jun 29, 2015 at 1:05 PM, Krishna Sankar ksanka...@gmail.com wrote: +1 (non-binding, of course) 1. Compiled OSX 10.10 (Yosemite) OK Total time: 13:26 min mvn clean package -Pyarn -Phadoop-2.6 -DskipTests 2. Tested pyspark, mllib 2.1. statistics (min,max,mean,Pearson,Spearman) OK 2.2. Linear/Ridge/Laso Regression OK 2.3. Decision Tree, Naive Bayes OK 2.4. KMeans OK Center And Scale OK 2.5. RDD operations OK State of the Union Texts - MapReduce, Filter,sortByKey (word count) 2.6. Recommendation (Movielens medium dataset ~1 M ratings) OK Model evaluation/optimization (rank, numIter, lambda) with itertools OK 3. Scala - MLlib 3.1. statistics (min,max,mean,Pearson,Spearman) OK 3.2. LinearRegressionWithSGD OK 3.3. Decision Tree OK 3.4. KMeans OK 3.5. Recommendation (Movielens medium dataset ~1 M ratings) OK 3.6. saveAsParquetFile OK 3.7. Read and verify the 4.3 save(above) - sqlContext.parquetFile, registerTempTable, sql OK 3.8. result = sqlContext.sql(SELECT OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders INNER JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID) OK 4.0. Spark SQL from Python OK 4.1. result = sqlContext.sql(SELECT * from people WHERE State = 'WA') OK 5.0. Packages 5.1. com.databricks.spark.csv - read/write OK Cheers k/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
@Ted, could you elaborate more on what was the test command that you ran? What profiles, using SBT or Maven? TD On Sun, Jun 28, 2015 at 12:21 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Krishna - this is still the current release candidate. - Patrick On Sun, Jun 28, 2015 at 12:14 PM, Krishna Sankar ksanka...@gmail.com wrote: Patrick, Haven't seen any replies on test results. I will byte ;o) - Should I test this version or is another one in the wings ? Cheers k/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
Patrick, Haven't seen any replies on test results. I will byte ;o) - Should I test this version or is another one in the wings ? Cheers k/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
Hey Krishna - this is still the current release candidate. - Patrick On Sun, Jun 28, 2015 at 12:14 PM, Krishna Sankar ksanka...@gmail.com wrote: Patrick, Haven't seen any replies on test results. I will byte ;o) - Should I test this version or is another one in the wings ? Cheers k/ On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
Hey Tom - no one voted on this yet, so I need to keep it open until people vote. But I'm not aware of specific things we are waiting for. Anyone else? - Patrick On Fri, Jun 26, 2015 at 7:10 AM, Tom Graves tgraves...@yahoo.com wrote: So is this open for vote then or are we waiting on other things? Tom On Thursday, June 25, 2015 10:32 AM, Andrew Ash and...@andrewash.com wrote: I would guess that many tickets targeted at 1.4.1 were set that way during the tail end of the 1.4.0 voting process as people realized they wouldn't make the .0 release in time. In that case, they were likely aiming for a 1.4.x release, not necessarily 1.4.1 specifically. Maybe creating a 1.4.x target in Jira in addition to 1.4.0, 1.4.1, 1.4.2, etc would make it more clear that these tickets are targeted at some 1.4 update release rather than specifically the 1.4.1 update. On Thu, Jun 25, 2015 at 5:38 AM, Sean Owen so...@cloudera.com wrote: That makes sense to me -- there's an urgent fix to get out. I missed that part. Not that it really matters but was that expressed elsewhere? I know we tend to start the RC process even when a few more changes are still in progress, to get a first wave or two of testing done early, knowing that the RC won't be the final one. It makes sense for some issues for X to be open when an RC is cut, if they are actually truly intended for X. 44 seems like a lot, and I don't think it's good practice just because that's how it's happened before. It looks like half of them weren't actually important for 1.4.x as we're now down to 21. I don't disagree with the idea that only most of the issues targeted for version X will be in version X; the target expresses a stretch goal. Given the fast pace of change that's probably the only practical view. I think we're just missing a step then: before RC of X, ask people to review and update the target of JIRAs for X? In this case, it was a good point to untarget stuff from 1.4.x entirely; I suspect everything else should then be targeted at 1.4.2 by default with the exception of a handful that people really do intend to work in for 1.4.1 before its final release. I know it sounds like pencil-pushing, but it's a cheap way to bring some additional focus to release planning. RC time has felt like a last-call to *begin* changes ad-hoc when it would go faster if it were more intentional and constrained. Meaning faster RCs, meaning getting back to a 3-month release cycle or less, and meaning less rush to push stuff into a .0 release and less frequent need for a maintenance .1 version. So what happens if all 1.4.1-targeted JIRAs are targeted to 1.4.2? would that miss something that is definitely being worked on for 1.4.1? On Wed, Jun 24, 2015 at 6:56 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean, This is being shipped now because there is a severe bug in 1.4.0 that can cause data corruption for Parquet users. There are no blockers targeted for 1.4.1 - so I don't see that JIRA is inconsistent with shipping a release now. The goal of having every single targeted JIRA cleared by the time we start voting, I don't think there is broad consensus and cultural adoption of that principle yet. So I do not take it as a signal this release is premature (the story has been the same for every previous release we've ever done). The fact that we hit 90/124 of issues targeted at this release means we are targeting such that we get around 70% of issues merged. That actually doesn't seem so bad to me since there is some uncertainty in the process. B - Patrick On Wed, Jun 24, 2015 at 1:54 AM, Sean Owen so...@cloudera.com wrote: There are 44 issues still targeted for 1.4.1. None are Blockers; 12 are Critical. ~80% were opened and/or set by committers. Compare with 90 issues resolved for 1.4.1. I'm concerned that committers are targeting lots more for a release even in the short term than realistically can go in. On its face, it suggests that an RC is premature. Why is 1.4.1 being put forth for release now? It seems like people are saying they want a fair bit more time to work on 1.4.1. I suspect that in fact people would rather untarget / slip (again) these JIRAs, but it calls into question again how the targeting is consistently off by this much. What unresolved JIRAs targeted for 1.4.1 are *really* still open for 1.4.1? like, what would go badly if all 32 non-Critical JIRAs were untargeted now? is the reality that there are a handful of items to get in before the final release, and those are hopefully the ~12 critical ones? How about some review of that before we ask people to seriously test these bits? On Wed, Jun 24, 2015 at 8:37 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted
Re: [VOTE] Release Apache Spark 1.4.1
So is this open for vote then or are we waiting on other things? Tom On Thursday, June 25, 2015 10:32 AM, Andrew Ash and...@andrewash.com wrote: I would guess that many tickets targeted at 1.4.1 were set that way during the tail end of the 1.4.0 voting process as people realized they wouldn't make the .0 release in time. In that case, they were likely aiming for a 1.4.x release, not necessarily 1.4.1 specifically. Maybe creating a 1.4.x target in Jira in addition to 1.4.0, 1.4.1, 1.4.2, etc would make it more clear that these tickets are targeted at some 1.4 update release rather than specifically the 1.4.1 update. On Thu, Jun 25, 2015 at 5:38 AM, Sean Owen so...@cloudera.com wrote: That makes sense to me -- there's an urgent fix to get out. I missed that part. Not that it really matters but was that expressed elsewhere? I know we tend to start the RC process even when a few more changes are still in progress, to get a first wave or two of testing done early, knowing that the RC won't be the final one. It makes sense for some issues for X to be open when an RC is cut, if they are actually truly intended for X. 44 seems like a lot, and I don't think it's good practice just because that's how it's happened before. It looks like half of them weren't actually important for 1.4.x as we're now down to 21. I don't disagree with the idea that only most of the issues targeted for version X will be in version X; the target expresses a stretch goal. Given the fast pace of change that's probably the only practical view. I think we're just missing a step then: before RC of X, ask people to review and update the target of JIRAs for X? In this case, it was a good point to untarget stuff from 1.4.x entirely; I suspect everything else should then be targeted at 1.4.2 by default with the exception of a handful that people really do intend to work in for 1.4.1 before its final release. I know it sounds like pencil-pushing, but it's a cheap way to bring some additional focus to release planning. RC time has felt like a last-call to *begin* changes ad-hoc when it would go faster if it were more intentional and constrained. Meaning faster RCs, meaning getting back to a 3-month release cycle or less, and meaning less rush to push stuff into a .0 release and less frequent need for a maintenance .1 version. So what happens if all 1.4.1-targeted JIRAs are targeted to 1.4.2? would that miss something that is definitely being worked on for 1.4.1? On Wed, Jun 24, 2015 at 6:56 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean, This is being shipped now because there is a severe bug in 1.4.0 that can cause data corruption for Parquet users. There are no blockers targeted for 1.4.1 - so I don't see that JIRA is inconsistent with shipping a release now. The goal of having every single targeted JIRA cleared by the time we start voting, I don't think there is broad consensus and cultural adoption of that principle yet. So I do not take it as a signal this release is premature (the story has been the same for every previous release we've ever done). The fact that we hit 90/124 of issues targeted at this release means we are targeting such that we get around 70% of issues merged. That actually doesn't seem so bad to me since there is some uncertainty in the process. B - Patrick On Wed, Jun 24, 2015 at 1:54 AM, Sean Owen so...@cloudera.com wrote: There are 44 issues still targeted for 1.4.1. None are Blockers; 12 are Critical. ~80% were opened and/or set by committers. Compare with 90 issues resolved for 1.4.1. I'm concerned that committers are targeting lots more for a release even in the short term than realistically can go in. On its face, it suggests that an RC is premature. Why is 1.4.1 being put forth for release now? It seems like people are saying they want a fair bit more time to work on 1.4.1. I suspect that in fact people would rather untarget / slip (again) these JIRAs, but it calls into question again how the targeting is consistently off by this much. What unresolved JIRAs targeted for 1.4.1 are *really* still open for 1.4.1? like, what would go badly if all 32 non-Critical JIRAs were untargeted now? is the reality that there are a handful of items to get in before the final release, and those are hopefully the ~12 critical ones? How about some review of that before we ask people to seriously test these bits? On Wed, Jun 24, 2015 at 8:37 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at:
Re: [VOTE] Release Apache Spark 1.4.1
I got the following when running test suite: [INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null) ^[[0m[^[[0minfo^[[0m] ^[[0mCompiling 2 Scala sources and 1 Java source to /home/hbase/spark-1.4.1/streaming/target/scala-2.10/test-classes...^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m/home/hbase/spark-1.4.1/streaming/src/test/scala/org/apache/spark/streaming/DStreamClosureSuite.scala:82: not found: type TestException^[[0m ^[[0m[^[[31merror^[[0m] ^[[0mthrow new TestException(^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m ^^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m/home/hbase/spark-1.4.1/streaming/src/test/scala/org/apache/spark/streaming/scheduler/JobGeneratorSuite.scala:73: not found: type TestReceiver^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m val inputStream = ssc.receiverStream(new TestReceiver)^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m ^^[[0m ^[[0m[^[[31merror^[[0m] ^[[0mtwo errors found^[[0m ^[[0m[^[[31merror^[[0m] ^[[0mCompile failed at Jun 25, 2015 5:12:24 PM [1.492s]^[[0m Has anyone else seen similar error ? Thanks On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
Pardon. During earlier test run, I got: ^[[32mStreamingContextSuite:^[[0m ^[[32m- from no conf constructor^[[0m ^[[32m- from no conf + spark home^[[0m ^[[32m- from no conf + spark home + env^[[0m ^[[32m- from conf with settings^[[0m ^[[32m- from existing SparkContext^[[0m ^[[32m- from existing SparkContext with settings^[[0m ^[[31m*** RUN ABORTED ***^[[0m ^[[31m java.lang.NoSuchMethodError: org.apache.spark.ui.JettyUtils$.createStaticHandler(Ljava/lang/String;Ljava/lang/String;)Lorg/eclipse/jetty/servlet/ServletContextHandler;^[[0m ^[[31m at org.apache.spark.streaming.ui.StreamingTab.attach(StreamingTab.scala:49)^[[0m ^[[31m at org.apache.spark.streaming.StreamingContext$$anonfun$start$2.apply(StreamingContext.scala:601)^[[0m ^[[31m at org.apache.spark.streaming.StreamingContext$$anonfun$start$2.apply(StreamingContext.scala:601)^[[0m ^[[31m at scala.Option.foreach(Option.scala:236)^[[0m ^[[31m at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:601)^[[0m ^[[31m at org.apache.spark.streaming.StreamingContextSuite$$anonfun$8.apply$mcV$sp(StreamingContextSuite.scala:101)^[[0m ^[[31m at org.apache.spark.streaming.StreamingContextSuite$$anonfun$8.apply(StreamingContextSuite.scala:96)^[[0m ^[[31m at org.apache.spark.streaming.StreamingContextSuite$$anonfun$8.apply(StreamingContextSuite.scala:96)^[[0m ^[[31m at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)^[[0m ^[[31m at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)^[[0m The error from previous email was due to absence of StreamingContextSuite.scala On Fri, Jun 26, 2015 at 1:27 PM, Ted Yu yuzhih...@gmail.com wrote: I got the following when running test suite: [INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null) ^[[0m[^[[0minfo^[[0m] ^[[0mCompiling 2 Scala sources and 1 Java source to /home/hbase/spark-1.4.1/streaming/target/scala-2.10/test-classes...^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m/home/hbase/spark-1.4.1/streaming/src/test/scala/org/apache/spark/streaming/DStreamClosureSuite.scala:82: not found: type TestException^[[0m ^[[0m[^[[31merror^[[0m] ^[[0mthrow new TestException(^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m ^^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m/home/hbase/spark-1.4.1/streaming/src/test/scala/org/apache/spark/streaming/scheduler/JobGeneratorSuite.scala:73: not found: type TestReceiver^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m val inputStream = ssc.receiverStream(new TestReceiver)^[[0m ^[[0m[^[[31merror^[[0m] ^[[0m ^^[[0m ^[[0m[^[[31merror^[[0m] ^[[0mtwo errors found^[[0m ^[[0m[^[[31merror^[[0m] ^[[0mCompile failed at Jun 25, 2015 5:12:24 PM [1.492s]^[[0m Has anyone else seen similar error ? Thanks On Tue, Jun 23, 2015 at 10:37 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
I would guess that many tickets targeted at 1.4.1 were set that way during the tail end of the 1.4.0 voting process as people realized they wouldn't make the .0 release in time. In that case, they were likely aiming for a 1.4.x release, not necessarily 1.4.1 specifically. Maybe creating a 1.4.x target in Jira in addition to 1.4.0, 1.4.1, 1.4.2, etc would make it more clear that these tickets are targeted at some 1.4 update release rather than specifically the 1.4.1 update. On Thu, Jun 25, 2015 at 5:38 AM, Sean Owen so...@cloudera.com wrote: That makes sense to me -- there's an urgent fix to get out. I missed that part. Not that it really matters but was that expressed elsewhere? I know we tend to start the RC process even when a few more changes are still in progress, to get a first wave or two of testing done early, knowing that the RC won't be the final one. It makes sense for some issues for X to be open when an RC is cut, if they are actually truly intended for X. 44 seems like a lot, and I don't think it's good practice just because that's how it's happened before. It looks like half of them weren't actually important for 1.4.x as we're now down to 21. I don't disagree with the idea that only most of the issues targeted for version X will be in version X; the target expresses a stretch goal. Given the fast pace of change that's probably the only practical view. I think we're just missing a step then: before RC of X, ask people to review and update the target of JIRAs for X? In this case, it was a good point to untarget stuff from 1.4.x entirely; I suspect everything else should then be targeted at 1.4.2 by default with the exception of a handful that people really do intend to work in for 1.4.1 before its final release. I know it sounds like pencil-pushing, but it's a cheap way to bring some additional focus to release planning. RC time has felt like a last-call to *begin* changes ad-hoc when it would go faster if it were more intentional and constrained. Meaning faster RCs, meaning getting back to a 3-month release cycle or less, and meaning less rush to push stuff into a .0 release and less frequent need for a maintenance .1 version. So what happens if all 1.4.1-targeted JIRAs are targeted to 1.4.2? would that miss something that is definitely being worked on for 1.4.1? On Wed, Jun 24, 2015 at 6:56 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean, This is being shipped now because there is a severe bug in 1.4.0 that can cause data corruption for Parquet users. There are no blockers targeted for 1.4.1 - so I don't see that JIRA is inconsistent with shipping a release now. The goal of having every single targeted JIRA cleared by the time we start voting, I don't think there is broad consensus and cultural adoption of that principle yet. So I do not take it as a signal this release is premature (the story has been the same for every previous release we've ever done). The fact that we hit 90/124 of issues targeted at this release means we are targeting such that we get around 70% of issues merged. That actually doesn't seem so bad to me since there is some uncertainty in the process. B - Patrick On Wed, Jun 24, 2015 at 1:54 AM, Sean Owen so...@cloudera.com wrote: There are 44 issues still targeted for 1.4.1. None are Blockers; 12 are Critical. ~80% were opened and/or set by committers. Compare with 90 issues resolved for 1.4.1. I'm concerned that committers are targeting lots more for a release even in the short term than realistically can go in. On its face, it suggests that an RC is premature. Why is 1.4.1 being put forth for release now? It seems like people are saying they want a fair bit more time to work on 1.4.1. I suspect that in fact people would rather untarget / slip (again) these JIRAs, but it calls into question again how the targeting is consistently off by this much. What unresolved JIRAs targeted for 1.4.1 are *really* still open for 1.4.1? like, what would go badly if all 32 non-Critical JIRAs were untargeted now? is the reality that there are a handful of items to get in before the final release, and those are hopefully the ~12 critical ones? How about some review of that before we ask people to seriously test these bits? On Wed, Jun 24, 2015 at 8:37 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key:
Re: [VOTE] Release Apache Spark 1.4.1
That makes sense to me -- there's an urgent fix to get out. I missed that part. Not that it really matters but was that expressed elsewhere? I know we tend to start the RC process even when a few more changes are still in progress, to get a first wave or two of testing done early, knowing that the RC won't be the final one. It makes sense for some issues for X to be open when an RC is cut, if they are actually truly intended for X. 44 seems like a lot, and I don't think it's good practice just because that's how it's happened before. It looks like half of them weren't actually important for 1.4.x as we're now down to 21. I don't disagree with the idea that only most of the issues targeted for version X will be in version X; the target expresses a stretch goal. Given the fast pace of change that's probably the only practical view. I think we're just missing a step then: before RC of X, ask people to review and update the target of JIRAs for X? In this case, it was a good point to untarget stuff from 1.4.x entirely; I suspect everything else should then be targeted at 1.4.2 by default with the exception of a handful that people really do intend to work in for 1.4.1 before its final release. I know it sounds like pencil-pushing, but it's a cheap way to bring some additional focus to release planning. RC time has felt like a last-call to *begin* changes ad-hoc when it would go faster if it were more intentional and constrained. Meaning faster RCs, meaning getting back to a 3-month release cycle or less, and meaning less rush to push stuff into a .0 release and less frequent need for a maintenance .1 version. So what happens if all 1.4.1-targeted JIRAs are targeted to 1.4.2? would that miss something that is definitely being worked on for 1.4.1? On Wed, Jun 24, 2015 at 6:56 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean, This is being shipped now because there is a severe bug in 1.4.0 that can cause data corruption for Parquet users. There are no blockers targeted for 1.4.1 - so I don't see that JIRA is inconsistent with shipping a release now. The goal of having every single targeted JIRA cleared by the time we start voting, I don't think there is broad consensus and cultural adoption of that principle yet. So I do not take it as a signal this release is premature (the story has been the same for every previous release we've ever done). The fact that we hit 90/124 of issues targeted at this release means we are targeting such that we get around 70% of issues merged. That actually doesn't seem so bad to me since there is some uncertainty in the process. B - Patrick On Wed, Jun 24, 2015 at 1:54 AM, Sean Owen so...@cloudera.com wrote: There are 44 issues still targeted for 1.4.1. None are Blockers; 12 are Critical. ~80% were opened and/or set by committers. Compare with 90 issues resolved for 1.4.1. I'm concerned that committers are targeting lots more for a release even in the short term than realistically can go in. On its face, it suggests that an RC is premature. Why is 1.4.1 being put forth for release now? It seems like people are saying they want a fair bit more time to work on 1.4.1. I suspect that in fact people would rather untarget / slip (again) these JIRAs, but it calls into question again how the targeting is consistently off by this much. What unresolved JIRAs targeted for 1.4.1 are *really* still open for 1.4.1? like, what would go badly if all 32 non-Critical JIRAs were untargeted now? is the reality that there are a handful of items to get in before the final release, and those are hopefully the ~12 critical ones? How about some review of that before we ask people to seriously test these bits? On Wed, Jun 24, 2015 at 8:37 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes
Re: [VOTE] Release Apache Spark 1.4.1
There are 44 issues still targeted for 1.4.1. None are Blockers; 12 are Critical. ~80% were opened and/or set by committers. Compare with 90 issues resolved for 1.4.1. I'm concerned that committers are targeting lots more for a release even in the short term than realistically can go in. On its face, it suggests that an RC is premature. Why is 1.4.1 being put forth for release now? It seems like people are saying they want a fair bit more time to work on 1.4.1. I suspect that in fact people would rather untarget / slip (again) these JIRAs, but it calls into question again how the targeting is consistently off by this much. What unresolved JIRAs targeted for 1.4.1 are *really* still open for 1.4.1? like, what would go badly if all 32 non-Critical JIRAs were untargeted now? is the reality that there are a handful of items to get in before the final release, and those are hopefully the ~12 critical ones? How about some review of that before we ask people to seriously test these bits? On Wed, Jun 24, 2015 at 8:37 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [VOTE] Release Apache Spark 1.4.1
Hey Sean, This is being shipped now because there is a severe bug in 1.4.0 that can cause data corruption for Parquet users. There are no blockers targeted for 1.4.1 - so I don't see that JIRA is inconsistent with shipping a release now. The goal of having every single targeted JIRA cleared by the time we start voting, I don't think there is broad consensus and cultural adoption of that principle yet. So I do not take it as a signal this release is premature (the story has been the same for every previous release we've ever done). The fact that we hit 90/124 of issues targeted at this release means we are targeting such that we get around 70% of issues merged. That actually doesn't seem so bad to me since there is some uncertainty in the process. B - Patrick On Wed, Jun 24, 2015 at 1:54 AM, Sean Owen so...@cloudera.com wrote: There are 44 issues still targeted for 1.4.1. None are Blockers; 12 are Critical. ~80% were opened and/or set by committers. Compare with 90 issues resolved for 1.4.1. I'm concerned that committers are targeting lots more for a release even in the short term than realistically can go in. On its face, it suggests that an RC is premature. Why is 1.4.1 being put forth for release now? It seems like people are saying they want a fair bit more time to work on 1.4.1. I suspect that in fact people would rather untarget / slip (again) these JIRAs, but it calls into question again how the targeting is consistently off by this much. What unresolved JIRAs targeted for 1.4.1 are *really* still open for 1.4.1? like, what would go badly if all 32 non-Critical JIRAs were untargeted now? is the reality that there are a handful of items to get in before the final release, and those are hopefully the ~12 critical ones? How about some review of that before we ask people to seriously test these bits? On Wed, Jun 24, 2015 at 8:37 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
[VOTE] Release Apache Spark 1.4.1
Please vote on releasing the following candidate as Apache Spark version 1.4.1! This release fixes a handful of known issues in Spark 1.4.0, listed here: http://s.apache.org/spark-1.4.1 The tag to be voted on is v1.4.1-rc1 (commit 60e08e5): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h= 60e08e50751fe3929156de956d62faea79f5b801 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-bin/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: [published as version: 1.4.1] https://repository.apache.org/content/repositories/orgapachespark-1118/ [published as version: 1.4.1-rc1] https://repository.apache.org/content/repositories/orgapachespark-1119/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-releases/spark-1.4.1-rc1-docs/ Please vote on releasing this package as Apache Spark 1.4.1! The vote is open until Saturday, June 27, at 06:32 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.4.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
