Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
I assume we will have an rc10 to fix the issues Matei found? Tom On Sunday, May 18, 2014 9:08 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Matei - the issue you found is not related to security. This patch a few days ago broke builds for Hadoop 1 with YARN support enabled. The patch directly altered the way we deal with commons-lang dependency, which is what is at the base of this stack trace. https://github.com/apache/spark/pull/754 - Patrick On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Alright, I've opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1's security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this? Matei On May 18, 2014, at 12:33 PM, Matei Zaharia matei.zaha...@gmail.com wrote: I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't quite match the master-setting behavior of the Unix based one. I'll send a pull request to fix them soon. Matei On May 17, 2014, at 11:32 AM, Sandy Ryza sandy.r...@cloudera.com wrote: +1 Reran my tests from rc5: * Built the release from source. * Compiled Java and Scala apps that interact with HDFS against it. * Ran them in local mode. * Ran them against a pseudo-distributed YARN cluster in both yarn-client mode and yarn-cluster mode. On Sat, May 17, 2014 at 10:08 AM, Andrew Or and...@databricks.com wrote: +1 2014-05-17 8:53 GMT-07:00 Mark Hamstra m...@clearstorydata.com: +1 On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: I'll start the voting with a +1. On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.0! This has one bug fix and one minor feature on top of rc8: SPARK-1864: https://github.com/apache/spark/pull/808 SPARK-1808: https://github.com/apache/spark/pull/799 The tag to be voted on is v1.0.0-rc9 (commit 920f947): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1017/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Tuesday, May 20, at 08:56 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ == API Changes == We welcome users to compile Spark applications against 1.0. There are a few API changes in this release. Here are links to the associated upgrade guides - user facing changes have been kept as small as possible. changes to ML vector specification: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10 changes to the Java API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark changes to the streaming API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x changes to the GraphX API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091 coGroup and related functions now return Iterable[T] instead of Seq[T] == Call toSeq on the result to restore the old behavior SparkContext.jarOfClass returns Option[String] instead of Seq[String] == Call toSeq on the result to restore old behavior
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
just rerun my test on rc5 everything works build applications with sbt and the spark-*.jar which is compiled with Hadoop 2.3 +1 -- Nan Zhu On Sunday, May 18, 2014 at 11:07 PM, witgo wrote: How to reproduce this bug? -- Original -- From: Patrick Wendell;pwend...@gmail.com (mailto:pwend...@gmail.com); Date: Mon, May 19, 2014 10:08 AM To: dev@spark.apache.org (mailto:dev@spark.apache.org)dev@spark.apache.org (mailto:dev@spark.apache.org); Cc: Tom Gravestgraves...@yahoo.com (mailto:tgraves...@yahoo.com); Subject: Re: [VOTE] Release Apache Spark 1.0.0 (rc9) Hey Matei - the issue you found is not related to security. This patch a few days ago broke builds for Hadoop 1 with YARN support enabled. The patch directly altered the way we deal with commons-lang dependency, which is what is at the base of this stack trace. https://github.com/apache/spark/pull/754 - Patrick On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia matei.zaha...@gmail.com (mailto:matei.zaha...@gmail.com) wrote: Alright, I've opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1's security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this? Matei On May 18, 2014, at 12:33 PM, Matei Zaharia matei.zaha...@gmail.com (mailto:matei.zaha...@gmail.com) wrote: I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't quite match the master-setting behavior of the Unix based one. I'll send a pull request to fix them soon. Matei On May 17, 2014, at 11:32 AM, Sandy Ryza sandy.r...@cloudera.com (mailto:sandy.r...@cloudera.com) wrote: +1 Reran my tests from rc5: * Built the release from source. * Compiled Java and Scala apps that interact with HDFS against it. * Ran them in local mode. * Ran them against a pseudo-distributed YARN cluster in both yarn-client mode and yarn-cluster mode. On Sat, May 17, 2014 at 10:08 AM, Andrew Or and...@databricks.com (mailto:and...@databricks.com) wrote: +1 2014-05-17 8:53 GMT-07:00 Mark Hamstra m...@clearstorydata.com (mailto:m...@clearstorydata.com): +1 On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com (mailto:pwend...@gmail.com) wrote: I'll start the voting with a +1. On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com (mailto:pwend...@gmail.com) wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.0! This has one bug fix and one minor feature on top of rc8: SPARK-1864: https://github.com/apache/spark/pull/808 SPARK-1808: https://github.com/apache/spark/pull/799 The tag to be voted on is v1.0.0-rc9 (commit 920f947): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1017/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Tuesday, May 20, at 08:56 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ == API Changes == We welcome users to compile Spark applications against 1.0. There are a few API changes in this release. Here are links to the associated upgrade guides - user facing changes have been kept as small as possible. changes to ML vector specification: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
We're cancelling this RC in favor of rc10. There were two blockers: an issue with Windows run scripts and an issue with the packaging for Hadoop 1 when hive support is bundled. https://issues.apache.org/jira/browse/SPARK-1875 https://issues.apache.org/jira/browse/SPARK-1876 Thanks everyone for the testing. TD will be cutting rc10, since I'm travelling this week (thanks TD!). - Patrick On Mon, May 19, 2014 at 7:06 PM, Nan Zhu zhunanmcg...@gmail.com wrote: just rerun my test on rc5 everything works build applications with sbt and the spark-*.jar which is compiled with Hadoop 2.3 +1 -- Nan Zhu On Sunday, May 18, 2014 at 11:07 PM, witgo wrote: How to reproduce this bug? -- Original -- From: Patrick Wendell;pwend...@gmail.com (mailto:pwend...@gmail.com); Date: Mon, May 19, 2014 10:08 AM To: dev@spark.apache.org (mailto:dev@spark.apache.org)dev@spark.apache.org (mailto:dev@spark.apache.org); Cc: Tom Gravestgraves...@yahoo.com (mailto:tgraves...@yahoo.com); Subject: Re: [VOTE] Release Apache Spark 1.0.0 (rc9) Hey Matei - the issue you found is not related to security. This patch a few days ago broke builds for Hadoop 1 with YARN support enabled. The patch directly altered the way we deal with commons-lang dependency, which is what is at the base of this stack trace. https://github.com/apache/spark/pull/754 - Patrick On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia matei.zaha...@gmail.com (mailto:matei.zaha...@gmail.com) wrote: Alright, I've opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1's security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this? Matei On May 18, 2014, at 12:33 PM, Matei Zaharia matei.zaha...@gmail.com (mailto:matei.zaha...@gmail.com) wrote: I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't quite match the master-setting behavior of the Unix based one. I'll send a pull request to fix them soon. Matei On May 17, 2014, at 11:32 AM, Sandy Ryza sandy.r...@cloudera.com (mailto:sandy.r...@cloudera.com) wrote: +1 Reran my tests from rc5: * Built the release from source. * Compiled Java and Scala apps that interact with HDFS against it. * Ran them in local mode. * Ran them against a pseudo-distributed YARN cluster in both yarn-client mode and yarn-cluster mode. On Sat, May 17, 2014 at 10:08 AM, Andrew Or and...@databricks.com (mailto:and...@databricks.com) wrote: +1 2014-05-17 8:53 GMT-07:00 Mark Hamstra m...@clearstorydata.com (mailto:m...@clearstorydata.com): +1 On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com (mailto:pwend...@gmail.com) wrote: I'll start the voting with a +1. On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com (mailto:pwend...@gmail.com) wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.0! This has one bug fix and one minor feature on top of rc8: SPARK-1864: https://github.com/apache/spark/pull/808 SPARK-1808: https://github.com/apache/spark/pull/799 The tag to be voted on is v1.0.0-rc9 (commit 920f947): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1017/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Tuesday, May 20, at 08:56 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn’t quite match the master-setting behavior of the Unix based one. I’ll send a pull request to fix them soon. Matei On May 17, 2014, at 11:32 AM, Sandy Ryza sandy.r...@cloudera.com wrote: +1 Reran my tests from rc5: * Built the release from source. * Compiled Java and Scala apps that interact with HDFS against it. * Ran them in local mode. * Ran them against a pseudo-distributed YARN cluster in both yarn-client mode and yarn-cluster mode. On Sat, May 17, 2014 at 10:08 AM, Andrew Or and...@databricks.com wrote: +1 2014-05-17 8:53 GMT-07:00 Mark Hamstra m...@clearstorydata.com: +1 On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: I'll start the voting with a +1. On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.0! This has one bug fix and one minor feature on top of rc8: SPARK-1864: https://github.com/apache/spark/pull/808 SPARK-1808: https://github.com/apache/spark/pull/799 The tag to be voted on is v1.0.0-rc9 (commit 920f947): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1017/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Tuesday, May 20, at 08:56 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ == API Changes == We welcome users to compile Spark applications against 1.0. There are a few API changes in this release. Here are links to the associated upgrade guides - user facing changes have been kept as small as possible. changes to ML vector specification: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10 changes to the Java API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark changes to the streaming API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x changes to the GraphX API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091 coGroup and related functions now return Iterable[T] instead of Seq[T] == Call toSeq on the result to restore the old behavior SparkContext.jarOfClass returns Option[String] instead of Seq[String] == Call toSeq on the result to restore old behavior
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Alright, I’ve opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1’s security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this? Matei On May 18, 2014, at 12:33 PM, Matei Zaharia matei.zaha...@gmail.com wrote: I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn’t quite match the master-setting behavior of the Unix based one. I’ll send a pull request to fix them soon. Matei On May 17, 2014, at 11:32 AM, Sandy Ryza sandy.r...@cloudera.com wrote: +1 Reran my tests from rc5: * Built the release from source. * Compiled Java and Scala apps that interact with HDFS against it. * Ran them in local mode. * Ran them against a pseudo-distributed YARN cluster in both yarn-client mode and yarn-cluster mode. On Sat, May 17, 2014 at 10:08 AM, Andrew Or and...@databricks.com wrote: +1 2014-05-17 8:53 GMT-07:00 Mark Hamstra m...@clearstorydata.com: +1 On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: I'll start the voting with a +1. On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.0! This has one bug fix and one minor feature on top of rc8: SPARK-1864: https://github.com/apache/spark/pull/808 SPARK-1808: https://github.com/apache/spark/pull/799 The tag to be voted on is v1.0.0-rc9 (commit 920f947): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1017/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Tuesday, May 20, at 08:56 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ == API Changes == We welcome users to compile Spark applications against 1.0. There are a few API changes in this release. Here are links to the associated upgrade guides - user facing changes have been kept as small as possible. changes to ML vector specification: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10 changes to the Java API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark changes to the streaming API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x changes to the GraphX API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091 coGroup and related functions now return Iterable[T] instead of Seq[T] == Call toSeq on the result to restore the old behavior SparkContext.jarOfClass returns Option[String] instead of Seq[String] == Call toSeq on the result to restore old behavior
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
no ideas off hand, I'll take a look tomorrow. Tom On Sunday, May 18, 2014 7:28 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Alright, I’ve opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1’s security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this? Matei On May 18, 2014, at 12:33 PM, Matei Zaharia matei.zaha...@gmail.com wrote: I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn’t quite match the master-setting behavior of the Unix based one. I’ll send a pull request to fix them soon. Matei On May 17, 2014, at 11:32 AM, Sandy Ryza sandy.r...@cloudera.com wrote: +1 Reran my tests from rc5: * Built the release from source. * Compiled Java and Scala apps that interact with HDFS against it. * Ran them in local mode. * Ran them against a pseudo-distributed YARN cluster in both yarn-client mode and yarn-cluster mode. On Sat, May 17, 2014 at 10:08 AM, Andrew Or and...@databricks.com wrote: +1 2014-05-17 8:53 GMT-07:00 Mark Hamstra m...@clearstorydata.com: +1 On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: I'll start the voting with a +1. On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.0! This has one bug fix and one minor feature on top of rc8: SPARK-1864: https://github.com/apache/spark/pull/808 SPARK-1808: https://github.com/apache/spark/pull/799 The tag to be voted on is v1.0.0-rc9 (commit 920f947): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1017/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Tuesday, May 20, at 08:56 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ == API Changes == We welcome users to compile Spark applications against 1.0. There are a few API changes in this release. Here are links to the associated upgrade guides - user facing changes have been kept as small as possible. changes to ML vector specification: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10 changes to the Java API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark changes to the streaming API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x changes to the GraphX API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091 coGroup and related functions now return Iterable[T] instead of Seq[T] == Call toSeq on the result to restore the old behavior SparkContext.jarOfClass returns Option[String] instead of Seq[String] == Call toSeq on the result to restore old behavior
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Hey Matei - the issue you found is not related to security. This patch a few days ago broke builds for Hadoop 1 with YARN support enabled. The patch directly altered the way we deal with commons-lang dependency, which is what is at the base of this stack trace. https://github.com/apache/spark/pull/754 - Patrick On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Alright, I've opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1's security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this? Matei On May 18, 2014, at 12:33 PM, Matei Zaharia matei.zaha...@gmail.com wrote: I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't quite match the master-setting behavior of the Unix based one. I'll send a pull request to fix them soon. Matei On May 17, 2014, at 11:32 AM, Sandy Ryza sandy.r...@cloudera.com wrote: +1 Reran my tests from rc5: * Built the release from source. * Compiled Java and Scala apps that interact with HDFS against it. * Ran them in local mode. * Ran them against a pseudo-distributed YARN cluster in both yarn-client mode and yarn-cluster mode. On Sat, May 17, 2014 at 10:08 AM, Andrew Or and...@databricks.com wrote: +1 2014-05-17 8:53 GMT-07:00 Mark Hamstra m...@clearstorydata.com: +1 On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: I'll start the voting with a +1. On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.0! This has one bug fix and one minor feature on top of rc8: SPARK-1864: https://github.com/apache/spark/pull/808 SPARK-1808: https://github.com/apache/spark/pull/799 The tag to be voted on is v1.0.0-rc9 (commit 920f947): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1017/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Tuesday, May 20, at 08:56 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ == API Changes == We welcome users to compile Spark applications against 1.0. There are a few API changes in this release. Here are links to the associated upgrade guides - user facing changes have been kept as small as possible. changes to ML vector specification: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10 changes to the Java API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark changes to the streaming API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x changes to the GraphX API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091 coGroup and related functions now return Iterable[T] instead of Seq[T] == Call toSeq on the result to restore the old behavior SparkContext.jarOfClass returns Option[String] instead of Seq[String] == Call toSeq on the result to restore old behavior
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
I'll start the voting with a +1. On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.0! This has one bug fix and one minor feature on top of rc8: SPARK-1864: https://github.com/apache/spark/pull/808 SPARK-1808: https://github.com/apache/spark/pull/799 The tag to be voted on is v1.0.0-rc9 (commit 920f947): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1017/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Tuesday, May 20, at 08:56 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ == API Changes == We welcome users to compile Spark applications against 1.0. There are a few API changes in this release. Here are links to the associated upgrade guides - user facing changes have been kept as small as possible. changes to ML vector specification: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10 changes to the Java API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark changes to the streaming API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x changes to the GraphX API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091 coGroup and related functions now return Iterable[T] instead of Seq[T] == Call toSeq on the result to restore the old behavior SparkContext.jarOfClass returns Option[String] instead of Seq[String] == Call toSeq on the result to restore old behavior
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
+1 2014-05-17 8:53 GMT-07:00 Mark Hamstra m...@clearstorydata.com: +1 On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: I'll start the voting with a +1. On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 1.0.0! This has one bug fix and one minor feature on top of rc8: SPARK-1864: https://github.com/apache/spark/pull/808 SPARK-1808: https://github.com/apache/spark/pull/799 The tag to be voted on is v1.0.0-rc9 (commit 920f947): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75 The release files, including signatures, digests, etc. can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1017/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/ Please vote on releasing this package as Apache Spark 1.0.0! The vote is open until Tuesday, May 20, at 08:56 UTC and passes if amajority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ == API Changes == We welcome users to compile Spark applications against 1.0. There are a few API changes in this release. Here are links to the associated upgrade guides - user facing changes have been kept as small as possible. changes to ML vector specification: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10 changes to the Java API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark changes to the streaming API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x changes to the GraphX API: http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091 coGroup and related functions now return Iterable[T] instead of Seq[T] == Call toSeq on the result to restore the old behavior SparkContext.jarOfClass returns Option[String] instead of Seq[String] == Call toSeq on the result to restore old behavior
