Re: Spark 0.9.1 release
@evan From the discussion in the JIRA, it seems that we still dont have a clear solution for SPARK-1138. Nor do we have a sense of whether the solution is going to small enough for a maintenance release. So I dont think we should block the release of Spark 0.9.1 for this. We can make another Spark 0.9.2 release once the correct solution has been figured out. @kevin I understand the problem. I will try to port the solution for master inthis PR https://github.com/apache/spark/pull/100/ into branch 0.9. Lets see if it works out. On Tue, Mar 25, 2014 at 10:19 AM, Kevin Markey kevin.mar...@oracle.comwrote: TD: A correct shading of ASM should only affect Spark code unless someone is relying on ASM 4.0 in unrelated project code, in which case they can add org.ow2.asm:asm:4.x as a dependency. Our short term solution has been to repackage other libraries with a 3.2 dependency or to exclude ASM when our use of a dependent library really doesn't need it. As you probably know, the real problem arises in ClassVisitor, which is an Interface in 3.x and before, but in 4.x it is an abstract class that takes a version constant as its constructor. The ASM folks of course had our best interests in mind when they did this, attempting to deal with the Java-version dependent changes from one ASM release to the next. Unfortunately, they didn't change the names or locations of their classes and interfaces, which would have helped. In our particular case, the only library from which we couldn't exclude ASM was org.glassfish.jersey.containers:jersey-container-servlet:jar:2.5.1. I added a new module to our project, including some dummy source code, because we needed the library to be self contained, made the servlet -- minus some unrelated transitive dependencies -- the only module dependency, then used the Maven shade plugin to relocate org.objectweb.asm to an arbitrary target. We added the new shaded module as a new project dependency, plus the unrelated transitive dependencies excluded above. This solved the problem. At least until we added WADL to the project. Then we needed to deal with it on its own terms. As you can see, we left Spark alone in all its ASM 4.0 glory. Why? Spark is more volatile than the other libraries. Also, the way in which we needed to deploy Spark and other resources on our (Yarn) clusters suggested that it would be easier to shade the other libraries. I wanted to avoid having to install a locally patched Spark library into our build, updating the cluster and individual developers whenever there's a new patch. Individual developers such as me who are testing the impact of patches can handle it, but the main build goes to Maven Central via our corporate Artifactory mirror. If suddenly we had a Spark 0.9.1 with a shaded ASM, it would have no negative impact on us. Only a positive impact. I just wish that all users of ASM would read FAQ entry 15!!! Thanks Kevin On 03/24/2014 06:30 PM, Tathagata Das wrote: Hello Kevin, A fix for SPARK-782 would definitely simplify building against Spark. However, its possible that a fix for this issue in 0.9.1 will break the builds (that reference spark) of existing 0.9 users, either due to a change in the ASM version, or for being incompatible with their current workarounds for this issue. That is not a good idea for a maintenance release, especially when 1.0 is not too far away. Can you (and others) elaborate more on the current workarounds that you have for this issue? Its best to understand all the implications of this fix. Note that in branch 0.9, it is not fixed, neither in SBT nor in Maven. TD On Mon, Mar 24, 2014 at 4:38 PM, Kevin Markey kevin.mar...@oracle.com wrote: Is there any way that [SPARK-782] (Shade ASM) can be included? I see that it is not currently backported to 0.9. But there is no single issue that has caused us more grief as we integrate spark-core with other project dependencies. There are way too many libraries out there in addition to Spark 0.9 and before that are not well-behaved (ASM FAQ recommends shading), including some Hive and Hadoop libraries and a number of servlet libraries. We can't control those, but if Spark were well behaved in this regard, it would help. Even for a maintenance release, and even if 1.0 is only 6 weeks away! (For those not following 782, according to Jira comments, the SBT build shades it, but it is the Maven build that ends up in Maven Central.) Thanks Kevin Markey On 03/19/2014 06:07 PM, Tathagata Das wrote: Hello everyone, Since the release of Spark 0.9, we have received a number of important bug fixes and we would like to make a bug-fix release of Spark 0.9.1. We are going to cut a release candidate soon and we would love it if people test it out. We have backported several bug fixes into the 0.9 and updated JIRA accordinglyhttps://spark-project.atlassian.net/browse/
Travis CI
Just a quick note to everyone that Patrick and I are playing around with Travis CI on the Spark github repository. For now, travis does not run all of the test cases, so will only be turned on experimentally. Long term it looks like Travis might give better integration with github, so we are going to see if it is feasible to get all of our tests running on it. *Jenkins remains the reference CI and should be consulted before merging pull requests, independent of what Travis says.* If you have any questions or want to help out with the investigation, let me know! Michael
Re: Travis CI
I assume the Jenkins is not working now? Best, -- Nan Zhu On Tuesday, March 25, 2014 at 6:42 PM, Michael Armbrust wrote: Just a quick note to everyone that Patrick and I are playing around with Travis CI on the Spark github repository. For now, travis does not run all of the test cases, so will only be turned on experimentally. Long term it looks like Travis might give better integration with github, so we are going to see if it is feasible to get all of our tests running on it. *Jenkins remains the reference CI and should be consulted before merging pull requests, independent of what Travis says.* If you have any questions or want to help out with the investigation, let me know! Michael
Re: Travis CI
That's not correct - like Michael said the Jenkins build remains the reference build for now. On Tue, Mar 25, 2014 at 7:03 PM, Nan Zhu zhunanmcg...@gmail.com wrote: I assume the Jenkins is not working now? Best, -- Nan Zhu On Tuesday, March 25, 2014 at 6:42 PM, Michael Armbrust wrote: Just a quick note to everyone that Patrick and I are playing around with Travis CI on the Spark github repository. For now, travis does not run all of the test cases, so will only be turned on experimentally. Long term it looks like Travis might give better integration with github, so we are going to see if it is feasible to get all of our tests running on it. *Jenkins remains the reference CI and should be consulted before merging pull requests, independent of what Travis says.* If you have any questions or want to help out with the investigation, let me know! Michael
Re: Travis CI
I just found that the Jenkins is not working from this afternoon for one PR, the first time build failed after 90 minutes, the second time it has run for more than 2 hours, no result is returned Best, -- Nan Zhu On Tuesday, March 25, 2014 at 10:06 PM, Patrick Wendell wrote: That's not correct - like Michael said the Jenkins build remains the reference build for now. On Tue, Mar 25, 2014 at 7:03 PM, Nan Zhu zhunanmcg...@gmail.com (mailto:zhunanmcg...@gmail.com) wrote: I assume the Jenkins is not working now? Best, -- Nan Zhu On Tuesday, March 25, 2014 at 6:42 PM, Michael Armbrust wrote: Just a quick note to everyone that Patrick and I are playing around with Travis CI on the Spark github repository. For now, travis does not run all of the test cases, so will only be turned on experimentally. Long term it looks like Travis might give better integration with github, so we are going to see if it is feasible to get all of our tests running on it. *Jenkins remains the reference CI and should be consulted before merging pull requests, independent of what Travis says.* If you have any questions or want to help out with the investigation, let me know! Michael
Re: Travis CI
Ya It's been a little bit slow lately because of a high error rate in interactions with the git-hub API. Unfortunately we are pretty slammed for the release and haven't had a ton of time to do further debugging. - Patrick On Tue, Mar 25, 2014 at 7:13 PM, Nan Zhu zhunanmcg...@gmail.com wrote: I just found that the Jenkins is not working from this afternoon for one PR, the first time build failed after 90 minutes, the second time it has run for more than 2 hours, no result is returned Best, -- Nan Zhu On Tuesday, March 25, 2014 at 10:06 PM, Patrick Wendell wrote: That's not correct - like Michael said the Jenkins build remains the reference build for now. On Tue, Mar 25, 2014 at 7:03 PM, Nan Zhu zhunanmcg...@gmail.com wrote: I assume the Jenkins is not working now? Best, -- Nan Zhu On Tuesday, March 25, 2014 at 6:42 PM, Michael Armbrust wrote: Just a quick note to everyone that Patrick and I are playing around with Travis CI on the Spark github repository. For now, travis does not run all of the test cases, so will only be turned on experimentally. Long term it looks like Travis might give better integration with github, so we are going to see if it is feasible to get all of our tests running on it. *Jenkins remains the reference CI and should be consulted before merging pull requests, independent of what Travis says.* If you have any questions or want to help out with the investigation, let me know! Michael
Suggest to workaround the org.eclipse.jetty.orbit problem with SBT 0.13.2-RC1
Hi all, Due to a bug https://issues.apache.org/jira/browse/IVY-899 of Ivy, SBT tries to download .orbit instead of .jar files and causing problems. This bug has been fixed in Ivy 2.3.0, but SBT 0.13.1 still uses Ivy 2.0. Aaron had kindly provided a workaround in PR #183https://github.com/apache/incubator-spark/pull/183, but I'm afraid only explicitly depend on javax.servlet only is not enough. I'm not pretty sure about this because I'm facing both this issue and ridiculously unstable network environment, which makes reproducing the bug extremely time consuming (sbt gen-idea costs at least half an hour to complete, and the generated result is broken. Most of the time was spent in dependency resolution). At last, I worked around this issue by updating my local SBT to 0.13.2-RC1. If any of you are experiencing similar problem, I suggest you upgrade your local SBT version. Since SBT 0.13.2-RC1 is not an official release, we have to build it from scratch: $ git clone g...@github.com:sbt/sbt.git sbt-0.13.2-rc1-src-home $ cd sbt-0.13.2-rc1-src-home $ git checkout v0.13.2-RC1 Now ensure you have SBT 0.13.1 installed as the latest stable version is required for bootstrapping: $ sbt publishLocal $ mv ~/.sbt/boot /tmp $ cd sbt-0.13.1-home/bin $ mv sbt-launch.jar sbt-launch-0.13.1.jar $ ln -sf sbt-0.13.2-rc1-src-home/target/sbt-launch-0.13.2-RC1.jar sbt-launch.jar Now you should be able to build Spark without worrying .orbit files. Hope it helps. Cheng
Re: Suggest to workaround the org.eclipse.jetty.orbit problem with SBT 0.13.2-RC1
- Original Message - At last, I worked around this issue by updating my local SBT to 0.13.2-RC1. If any of you are experiencing similar problem, I suggest you upgrade your local SBT version. If this issue is causing grief for anyone on Fedora 20, know that you can install sbt via yum and get a sbt 0.13.1 that has been patched to use Ivy 2.3.0 instead of Ivy 2.3.0-rc1. Obviously, this isn't a solution for everyone, but it's certainly cleaner than building your own sbt locally if you're using Fedora. (If you try this out and run in to any trouble, please let me know off-list and I'll help out.) best, wb
Re: Suggest to workaround the org.eclipse.jetty.orbit problem with SBT 0.13.2-RC1
I think we should upgrade sbt, I have been using sbt since 13.2-M1 and have not spotted any issues. So RC1 should be good + it has the fast incremental compilation. Prashant Sharma On Wed, Mar 26, 2014 at 10:41 AM, Will Benton wi...@redhat.com wrote: - Original Message - At last, I worked around this issue by updating my local SBT to 0.13.2-RC1. If any of you are experiencing similar problem, I suggest you upgrade your local SBT version. If this issue is causing grief for anyone on Fedora 20, know that you can install sbt via yum and get a sbt 0.13.1 that has been patched to use Ivy 2.3.0 instead of Ivy 2.3.0-rc1. Obviously, this isn't a solution for everyone, but it's certainly cleaner than building your own sbt locally if you're using Fedora. (If you try this out and run in to any trouble, please let me know off-list and I'll help out.) best, wb
Re: Spark 0.9.1 release
PR 159 seems like a fairly big patch to me. And quite recent, so its impact on the scheduling is not clear. It may also depend on other changes that may have gotten into the DAGScheduler but not pulled into branch 0.9. I am not sure it is a good idea to pull that in. We can pull those changes later for 0.9.2 if required. TD On Tue, Mar 25, 2014 at 8:44 PM, Mridul Muralidharan mri...@gmail.comwrote: Forgot to mention this in the earlier request for PR's. If there is another RC being cut, please add https://github.com/apache/spark/pull/159 to it too (if not done already !). Thanks, Mridul On Thu, Mar 20, 2014 at 5:37 AM, Tathagata Das tathagata.das1...@gmail.com wrote: Hello everyone, Since the release of Spark 0.9, we have received a number of important bug fixes and we would like to make a bug-fix release of Spark 0.9.1. We are going to cut a release candidate soon and we would love it if people test it out. We have backported several bug fixes into the 0.9 and updated JIRA accordingly https://spark-project.atlassian.net/browse/SPARK-1275?jql=project%20in%20(SPARK%2C%20BLINKDB%2C%20MLI%2C%20MLLIB%2C%20SHARK%2C%20STREAMING%2C%20GRAPH%2C%20TACHYON)%20AND%20fixVersion%20%3D%200.9.1%20AND%20status%20in%20(Resolved%2C%20Closed) . Please let me know if there are fixes that were not backported but you would like to see them in 0.9.1. Thanks! TD
Re: [VOTE] Release Apache Spark 0.9.1 (rc1)
+1 looks good to me. I tried both the source and CDH4 versions and looked at the new streaming docs. The release notes seem slightly incomplete, but I guess you’re still working on them? Anyway those don’t go into the release tarball so it’s okay. Matei On Mar 24, 2014, at 2:01 PM, Tathagata Das tathagata.das1...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 0.9.1 A draft of the release notes along with the CHANGES.txt file is attached to this e-mail. The tag to be voted on is v0.9.1 (commit 81c6a06c): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=81c6a06c796a87aaeb5f129f36e4c3396e27d652 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~tdas/spark-0.9.1-rc1/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/tdas.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1007/ The documentation corresponding to this release can be found at: http://people.apache.org/~tdas/spark-0.9.1-rc1-docs/ Please vote on releasing this package as Apache Spark 0.9.1! The vote is open until Thursday, March 27, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 0.9.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ CHANGES.txtRELEASE_NOTES.txt
Re: Spark 0.9.1 release
I don't think the blacklisting is a priority and the CPUS_PER_TASK issue was still broken after this patch (so broken that I'm convinced no one actually uses this feature!!), so agree with TD's sentiment that this shouldn't go into 0.9.1. On Tue, Mar 25, 2014 at 10:23 PM, Tathagata Das tathagata.das1...@gmail.com wrote: PR 159 seems like a fairly big patch to me. And quite recent, so its impact on the scheduling is not clear. It may also depend on other changes that may have gotten into the DAGScheduler but not pulled into branch 0.9. I am not sure it is a good idea to pull that in. We can pull those changes later for 0.9.2 if required. TD On Tue, Mar 25, 2014 at 8:44 PM, Mridul Muralidharan mri...@gmail.com wrote: Forgot to mention this in the earlier request for PR's. If there is another RC being cut, please add https://github.com/apache/spark/pull/159 to it too (if not done already !). Thanks, Mridul On Thu, Mar 20, 2014 at 5:37 AM, Tathagata Das tathagata.das1...@gmail.com wrote: Hello everyone, Since the release of Spark 0.9, we have received a number of important bug fixes and we would like to make a bug-fix release of Spark 0.9.1. We are going to cut a release candidate soon and we would love it if people test it out. We have backported several bug fixes into the 0.9 and updated JIRA accordingly https://spark-project.atlassian.net/browse/SPARK-1275?jql=project%20in%20(SPARK%2C%20BLINKDB%2C%20MLI%2C%20MLLIB%2C%20SHARK%2C%20STREAMING%2C%20GRAPH%2C%20TACHYON)%20AND%20fixVersion%20%3D%200.9.1%20AND%20status%20in%20(Resolved%2C%20Closed) . Please let me know if there are fixes that were not backported but you would like to see them in 0.9.1. Thanks! TD
Re: [VOTE] Release Apache Spark 0.9.1 (rc1)
Actually I found one minor issue, which is that the support for Tachyon in make-distribution.sh seems to rely on GNU sed flags and doesn’t work on Mac OS X. But I’d be okay pushing that to a later release since this is a packaging operation that you do only once, and presumably you’d do it on a Linux cluster. I opened https://spark-project.atlassian.net/browse/SPARK-1326 to track it. We can put it in another RC if we find bigger issues. Matei On Mar 25, 2014, at 10:31 PM, Matei Zaharia matei.zaha...@gmail.com wrote: +1 looks good to me. I tried both the source and CDH4 versions and looked at the new streaming docs. The release notes seem slightly incomplete, but I guess you’re still working on them? Anyway those don’t go into the release tarball so it’s okay. Matei On Mar 24, 2014, at 2:01 PM, Tathagata Das tathagata.das1...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark version 0.9.1 A draft of the release notes along with the CHANGES.txt file is attached to this e-mail. The tag to be voted on is v0.9.1 (commit 81c6a06c): https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=81c6a06c796a87aaeb5f129f36e4c3396e27d652 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~tdas/spark-0.9.1-rc1/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/tdas.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1007/ The documentation corresponding to this release can be found at: http://people.apache.org/~tdas/spark-0.9.1-rc1-docs/ Please vote on releasing this package as Apache Spark 0.9.1! The vote is open until Thursday, March 27, at 22:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Spark 0.9.1 [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.apache.org/ CHANGES.txtRELEASE_NOTES.txt
Re: Spark 0.9.1 release
On Wed, Mar 26, 2014 at 10:53 AM, Tathagata Das tathagata.das1...@gmail.com wrote: PR 159 seems like a fairly big patch to me. And quite recent, so its impact on the scheduling is not clear. It may also depend on other changes that may have gotten into the DAGScheduler but not pulled into branch 0.9. I am not sure it is a good idea to pull that in. We can pull those changes later for 0.9.2 if required. There is no impact on scheduling : it only has an impact on error handling - it ensures that you can actually use spark on yarn in multi-tennent clusters more reliably. Currently, any reasonably long running job (30 mins+) working on non trivial dataset will fail due to accumulated failures in spark. Regards, Mridul TD On Tue, Mar 25, 2014 at 8:44 PM, Mridul Muralidharan mri...@gmail.comwrote: Forgot to mention this in the earlier request for PR's. If there is another RC being cut, please add https://github.com/apache/spark/pull/159 to it too (if not done already !). Thanks, Mridul On Thu, Mar 20, 2014 at 5:37 AM, Tathagata Das tathagata.das1...@gmail.com wrote: Hello everyone, Since the release of Spark 0.9, we have received a number of important bug fixes and we would like to make a bug-fix release of Spark 0.9.1. We are going to cut a release candidate soon and we would love it if people test it out. We have backported several bug fixes into the 0.9 and updated JIRA accordingly https://spark-project.atlassian.net/browse/SPARK-1275?jql=project%20in%20(SPARK%2C%20BLINKDB%2C%20MLI%2C%20MLLIB%2C%20SHARK%2C%20STREAMING%2C%20GRAPH%2C%20TACHYON)%20AND%20fixVersion%20%3D%200.9.1%20AND%20status%20in%20(Resolved%2C%20Closed) . Please let me know if there are fixes that were not backported but you would like to see them in 0.9.1. Thanks! TD
