Re: Spark 0.9.1 release

2014-03-25 Thread Tathagata Das
@evan From the discussion in the JIRA, it seems that we still dont have a clear solution for SPARK-1138. Nor do we have a sense of whether the solution is going to small enough for a maintenance release. So I dont think we should block the release of Spark 0.9.1 for this. We can make another Spark

Travis CI

2014-03-25 Thread Michael Armbrust
Just a quick note to everyone that Patrick and I are playing around with Travis CI on the Spark github repository. For now, travis does not run all of the test cases, so will only be turned on experimentally. Long term it looks like Travis might give better integration with github, so we are

Re: Travis CI

2014-03-25 Thread Nan Zhu
I assume the Jenkins is not working now? Best, -- Nan Zhu On Tuesday, March 25, 2014 at 6:42 PM, Michael Armbrust wrote: Just a quick note to everyone that Patrick and I are playing around with Travis CI on the Spark github repository. For now, travis does not run all of the test

Re: Travis CI

2014-03-25 Thread Patrick Wendell
That's not correct - like Michael said the Jenkins build remains the reference build for now. On Tue, Mar 25, 2014 at 7:03 PM, Nan Zhu zhunanmcg...@gmail.com wrote: I assume the Jenkins is not working now? Best, -- Nan Zhu On Tuesday, March 25, 2014 at 6:42 PM, Michael Armbrust wrote:

Re: Travis CI

2014-03-25 Thread Nan Zhu
I just found that the Jenkins is not working from this afternoon for one PR, the first time build failed after 90 minutes, the second time it has run for more than 2 hours, no result is returned Best, -- Nan Zhu On Tuesday, March 25, 2014 at 10:06 PM, Patrick Wendell wrote: That's not

Re: Travis CI

2014-03-25 Thread Patrick Wendell
Ya It's been a little bit slow lately because of a high error rate in interactions with the git-hub API. Unfortunately we are pretty slammed for the release and haven't had a ton of time to do further debugging. - Patrick On Tue, Mar 25, 2014 at 7:13 PM, Nan Zhu zhunanmcg...@gmail.com wrote: I

Suggest to workaround the org.eclipse.jetty.orbit problem with SBT 0.13.2-RC1

2014-03-25 Thread Cheng Lian
Hi all, Due to a bug https://issues.apache.org/jira/browse/IVY-899 of Ivy, SBT tries to download .orbit instead of .jar files and causing problems. This bug has been fixed in Ivy 2.3.0, but SBT 0.13.1 still uses Ivy 2.0. Aaron had kindly provided a workaround in PR

Re: Suggest to workaround the org.eclipse.jetty.orbit problem with SBT 0.13.2-RC1

2014-03-25 Thread Will Benton
- Original Message - At last, I worked around this issue by updating my local SBT to 0.13.2-RC1. If any of you are experiencing similar problem, I suggest you upgrade your local SBT version. If this issue is causing grief for anyone on Fedora 20, know that you can install sbt via yum

Re: Suggest to workaround the org.eclipse.jetty.orbit problem with SBT 0.13.2-RC1

2014-03-25 Thread Prashant Sharma
I think we should upgrade sbt, I have been using sbt since 13.2-M1 and have not spotted any issues. So RC1 should be good + it has the fast incremental compilation. Prashant Sharma On Wed, Mar 26, 2014 at 10:41 AM, Will Benton wi...@redhat.com wrote: - Original Message - At last, I

Re: Spark 0.9.1 release

2014-03-25 Thread Tathagata Das
PR 159 seems like a fairly big patch to me. And quite recent, so its impact on the scheduling is not clear. It may also depend on other changes that may have gotten into the DAGScheduler but not pulled into branch 0.9. I am not sure it is a good idea to pull that in. We can pull those changes

Re: [VOTE] Release Apache Spark 0.9.1 (rc1)

2014-03-25 Thread Matei Zaharia
+1 looks good to me. I tried both the source and CDH4 versions and looked at the new streaming docs. The release notes seem slightly incomplete, but I guess you’re still working on them? Anyway those don’t go into the release tarball so it’s okay. Matei On Mar 24, 2014, at 2:01 PM, Tathagata

Re: Spark 0.9.1 release

2014-03-25 Thread Kay Ousterhout
I don't think the blacklisting is a priority and the CPUS_PER_TASK issue was still broken after this patch (so broken that I'm convinced no one actually uses this feature!!), so agree with TD's sentiment that this shouldn't go into 0.9.1. On Tue, Mar 25, 2014 at 10:23 PM, Tathagata Das

Re: [VOTE] Release Apache Spark 0.9.1 (rc1)

2014-03-25 Thread Matei Zaharia
Actually I found one minor issue, which is that the support for Tachyon in make-distribution.sh seems to rely on GNU sed flags and doesn’t work on Mac OS X. But I’d be okay pushing that to a later release since this is a packaging operation that you do only once, and presumably you’d do it on a

Re: Spark 0.9.1 release

2014-03-25 Thread Mridul Muralidharan
On Wed, Mar 26, 2014 at 10:53 AM, Tathagata Das tathagata.das1...@gmail.com wrote: PR 159 seems like a fairly big patch to me. And quite recent, so its impact on the scheduling is not clear. It may also depend on other changes that may have gotten into the DAGScheduler but not pulled into