Re: Compiling Spark with a local hadoop profile

> On 8 Oct 2015, at 19:31, sbiookag wrote: > > Thanks Ted for reply. > > But this is not what I want. This would tell spark to read hadoop dependency > from maven repository, which is the original version of hadoop. I myslef is > modifying the hadoop code, and wanted to

Operations with cached RDD

Dear Spark developers, I am trying to understand how Spark UI displays operation with the cached RDD. For example, the following code caches an rdd: >> val rdd = sc.parallelize(1 to 5, 5).zipWithIndex.cache >> rdd.count The Jobs tab shows me that the RDD is evaluated: : 1 count at :24

passing a AbstractFunction1 to sparkContext().runJob instead of a Closure

hi all, I want to run a job in the spark context and since I am running the system in the java environment, I can not use a closure in the sparkContext().runJob. Instead, I am passing an AbstractFunction1 extension. while I get the jobs run without an issue, I constantly get the following WARN

Re: Scala 2.11 builds broken/ Can the PR build run also 2.11?

I would push back slightly. The reason we have the PR builds taking so long is death by a million small things that we add. Doing a full 2.11 compile is order minutes... it's a nontrivial increase to the build times. It doesn't seem that bad to me to go back post-hoc once in a while and fix 2.11

Re: Scala 2.11 builds broken/ Can the PR build run also 2.11?

That is correct !, I have thought about this a lot of times. The only solution is to implement a "real" cross build for both version. I am going to think more in this. :) Prashant Sharma On Sat, Oct 10, 2015 at 2:04 AM, Patrick Wendell wrote: > I would push back slightly.

sbt test error -- "Could not reserve enough space"

Hi, I am trying to build and test the current master. My system is Ubuntu 14.04 with 4 G physical memory with Oracle Java 8. I have been running into various out-of-memory errors. I tried building with Maven but couldn't get all the way through compile and package. I'm having better luck with

Re: Scala 2.11 builds broken/ Can the PR build run also 2.11?

Sorry for not being clear, yes, that's about the Sbt build and treating warnings as errors. Warnings in 2.11 are useful, though, it'd be a pity to keep introducing potential issues. As a stop-gap measure I can disable them in the Sbt build, is it hard to run the CI test with 2.11/sbt? iulian

Re: Scala 2.11 builds broken/ Can the PR build run also 2.11?

How about just fixing the warning? I get it; it doesn't stop this from happening again, but still seems less drastic than tossing out the whole mechanism. On Fri, Oct 9, 2015 at 3:18 PM, Iulian Dragoș wrote: > Sorry for not being clear, yes, that's about the Sbt build

Re: Scala 2.11 builds broken/ Can the PR build run also 2.11?

> > How about just fixing the warning? I get it; it doesn't stop this from > happening again, but still seems less drastic than tossing out the > whole mechanism. > +1 It also does not seem that expensive to test only compilation for Scala 2.11 on PR builds.

Re: Scala 2.11 builds broken/ Can the PR build run also 2.11?

+1, much better than having a new PR each time to fix something for scala-2.11 every time a patch breaks it. Thanks, Hari Shreedharan > On Oct 9, 2015, at 11:47 AM, Michael Armbrust wrote: > > How about just fixing the warning? I get it; it doesn't stop this from >