Hi Lars, Unfortunately, due to some incompatible changes we pulled in to be closer to YARN trunk, Spark-on-YARN does not work against CDH 4.4+ (but does work against CDH5)
-Sandy On Tue, Mar 4, 2014 at 6:33 AM, Tom Graves <tgraves...@yahoo.com> wrote: > What is your question about Any hints? > The maven build worked for me yesterday again fine. > > You should create a jira for any pull request like the documentation > states. The jira thing is new so I think people are still getting used to > it. > > Tom > > > > On Tuesday, March 4, 2014 2:51 AM, Lars Francke <lars.fran...@gmail.com> > wrote: > > Hi, > > sorry to bother again. > > As a newbie to the project it's hard to judge whether I'm doing > anything wrong, the documentation is outdated or the Maven/SBT files > have diverged from the actual code by defining older/now incompatible > versions or something else going wrong. > > Any hints? > > Also an unrelated note/question: I see tons of pull requests being > accepted without a JIRA but the documentation says to create a JIRA > issue first[1]. So I assume it's okay to just send pull requests? > > Thanks for your help. > > Cheers, > Lars > > [1] < > https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark> > > > On Fri, Feb 28, 2014 at 6:41 PM, Lars Francke <lars.fran...@gmail.com> > wrote: > > Hey, > > > > so currently it doesn't work because of > > <https://github.com/apache/spark/pull/6#issuecomment-36343187> > > > > IntelliJ reports a lot of warnings with default settings and I haven't > > found a way to tell IntellJ to use different Hadoop versions yet. > > mvn clean compile -Pyarn fails as well (compilation errror > > > > Your command works indeed. Default yarn version is 0.23.7 which > > doesn't seem to work with the default 2.2.0 Hadoop version (anymore?) > > > > I was basically trying to follow the documentation: > > <http://spark.incubator.apache.org/docs/latest/building-with-maven.html> > > > > mvn clean compile -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.5.0 > > -Dyarn.version=2.0.0-cdh4.5.0 fails as well as does mvn clean compile > > -Pyarn-alpha > > > > Thanks for showing me a configuration that works. Unfortunately the > > default ones and at least one of the documented ones fail. > > > > Cheers, > > Lars > > > > > > On Fri, Feb 28, 2014 at 3:05 PM, Tom Graves <tgraves...@yahoo.com> > wrote: > >> what build command are you using? What do you mean when you say YARN > branch? > >> > >> The yarn builds have been working fine for me with maven. Build > command I use against hadoop 2.2 or higher: mvn -Dyarn.version=2.2.0 > -Dhadoop.version=2.2.0 -Pyarn clean package -DskipTests > >> > >> Tom > >> > >> > >> > >> On Friday, February 28, 2014 6:14 AM, Lars Francke < > lars.fran...@gmail.com> wrote: > >> > >> Hey, > >> > >> I'm trying to dig into Spark's code but am running into a couple of > problems. > >> > >> 1) The yarn-common directory is not included in the Maven build > >> causing things to fail because the dependency is missing. If I see the > >> history correct it used to be a Maven module but is not anymore. > >> > >> 2) When I try to include the yarn-common directory in the build things > >> start going bad. Compilation failures all over the place and I think > >> there are some dependency issues in there as well. > >> > >> This leads me to believe that either the Maven build system isn't > >> maintained for YARN or the whole YARN branch isn't. What's the status > >> here? > >> > >> Without YARN things build fine for me using Maven. > >> > >> Thanks for your help. > >> > >> Cheers, > >> Lars >
