You may need to add the -Phadoop-2.4 profile. When building or release packages for Hadoop 2.4 we use the following flags:
-Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn - Patrick On Thu, Mar 5, 2015 at 12:47 PM, Kelly, Jonathan <jonat...@amazon.com> wrote: > I confirmed that this has nothing to do with BigTop by running the same mvn > command directly in a fresh clone of the Spark package at the v1.2.1 tag. I > got the same exact error. > > > Jonathan Kelly > > Elastic MapReduce - SDE > > Port 99 (SEA35) 08.220.C2 > > > From: <Kelly>, Jonathan Kelly <jonat...@amazon.com> > Date: Thursday, March 5, 2015 at 10:39 AM > To: "user@spark.apache.org" <user@spark.apache.org> > Subject: Spark v1.2.1 failing under BigTop build in External Flume Sink (due > to missing Netty library) > > I'm running into an issue building Spark v1.2.1 (as well as the latest in > branch-1.2 and v1.3.0-rc2 and the latest in branch-1.3) with BigTop (v0.9, > which is not quite released yet). The build fails in the External Flume > Sink subproject with the following error: > > [INFO] Compiling 5 Scala sources and 3 Java sources to > /workspace/workspace/bigtop.spark-rpm/build/spark/rpm/BUILD/spark-1.3.0/external/flume-sink/target/scala-2.10/classes... > [WARNING] Class org.jboss.netty.channel.ChannelFactory not found - > continuing with a stub. > [ERROR] error while loading NettyServer, class file > '/home/ec2-user/.m2/repository/org/apache/avro/avro-ipc/1.7.6/avro-ipc-1.7.6.jar(org/apache/avro/ipc/NettyServer.class)' > is broken > (class java.lang.NullPointerException/null) > [WARNING] one warning found > [ERROR] one error found > > It seems like what is happening is that the Netty library is missing at > build time, which happens because it is explicitly excluded in the pom.xml > (see > https://github.com/apache/spark/blob/v1.2.1/external/flume-sink/pom.xml#L42). > I attempted removing the exclusions and the explicit re-add for the test > scope on lines 77-88, and that allowed the build to succeed, though I don't > know if that will cause problems at runtime. I don't have any experience > with the Flume Sink, so I don't really know how to test it. (And, to be > clear, I'm not necessarily trying to get the Flume Sink to work-- I just > want the project to build successfully, though of course I'd still want the > Flume Sink to work for whomever does need it.) > > Does anybody have any idea what's going on here? Here is the command BigTop > is running to build Spark: > > mvn -Pbigtop-dist -Pyarn -Phive -Phive-thriftserver -Pkinesis-asl > -Divy.home=/home/ec2-user/.ivy2 -Dsbt.ivy.home=/home/ec2-user/.ivy2 > -Duser.home=/home/ec2-user -Drepo.maven.org= > -Dreactor.repo=file:///home/ec2-user/.m2/repository > -Dhadoop.version=2.4.0-amzn-3-SNAPSHOT -Dyarn.version=2.4.0-amzn-3-SNAPSHOT > -Dprotobuf.version=2.5.0 -Dscala.version=2.10.3 -Dscala.binary.version=2.10 > -DskipTests -DrecompileMode=all install > > As I mentioned above, if I switch to the latest in branch-1.2, to > v1.3.0-rc2, or to the latest in branch-1.3, I get the same exact error. I > was not getting the error with Spark v1.1.0, though there weren't any > changes to the external/flume-sink/pom.xml between v1.1.0 and v1.2.1. > > > ~ Jonathan Kelly --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
