I confirmed that this has nothing to do with BigTop by running the same mvn command directly in a fresh clone of the Spark package at the v1.2.1 tag. I got the same exact error.
Jonathan Kelly Elastic MapReduce - SDE Port 99 (SEA35) 08.220.C2 From: <Kelly>, Jonathan Kelly <jonat...@amazon.com<mailto:jonat...@amazon.com>> Date: Thursday, March 5, 2015 at 10:39 AM To: "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Spark v1.2.1 failing under BigTop build in External Flume Sink (due to missing Netty library) I'm running into an issue building Spark v1.2.1 (as well as the latest in branch-1.2 and v1.3.0-rc2 and the latest in branch-1.3) with BigTop (v0.9, which is not quite released yet). The build fails in the External Flume Sink subproject with the following error: [INFO] Compiling 5 Scala sources and 3 Java sources to /workspace/workspace/bigtop.spark-rpm/build/spark/rpm/BUILD/spark-1.3.0/external/flume-sink/target/scala-2.10/classes... [WARNING] Class org.jboss.netty.channel.ChannelFactory not found - continuing with a stub. [ERROR] error while loading NettyServer, class file '/home/ec2-user/.m2/repository/org/apache/avro/avro-ipc/1.7.6/avro-ipc-1.7.6.jar(org/apache/avro/ipc/NettyServer.class)' is broken (class java.lang.NullPointerException/null) [WARNING] one warning found [ERROR] one error found It seems like what is happening is that the Netty library is missing at build time, which happens because it is explicitly excluded in the pom.xml (see https://github.com/apache/spark/blob/v1.2.1/external/flume-sink/pom.xml#L42). I attempted removing the exclusions and the explicit re-add for the test scope on lines 77-88, and that allowed the build to succeed, though I don't know if that will cause problems at runtime. I don't have any experience with the Flume Sink, so I don't really know how to test it. (And, to be clear, I'm not necessarily trying to get the Flume Sink to work-- I just want the project to build successfully, though of course I'd still want the Flume Sink to work for whomever does need it.) Does anybody have any idea what's going on here? Here is the command BigTop is running to build Spark: mvn -Pbigtop-dist -Pyarn -Phive -Phive-thriftserver -Pkinesis-asl -Divy.home=/home/ec2-user/.ivy2 -Dsbt.ivy.home=/home/ec2-user/.ivy2 -Duser.home=/home/ec2-user -Drepo.maven.org= -Dreactor.repo=file:///home/ec2-user/.m2/repository -Dhadoop.version=2.4.0-amzn-3-SNAPSHOT -Dyarn.version=2.4.0-amzn-3-SNAPSHOT -Dprotobuf.version=2.5.0 -Dscala.version=2.10.3 -Dscala.binary.version=2.10 -DskipTests -DrecompileMode=all install As I mentioned above, if I switch to the latest in branch-1.2, to v1.3.0-rc2, or to the latest in branch-1.3, I get the same exact error. I was not getting the error with Spark v1.1.0, though there weren't any changes to the external/flume-sink/pom.xml between v1.1.0 and v1.2.1. ~ Jonathan Kelly
