It took me hours to debug a problem yesterday on the latest master branch (0.9.0-SNAPSHOT), and I would like to share with the dev list in case anybody runs into this Akka problem.
A little background for those of you who haven't followed closely the development of Spark and YARN 2.2: YARN 2.2 uses protobuf 2.5, and Akka uses an older version of protobuf that is not binary compatible. In order to have a single build that is compatible for both YARN 2.2 and pre-2.2 YARN/Hadoop, we published a special version of Akka that builds with protobuf shaded (i.e. using a different package name for the protobuf stuff). However, it turned out Scala 2.10 includes a version of Akka jar in its default classpath (look at the lib folder in Scala 2.10 binary distribution). If you use the scala command to launch any Spark application on the current master branch, there is a pretty high chance that you wouldn't be able to create the SparkContext (stack trace at the end of the email). The problem is that the Akka packaged with Scala 2.10 takes precedence in the classloader over the special Akka version Spark includes. Before we have a good solution for this, the workaround is to use java to launch the application instead of scala. All you need to do is to include the right Scala jars (scala-library and scala-compiler) in the classpath. Note that the scala command is really just a simple script that calls java with the right classpath. Stack trace: java.lang.NoSuchMethodException: akka.remote.RemoteActorRefProvider.<init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.actor.DynamicAccess) at java.lang.Class.getConstructor0(Class.java:2763) at java.lang.Class.getDeclaredConstructor(Class.java:2021) at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:77) at scala.util.Try$.apply(Try.scala:161) at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:74) at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85) at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85) at scala.util.Success.flatMap(Try.scala:200) at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:85) at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:546) at akka.actor.ActorSystem$.apply(ActorSystem.scala:111) at akka.actor.ActorSystem$.apply(ActorSystem.scala:104) at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:79) at org.apache.spark.SparkEnv$.createFromSystemProperties(SparkEnv.scala:120) at org.apache.spark.SparkContext.<init>(SparkContext.scala:106)
