There seems to be a version mismatch somewhere. You can try and find out the cause with debug serialization information. I think the jvm flag -Dsun.io.serialization.extendedDebugInfo=true should help.
Best Regards, Sonal Founder, Nube Technologies <http://www.nubetech.co> Check out Reifier at Spark Summit 2015 <https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/> <http://in.linkedin.com/in/sonalgoyal> On Fri, Aug 7, 2015 at 4:42 AM, Jeff Jones <jjo...@adaptivebiotech.com> wrote: > I wrote a very simple Spark 1.4.1 app that I can run through a local > driver program just fine using setMaster(“local[*]”). The app is as > follows: > > > > import org.apache.spark.SparkContext > > import org.apache.spark.SparkContext._ > > import org.apache.spark.SparkConf > > import org.apache.spark.rdd.RDD > > > > object Hello { > > def main(args: Array[String]): Unit = { > > val conf = new SparkConf().setAppName("Simple > Application").setMaster("local[*]") > > val sc = new SparkContext(conf) > > val data:RDD[Int] = sc.parallelize(Seq(1,2,12,34,2354,123,100), 2) > > println("Max: " + data.max) > > println("Min: " + data.min) > > } > > } > > > > I compile this using the following build.sbt which will pull the needed > Spark libraries for me. > > > > name := """SparkyJeff""" > > > > version := "1.0" > > > > scalaVersion := "2.11.6" > > > > // Change this to another test framework if you prefer > > libraryDependencies ++= Seq( > > "org.apache.spark" %% "spark-core" % "1.4.1", > > "org.apache.spark" %% "spark-sql" % "1.4.1") > > > > // Uncomment to use Akka > > //libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.11" > > > > fork := true > > > > Now I’m trying to run this against a standalone cluster by changing the > setMaster(“local[*]”) to setMaster(“spark://p3.ourdomain.com:7077”). I > downloaded Spark 1.4.1 for Hadoop 2.6 or greater. Set the SPARK_MASTER_IP=” > p3.ourdomain.com”, SPARK_WORKER_CORES=”1000”,SPARK_WORKER_MEMORY=”500g” > and then started the cluster using run-all.sh. The cluster appears to start > fine. I can hit cluster UI at p3.ourdomain.com:8080 and see the same > master URL as mentioned above. > > > > Now when I run my little app I get the following client error: > > > > … > > [error] 15/08/05 16:03:40 INFO AppClient$ClientActor: Connecting to master > akka.tcp://sparkmas...@p3.ourdomain.com:7077/user/Master... > > [error] 15/08/05 16:03:40 WARN ReliableDeliverySupervisor: Association > with remote system [akka.tcp://sparkmas...@p3.ourdomain.com:7077] has > failed, address is now gated for [5000] ms. Reason is: [Disassociated]. > > [error] 15/08/05 16:04:00 INFO AppClient$ClientActor: Connecting to master > akka.tcp://sparkmas...@p3.ourdomain.com:7077/user/Master... > > [error] 15/08/05 16:04:00 WARN ReliableDeliverySupervisor: Association > with remote system [akka.tcp://sparkmas...@p3.ourdomain.com:7077] has > failed, address is now gated for [5000] ms. Reason is: [Disassociated]. > > [error] 15/08/05 16:04:20 INFO AppClient$ClientActor: Connecting to master > akka.tcp://sparkmas...@p3.ourdomain.com:7077/user/Master... > > [error] 15/08/05 16:04:20 WARN ReliableDeliverySupervisor: Association > with remote system [akka.tcp://sparkmas...@p3.ourdomain.com:7077] has > failed, address is now gated for [5000] ms. Reason is: [Disassociated]. > > [error] 15/08/05 16:04:40 ERROR SparkDeploySchedulerBackend: Application > has been killed. Reason: All masters are unresponsive! Giving up. > > … > > > > Looking into the master logs I find: > > > > 15/08/06 22:52:28 INFO Master: akka.tcp://sparkDriver@192.168.137.41:48877 > got disassociated, removing it. > > 15/08/06 22:52:46 ERROR Remoting: org.apache.spark.deploy.Command; local > class incompatible: stream classdesc serialVersionUID = > -7098307370860582211, local class serialVersionUID = -3335312719467547622 > > java.io.InvalidClassException: org.apache.spark.deploy.Command; local > class incompatible: stream classdesc serialVersionUID = > -7098307370860582211, local class serialVersionUID = -3335312719467547622 > > at > java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621) > > at > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623) > > at > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774) > > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993) > > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918) > > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) > > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) > > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) > > at > akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136) > > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > > at > akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136) > > at > akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104) > > at scala.util.Try$.apply(Try.scala:161) > > at > akka.serialization.Serialization.deserialize(Serialization.scala:98) > > at > akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63) > > at > akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104) > > at scala.util.Try$.apply(Try.scala:161) > > at > akka.serialization.Serialization.deserialize(Serialization.scala:98) > > at > akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23) > > at > akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58) > > at > akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58) > > at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76) > > at > akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937) > > at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > > at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415) > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > > at akka.actor.ActorCell.invoke(ActorCell.scala:487) > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > > at akka.dispatch.Mailbox.run(Mailbox.scala:220) > > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > > at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > > Any help would be greatly appreciated. > > > > Thanks, > Jeff > > > This message (and any attachments) is intended only for the designated > recipient(s). It > may contain confidential or proprietary information, or have other > limitations on use as > indicated by the sender. If you are not a designated recipient, you may > not review, use, > copy or distribute this message. If you received this in error, please > notify the sender by > reply e-mail and delete this message. >
