The unit of "spark.akka.frameSize" is MB. The max value is 2047. Best Regards, Shixiong Zhu
2015-02-05 1:16 GMT+08:00 sahanbull <sa...@skimlinks.com>: > I am trying to run a spark application with > > -Dspark.executor.memory=30g -Dspark.kryoserializer.buffer.max.mb=2000 > -Dspark.akka.frameSize=10000 > > and the job fails because one or more of the akka frames are larger than > 10000mb (12000 ish). > > When I change the Dspark.akka.frameSize=10000 to 12000,15000 and 20000 and > RUN: > > ./spark/bin/spark-submit --driver-memory 30g --executor-memory 30g > mySparkCode.py > > I get an error in the startup as : > > > ERROR OneForOneStrategy: Cannot instantiate transport > [akka.remote.transport.netty.NettyTransport]. Make sure it extends > [akka.remote.transport.Transport] and ha > s constructor with [akka.actor.ExtendedActorSystem] and > [com.typesafe.config.Config] parameters > java.lang.IllegalArgumentException: Cannot instantiate transport > [akka.remote.transport.netty.NettyTransport]. Make sure it extends > [akka.remote.transport.Transport] and has const > ructor with [akka.actor.ExtendedActorSystem] and > [com.typesafe.config.Config] parameters > at > > akka.remote.EndpointManager$$anonfun$8$$anonfun$3.applyOrElse(Remoting.scala:620) > at > > akka.remote.EndpointManager$$anonfun$8$$anonfun$3.applyOrElse(Remoting.scala:618) > at > > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) > at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185) > at scala.util.Try$.apply(Try.scala:161) > at scala.util.Failure.recover(Try.scala:185) > at akka.remote.EndpointManager$$anonfun$8.apply(Remoting.scala:618) > at akka.remote.EndpointManager$$anonfun$8.apply(Remoting.scala:610) > at > > scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at > scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721) > at > > akka.remote.EndpointManager.akka$remote$EndpointManager$$listens(Remoting.scala:610) > at > > akka.remote.EndpointManager$$anonfun$receive$2.applyOrElse(Remoting.scala:450) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) > at akka.actor.ActorCell.invoke(ActorCell.scala:456) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > at > > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) > at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.lang.IllegalArgumentException: requirement failed: Setting > 'maximum-frame-size' must be at least 32000 bytes > at scala.Predef$.require(Predef.scala:233) > at > akka.util.Helpers$Requiring$.requiring$extension1(Helpers.scala:104) > at > > org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121) > at > org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) > at > org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) > at > > org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1446) > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) > at > org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1442) > at > org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56) > at org.apache.spark.SparkEnv$.create(SparkEnv.scala:153) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:203) > at > > org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:53) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) > at > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) > at py4j.Gateway.invoke(Gateway.java:214) > at > > py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) > at > py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) > at py4j.GatewayConnection.run(GatewayConnection.java:207) > at java.lang.Thread.run(Thread.java:745) > > Can anyone give me a clue about whats going wrong here?? > > I am running spark 1.1.0 in r3.2xlarge EC2 instances that come with 60GB > RAM. > > Many thanks > Sahan > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-changing-the-akka-framesize-parameter-tp21497.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
