Re: spark.akka.frameSize setting problem
I meet the same problem, did you solve it ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-akka-frameSize-setting-problem-tp3416p20063.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: spark.akka.frameSize setting problem
4096MB is greater than Int.MaxValue and it will be overflow in Spark. Please set it less then 4096. Best Regards, Shixiong Zhu 2014-12-01 13:14 GMT+08:00 Ke Wang jkx...@gmail.com: I meet the same problem, did you solve it ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-akka-frameSize-setting-problem-tp3416p20063.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: spark.akka.frameSize setting problem
Sorry. Should be not greater than 2048. 2047 is the greatest value. Best Regards, Shixiong Zhu 2014-12-01 13:20 GMT+08:00 Shixiong Zhu zsxw...@gmail.com: 4096MB is greater than Int.MaxValue and it will be overflow in Spark. Please set it less then 4096. Best Regards, Shixiong Zhu 2014-12-01 13:14 GMT+08:00 Ke Wang jkx...@gmail.com: I meet the same problem, did you solve it ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-akka-frameSize-setting-problem-tp3416p20063.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: spark.akka.frameSize setting problem
Created a JIRA to track it: https://issues.apache.org/jira/browse/SPARK-4664 Best Regards, Shixiong Zhu 2014-12-01 13:22 GMT+08:00 Shixiong Zhu zsxw...@gmail.com: Sorry. Should be not greater than 2048. 2047 is the greatest value. Best Regards, Shixiong Zhu 2014-12-01 13:20 GMT+08:00 Shixiong Zhu zsxw...@gmail.com: 4096MB is greater than Int.MaxValue and it will be overflow in Spark. Please set it less then 4096. Best Regards, Shixiong Zhu 2014-12-01 13:14 GMT+08:00 Ke Wang jkx...@gmail.com: I meet the same problem, did you solve it ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-akka-frameSize-setting-problem-tp3416p20063.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
spark.akka.frameSize setting problem
Hi, I just run a simple example to generate some data for the ALS algorithm. my spark version is 0.9, and in local mode, the memory of my node is 108G but when I set conf.set(spark.akka.frameSize, 4096), it then occurred the following problem, and when I do not set this, it runs well . I see the Meaning of the ***spark.akka.frameSize is the maximum message size, and it in MB.* *From the error info, I guess it is the frameSize too large that the startup timed out, but the conf meaning is the Maximum, not the init, and what confuse me is the Setting 'maximum-frame-size' must be at least 32000 bytes info, because my frame-size is greater than 32000. can anyone explain this? or maybe the conf document should specify more clearly?* *ERROR OneForOneStrategy: Cannot instantiate transport [akka.remote.transport.netty.NettyTransport]. Make sure it extends [akka.remote.transport.Transport] and has constructor with [akka.actor.ExtendedActorSystem] and [com.typesafe.config.Config] parameters* *java.lang.IllegalArgumentException: Cannot instantiate transport [akka.remote.transport.netty.NettyTransport]. Make sure it extends [akka.remote.transport.Transport] and has constructor with [akka.actor.ExtendedActorSystem] and [com.typesafe.config.Config] parameters* * at akka.remote.EndpointManager$$anonfun$8$$anonfun$3.applyOrElse(Remoting.scala:620)* * at akka.remote.EndpointManager$$anonfun$8$$anonfun$3.applyOrElse(Remoting.scala:618)* *...* * at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)* * at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)* *Caused by: java.lang.IllegalArgumentException: requirement failed: Setting 'maximum-frame-size' must be at least 32000 bytes* * at scala.Predef$.require(Predef.scala:233)* * at akka.util.Helpers$Requiring$.requiring$extension1(Helpers.scala:104)* * ...* * at scala.util.Success.flatMap(Try.scala:200)* * at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)* * ... 19 more* *14/03/28 07:25:27 ERROR Remoting: Remoting error: [Startup timed out] [* *akka.remote.RemoteTransportException: Startup timed out* * at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129)* * at akka.remote.Remoting.start(Remoting.scala:191)* * ...* * at sbt.Logger$$anon$4.apply(Logger.scala:90)* * at sbt.TrapExit$App.run(TrapExit.scala:244)* * at java.lang.Thread.run(Thread.java:744)* *Caused by: java.util.concurrent.TimeoutException: Futures timed out after [1 milliseconds]* * at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)* * ...* * at akka.remote.Remoting.start(Remoting.scala:173)*
