Re: Terminal freeze during SVM
so I need to reconfigure my sparkcontext this way: val conf = new SparkConf() .setMaster(local) .setAppName(CountingSheep) .set(spark.executor.memory, 1g) .set(spark.akka.frameSize,20) val sc = new SparkContext(conf) And start a new cluster with the setup scripts from Spark 1.0.1. Is this the right approach? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Terminal-freeze-during-SVM-Broken-pipe-tp9022p9941.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Terminal freeze during SVM
Tried the newest branch, but still get stuck on the same task: (kill) runJob at SlidingRDD.scala:74 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Terminal-freeze-during-SVM-Broken-pipe-tp9022p9304.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Terminal freeze during SVM
news20.binary's feature dimension is 1.35M. So the serialized task size is above the default limit 10M. You need to set spark.akka.frameSize to, e.g, 20. Due to a bug SPARK-1112, this parameter is not passed to executors automatically, which causes Spark freezes. This was fixed in the latest master and v1.0.1-rc2. If you rebuild spark, remember to sync the assembly jar to workers. -Xiangrui On Thu, Jul 10, 2014 at 7:56 AM, AlexanderRiggers alexander.rigg...@gmail.com wrote: Tried the newest branch, but still get stuck on the same task: (kill) runJob at SlidingRDD.scala:74 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Terminal-freeze-during-SVM-Broken-pipe-tp9022p9304.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Terminal freeze during SVM
By latest branch you mean Apache Spark 1.0.0 ? and what do you mean by master? Because I am using v 1.0.0 - Alex -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Terminal-freeze-during-SVM-Broken-pipe-tp9022p9208.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Terminal freeze during SVM
It means pulling the code from latest development branch from git repository. On Jul 9, 2014 9:45 AM, AlexanderRiggers alexander.rigg...@gmail.com wrote: By latest branch you mean Apache Spark 1.0.0 ? and what do you mean by master? Because I am using v 1.0.0 - Alex -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Terminal-freeze-during-SVM-Broken-pipe-tp9022p9208.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
