@patrick I think there is a bug...when this timeout happens then suddenly I see some negative ms numbers in spark ui....I tried to send a pic showing the negative ms numbers but it was rejected by mailing list...I will send it your gmail...
>From the archive I saw some more suggestions: >> It seems that these tuning have solved the problem (in spark-env.sh): export SPARK_DAEMON_JAVA_OPTS="-Dspark.worker.timeout=600 -Dspark.akka.timeout=200 -Dspark.shuffle.consolidateFiles=true" export SPARK_JAVA_OPTS="-Dspark.worker.timeout=600 -Dspark.akka.timeout=200 -Dspark.shuffle.consolidateFiles=true" >> For more understanding, it will be great if you could explain how are these two -Dspark.worker.timeout=600 -Dspark.akka.timeout=200 different than spark.storage.blockManagerSlaveTimeoutsMs Also what's the difference between worker timeout and akka timeout ? Thanks. Deb On Fri, Apr 4, 2014 at 10:07 PM, Patrick Wendell <[email protected]> wrote: > If you look in the Spark UI, do you see any garbage collection happening? > My best guess is that some of the executors are going into GC and they are > timing out. You can manually increase the timeout by setting the Spark conf: > > spark.storage.blockManagerSlaveTimeoutMs > > to a higher value. In your case it's setting this to 45000 or 45 seconds. > > > > > On Fri, Apr 4, 2014 at 5:52 PM, Debasish Das <[email protected]>wrote: > >> Hi, >> >> In my ALS runs I am noticing messages that complain about heart beats: >> >> 14/04/04 20:43:09 WARN BlockManagerMasterActor: Removing BlockManager >> BlockManagerId(17, machine1, 53419, 0) with no recent heart beats: 48476ms >> exceeds 45000ms >> 14/04/04 20:43:09 WARN BlockManagerMasterActor: Removing BlockManager >> BlockManagerId(12, machine2, 60714, 0) with no recent heart beats: 45328ms >> exceeds 45000ms >> 14/04/04 20:43:09 WARN BlockManagerMasterActor: Removing BlockManager >> BlockManagerId(19, machine3, 39496, 0) with no recent heart beats: 53259ms >> exceeds 45000ms >> >> Is this some issue with the underlying jvm over which akka is run ? Can I >> increase the heartbeat somehow to get these messages resolved ? >> >> Any more insight about the possible cause for the heartbeat will be >> helpful... >> >> It tried to re-run the job but it ultimately failed... >> >> Also I am noticing negative numbers in the stage duration: >> >> >> >> >> Any insights into the problem will be very helpful... >> >> Thanks. >> Deb >> > >
