This is all that I see related to spark.MapOutputTrackerMaster in the master logs after OOME
14/08/21 13:24:45 ERROR ActorSystemImpl: Uncaught fatal error from thread [spark-akka.actor.default-dispatcher-27] shutting down ActorSystem [spark] java.lang.OutOfMemoryError: Java heap space Exception in thread "Thread-59" org.apache.spark.SparkException: Error communicating with MapOutputTracker at org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:108) at org.apache.spark.MapOutputTracker.sendTracker(MapOutputTracker.scala:114) at org.apache.spark.MapOutputTrackerMaster.stop(MapOutputTracker.scala:319) at org.apache.spark.SparkEnv.stop(SparkEnv.scala:82) at org.apache.spark.SparkContext.stop(SparkContext.scala:984) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:449) Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://spark/user/MapOutputTracker#112553370]] had already been terminated. at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134) at org.apache.spark.MapOutputTracker.askTracker(MapOutputTracker.scala:104) > 2.Erery excutor will processing 10+TB/2000 = 5G data. ReduceByKey will > create a hashtable of unique lines form this 5G data and keep it in memory. > it is maybe exceeed 16G . So you mean the master gets that information from individual nodes and keeps it in memory? On Aug 21, 2014, at 8:18 PM, Nieyuan <qiushuiwuh...@gmail.com> wrote: > 1.At begining of reduce task , mask will deliver map output info to every > excutor. You can check stderr to find size of map output info . It should be > : > "spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is > xxx bytes" > > 2.Erery excutor will processing 10+TB/2000 = 5G data. ReduceByKey will > create a hashtable of unique lines form this 5G data and keep it in memory. > it is maybe exceeed 16G . > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/AppMaster-OOME-on-YARN-tp12612p12627.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org