Hi Utkarsh, I replied earlier asking what is your task assignment like with fine vs coarse grain mode look like?
Tim On Thu, Oct 1, 2015 at 4:05 PM, Utkarsh Sengar <utkarsh2...@gmail.com> wrote: > Bumping it up, its not really a blocking issue. > But fine grain mode eats up uncertain number of resources in mesos and > launches tons of tasks, so I would prefer using the coarse grained mode if > only it didn't run out of memory. > > Thanks, > -Utkarsh > > On Mon, Sep 28, 2015 at 2:24 PM, Utkarsh Sengar <utkarsh2...@gmail.com> > wrote: > >> Hi Tim, >> >> 1. spark.mesos.coarse:false (fine grain mode) >> This is the data dump for config and executors assigned: >> https://gist.github.com/utkarsh2012/6401d5526feccab14687 >> >> 2. spark.mesos.coarse:true (coarse grain mode) >> Dump for coarse mode: >> https://gist.github.com/utkarsh2012/918cf6f8ed5945627188 >> >> As you can see, exactly the same code works fine in fine grained, goes >> out of memory in coarse grained mode. First an executor was lost and then >> the driver went out of memory. >> So I am trying to understand what is different in fine grained vs coarse >> mode other than allocation of multiple mesos tasks vs 1 mesos task. Clearly >> spark is not managing memory in the same way. >> >> Thanks, >> -Utkarsh >> >> >> On Fri, Sep 25, 2015 at 9:17 AM, Tim Chen <t...@mesosphere.io> wrote: >> >>> Hi Utkarsh, >>> >>> What is your job placement like when you run fine grain mode? You said >>> coarse grain mode only ran with one node right? >>> >>> And when the job is running could you open the Spark webui and get stats >>> about the heap size and other java settings? >>> >>> Tim >>> >>> On Thu, Sep 24, 2015 at 10:56 PM, Utkarsh Sengar <utkarsh2...@gmail.com> >>> wrote: >>> >>>> Bumping this one up, any suggestions on the stacktrace? >>>> spark.mesos.coarse=true is not working and the driver crashed with the >>>> error. >>>> >>>> On Wed, Sep 23, 2015 at 3:29 PM, Utkarsh Sengar <utkarsh2...@gmail.com> >>>> wrote: >>>> >>>>> Missed to do a reply-all. >>>>> >>>>> Tim, >>>>> >>>>> spark.mesos.coarse = true doesn't work and spark.mesos.coarse = false >>>>> works (sorry there was a typo in my last email, I meant "when I do >>>>> "spark.mesos.coarse=false", the job works like a charm. "). >>>>> >>>>> I get this exception with spark.mesos.coarse = true: >>>>> >>>>> 15/09/22 20:18:05 INFO MongoCollectionSplitter: Created split: min={ >>>>> "_id" : "55af4bf26750ad38a444d7cf"}, max= { "_id" : >>>>> "55af5a61e8a42806f47546c1"} >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611337>15/09/22 >>>>> 20:18:05 INFO MongoCollectionSplitter: Created split: min={ "_id" : >>>>> "55af5a61e8a42806f47546c1"}, max= null >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611453>Exception >>>>> in thread "main" java.lang.OutOfMemoryError: Java heap space >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611524> >>>>> at org.apache.spark.rdd.CartesianRDD.getPartitions(CartesianRDD.scala:60) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611599> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611671> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611743> >>>>> at scala.Option.getOrElse(Option.scala:120) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611788> >>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611843> >>>>> at org.apache.spark.rdd.CartesianRDD.getPartitions(CartesianRDD.scala:60) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611918> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#611990> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612062> >>>>> at scala.Option.getOrElse(Option.scala:120) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612107> >>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612162> >>>>> at >>>>> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612245> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612317> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612389> >>>>> at scala.Option.getOrElse(Option.scala:120) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612434> >>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612489> >>>>> at >>>>> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612572> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612644> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612716> >>>>> at scala.Option.getOrElse(Option.scala:120) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612761> >>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612816> >>>>> at >>>>> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612899> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#612971> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613043> >>>>> at scala.Option.getOrElse(Option.scala:120) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613088> >>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613143> >>>>> at >>>>> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613226> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613298> >>>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613370> >>>>> at scala.Option.getOrElse(Option.scala:120) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613415> >>>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613470> >>>>> at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:82) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613537> >>>>> at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:78) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613612>15/09/22 >>>>> 20:18:17 INFO SparkContext: Invoking stop() from shutdown hook >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613684>15/09/22 >>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_2_piece0 on >>>>> some-ip-here:37706 in memory (size: 1964.0 B, free: 2.8 GB) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613814>15/09/22 >>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_2_piece0 on >>>>> mesos-slave10 >>>>> in memory (size: 1964.0 B, free: 5.2 GB) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#613977>15/09/22 >>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on >>>>> some-ip-here:37706 in memory (size: 17.2 KB, free: 2.8 GB) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614106>15/09/22 >>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on >>>>> mesos-slave105 in memory (size: 17.2 KB, free: 5.2 GB) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614268>15/09/22 >>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on mesos-slave1 >>>>> in memory (size: 17.2 KB, free: 5.2 GB) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614429>15/09/22 >>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on mesos-slave9 >>>>> in memory (size: 17.2 KB, free: 5.2 GB) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614590>15/09/22 >>>>> 20:18:17 INFO BlockManagerInfo: Removed broadcast_1_piece0 on mesos-slave3 >>>>> in memory (size: 17.2 KB, free: 5.2 GB) >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614751>15/09/22 >>>>> 20:18:17 INFO SparkUI: Stopped Spark web UI at >>>>> http://some-ip-here:4040 >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614831>15/09/22 >>>>> 20:18:17 INFO DAGScheduler: Stopping DAGScheduler >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614890>15/09/22 >>>>> 20:18:17 INFO CoarseMesosSchedulerBackend: Shutting down all executors >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#614970>15/09/22 >>>>> 20:18:17 INFO CoarseMesosSchedulerBackend: Asking each executor to shut >>>>> down >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615056>I0922 >>>>> 20:18:17.794598 171 sched.cpp:1591] Asked to stop the driver >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615125>I0922 >>>>> 20:18:17.794739 143 sched.cpp:835] Stopping framework >>>>> '20150803-224832-1577534986-5050-1614-0016' >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615231>15/09/22 >>>>> 20:18:17 INFO CoarseMesosSchedulerBackend: driver.run() returned with code >>>>> DRIVER_STOPPED >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615330>15/09/22 >>>>> 20:18:17 INFO MapOutputTrackerMasterEndpoint: >>>>> MapOutputTrackerMasterEndpoint stopped! >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615425>15/09/22 >>>>> 20:18:17 INFO Utils: path = >>>>> /tmp/spark-98801318-9c49-473b-bf2f-07ea42187252/blockmgr-0e0e1a1c-894e-4e79-beac-ead0dff43166, >>>>> already present as root for deletion. >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615595>15/09/22 >>>>> 20:18:17 INFO MemoryStore: MemoryStore cleared >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615651>15/09/22 >>>>> 20:18:17 INFO BlockManager: BlockManager stopped >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615709>15/09/22 >>>>> 20:18:17 INFO BlockManagerMaster: BlockManagerMaster stopped >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615779>15/09/22 >>>>> 20:18:17 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: >>>>> OutputCommitCoordinator stopped! >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615892>15/09/22 >>>>> 20:18:17 INFO SparkContext: Successfully stopped SparkContext >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#615963>15/09/22 >>>>> 20:18:17 INFO Utils: Shutdown hook called >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#616014>15/09/22 >>>>> 20:18:17 INFO Utils: Deleting directory >>>>> /tmp/spark-98801318-9c49-473b-bf2f-07ea42187252 >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#616111>15/09/22 >>>>> 20:18:17 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down >>>>> remote daemon. >>>>> >>>>> <http://singularity-qa-uswest2.otenv.com/task/ds-tetris-simspark-usengar.2015.09.22T20.14.36-1442952963980-1-mesos_slave1_qa_uswest2.qasql.opentable.com-us_west_2a/tail/stderr#616206>15/09/22 >>>>> 20:18:17 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon >>>>> shut >>>>> down; proceeding with flushing remote transports. >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Sep 22, 2015 at 1:26 AM, Tim Chen <t...@mesosphere.io> wrote: >>>>> >>>>>> Hi Utkarsh, >>>>>> >>>>>> Just to be sure you originally set coarse to false but then to true? >>>>>> Or is it the other way around? >>>>>> >>>>>> Also what's the exception/stack trace when the driver crashed? >>>>>> >>>>>> Coarse grain mode per-starts all the Spark executor backends, so has >>>>>> the least overhead comparing to fine grain. There is no single answer for >>>>>> which mode you should use, otherwise we would have removed one of those >>>>>> modes since it depends on your use case. >>>>>> >>>>>> There are quite some factor why there could be huge GC pauses, but I >>>>>> don't think if you switch to standalone your GC pauses go away. >>>>>> >>>>>> Tim >>>>>> >>>>>> On Mon, Sep 21, 2015 at 5:18 PM, Utkarsh Sengar < >>>>>> utkarsh2...@gmail.com> wrote: >>>>>> >>>>>>> I am running Spark 1.4.1 on mesos. >>>>>>> >>>>>>> The spark job does a "cartesian" of 4 RDDs (aRdd, bRdd, cRdd, dRdd) >>>>>>> of size 100, 100, 7 and 1 respectively. Lets call it prouctRDD. >>>>>>> >>>>>>> Creation of "aRdd" needs data pull from multiple data sources, >>>>>>> merging it and creating a tuple of JavaRdd, finally aRDD looks something >>>>>>> like this: JavaRDD<Tuple4<A1, A2>> >>>>>>> bRdd, cRdd and dRdds are just List<> of values. >>>>>>> >>>>>>> Then apply a transformation on prouctRDD and finally call >>>>>>> "saveAsTextFile" to save the result of my transformation. >>>>>>> >>>>>>> Problem: >>>>>>> By setting "spark.mesos.coarse=true", creation of "aRdd" works fine >>>>>>> but driver crashes while doing the cartesian but when I do >>>>>>> "spark.mesos.coarse=true", the job works like a charm. I am running >>>>>>> spark >>>>>>> on mesos. >>>>>>> >>>>>>> Comments: >>>>>>> So I wanted to understand what role does "spark.mesos.coarse=true" >>>>>>> plays in terms of memory and compute performance. My findings look >>>>>>> counter >>>>>>> intuitive since: >>>>>>> >>>>>>> 1. "spark.mesos.coarse=true" just runs on 1 mesos task, so there >>>>>>> should be an overhead of spinning up mesos tasks which should impact >>>>>>> the >>>>>>> performance. >>>>>>> 2. What config for "spark.mesos.coarse" recommended for running >>>>>>> spark on mesos? Or there is no best answer and it depends on usecase? >>>>>>> 3. Also by setting "spark.mesos.coarse=true", I notice that I >>>>>>> get huge GC pauses even with small dataset but a long running job >>>>>>> (but this >>>>>>> can be a separate discussion). >>>>>>> >>>>>>> Let me know if I am missing something obvious, we are learning spark >>>>>>> tuning as we move forward :) >>>>>>> >>>>>>> -- >>>>>>> Thanks, >>>>>>> -Utkarsh >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> -Utkarsh >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> -Utkarsh >>>> >>> >>> >> >> >> -- >> Thanks, >> -Utkarsh >> > > > > -- > Thanks, > -Utkarsh >
