Hi Ted, That looks exactly what happens. It has been 5 hrs now. The code was built for 1.4. Thank you very much!
Best Regards, Jerry Sent from my iPhone > On 14 Nov, 2015, at 11:21 pm, Ted Yu <yuzhih...@gmail.com> wrote: > > Which release are you using ? > If older than 1.5.0, you miss some fixes such as SPARK-9952 > > Cheers > >> On Sat, Nov 14, 2015 at 6:35 PM, Jerry Lam <chiling...@gmail.com> wrote: >> Hi spark users and developers, >> >> Have anyone experience the slow startup of a job when it contains a stage >> with over 4 millions of tasks? >> The job has been pending for 1.4 hours without doing anything (please refer >> to the attached pictures). However, the driver is busy doing something. >> jstack the driver and I found the following relevant: >> >> ``` >> "dag-scheduler-event-loop" daemon prio=10 tid=0x00007f24a8c59800 nid=0x454 >> runnable [0x00007f23b3e29000] >> java.lang.Thread.State: RUNNABLE >> at >> org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:231) >> at >> org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:231) >> at scala.Option.getOrElse(Option.scala:120) >> at org.apache.spark.rdd.RDD.preferredLocations(RDD.scala:230) >> at >> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:1399) >> at >> org.apache.spark.scheduler.DAGScheduler.getPreferredLocs(DAGScheduler.scala:1373) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$16.apply(DAGScheduler.scala:911) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$16.apply(DAGScheduler.scala:910) >> at >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at scala.collection.Iterator$class.foreach(Iterator.scala:727) >> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) >> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) >> at scala.collection.AbstractIterable.foreach(Iterable.scala:54) >> at >> scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >> at scala.collection.AbstractTraversable.map(Traversable.scala:105) >> at >> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:910) >> at >> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:834) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:837) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:836) >> at scala.collection.immutable.List.foreach(List.scala:318) >> at >> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:836) >> at >> org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:818) >> at >> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1453) >> at >> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1445) >> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) >> ``` >> >> It seems that it takes long time for the driver to create/schedule the DAG >> for that many tasks. Is there a way to speed it up? >> >> Best Regards, >> >> Jerry >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >
