Re: spark 1.1.0/yarn hang

[Tian Zhang]

We have narrowed this hanging issue down to the calliope package 
that we used to create RDD from reading cassandra table.
The calliope native RDD interface seems hanging and I have decided to switch
to the calliope cql3 RDD interface.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-yarn-hang-tp16396p17087.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

spark 1.1.0/yarn hang

[tian zhang]

Hi, I have spark 1.1.0 yarn installation. I am using spark-submit to run a 
simple application.
From the console output, I have 769 partitions and after task 768 in stage 0 
(count) finished,
it hangs. I used jstack to dump the stacktop and it shows it is waiting ...

Any suggestion what might go wrong and how to debug this kind of hanging?

Thanks.

Tian


main prio=10 tid=0x7f6058009000 nid=0x7ecd in Object.wait() 
[0x7f605e4d9000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xfdd30500 (a 
org.apache.spark.scheduler.JobWaiter)
at java.lang.Object.wait(Object.java:503)
at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
- locked 0xfdd30500 (a org.apache.spark.scheduler.JobWaiter)
at 
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:511)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1088)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1107)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1121)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1135)
at org.apache.spark.rdd.RDD.count(RDD.scala:904)
at 
com.oncue.rna.realtime.streaming.spark.TopShowsToKafkaJob.getTopShows(TopShowsToKafkaJob.scala:29)
at 
com.oncue.rna.realtime.streaming.spark.TopShowsToKafkaJob.getTopShows(TopShowsToKafkaJob.scala:45)
at 
com.oncue.rna.realtime.streaming.spark.TopShowsToKafkaJob$$anonfun$5.apply(TopShowsToKafkaJob.scala:79)
at 
com.oncue.rna.realtime.streaming.spark.TopShowsToKafkaJob$$anonfun$5.apply(TopShowsToKafkaJob.scala:76)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at 
com.oncue.rna.realtime.streaming.spark.TopShowsToKafkaJob.processRecentWindow(TopShowsToKafkaJob.scala:76)
at 
com.oncue.rna.realtime.streaming.spark.TopShowsToKafkaJob$.processRecentWindow(TopShowsToKafkaJob.scala:98)
at 
com.oncue.rna.realtime.streaming.spark.TopShowsToKafkaJob$.main(TopShowsToKafkaJob.scala:112)
at 
com.oncue.rna.realtime.streaming.spark.TopShowsToKafkaJob.main(TopShowsToKafkaJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

   Locked ownable synchronizers:
- None