[
https://issues.apache.org/jira/browse/SPARK-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-16811.
-------------------------------
Resolution: Invalid
Read https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
first please
> spark streaming-Exception in thread “submit-job-thread-pool-40”
> ---------------------------------------------------------------
>
> Key: SPARK-16811
> URL: https://issues.apache.org/jira/browse/SPARK-16811
> Project: Spark
> Issue Type: Question
> Components: Streaming
> Affects Versions: 1.6.1
> Environment: spark 1.6.1
> Reporter: BinXu
> Priority: Blocker
> Labels: maven
> Original Estimate: 1,176h
> Remaining Estimate: 1,176h
>
> {color:red}I am using spark 1.6. When I run streaming application for a few
> minutes,it cause such Exception Like this:{color}
> Exception in thread "submit-job-thread-pool-40" Exception in thread
> "submit-job-thread-pool-33" Exception in thread "submit-job-thread-pool-23"
> Exception in thread "submit-job-thread-pool-14" Exception in thread
> "submit-job-thread-pool-29" Exception in thread "submit-job-thread-pool-39"
> Exception in thread "submit-job-thread-pool-2" java.lang.Error:
> java.lang.InterruptedException
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> at
> org.apache.spark.SimpleFutureAction.org$apache$spark$SimpleFutureAction$$awaitResult(FutureAction.scala:165)
> at org.apache.spark.SimpleFutureAction$$anon$1.run(FutureAction.scala:147)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ... 2 more
> java.lang.Error: java.lang.InterruptedException
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> at
> org.apache.spark.SimpleFutureAction.org$apache$spark$SimpleFutureAction$$awaitResult(FutureAction.scala:165)
> at org.apache.spark.SimpleFutureAction$$anon$1.run(FutureAction.scala:147)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ... 2 more
> java.lang.Error: java.lang.InterruptedException
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> at
> org.apache.spark.SimpleFutureAction.org$apache$spark$SimpleFutureAction$$awaitResult(FutureAction.scala:165)
> at org.apache.spark.SimpleFutureAction$$anon$1.run(FutureAction.scala:147)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ... 2 more
> java.lang.Error: java.lang.InterruptedException
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> at
> org.apache.spark.SimpleFutureAction.org$apache$spark$SimpleFutureAction$$awaitResult(FutureAction.scala:165)
> at org.apache.spark.SimpleFutureAction$$anon$1.run(FutureAction.scala:147)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ... 2 more
> java.lang.Error: java.lang.InterruptedException
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
> at
> org.apache.spark.SimpleFutureAction.org$apache$spark$SimpleFutureAction$$awaitResult(FutureAction.scala:165)
> at org.apache.spark.SimpleFutureAction$$anon$1.run(FutureAction.scala:147)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ... 2 more
> {color:red}
> This problem is so wield,the consume speed is enough.My batch time is 20s,
> and the first few minutes(app is ok),the process time is 8s. But After
> that,the Exception will appear. The input partitions is 1500, the executor is
> 400,and spark cores is 4. Submit job with cluster mode like this:{color}
> *
> ../../bin/spark-submit --master yarn-cluster \
> --files logback.xml \
> --driver-memory 20g \
> --num-executors 400 \
> --executor-memory 8g \
> --executor-cores 4 \
> --driver-cores 6 \
> --queue spark-test \
> --class com.baidu.di.MapFlow
> ./target/di.core.index-1.0-SNAPSHOT.jar
> *
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
