[ https://issues.apache.org/jira/browse/SPARK-26152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902477#comment-16902477 ]
Shixiong Zhu commented on SPARK-26152: -------------------------------------- [~ajithshetty] does your PR fix the flaky test? If I read correctly, the failure is because of OOM. RejectedExecutionException is just side effect because OOM triggers the executor shut down. Should we try to figure out why OOM happens? Maybe we have some memory leak. > Synchronize Worker Cleanup with Worker Shutdown > ----------------------------------------------- > > Key: SPARK-26152 > URL: https://issues.apache.org/jira/browse/SPARK-26152 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.4.0, 2.4.1, 2.4.2, 3.0.0, 2.4.3 > Reporter: Dongjoon Hyun > Assignee: Ajith S > Priority: Critical > Fix For: 2.4.4, 3.0.0 > > Attachments: Screenshot from 2019-03-11 17-03-40.png > > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/5627 > (2018-11-16) > {code} > BroadcastSuite: > - Using TorrentBroadcast locally > - Accessing TorrentBroadcast variables from multiple threads > - Accessing TorrentBroadcast variables in a local cluster (encryption = off) > java.util.concurrent.RejectedExecutionException: Task > scala.concurrent.impl.CallbackRunnable@59428a1 rejected from > java.util.concurrent.ThreadPoolExecutor@4096a677[Shutting down, pool size = > 1, active threads = 1, queued tasks = 0, completed tasks = 0] > at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > at > java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:134) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:63) > at > scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:78) > at > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) > at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) > at > scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:55) > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870) > at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:106) > at > scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > java.util.concurrent.RejectedExecutionException: Task > scala.concurrent.impl.CallbackRunnable@40a5bf17 rejected from > java.util.concurrent.ThreadPoolExecutor@5a73967[Shutting down, pool size = 1, > active threads = 1, queued tasks = 0, completed tasks = 0] > at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > at > java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:134) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:63) > at > scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:78) > at > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) > at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) > at > scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:55) > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870) > at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:106) > at > scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > java.util.concurrent.RejectedExecutionException: Task > scala.concurrent.impl.CallbackRunnable@7072dd4f rejected from > java.util.concurrent.ThreadPoolExecutor@35705d25[Shutting down, pool size = > 1, active threads = 1, queued tasks = 0, completed tasks = 0] > at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > at > java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:134) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:63) > at > scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:78) > at > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) > at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) > at > scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:55) > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870) > at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:106) > at > scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Exception in thread "RemoteBlock-temp-file-clean-thread" > java.lang.OutOfMemoryError: GC overhead limit exceeded > Exception in thread "dispatcher-event-loop-5" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "dispatcher-event-loop-12" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "dispatcher-event-loop-4" Exception in thread > "dispatcher-event-loop-9" java.lang.OutOfMemoryError: GC overhead limit > exceeded > java.lang.OutOfMemoryError: GC overhead limit exceeded > Exception in thread "dispatcher-event-loop-6" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "dispatcher-event-loop-12" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "dispatcher-event-loop-11" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "dispatcher-event-loop-13" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "dispatcher-event-loop-27" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "netty-rpc-env-timeout" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "Timer-1568" java.lang.OutOfMemoryError: GC overhead > limit exceeded > Exception in thread "dispatcher-event-loop-17" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "Spark Context Cleaner" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "dispatcher-event-loop-16" java.lang.OutOfMemoryError: GC > overhead limit exceeded > Exception in thread "stop-spark-context" java.lang.OutOfMemoryError: GC > overhead limit exceeded > *** RUN ABORTED *** > java.lang.OutOfMemoryError: GC overhead limit exceeded > ... > {code} > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/5645/ > {code} > BroadcastSuite: > - Using TorrentBroadcast locally > - Accessing TorrentBroadcast variables from multiple threads > - Accessing TorrentBroadcast variables in a local cluster (encryption = off) > *** FAILED *** > org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 > in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 > (TID 7, localhost, executor 1): java.io.IOException: > org.apache.spark.SparkException: Failed to get broadcast_1_piece0 of > broadcast_1 > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1333) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:208) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:84) > at org.apache.spark.scheduler.Task.run(Task.scala:121) > at > org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:425) > at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:428) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.spark.SparkException: Failed to get broadcast_1_piece0 > of broadcast_1 > at > org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBlocks$1(TorrentBroadcast.scala:179) > at > scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:12) > at scala.collection.immutable.List.foreach(List.scala:388) > at > org.apache.spark.broadcast.TorrentBroadcast.readBlocks(TorrentBroadcast.scala:151) > at > org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$3(TorrentBroadcast.scala:231) > at scala.Option.getOrElse(Option.scala:121) > at > org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$1(TorrentBroadcast.scala:211) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1326) > ... 13 more > Driver stacktrace: > at > org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:1926) > at > org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1914) > at > org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1913) > at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58) > at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1913) > at > org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:929) > at > org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:929) > at scala.Option.foreach(Option.scala:257) > ... > Cause: java.io.IOException: org.apache.spark.SparkException: Failed to get > broadcast_1_piece0 of broadcast_1 > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1333) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:208) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:84) > at org.apache.spark.scheduler.Task.run(Task.scala:121) > at > org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:425) > at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) > ... > Cause: org.apache.spark.SparkException: Failed to get broadcast_1_piece0 of > broadcast_1 > at > org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBlocks$1(TorrentBroadcast.scala:179) > at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:12) > at scala.collection.immutable.List.foreach(List.scala:388) > at > org.apache.spark.broadcast.TorrentBroadcast.readBlocks(TorrentBroadcast.scala:151) > at > org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$3(TorrentBroadcast.scala:231) > at scala.Option.getOrElse(Option.scala:121) > at > org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$1(TorrentBroadcast.scala:211) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1326) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:208) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66) > ... > java.util.concurrent.RejectedExecutionException: Task > scala.concurrent.impl.CallbackRunnable@68c6a53e rejected from > java.util.concurrent.ThreadPoolExecutor@2dc42642[Shutting down, pool size = > 1, active threads = 1, queued tasks = 0, completed tasks = 0] > at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > at > java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:134) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:63) > at > scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:78) > at > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) > at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) > at > scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:55) > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870) > at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:106) > at > scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > java.util.concurrent.RejectedExecutionException: Task > scala.concurrent.impl.CallbackRunnable@7e69ed97 rejected from > java.util.concurrent.ThreadPoolExecutor@3b6b7c3e[Shutting down, pool size = > 1, active threads = 1, queued tasks = 0, completed tasks = 0] > at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > at > java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:134) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:63) > at > scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:78) > at > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) > at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) > at > scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:55) > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870) > at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:106) > at > scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > java.util.concurrent.RejectedExecutionException: Task > scala.concurrent.impl.CallbackRunnable@2e07a335 rejected from > java.util.concurrent.ThreadPoolExecutor@6b052891[Shutting down, pool size = > 1, active threads = 1, queued tasks = 0, completed tasks = 0] > at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > at > java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) > at > scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:134) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:63) > at > scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:78) > at > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) > at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) > at > scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:55) > at > scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870) > at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:106) > at > scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) > at > scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) > at scala.concurrent.Promise.complete(Promise.scala:49) > at scala.concurrent.Promise.complete$(Promise.scala:48) > at > scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:183) > at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 11/20/18 5:35:25 PM > ============================================================ > -- Gauges > ---------------------------------------------------------------------- > master.aliveWorkers > 11/20/18 5:55:13 PM > ============================================================ > -- Gauges > ---------------------------------------------------------------------- > master.aliveWorkers > 11/20/18 6:14:49 PM > ============================================================ > -- Gauges > ---------------------------------------------------------------------- > master.aliveWorkers > 11/20/18 6:34:28 PM > ============================================================ > -- Gauges > ---------------------------------------------------------------------- > master.aliveWorkers > 11/20/18 6:54:30 PM > ============================================================ > -- Gauges > ---------------------------------------------------------------------- > master.aliveWorkers > 11/20/18 7:14:35 PM > ============================================================ > -- Gauges > ---------------------------------------------------------------------- > master.aliveWorkers > 11/20/18 7:34:31 PM > ============================================================ > -- Gauges > ---------------------------------------------------------------------- > master.aliveWorkers > - Accessing TorrentBroadcast variables in a local cluster (encryption = on) > *** FAILED *** > org.apache.spark.SparkException: Job 0 cancelled as part of cancellation of > all jobs > ... > {code} > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/5809/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org