sharkd tu created SPARK-17664: --------------------------------- Summary: Failed to saveAsHadoop when speculate is enabled Key: SPARK-17664 URL: https://issues.apache.org/jira/browse/SPARK-17664 Project: Spark Issue Type: Bug Affects Versions: 1.6.1 Reporter: sharkd tu
>From follow logs, task 22 has failed 4 times because of "the driver did not >authorize commit". But the strange thing was thst I could't find task 22.1. >Why? Maybe some synchronization error? ---------------------------------------------------------------------------- 16/09/26 02:14:18 INFO TaskSetManager: Lost task 22.0 in stage 1856.0 (TID 953902) on executor 10.196.131.13: java.security.PrivilegedActionException (null) [duplicate 4] 16/09/26 02:14:18 INFO TaskSetManager: Marking task 22 in stage 1856.0 (on 10.196.131.13) as speculatable because it ran more than 5601 ms 16/09/26 02:14:18 INFO TaskSetManager: Starting task 22.2 in stage 1856.0 (TID 954074, 10.215.143.14, partition 22,PROCESS_LOCAL, 2163 bytes) 16/09/26 02:14:18 INFO TaskSetManager: Lost task 22.2 in stage 1856.0 (TID 954074) on executor 10.215.143.14: java.security.PrivilegedActionException (null) [duplicate 5] 16/09/26 02:14:18 INFO TaskSetManager: Marking task 22 in stage 1856.0 (on 10.196.131.13) as speculatable because it ran more than 5601 ms 16/09/26 02:14:18 INFO TaskSetManager: Starting task 22.3 in stage 1856.0 (TID 954075, 10.196.131.28, partition 22,PROCESS_LOCAL, 2163 bytes) 16/09/26 02:14:19 INFO TaskSetManager: Lost task 22.3 in stage 1856.0 (TID 954075) on executor 10.196.131.28: java.security.PrivilegedActionException (null) [duplicate 6] 16/09/26 02:14:19 INFO TaskSetManager: Marking task 22 in stage 1856.0 (on 10.196.131.13) as speculatable because it ran more than 5601 ms 16/09/26 02:14:19 INFO TaskSetManager: Starting task 22.4 in stage 1856.0 (TID 954076, 10.215.153.225, partition 22,PROCESS_LOCAL, 2163 bytes) 16/09/26 02:14:19 INFO TaskSetManager: Lost task 22.4 in stage 1856.0 (TID 954076) on executor 10.215.153.225: java.security.PrivilegedActionException (null) [duplicate 7] 16/09/26 02:14:19 ERROR TaskSetManager: Task 22 in stage 1856.0 failed 4 times; aborting job 16/09/26 02:14:19 INFO YarnClusterScheduler: Cancelling stage 1856 16/09/26 02:14:19 INFO YarnClusterScheduler: Stage 1856 was cancelled 16/09/26 02:14:19 INFO DAGScheduler: ResultStage 1856 (saveAsHadoopFile at TDWProvider.scala:514) failed in 23.049 s 16/09/26 02:14:19 INFO DAGScheduler: Job 76 failed: saveAsHadoopFile at TDWProvider.scala:514, took 69.865181 s 16/09/26 02:14:19 ERROR ApplicationMaster: User class threw exception: java.security.PrivilegedActionException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 22 in stage 1856.0 failed 4 times, most recent failure: Lost task 22.4 in stage 1856.0 (TID 954076, 10.215.153.225): java.security.PrivilegedActionException: org.apache.spark.executor.CommitDeniedException: attempt_201609260213_1856_m_000022_954076: Not committed because the driver did not authorize commit at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1723) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1284) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1282) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.spark.executor.CommitDeniedException: attempt_201609260213_1856_m_000022_954076: Not committed because the driver did not authorize commit at org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:135) at org.apache.spark.SparkHadoopWriter.commit(SparkHadoopWriter.scala:142) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anon$4.run(PairRDDFunctions.scala:1311) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anon$4.run(PairRDDFunctions.scala:1284) ... 11 more -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org