LantaoJin commented on pull request #31119: URL: https://github.com/apache/spark/pull/31119#issuecomment-757647778
Without the patch, the driver log ``` 21/01/10 21:02:33,406 INFO [HiveServer2-Handler-Pool: Thread-255] thriftserver.SparkExecuteStatementOperation:57 : Submitting query 'select s... where rn<=2' with e5a71a11-817e-43f0-b0ae-083313d71a4b 21/01/10 21:02:33,411 INFO [HiveServer2-Background-Pool: Thread-260] query:258 : Running query [e5a71a11-817e-43f0-b0ae-083313d71a4b] in session [36ded59a-d5c6-45f0-b83b-d83034085d13] 'select ... where rn<=2' 21/01/10 21:02:43,269 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler:57 : Submitting ResultStage 0 (MapPartitionsRDD[7] at $anonfun$withThreadLocalCaptured$1 at FutureTask.java:266), which has no missing parents 21/01/10 21:02:43,362 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler:57 : Submitting ResultStage 1 (MapPartitionsRDD[6] at $anonfun$withThreadLocalCaptured$1 at FutureTask.java:266), which has no missing parents 21/01/10 21:02:53,543 INFO [HiveServer2-Handler-Pool: Thread-255] thriftserver.SparkExecuteStatementOperation:57 : Cancel query with e5a71a11-817e-43f0-b0ae-083313d71a4b 21/01/10 21:02:53,543 INFO [spark-listener-group-appStatus] ui.SQLAppStatusListener:434 : [TaskProgress] execution ends active tasks: 0, completed tasks: 0, total tasks: 462. 21/01/10 21:02:53,543 INFO [HiveServer2-Handler-Pool: Thread-255] scheduler.DAGScheduler:57 : Asked to cancel job group e5a71a11-817e-43f0-b0ae-083313d71a4b 21/01/10 21:02:53,545 INFO [HiveServer2-Background-Pool: Thread-260] scheduler.DAGScheduler:57 : Asked to cancel job group e5a71a11-817e-43f0-b0ae-083313d71a4b ``` With the patch, the related driver log, we can see the Job 0 and Job 1 failed due to cancelling. ``` 21/01/10 20:43:13,250 INFO [HiveServer2-Handler-Pool: Thread-257] thriftserver.SparkExecuteStatementOperation:57 : Submitting query 'select ... where rn<=2' with c971270f-231d-42d5-af35-86463416861d 21/01/10 20:43:13,252 INFO [HiveServer2-Background-Pool: Thread-263] query:258 : Running query [c971270f-231d-42d5-af35-86463416861d] in session [692f89f2-087d-4879-94d9-605c7c6e6c6c] 'select ... where rn<=2' 21/01/10 20:43:19,063 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler:57 : Submitting ResultStage 0 (MapPartitionsRDD[7] at $anonfun$withThreadLocalCaptured$1 at FutureTask.java:266), which has no missing parents 21/01/10 20:43:19,175 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler:57 : Submitting ResultStage 1 (MapPartitionsRDD[6] at $anonfun$withThreadLocalCaptured$1 at FutureTask.java:266), which has no missing parents 21/01/10 20:43:23,273 INFO [HiveServer2-Handler-Pool: Thread-257] thriftserver.SparkExecuteStatementOperation:57 : Cancel query with c971270f-231d-42d5-af35-86463416861d 21/01/10 20:43:23,273 INFO [HiveServer2-Handler-Pool: Thread-257] scheduler.DAGScheduler:57 : Asked to cancel job group c971270f-231d-42d5-af35-86463416861d 21/01/10 20:43:23,274 INFO [spark-listener-group-appStatus] ui.SQLAppStatusListener:434 : [TaskProgress] execution ends active tasks: 0, completed tasks: 0, total tasks: 462. 21/01/10 20:43:23,276 INFO [HiveServer2-Background-Pool: Thread-263] scheduler.DAGScheduler:57 : Asked to cancel job group c971270f-231d-42d5-af35-86463416861d 21/01/10 20:43:23,278 INFO [dag-scheduler-event-loop] cluster.YarnClusterScheduler:57 : Cancelling stage 0 21/01/10 20:43:23,279 INFO [dag-scheduler-event-loop] cluster.YarnClusterScheduler:57 : Killing all running tasks in stage 0: Stage cancelled 21/01/10 20:43:23,280 INFO [HiveServer2-Handler-Pool: Thread-257] scheduler.DAGScheduler:57 : Asked to cancel job group c971270f-231d-42d5-af35-86463416861d 21/01/10 20:43:23,280 INFO [HiveServer2-Handler-Pool: Thread-257] thriftserver.SparkExecuteStatementOperation:57 : Close statement with c971270f-231d-42d5-af35-86463416861d 21/01/10 20:43:23,284 INFO [dag-scheduler-event-loop] cluster.YarnClusterScheduler:57 : Removed TaskSet 0.0, whose tasks have all completed, from pool default 21/01/10 20:43:23,285 INFO [dag-scheduler-event-loop] cluster.YarnClusterScheduler:57 : Stage 0 was cancelled 21/01/10 20:43:23,285 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler:57 : ResultStage 0 ($anonfun$withThreadLocalCaptured$1 at FutureTask.java:266) failed in 4.194 s due to Job 0 cancelled part of cancelled job group c971270f-231d -42d5-af35-86463416861d. 21/01/10 20:43:23,291 INFO [broadcast-exchange-0] scheduler.DAGScheduler:57 : Job 0 failed: $anonfun$withThreadLocalCaptured$1 at FutureTask.java:266, took 4.257828 s 21/01/10 20:43:23,292 INFO [dag-scheduler-event-loop] cluster.YarnClusterScheduler:57 : Cancelling stage 1 21/01/10 20:43:23,293 INFO [dag-scheduler-event-loop] cluster.YarnClusterScheduler:57 : Killing all running tasks in stage 1: Stage cancelled 21/01/10 20:43:23,293 INFO [dag-scheduler-event-loop] cluster.YarnClusterScheduler:57 : Removed TaskSet 1.0, whose tasks have all completed, from pool default 21/01/10 20:43:23,293 INFO [dag-scheduler-event-loop] cluster.YarnClusterScheduler:57 : Stage 1 was cancelled 21/01/10 20:43:23,294 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler:57 : ResultStage 1 ($anonfun$withThreadLocalCaptured$1 at FutureTask.java:266) failed in 4.108 s due to Job 1 cancelled part of cancelled job group c971270f-231d -42d5-af35-86463416861d. 21/01/10 20:43:23,295 INFO [broadcast-exchange-1] scheduler.DAGScheduler:57 : Job 1 failed: $anonfun$withThreadLocalCaptured$1 at FutureTask.java:266, took 4.261140 s ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
