Github user markgrover commented on a diff in the pull request: https://github.com/apache/spark/pull/8093#discussion_r36783358 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala --- @@ -110,6 +114,7 @@ private[spark] abstract class YarnSchedulerBackend( addWebUIFilter(filterName, filterParams, proxyBase) case RemoveExecutor(executorId, reason) => + logWarning(reason) --- End diff -- This isn't actually. That's really the line that shows us that the Container was exited due to YARN killing the container. If you go debug, you can see all the Akka communication and find it out that way, but there's nothing info or harsher that logs this. I thought about warning being a little too much here but the only WARN messages being logged here, as far as I can tell are legitimate kills by YARN (including a snippet below) so unless you disagree, I am inclined to keep it at WARN. ```` Marks-MacBook-Pro-2:new mgrover$ grep -ri "WARN YarnSchedulerBackend$YarnSchedulerEndpoint" c.out c.out:15/08/09 08:24:39 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 1.4 GB of 1.4 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. c.out:15/08/09 08:24:39 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Yarn deallocated the executor 1 (container container_1438872994881_0029_01_000002). Reason: Container [pid=11659,containerID=container_1438872994881_0029_01_000002] is running beyond physical memory limits. Current usage: 1.4 GB of 1.4 GB physical memory used; 2.5 GB of 2.9 GB virtual memory used. Killing container. c.out:15/08/09 08:24:42 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 1.6 GB of 1.4 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. c.out:15/08/09 08:24:42 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Yarn deallocated the executor 2 (container container_1438872994881_0029_01_000003). Reason: Container [pid=29045,containerID=container_1438872994881_0029_01_000003] is running beyond physical memory limits. Current usage: 1.6 GB of 1.4 GB physical memory used; 2.6 GB of 2.9 GB virtual memory used. Killing container. c.out:15/08/09 08:24:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 1.6 GB of 1.4 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. c.out:15/08/09 08:24:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Yarn deallocated the executor 4 (container container_1438872994881_0029_01_000005). Reason: Container [pid=29121,containerID=container_1438872994881_0029_01_000005] is running beyond physical memory limits. Current usage: 1.6 GB of 1.4 GB physical memory used; 2.7 GB of 2.9 GB virtual memory used. Killing container. ````
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org