[
https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370762#comment-15370762
]
Thomas Graves commented on SPARK-15703:
---------------------------------------
sorry for my delay in responding, we reproduced this again and we aren't seeing
the "Dropping SparkListenerEvent".. message. [~Dhruve Ashar] is investigating
some more.
In 2.x EVENT_QUEUE_CAPACITY is 10000 and is hardcoded, I'm surprised this isn't
configurable. I'm actually a little concerned about this setting because in the
past if events weren't publish properly things got out of sync because other
components do their own tracking and rely on getting the events. Maybe that
has been fixed though.
> Spark UI doesn't show all tasks as completed when it should
> -----------------------------------------------------------
>
> Key: SPARK-15703
> URL: https://issues.apache.org/jira/browse/SPARK-15703
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.0.0
> Reporter: Thomas Graves
> Priority: Critical
> Attachments: Screen Shot 2016-06-01 at 11.21.32 AM.png, Screen Shot
> 2016-06-01 at 11.23.48 AM.png
>
>
> The Spark UI doesn't seem to be showing all the tasks and metrics.
> I ran a job with 100000 tasks but Detail stage page says it completed 93029:
> Summary Metrics for 93029 Completed Tasks
> The Stages for all jobs pages list that only 89519/100000 tasks finished but
> its completed. The metrics for shuffled write and input are also incorrect.
> I will attach screen shots.
> I checked the logs and it does show that all the tasks actually finished.
> 16/06/01 16:15:42 INFO TaskSetManager: Finished task 59880.0 in stage 2.0
> (TID 54038) in 265309 ms on 10.213.45.51 (100000/100000)
> 16/06/01 16:15:42 INFO YarnClusterScheduler: Removed TaskSet 2.0, whose tasks
> have all completed, from pool
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
