[
https://issues.apache.org/jira/browse/SPARK-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068147#comment-16068147
]
igor mazor edited comment on SPARK-21252 at 6/29/17 10:42 AM:
--------------------------------------------------------------
Yes, its indeed possible to drill to individual stages, however the scenario I
described on my last comment is in the same stage.
The first attempt following by timeout after 20 seconds and then the second
successful attempt are all in the same stage and therefore the UI shows 20
seconds duration, although thats not true.
Also, if the stage was indeed 40 ms, the UI shows 40 ms as the duration, but
with seconds there is rounding which also causing for inconsistent result
presentation.
If you already do the rounding, why then the 40 ms is not rounded to 0 seconds ?
Also I noticed now that for example stage that took 42 ms (Job Duration), when
I click on that stage I see Duration = 38 ms and when going deeper into the
Summary Metrics for Tasks, I see that the longest task was 35 ms, so not sure
where to all the gaps went.
was (Author: mazor.igal):
Yes, its indeed possible to drill to individual stages, however the scenario I
described on my last comment is happens in the same stage.
The first attempt following by timeout after 20 seconds and then the second
successful attempt are all in the same stage and therefore the UI shows 20
seconds duration, although thats not true.
Also, if the stage was indeed 40 ms, the UI shows 40 ms as the duration, but
with seconds there is rounding which also causing for inconsistent result
presentation.
If you already do the rounding, why then the 40 ms is not rounded to 0 seconds ?
Also I noticed now that for example stage that took 42 ms (Job Duration), when
I click on that stage I see Duration = 38 ms and when going deeper into the
Summary Metrics for Tasks, I see that the longest task was 35 ms, so not sure
where to all the gaps went.
> The duration times showed by spark web UI are inaccurate
> --------------------------------------------------------
>
> Key: SPARK-21252
> URL: https://issues.apache.org/jira/browse/SPARK-21252
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1
> Reporter: igor mazor
>
> The duration times showed by spark UI are inaccurate and seems to be rounded.
> For example when a job had 2 stages, first stage executed in 47 ms and second
> stage in 3 seconds, the total execution time showed by the UI is 4 seconds.
> Another example, first stage was executed in 20 ms and second stage in 4
> seconds, the total execution time showed by the UI would be in that case also
> 4 seconds.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
