[
https://issues.apache.org/jira/browse/SPARK-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14263664#comment-14263664
]
Josh Rosen edited comment on SPARK-5023 at 1/3/15 10:57 PM:
------------------------------------------------------------
This sounds superficially similar to SPARK-4836; were there any failed stages
in your logs?
was (Author: joshrosen):
This sounds superficially similar to SPARK-4836; were there any failed stages
in your los?
> In Web UI job history, the total job duration is incorrect (much smaller than
> the sum of its stages)
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-5023
> URL: https://issues.apache.org/jira/browse/SPARK-5023
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.1.1, 1.2.0
> Environment: Amazon EC2 AMI r3.2xlarge, cluster of 20 to 50 nodes,
> running the ec2 provided scripts to create.
> Reporter: Eran Medan
>
> I'm running a long process using Spark + Graph and things look good on the
> 4040 job status UI, but when the job is done, when going to the history then
> the job total duration is much, much smaller than the total of its stages.
> The way I set logs up is this:
> val homeDir = sys.props("user.home")
> val logsPath = new File(homeDir,"sparkEventLogs")
> val conf = new SparkConf().setAppName("...")
> conf.set("spark.eventLog.enabled", "true").set("spark.eventLog.dir",
> logsPath.getCanonicalPath)
> for example job ID X - duration 0.2 s, but when I click the job and look at
> its stages, the sum of their duration is more than 15 minutes!
> (before the job was over, in the 4040 job status, the job duration was
> correct, it is only incorrect when its done and going to the logs)
> I hope I didn't configure something because I was very surprised no one
> reported it yet (I searched, but perhaps I missed it)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
