[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[ https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137336#comment-15137336 ] Apache Spark commented on SPARK-7889: - User 'squito' has created a pull request for this issue: https://github.com/apache/spark/pull/8 > Jobs progress of apps on complete page of HistoryServer shows uncompleted > - > > Key: SPARK-7889 > URL: https://issues.apache.org/jira/browse/SPARK-7889 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: meiyoula >Priority: Minor > > When running a SparkPi with 2000 tasks, cliking into the app on incomplete > page, the job progress shows 400/2000. After the app is completed, the app > goes to complete page from incomplete, and now cliking into the app, the job > progress still shows 400/2000. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[ https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15117288#comment-15117288 ] Steve Loughran commented on SPARK-7889: --- I'm going to note something something problematic about probing HDFS for changes, HDFS-5478 reports that flushing fs data doesn't immediately update fs length. So detecting changes in log files saved to HDFS isn't likely to be that immediate. the SPARK-1537 timeline binding won't have this problem, nor will logs saved to other filesystems (object stores have different issues) > Jobs progress of apps on complete page of HistoryServer shows uncompleted > - > > Key: SPARK-7889 > URL: https://issues.apache.org/jira/browse/SPARK-7889 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: meiyoula >Priority: Minor > > When running a SparkPi with 2000 tasks, cliking into the app on incomplete > page, the job progress shows 400/2000. After the app is completed, the app > goes to complete page from incomplete, and now cliking into the app, the job > progress still shows 400/2000. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[
https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035678#comment-15035678
]
Steve Loughran commented on SPARK-7889:
---
While I work on this, I suspect one of the issues is the reliance on
{{FileStatus..getModificationTime()}} as the metric for a file being updated.
It's not enough to track changes of incomplete apps
# when data is appended to an in-progress app through an open output stream,
the modification time does not change: hence, the log is not considered updates
# when a log file is renamed from {{$name.inprogress}} to {{$name}}, the
modtime is changed in HDFS, _but not, apparently, in a POSIX fs.
Issue #1 is stopping probes for incompletness; The query for incomplete apps is
going to have to probe for an update using file size.
For issue #2, a call to FileSystem.setTimes() on the renamed file, will
guarantee that the change is picked up; HADOOP-12612 filed to cover the issue
that the modtime semantics of rename are undefined and clearly inconsistent.
> Jobs progress of apps on complete page of HistoryServer shows uncompleted
> -
>
> Key: SPARK-7889
> URL: https://issues.apache.org/jira/browse/SPARK-7889
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Reporter: meiyoula
>Priority: Minor
>
> When running a SparkPi with 2000 tasks, cliking into the app on incomplete
> page, the job progress shows 400/2000. After the app is completed, the app
> goes to complete page from incomplete, and now cliking into the app, the job
> progress still shows 400/2000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[ https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15022750#comment-15022750 ] Apache Spark commented on SPARK-7889: - User 'steveloughran' has created a pull request for this issue: https://github.com/apache/spark/pull/9913 > Jobs progress of apps on complete page of HistoryServer shows uncompleted > - > > Key: SPARK-7889 > URL: https://issues.apache.org/jira/browse/SPARK-7889 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: meiyoula >Priority: Minor > > When running a SparkPi with 2000 tasks, cliking into the app on incomplete > page, the job progress shows 400/2000. After the app is completed, the app > goes to complete page from incomplete, and now cliking into the app, the job > progress still shows 400/2000. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[ https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595996#comment-14595996 ] Apache Spark commented on SPARK-7889: - User 'steveloughran' has created a pull request for this issue: https://github.com/apache/spark/pull/6935 Jobs progress of apps on complete page of HistoryServer shows uncompleted - Key: SPARK-7889 URL: https://issues.apache.org/jira/browse/SPARK-7889 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: meiyoula Priority: Minor When running a SparkPi with 2000 tasks, cliking into the app on incomplete page, the job progress shows 400/2000. After the app is completed, the app goes to complete page from incomplete, and now cliking into the app, the job progress still shows 400/2000. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[ https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595998#comment-14595998 ] Steve Loughran commented on SPARK-7889: --- Added a [new pull request|https://github.com/apache/spark/pull/6935]; notes which uis do not contain a completed attempt -and will refresh those. Jobs progress of apps on complete page of HistoryServer shows uncompleted - Key: SPARK-7889 URL: https://issues.apache.org/jira/browse/SPARK-7889 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: meiyoula Priority: Minor When running a SparkPi with 2000 tasks, cliking into the app on incomplete page, the job progress shows 400/2000. After the app is completed, the app goes to complete page from incomplete, and now cliking into the app, the job progress still shows 400/2000. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[
https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590215#comment-14590215
]
Steve Loughran commented on SPARK-7889:
---
Is JIRA about
(a) the status on the listing of complete/uncomplete being wrong in some way
(b) the actual job view (history/some-app-id) being stale when a job completes.
(b) is consistent with what I observed in SPARK-8275
Looking at your patch, and comparing it with my proposal, I prefer mine. All
I'm proposing is invalidating the cache on work in progress, so that it is
retrieved again.
Thinking about it some more, we can go one better: rely on the
{{ApplicationHistoryInfo.lastUpdated}} field to tell us when the UI was last
updated. If we cache the update time with the UI, on any GET of an appUI, we
can look to see if the previous UI was not completed and if the lastupdated
time has changed...if so. that triggers a refresh.
with this approach the entry you see will always be the one most recently
published to the history store (of any implementation), and picked up by the
history provider in its getListing()/background refresh operation.
Jobs progress of apps on complete page of HistoryServer shows uncompleted
-
Key: SPARK-7889
URL: https://issues.apache.org/jira/browse/SPARK-7889
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: meiyoula
Priority: Minor
When running a SparkPi with 2000 tasks, cliking into the app on incomplete
page, the job progress shows 400/2000. After the app is completed, the app
goes to complete page from incomplete, and now cliking into the app, the job
progress still shows 400/2000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[ https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591141#comment-14591141 ] meiyoula commented on SPARK-7889: - [~ste...@apache.org] Can you realize your proposal with code, I think maybe you can create a new pull request. Jobs progress of apps on complete page of HistoryServer shows uncompleted - Key: SPARK-7889 URL: https://issues.apache.org/jira/browse/SPARK-7889 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: meiyoula Priority: Minor When running a SparkPi with 2000 tasks, cliking into the app on incomplete page, the job progress shows 400/2000. After the app is completed, the app goes to complete page from incomplete, and now cliking into the app, the job progress still shows 400/2000. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[
https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568650#comment-14568650
]
Masayoshi TSUZUKI commented on SPARK-7889:
--
As a workaround, try to set {{spark.history.retainedApplications}} as {{0}}.
Jobs progress of apps on complete page of HistoryServer shows uncompleted
-
Key: SPARK-7889
URL: https://issues.apache.org/jira/browse/SPARK-7889
Project: Spark
Issue Type: Bug
Components: Spark Core
Reporter: meiyoula
When running a SparkPi with 2000 tasks, cliking into the app on incomplete
page, the job progress shows 400/2000. After the app is completed, the app
goes to complete page from incomplete, and now cliking into the app, the job
progress still shows 400/2000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7889) Jobs progress of apps on complete page of HistoryServer shows uncompleted
[ https://issues.apache.org/jira/browse/SPARK-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566888#comment-14566888 ] Apache Spark commented on SPARK-7889: - User 'XuTingjun' has created a pull request for this issue: https://github.com/apache/spark/pull/6545 Jobs progress of apps on complete page of HistoryServer shows uncompleted - Key: SPARK-7889 URL: https://issues.apache.org/jira/browse/SPARK-7889 Project: Spark Issue Type: Bug Components: Spark Core Reporter: meiyoula When running a SparkPi with 2000 tasks, cliking into the app on incomplete page, the job progress shows 400/2000. After the app is completed, the app goes to complete page from incomplete, and now cliking into the app, the job progress still shows 400/2000. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
