[
https://issues.apache.org/jira/browse/SPARK-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941112#comment-14941112
]
Steve Loughran commented on SPARK-5925:
---------------------------------------
I've actually thought about how to do this as part of SPARK-1537, but was
waiting to get the core patch done first.
What needs to be done is *relatively* straightforward:
for every incomplete app loaded into the history server, something needs to
GETs updated events from the history server and forwards them to the app UI.
That's the core concept; implementation details need to consider
* completion of app
* removal of app from cache & need to ensure no leaks by retaining app ui links
elsewhere
* the fact that ATS doesn't do blocking reads, so polling on a schedule is
required
* the need to keep load on the history server down so as to avoid overloading
it on a large cluster.
> YARN - Spark progress bar stucks at 10% but after finishing shows 100%
> ----------------------------------------------------------------------
>
> Key: SPARK-5925
> URL: https://issues.apache.org/jira/browse/SPARK-5925
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.2.1
> Reporter: Laszlo Fesus
> Priority: Minor
>
> I did set up a yarn cluster (CDH5) and spark (1.2.1), and also started Spark
> History Server. Now I am able to click on more details on yarn's web
> interface and get redirected to the appropriate spark logs during both job
> execution and also after the job has finished.
> My only concern is that while a spark job is being executed (either
> yarn-client or yarn-cluster), the progress bar stucks at 10% and doesn't
> increase as for MapReduce jobs. After finishing, it shows 100% properly, but
> we are loosing the real-time tracking capability of the status bar.
> Also tested yarn restful web interface, and it retrieves again 10% during
> (yarn) spark job execution, and works well again after finishing. (I suppose
> for the while being I should have a look on Spark Job Server and see if it's
> possible to track the job via its restful web interface.)
> Did anyone else experience this behaviour? Thanks in advance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
