[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15937377#comment-15937377
]
Marcelo Vanzin commented on SPARK-18085:
----------------------------------------
Hi all, an update. I started working on this again and I was concerned about my
original approach; fixing conflicts would be hard and it would be easy to miss
changes in the UI while the work is going on. So I changed a bit the way I'm
working to make integration a little more gradual.
Basically, instead of forking UI code, I'm moving the old code into the new
module and making changes to it so tests work. So while this work is going on,
there's a "franken ui" where part of the UI is using the old backend, and part
of the UI is using the new backend. This will save time in the end, and will
catch errors in the new code sooner (as I found out when trying to fix all the
unit tests).
So I updated all the branches with the new approach:
https://github.com/vanzin/spark/tree/shs-ng/M4.4
https://github.com/vanzin/spark/tree/shs-ng/M4.3
https://github.com/vanzin/spark/tree/shs-ng/M4.2
https://github.com/vanzin/spark/tree/shs-ng/M4.1
https://github.com/vanzin/spark/tree/shs-ng/M4.0
https://github.com/vanzin/spark/tree/shs-ng/M3
https://github.com/vanzin/spark/tree/shs-ng/M2
https://github.com/vanzin/spark/tree/shs-ng/M1
There's a new branch (M4.0) where I do some initial integration to allow the
"franken ui" to be used. And each M4 branch contains more changes than before
because I had to make all existing unit tests pass after the data was served
from the backend. Current state is that all "core" pages are using the new
backend and pass unit tests.
Next I'll be re-basing everything to current master, and then I'll start to add
more things and clean up more old code that's not needed anymore.
> Better History Server scalability for many / large applications
> ---------------------------------------------------------------
>
> Key: SPARK-18085
> URL: https://issues.apache.org/jira/browse/SPARK-18085
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues
> when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues.
> I'll be attaching a document shortly describing the issues and suggesting a
> path to how to solve them.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
