[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084451#comment-16084451
]
Ye Zhou commented on SPARK-18085:
---------------------------------
I want to add my own testing experience with the codes from the HEAD of your
repo. I have built and deployed this Spark History Server 2 weeks ago and set
the log dir to our dev cluster where around 1.5k spark applications daily
application logs will be produced. And we continuously generate the loads to it
by rest call. It works well. Thanks for the work.
I have gone through the codes partially and have some minor improvement ideas
to fit our internal need especially if there are more than 50K log files in the
log dir. Some of them might be also useful to upstream.
> SPIP: Better History Server scalability for many / large applications
> ---------------------------------------------------------------------
>
> Key: SPARK-18085
> URL: https://issues.apache.org/jira/browse/SPARK-18085
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Labels: SPIP
> Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues
> when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues.
> I'll be attaching a document shortly describing the issues and suggesting a
> path to how to solve them.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
