[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894768#comment-15894768
]
Marcelo Vanzin commented on SPARK-18085:
----------------------------------------
bq. does this local db will delete the data as specified by the configuration?
The existing log cleaner functionality will be maintained, so the application
logs will be cleaned the same way they are today. For the new local DBs, I
kinda touch on that in the document. My current plan is to first have a
configuration for the maximum amount of data the SHS can use locally (and use a
LRU-style approach to delete local DBs), and eventually cache these DBs in
remote storage (e.g. HDFS) so that they don't need to be re-created (which can
be expensive).
> Better History Server scalability for many / large applications
> ---------------------------------------------------------------
>
> Key: SPARK-18085
> URL: https://issues.apache.org/jira/browse/SPARK-18085
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues
> when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues.
> I'll be attaching a document shortly describing the issues and suggesting a
> path to how to solve them.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
