[
https://issues.apache.org/jira/browse/SPARK-33215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218683#comment-17218683
]
Apache Spark commented on SPARK-33215:
--------------------------------------
User 'baohe-zhang' has created a pull request for this issue:
https://github.com/apache/spark/pull/30126
> Speed up event log download by skipping UI rebuild
> --------------------------------------------------
>
> Key: SPARK-33215
> URL: https://issues.apache.org/jira/browse/SPARK-33215
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 2.4.7, 3.0.1
> Reporter: Baohe Zhang
> Priority: Major
>
> Right now, when we want to download the event logs from the spark history
> server(SHS), SHS will need to parse entire the event log to rebuild UI, and
> this is just for view permission checks. UI rebuilding is a time-consuming
> and memory-intensive task, especially for large logs. However, this process
> is unnecessary for event log download.
> This patch enables SHS to check UI view permissions of a given app/attempt
> for a given user, without rebuilding the UI. This is achieved by adding a
> method "checkUIViewPermissions(appId: String, attemptId: Option[String],
> user: String): Boolean" to many layers of history server components.
> With this patch, UI rebuild can be skipped when downloading event logs from
> the history server. Thus the time of downloading a GB scale event log can be
> reduced from several minutes to several seconds, and the memory consumption
> of UI rebuilding can be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
