[ 
https://issues.apache.org/jira/browse/SPARK-33215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218683#comment-17218683
 ] 

Apache Spark commented on SPARK-33215:
--------------------------------------

User 'baohe-zhang' has created a pull request for this issue:
https://github.com/apache/spark/pull/30126

> Speed up event log download by skipping UI rebuild
> --------------------------------------------------
>
>                 Key: SPARK-33215
>                 URL: https://issues.apache.org/jira/browse/SPARK-33215
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>    Affects Versions: 2.4.7, 3.0.1
>            Reporter: Baohe Zhang
>            Priority: Major
>
> Right now, when we want to download the event logs from the spark history 
> server(SHS), SHS will need to parse entire the event log to rebuild UI, and 
> this is just for view permission checks. UI rebuilding is a time-consuming 
> and memory-intensive task, especially for large logs. However, this process 
> is unnecessary for event log download.
> This patch enables SHS to check UI view permissions of a given app/attempt 
> for a given user, without rebuilding the UI. This is achieved by adding a 
> method "checkUIViewPermissions(appId: String, attemptId: Option[String], 
> user: String): Boolean" to many layers of history server components.
> With this patch, UI rebuild can be skipped when downloading event logs from 
> the history server. Thus the time of downloading a GB scale event log can be 
> reduced from several minutes to several seconds, and the memory consumption 
> of UI rebuilding can be avoided.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to