Baohe Zhang created SPARK-33215:
-----------------------------------
Summary: Speed up event log download by skipping UI rebuild
Key: SPARK-33215
URL: https://issues.apache.org/jira/browse/SPARK-33215
Project: Spark
Issue Type: Improvement
Components: Web UI
Affects Versions: 3.0.1, 2.4.7
Reporter: Baohe Zhang
Right now, when we want to download the event logs from the spark history
server(SHS), SHS will need to parse entire the event log to rebuild UI, and
this is just for view permission checks. UI rebuilding is a time-consuming and
memory-intensive task, especially for large logs. However, this process is
unnecessary for event log download.
This patch enables SHS to check UI view permissions of a given app/attempt for
a given user, without rebuilding the UI. This is achieved by adding a method
"checkUIViewPermissions(appId: String, attemptId: Option[String], user:
String): Boolean" to many layers of history server components.
With this patch, UI rebuild can be skipped when downloading event logs from the
history server. Thus the time of downloading a GB scale event log can be
reduced from several minutes to several seconds, and the memory consumption of
UI rebuilding can be avoided.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
