[
https://issues.apache.org/jira/browse/SPARK-28069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noritaka Sekiyama updated SPARK-28069:
--------------------------------------
Description:
History server polls the directory specified in spark.history.fs.logDirectory,
and displays the event logs based on the data included in the directory.
In the Cloud, event logs might be written into different directories from
different clusters.
Currently it is not so easy to launch history server for multiple clusters
because there is no ways to switch log directory after starting the history
server.
If users can switch log directory from Spark UI without restarting history
server, it would be beneficial for users who wants to use Cloud storage for
event log directory.
Suggested implementation
* History server polls the directory specified in spark.history.fs.logDirectory
when it started.
* Add a button of "Switch log directory" at the top page of Web UI.
* The directory which users selected is stored only in memory. (It won't be
persistent configuration.)
* Next time when history server started, the directory specified in
spark.history.fs.logDirectory is used.
was:
History server polls the directory specified in spark.history.fs.logDirectory,
and displays the event logs based on the data included in the directory.
In the Cloud, event logs might be written into different directories from
different clusters.
Currently it is not so easy to launch history server for multiple clusters
because there is no ways to switch log directory after starting the history
server.
If users can switch log directory from Spark UI without restarting history
server, it would be beneficial for users who wants to use Cloud storage for
event log directory.
> Switch log directory from Spark UI without restarting history server
> --------------------------------------------------------------------
>
> Key: SPARK-28069
> URL: https://issues.apache.org/jira/browse/SPARK-28069
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 2.4.3
> Reporter: Noritaka Sekiyama
> Priority: Minor
>
> History server polls the directory specified in
> spark.history.fs.logDirectory, and displays the event logs based on the data
> included in the directory.
> In the Cloud, event logs might be written into different directories from
> different clusters.
> Currently it is not so easy to launch history server for multiple clusters
> because there is no ways to switch log directory after starting the history
> server.
> If users can switch log directory from Spark UI without restarting history
> server, it would be beneficial for users who wants to use Cloud storage for
> event log directory.
> Suggested implementation
> * History server polls the directory specified in
> spark.history.fs.logDirectory when it started.
> * Add a button of "Switch log directory" at the top page of Web UI.
> * The directory which users selected is stored only in memory. (It won't be
> persistent configuration.)
> * Next time when history server started, the directory specified in
> spark.history.fs.logDirectory is used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
