[
https://issues.apache.org/jira/browse/SPARK-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Agarwal updated SPARK-9806:
---------------------------------
Description:
Currently, we are sharing {{ReplayListenerBus}} for replaying the event logs of
various apps.
https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L226
While replaying the event log for a particular app, we add an
{{ApplicationEventListener}} to the bus.
https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L391
But we never remove it.
This results in one {{ReplayListenerBus}} being associated with multiple
{{ApplicationEventListener}}:
{code}
15/08/11 00:04:00 log-replay-executor INFO FsHistoryProvider: Replaying log
path:
hdfs://localhost:9000/spark-history/application_1438993108319_0146_1.snappy
15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Elapsed time:
39.730114407s
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Application log
application_1438993108319_0146_1.snappy loaded successfully.
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Replaying log
path:
hdfs://localhost:9000/spark-history/application_1438993108319_0126_1.snappy
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Elapsed time:
20.483128154s
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Application log
application_1438993108319_0126_1.snappy loaded successfully.
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Replaying log
path:
hdfs://localhost:9000/spark-history/application_1438993108319_0116_1.snappy
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Elapsed time:
29.110070845s
15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Application log
application_1438993108319_0116_1.snappy loaded successfully.
{code}
We should either remove the listener from the bus or create a new bus for each
app.
was:
Currently, we are sharing {{ReplayListenerBus}} for replaying the event logs of
various apps.
https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L226
While replaying the event log for a particular app, we add an
{{ApplicationEventListener}} to the bus.
https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L391
But we never remove it.
This result in something like the following:
{code}
15/08/11 00:04:00 log-replay-executor INFO FsHistoryProvider: Replaying log
path:
hdfs://localhost:9000/spark-history/application_1438993108319_0146_1.snappy
15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Elapsed time:
39.730114407s
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Application log
application_1438993108319_0146_1.snappy loaded successfully.
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Replaying log
path:
hdfs://localhost:9000/spark-history/application_1438993108319_0126_1.snappy
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Elapsed time:
20.483128154s
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Application log
application_1438993108319_0126_1.snappy loaded successfully.
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Replaying log
path:
hdfs://localhost:9000/spark-history/application_1438993108319_0116_1.snappy
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
onApplicationStart
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener:
onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Elapsed time:
29.110070845s
15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Application log
application_1438993108319_0116_1.snappy loaded successfully.
{code}
We should either remove the listener from the bus or create a new bus for each
app.
> Don't share ReplayListenerBus between multiple applications
> -----------------------------------------------------------
>
> Key: SPARK-9806
> URL: https://issues.apache.org/jira/browse/SPARK-9806
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.4.0
> Reporter: Rohit Agarwal
> Priority: Minor
>
> Currently, we are sharing {{ReplayListenerBus}} for replaying the event logs
> of various apps.
> https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L226
> While replaying the event log for a particular app, we add an
> {{ApplicationEventListener}} to the bus.
> https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L391
> But we never remove it.
> This results in one {{ReplayListenerBus}} being associated with multiple
> {{ApplicationEventListener}}:
> {code}
> 15/08/11 00:04:00 log-replay-executor INFO FsHistoryProvider: Replaying log
> path:
> hdfs://localhost:9000/spark-history/application_1438993108319_0146_1.snappy
> 15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener:
> onEnvUpdate
> 15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener:
> onApplicationStart
> 15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
> onApplicationEnd
> 15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Elapsed time:
> 39.730114407s
> 15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Application log
> application_1438993108319_0146_1.snappy loaded successfully.
> 15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Replaying log
> path:
> hdfs://localhost:9000/spark-history/application_1438993108319_0126_1.snappy
> 15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
> onEnvUpdate
> 15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
> onEnvUpdate
> 15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
> onApplicationStart
> 15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener:
> onApplicationStart
> 15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
> onApplicationEnd
> 15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
> onApplicationEnd
> 15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Elapsed time:
> 20.483128154s
> 15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Application log
> application_1438993108319_0126_1.snappy loaded successfully.
> 15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Replaying log
> path:
> hdfs://localhost:9000/spark-history/application_1438993108319_0116_1.snappy
> 15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
> onEnvUpdate
> 15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
> onEnvUpdate
> 15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
> onEnvUpdate
> 15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
> onApplicationStart
> 15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
> onApplicationStart
> 15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener:
> onApplicationStart
> 15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener:
> onApplicationEnd
> 15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener:
> onApplicationEnd
> 15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener:
> onApplicationEnd
> 15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Elapsed time:
> 29.110070845s
> 15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Application log
> application_1438993108319_0116_1.snappy loaded successfully.
> {code}
> We should either remove the listener from the bus or create a new bus for
> each app.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
