[ https://issues.apache.org/jira/browse/SPARK-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579273#comment-14579273 ]
Steve Loughran edited comment on SPARK-8275 at 6/10/15 11:43 AM: ----------------------------------------------------------------- The cache code is from google; history server provides a method to get the data for an entry, but there's no logic in the cache itself to have a refresh time on entries. One solution would be # cache entries to include a timestamp and completed flag alongside the SparkUI instances # direct all cache.get operations through a single method in HistoryServer # have that method do something like {code} def getUI(id: String): SparkUI = { var cacheEntry = cache.get(id) if (!cacheEntry.completed && (cacheEntry.timestamp + expiryTime) < now()) { cache.release(id) cache.get(id) } cacheEntry } {code} this will leave out of date entries in the cache, but on any retrieval trigger the rebuild. was (Author: ste...@apache.org): The cache code is from google; history server provides a method to get the data for an entry, but there's no logic in the cache itself to have a refresh time on entries. One solution would be # cache entries to include a timestamp and completed flag alongside the SparkUI instances # direct all cache.get operations through a single method in HistoryServer # have that method do something like {code} def getUI(id: String): SparkUI = { var cacheEntry = cache.get(id) if (!cacheEntry.completed && (cacheEntry.timestamp + expiryTime) < now()) { cache.release(id) cache.get(id) } cacheEntry } this will leave out of date entries in the cache, but on any retrieval trigger the rebuild. > HistoryServer caches incomplete App UIs > --------------------------------------- > > Key: SPARK-8275 > URL: https://issues.apache.org/jira/browse/SPARK-8275 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 1.3.1 > Reporter: Steve Loughran > > The history server caches applications retrieved from the > {{ApplicationHistoryProvider.getAppUI()}} call for performance: it's > expensive to rebuild. > However, this cache also includes incomplete applications, as well as > completed ones —and it never attempts to refresh the incomplete application. > As a result, if you do a GET of the history of a running application, even > after the application is finished, you'll still get the web UI/history as it > was when that first GET was issued. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org