GitHub user steveloughran opened a pull request: https://github.com/apache/spark/pull/9913
[SPARK-7889] [Core] WiP History Caching: xu tingjun's patch against master This is @XuTingjun's patch (#6545) reapplied to trunk with a bit of cleanup (the probe in the history provider is now `isComplete(String appId, Option[String] attemptId`; the conditions around its use and is internals slightly tweaked. Looking at this code now, I do agree the history provider should have a say in the state of the attempts, I'm not sure if this is the right approach. Having 3x different maps to track app state (including two that never get cleaned up) isn't ideal, and if it were to be so, then the two new ones would have to be thread safe, so as to handle the scenario of "multiple requests coming in at the same time" I'd rather retain the history server's information alongside the cached app, (as my patch does), though that still leaves the question of how to probe for a updated version and then reload it, especially in a way that is thread safe. Oh, and testing, obviously. (once this PR has been tested I'm going to close it; it's here as reference) You can merge this pull request into a Git repository by running: $ git pull https://github.com/steveloughran/spark history/SPARK-7889-XuTingjun Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9913.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9913 ---- commit 1add1ec9dd315e5b2cf9d2a77e95f7e47b3bf5c3 Author: Steve Loughran <ste...@hortonworks.com> Date: 2015-11-23T18:40:35Z SPARK-7889 apply XuTingjun's patch to trunk commit 865808e4f555396b91422780dd6267a0457d9d43 Author: Steve Loughran <ste...@hortonworks.com> Date: 2015-11-23T18:55:38Z SCALA-7889 ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org