GitHub user nfergu opened a pull request: https://github.com/apache/spark/pull/2438
[SPARK-3051] Support looking-up named accumulators in a registry This proposal builds on SPARK-2380 (Support displaying accumulator values in the web UI) to allow named accumulables to be looked-up in a "registry", as opposed to having to be passed to every method that need to access them. See the JIRA ticket (SPARK-3051) for some more details, including an example use case. An AccumulableRegistry object is provided to allow accumulables to be looked-up by name at task execution time. This requires named accumulables to be broadcast to the executors before the task is executed. This is taken care of in the DAGScheduler in much the same way as for Tasks. Accumulables were already stored in thread-local variables in the Accumulators object, so exposing these in the registry was simply a matter of wrapping this object, and keying the accumulables by name (they were previously keyed only by ID). Note that Accumulables cannot be looked-up from the registry in the driver program; they can only be obtained while an operation is being performed on an RDD (a task is executing). One important thing to note in the implementation is that it is important that we we deserialize any named accumulators that have been broadcast before those that are explicitly passed with the task as we want the explicitly-passed ones to override the broadcast ones (otherwise the explicitly passed ones would not work). This may be a little brittle, but there are tests (in AccumulatorSuite) that will break if this is ever changed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/nfergu/spark accumreg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2438.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2438 ---- commit 7fae6af05d0ca91ac5444161a1e16b716ac3af83 Author: Neil Ferguson <neil.fergu...@nice.com> Date: 2014-08-14T22:37:31Z First attempt at allowing named accumulators to be looked-up in an AccumulableRegistry. Needs clean-up, testing, and documentation. Conflicts: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala commit 7ae16d69c30baad4609c226c79625958170198be Author: Neil Ferguson <neil.fergu...@nice.com> Date: 2014-09-16T21:47:26Z Added documentation, testing, and some fixes for functionality to allow named accumulators to be looked-up in an AccumulableRegistry. Conflicts: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala commit a89c51daeb3f172d2a3116afff48dc034e4d50c9 Author: Neil Ferguson <neil.fergu...@nice.com> Date: 2014-09-17T21:53:13Z Small documentation tweaks for AccumulableRegistry functionality ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org