Hi, On Fri, Dec 5, 2014 at 3:56 AM, Akshat Aranya <aara...@gmail.com> wrote:
> Is it possible to have some state across multiple calls to mapPartitions > on each partition, for instance, if I want to keep a database connection > open? > If you're using Scala, you can use a singleton object, this will exist once per JVM (i.e., once per executor), like object DatabaseConnector { lazy val conn = ... } Please be aware that shutting down the connection is much harder than opening it, because you basically have no idea when processing is done for an executor, AFAIK. Tobias