It sounds like your code already does its initialization at most once per JVM, and that's about as good as it gets. Each partition asks for init in a thread-safe way and the first request succeeds.
On Thu, Oct 23, 2014 at 1:41 PM, Darin McBeath <[email protected]> wrote: > I have some code that I only need to be executed once per executor in my > spark application. My current approach is to do something like the > following: > > scala> xmlKeyPair.foreachPartition(i => XPathProcessor.init("ats", > "Namespaces/NamespaceContext")) > > So, If I understand correctly, the XPathProcessor.init will be called once > per partition. Since I have 48 partitions for this RDD and 2 million > documents, that seems acceptable. The downside is that I likely will have > fewer executors than 48 (each executor will handle more than 1 partition) so > the executor would be called more than once with XPathProcessor.init. I > have code in place to make sure this is not an issue. But, I was wondering > if there is a better way to accomplish something like this. > > Thanks. > > Darin. > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
