It sounds like your code already does its initialization at most once
per JVM, and that's about as good as it gets. Each partition asks for
init in a thread-safe way and the first request succeeds.

On Thu, Oct 23, 2014 at 1:41 PM, Darin McBeath
<[email protected]> wrote:
> I have some code that I only need to be executed once per executor in my
> spark application.  My current approach is to do something like the
> following:
>
> scala> xmlKeyPair.foreachPartition(i => XPathProcessor.init("ats",
> "Namespaces/NamespaceContext"))
>
> So, If I understand correctly, the XPathProcessor.init will be called once
> per partition.  Since I have 48 partitions for this RDD and 2 million
> documents, that seems acceptable.  The downside is that I likely will have
> fewer executors than 48 (each executor will handle more than 1 partition) so
> the executor would be called more than once with XPathProcessor.init.  I
> have code in place to make sure this is not an issue.  But, I was wondering
> if there is a better way to accomplish something like this.
>
> Thanks.
>
> Darin.
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to