I have some code that I only need to be executed once per executor in my spark 
application.  My current approach is to do something like the following:
scala> xmlKeyPair.foreachPartition(i => XPathProcessor.init("ats", 
"Namespaces/NamespaceContext"))
So, If I understand correctly, the XPathProcessor.init will be called once per 
partition.  Since I have 48 partitions for this RDD and 2 million documents, 
that seems acceptable.  The downside is that I likely will have fewer executors 
than 48 (each executor will handle more than 1 partition) so the executor would 
be called more than once with XPathProcessor.init.  I have code in place to 
make sure this is not an issue.  But, I was wondering if there is a better way 
to accomplish something like this.
Thanks.
Darin.

Reply via email to