I have some code that I only need to be executed once per executor in my spark
application. My current approach is to do something like the following:
scala> xmlKeyPair.foreachPartition(i => XPathProcessor.init("ats",
"Namespaces/NamespaceContext"))
So, If I understand correctly, the XPathProcessor.init will be called once per
partition. Since I have 48 partitions for this RDD and 2 million documents,
that seems acceptable. The downside is that I likely will have fewer executors
than 48 (each executor will handle more than 1 partition) so the executor would
be called more than once with XPathProcessor.init. I have code in place to
make sure this is not an issue. But, I was wondering if there is a better way
to accomplish something like this.
Thanks.
Darin.