Hello,
I was wondering what the best way is to have some form of 
singleton per worker...something that'll be instantiated once at the 
start of a job for each worker, and shut down when all the work on that 
node is completed. For instance, say I have a client library that 
initiates a single session to a central server, and you're meant to 
reuse that for all little bits of work, and finally close the session 
when all the work items are done. Opening and closing sessions for each 
data item would be wasteful (and in some cases, cause havoc for the 
client).

I've been doing this with foreachPartition (i.e. have 
the parameters for creating the singleton outside the loop, do a 
foreachPartition, create the instance, loop over entries in the 
partition, close the partition), but it's quite cludgy. Is there a 
pattern by which I can have an instance of something nonserializable on 
each worker?

Regards,
Ashic.
                                          

Reply via email to