Hello,
I'm looking for a solution to use jruby on top of spark. It looks that the work
required is quite small. The only tricky point is that I need to make sure that
every worker thread has a ruby interpreter initialized. Basically, I need to
register a function to be called when each worker thread is created : a thread
local variable must be set for the ruby interpreter so that ruby object can be
deserialized.
Is there any solution to setup the worker threads before any spark call is made
using this thread ?
I've found a workaround using a java object that wrap any ruby function like:
rdd.map( JavaFuncWrapper.new(ruby_function) ). In the java JavaFuncWrapper, the
readObject method used to serialize has been overridden to perform the
initialization. I must say that I'm not very confident with this workaround. I
would prefer a more straightforward solution.
Thanks for your help,
Regards,
Lionel
-
Moody's monitors email communications through its networks for regulatory
compliance purposes and to protect its clients, employees and business and
where allowed to do so by applicable law. Parties communicating with Moody's
consent to such monitoring by their use of the email communication. The
information contained in this e-mail message, and any attachment thereto, is
confidential and may not be disclosed without our express permission. If you
are not the intended recipient or an employee or agent responsible for
delivering this message to the intended recipient, you are hereby notified that
you have received this message in error and that any review, dissemination,
distribution or copying of this message, or any attachment thereto, in whole or
in part, is strictly prohibited. If you have received this message in error,
please immediately notify us by telephone, fax or e-mail and delete the message
and all of its attachments. Thank you. Every effort is made to keep our network
free from viruses. You should, however, review this e-mail message, as well as
any attachment thereto, for viruses. We take no responsibility and have no
liability for any computer virus which may be transferred via this e-mail
message.
-