Hi,
I have a question about how does "imports" get send to the python worker.
For example, I have
def foo(x):
return np.abs(x)
If I run this code directly, it obviously failed (because np is undefined
on the driver process):
sc.paralleilize([1, 2, 3]).map(foo).collect()
However, if I add the import statement "import numpy as np" on the driver,
it works. So somehow driver is sending that "imports" to the worker when
executing foo on the worker but I cannot seem t o find the code that does
this - Can someone please send me a pointer?
Thanks,
Li
