I think I found a way to use pycall in parallel: To get around the "cannot serialize pointer problem", I declared the corpus delicti everywhere and wrapped it inside a julia function that is also declared everywhere: It's also kind of clumsy and I'm not sure if it applies to your problem, but this worked for me:
@everywhere using PyCall @everywhere @pyimport a_python_module @everywhere pyfun = a_python_function @everywhere function wrap_pyfun(x) return pyfun(x) end r1 = remotecall(2, wrap_pyfun, arg1) r2 = remotecall(3, wrap_pyfun, arg2) It also works if you have a python object and you need to call one of it's methods @everywhere pyobj = a_python_object @everywhere function wrap_pymethod(x) return pyobj[:method](x) end hope this helps On Saturday, October 31, 2015 at 2:43:19 AM UTC+1, Yakir Gagnon wrote: > > OK, I found a horrible work around that might be good enough (for me): > Here is a mock python script: > > import math, time > x = math.sin(math.pi/4) > time.sleep(5) # mock local work > print(x) # spit it out to julia > > and here is the julia code that runs it: > > r = @spawn readall(`python example.py`) > sleep(2) # mock local work > wait(r) # wait on python > t = fetch(r) > x = parse(t) > x - sin(pi/4) # not zero but it works... > > This seems to work no matter what. It’s horrid, but better than nothing. > > On Friday, October 30, 2015 at 11:14:18 PM UTC+10, Matthew Pearce wrote: > > > So I got something working for my pylab example. > > julia> import PyCall > > julia> PyCall.@pyimport pylab > > julia> @everywhere import PyCall > > julia> @everywhere PyCall.@pyimport pylab > > julia> @everywhere A = pylab.cumsum(collect(1:10))*1. > > julia> fetch(@spawnat remotes[1] A) > 10-element Array{Float64,1}: > 1.0 > 3.0 > 6.0 > 10.0 > 15.0 > 21.0 > 28.0 > 36.0 > 45.0 > 55.0 > > > > > No luck with the math module I'm afraid. Two different types of errors > depending on style: > > julia> @spawnat remotes[1] PyCall.@pyimport math as pymath > RemoteRef{Channel{Any}}(2,1,305) > > julia> fetch(@spawnat remotes[1] (pymath.sin(pymath.pi / 4) - sin(pymath.pi > / 4)) ) > ERROR: On worker 2: > UndefVarError: pymath not defined > in anonymous at multi.jl:1330 > in anonymous at multi.jl:889 > in run_work_thunk at multi.jl:645 > in run_work_thunk at multi.jl:654 > in anonymous at task.jl:54 > in remotecall_fetch at multi.jl:731 > [inlined code] from multi.jl:368 > in call_on_owner at multi.jl:776 > in fetch at multi.jl:784 > > julia> @everywhere PyCall.@pyimport math as pymath > > julia> fetch(@spawnat remotes[1] (pymath.sin(pymath.pi / 4) - sin(pymath.pi > / 4)) ) > Worker 2 terminated.srun: error: mrc-bsu-tesla1: task 0: Exited with exit > code 1 > ERROR: ProcessExitedException() > in yieldto at ./task.jl:67 > in wait at ./task.jl:367 > in wait at ./task.jl:282 > in wait at ./channels.jl:97 > in take! at ./channels.jl:84 > in take! at ./multi.jl:792 > in remotecall_fetch at multi.jl:729 > [inlined code] from multi.jl:368 > in call_on_owner at multi.jl:776 > in fetch at multi.jl:784 > > > ERROR (unhandled task failure): EOFError: read end of file > > > > > > On Friday, October 30, 2015 at 1:28:21 AM UTC, Yakir Gagnon wrote: > > @Matthew: did you find a solution? > > On Tuesday, October 27, 2015 at 8:44:53 AM UTC+10, Yakir Gagnon wrote: > > Yea, right? So what’s the answer? How can we if at all do any PyCalls > parallely? > > On Monday, October 26, 2015 at 11:49:35 PM UTC+10, Matthew Pearce wrote: > > Thought I had an idea about this, I was wrong: > > ```julia > > julia> @everywhere using PyCall > > julia> @everywhere @pyimport pylab > > julia> remotecall_fetch(pylab.cumsum, 5, collect(1:10)) > ERROR: cannot serialize a pointer > [inlined code] from error.jl:21 > in serialize at serialize.jl:420 > [inlined code] from dict.jl:372 > in serialize at serialize.jl:428 > in serialize at serialize.jl:310 > in serialize at serialize.jl:420 (repeats 2 times) > in serialize at serialize.jl:302 > in serialize at serialize.jl:420 > [inlined code] from dict.jl:372 > in serialize at serialize.jl:428 > in serialize at serialize.jl:310 > in serialize at serialize.jl:420 (repeats 2 times) > in serialize at serialize.jl:302 > in serialize at serialize.jl:420 > [inlined code] from dict.jl:372 > in send_msg_ at multi.jl:222 > [inlined code] from multi.jl:177 > in remotecall_fetch at multi.jl:728 > [inlined code] from multi.jl:368 > in remotecall_fetch at multi.jl:734 > > julia> pylab.cumsum(collect(1:10)) > 10-element Array{Int64,1}: > 1 > 3 > 6 > 10 > 15 > 21 > 28 > 36 > 45 > 55 > > ``` > > > > ...