I am using rpyc to spawn a python process that uses joblib on a remote server. I am running:
#rpyc_classic -m forking This is the only way I have been able to get the remote process to use multiple CPUs. However after running for a while, I only get 2 or 3 CPUs active whereas initially I was using 48 CPUs. Looking at the list processes on the server, I do see about 200 processes like the one below 69976 67657 0 12:23 pts/1 00:00:00 /usr/bin/python /usr/bin/rpyc_classic.py -m forking Despite that only 2 or 3 CPUs are active and my job basically slows down. It is almost as if once the RPYC process is spawned it does not return. How do I get the forking processes to return on finish Is this the best way to do use rpyc for remotely executing my job. Note that when running the python program locally, it uses joblib likes it is supposed to and uses up as many CPU's as available. It is only when running through rpyc that it plateaus after about 200 rpyc_classic processes. Thank you -- --- You received this message because you are subscribed to the Google Groups "rpyc" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
