The multiprocessing module might rely on the fact that it is called by the main thread which did the initialization. But the rpyc server is always using a new thread for every connection.
A solution might be to run the Threaded Server in a new thread and operate a queue with the main thread. submitJob() will simply stuff the parameter in the queue. The mean thread fetches the stuff creates the object and calls execute which will spawn a new process. ... 2015-09-22 5:11 GMT+02:00 Michael <[email protected]>: > Great info, and explanation Thank you Rüdiger > > Regarding the error, to try narrow down where to focus my effort I have > tested the following: > > > - Using RPyC and multiprocessing passing Job object from client to > server --> FAIL > - Using RPyC and multiprocessing passing strings and lists from client > to server where the server creates Job Object --> FAIL > - Using RPyC and not using multiprocessing --> PASS > - Not Using RPyC and using multiprocessing --> PASS > > > The above behavior lead me to believe that the issue is neither RPyC or > Multiprocessing isolation but rather trying to use them together. > > > > > On Tue, Sep 22, 2015 at 1:48 AM, Rüdiger Kessel <[email protected] > > wrote: > >> I think the problem is not rpyc related. It seems that in the traceback >> no rpyc lib is envolved. >> >> You should test to call exposed_submitJob() directly on the server >> without using rpyc. Probably the problem will show up as well. >> >> If you create the Job object on the client side and you pass the object >> to the server then the server just gets a reference (called netref if I >> remember correctly) to the object. If you now call currJob.execute() on the >> server then the server gets a reference to the method execute() in the job >> objects and executes this method. Now, where does the object exist? It is >> on the client side. So where is the execute-method? It is also on the >> client side. So you end up running the job on the client PC that way. Why >> do you use the whole server hokus pokus to end up running a local job? >> If you want something to be executed on the server you need to make sure >> that it exists as a callable on the server (and not just a netref to the >> client) and that the server has all resources including file acces to run >> it. >> One way to achieve this is to create the job object on the server. There >> might also be a way of copying byte code but I do not know how to achieve >> this. >> You can pass the object (as netref) back to the client and the client can >> access the object via netref as if it would be existing locally. >> >> So rpyc allows to create objects anwhere and to use them anywhere. But >> the objects stay where they were created and execution is always happening >> where the objects were created. (The same is true for the object data by >> the way). rpyc is doing a very good job to hide the details and you can use >> remote objects as if they would be existing locally. >> >> So the usual problem is that the client knows what the server should do >> including some resources. The server has the computing power and other >> required resources. Now one needs to create a callable on the server side >> based on the specification from the client and execute it on the server. >> During the execution the client might supply some local resources (like >> files) to support the server. At the end some results and some status >> informations should be available on the client side. >> >> rpyc can organize the communication between client and server and it can >> zerodeploy (http://rpyc.readthedocs.org/en/latest/docs/zerodeploy.html) >> rpyc. You might need something similar to deploy the "what to do list >> (usually called program)" to the server. So you need to write a framework >> which deploys the "program" to the server where you want the execution to >> happen. You can use rpyc as a transport mechanism (e.g. remote file >> access). Then you can use rpyc to start the execution on the server >> (standard remote call). During execution the server can access all objects, >> which are not locally available, via netref. At the end the client can >> access results and data if you pass back a netref to the client. You need >> to make sure that as long as you want to access stuff via netref a thread >> handling rpyc requests is running on the server and the client. >> >> In my Monte Carlo tool I only needed the computing power from the server. >> The programs are numerical calculations (equations) encoded in xml which >> were parsed on the server side and compiled to a python callable. The >> client provides an interface object for file access, standard IO and >> status/control information. >> >> The limitation of this approach is that the "program" is limited to what >> the parser can handle. But for my tool this limitation existed also for the >> local simulation tool which existed before. If you use pythen as a parser >> then you could transport python source code to the server and compile it >> there which would be more generic. Or you could copy batch files and run >> them on the server (similar to zerodeploy). >> >> I hope this will help... >> >> Greetings >> Rüdiger >> >> >> >> . >> >> >> 2015-09-21 16:23 GMT+02:00 Michael Mann <[email protected]>: >> >>> I should add that I believe this is directly related to using >>> multiprocessing module in server because I do not get an error when I >>> execute the job directly (not using multiprocessing.process()) in function >>> exposed_submitJobObj(self,newJob) >>> using newJob.execute() as opposed to calling addJobToQueue(self,newJob) >>> >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "rpyc" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "rpyc" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/rpyc/BWIlXGaqlMk/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Michael Mann > > -- > > --- > You received this message because you are subscribed to the Google Groups > "rpyc" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "rpyc" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
