Re: multiprocessing: child process race to answer
On 03/11/2013 10:10, capple...@gmail.com wrote: On Friday, November 1, 2013 10:35:47 PM UTC-4, smhall05 wrote: I am using a basic multiprocessing snippet I found: #- from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': pool = Pool(processes=4) # start 4 worker processes result = pool.apply_async(f, [10])# evaluate "f(10)" asynchronously print result.get(timeout=1) print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" #- I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer. I have tried: sys.exit(0) #this causes the program to hang pool.close() pool.terminate These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you You could take a look at the Mutiprocessing module capabilities for exchanging objects between processes: http://docs.python.org/3.3/library/multiprocessing.html#exchanging-objects-between-processes Would you please read and action this as it prevents the double line spacing that you can observe above, thanks https://wiki.python.org/moin/GoogleGroupsPython -- Python is the second best programming language in the world. But the best has yet to be invented. Christian Tismer Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing: child process race to answer
On Friday, November 1, 2013 10:35:47 PM UTC-4, smhall05 wrote: > I am using a basic multiprocessing snippet I found: > > > > #- > > from multiprocessing import Pool > > > > def f(x): > > return x*x > > > > if __name__ == '__main__': > > pool = Pool(processes=4) # start 4 worker processes > > result = pool.apply_async(f, [10])# evaluate "f(10)" asynchronously > > print result.get(timeout=1) > > print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" > > #- > > > > I am using this code to have each process go off and solve the same problem, > just with different inputs to the problem. I need to be able to kill all > processes once 1 of n processes has come up with the solution. There will > only be one answer. > > > > I have tried: > > > > sys.exit(0) #this causes the program to hang > > pool.close() > > pool.terminate > > > > These still allow further processing before the program terminates. What else > can I try? I am not able to share the exact code at this time. I can provide > more detail if I am unclear. Thank you You could take a look at the Mutiprocessing module capabilities for exchanging objects between processes: http://docs.python.org/3.3/library/multiprocessing.html#exchanging-objects-between-processes -- https://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing: child process race to answer (forgot to Cc: the list)
On Nov 2, 2013, at 11:44 AM, Sherard Hall wrote: > Thank you for the response. Processing time is very important so I suspect > having to write to disk will take more time than letting the other processes > complete without finding the answer. So I did some profiling one process > finds the answer in about 250ms, but since I can't stop the other processes, > it takes about 800ms before I can use the answer. Do you recommend a global > variable flag? Any other suggestions? > > On Nov 2, 2013 8:17 AM, "William Ray Wing" wrote: > On Nov 2, 2013, at 1:03 AM, smhall05 wrote: > > > On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote: > >> On 02/11/2013 02:35, smhall05 wrote: > >> > >>> I am using a basic multiprocessing snippet I found: > >>> > >>> #- > >>> from multiprocessing import Pool > >>> > >>> def f(x): > >>> return x*x > >>> > >>> if __name__ == '__main__': > >>> pool = Pool(processes=4) # start 4 worker processes > >>> result = pool.apply_async(f, [10])# evaluate "f(10)" > >>> asynchronously > >>> print result.get(timeout=1) > >>> print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" > >>> #- > >>> > >>> I am using this code to have each process go off and solve the same > >>> problem, just with different inputs to the problem. I need to be able to > >>> kill all processes once 1 of n processes has come up with the solution. > >>> There will only be one answer. > >>> > >>> I have tried: > >>> > >>> sys.exit(0) #this causes the program to hang > >>> pool.close() > >>> pool.terminate > >>> > >> > >> Did you actually mean "pool.terminate", or is that a typo for > >> > >> "pool.terminate()"? > >> > >>> These still allow further processing before the program terminates. What > >>> else can I try? I am not able to share the exact code at this time. I can > >>> provide more detail if I am unclear. Thank you > >>> > > > > I am not sure to be honest, however it turns out that I can't use > > pool.terminate() because pool is defined in main and not accessible under > > my def in which I check for the correct answer. > > -- > > https://mail.python.org/mailman/listinfo/python-list > > So, the simplest solution to that situation is to have whichever subprocess > that finds the correct answer set a flag which the calling process can check. > Depending on your OS, that flag can be anything from setting a lock to > something as simple as creating a file which the calling process periodically > wakes up and looks for, maybe just a file in which the subprocess has written > the answer. > > Bill > > -- > https://mail.python.org/mailman/listinfo/python-list Well, the multiprocessing library provides listeners and clients that wrap BSD style sockets and allow you to send (push) arbitrary python objects to a listener, i.e., the master. There might be something better that was OS specific, but this will keep it pure python. I've not tested it, but there is a simple example here on Stackoverflow: http://stackoverflow.com/questions/6920858/interprocess-communication-in-python -Bill-- https://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing: child process race to answer
Thank you for the response. Processing time is very important so I suspect having to write to disk will take more time than letting the other processes complete without finding the answer. So I did some profiling one process finds the answer in about 250ms, but since I can't stop the other processes, it takes about 800ms before I can use the answer. Do you recommend a global variable flag? Any other suggestions? On Nov 2, 2013 8:17 AM, "William Ray Wing" wrote: > On Nov 2, 2013, at 1:03 AM, smhall05 wrote: > > > On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote: > >> On 02/11/2013 02:35, smhall05 wrote: > >> > >>> I am using a basic multiprocessing snippet I found: > >>> > >>> #- > >>> from multiprocessing import Pool > >>> > >>> def f(x): > >>> return x*x > >>> > >>> if __name__ == '__main__': > >>> pool = Pool(processes=4) # start 4 worker processes > >>> result = pool.apply_async(f, [10])# evaluate "f(10)" > asynchronously > >>> print result.get(timeout=1) > >>> print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" > >>> #- > >>> > >>> I am using this code to have each process go off and solve the same > problem, just with different inputs to the problem. I need to be able to > kill all processes once 1 of n processes has come up with the solution. > There will only be one answer. > >>> > >>> I have tried: > >>> > >>> sys.exit(0) #this causes the program to hang > >>> pool.close() > >>> pool.terminate > >>> > >> > >> Did you actually mean "pool.terminate", or is that a typo for > >> > >> "pool.terminate()"? > >> > >>> These still allow further processing before the program terminates. > What else can I try? I am not able to share the exact code at this time. I > can provide more detail if I am unclear. Thank you > >>> > > > > I am not sure to be honest, however it turns out that I can't use > pool.terminate() because pool is defined in main and not accessible under > my def in which I check for the correct answer. > > -- > > https://mail.python.org/mailman/listinfo/python-list > > So, the simplest solution to that situation is to have whichever > subprocess that finds the correct answer set a flag which the calling > process can check. Depending on your OS, that flag can be anything from > setting a lock to something as simple as creating a file which the calling > process periodically wakes up and looks for, maybe just a file in which the > subprocess has written the answer. > > Bill > > -- https://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing: child process race to answer
On Nov 2, 2013, at 1:03 AM, smhall05 wrote: > On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote: >> On 02/11/2013 02:35, smhall05 wrote: >> >>> I am using a basic multiprocessing snippet I found: >>> >>> #- >>> from multiprocessing import Pool >>> >>> def f(x): >>> return x*x >>> >>> if __name__ == '__main__': >>> pool = Pool(processes=4) # start 4 worker processes >>> result = pool.apply_async(f, [10])# evaluate "f(10)" asynchronously >>> print result.get(timeout=1) >>> print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" >>> #- >>> >>> I am using this code to have each process go off and solve the same >>> problem, just with different inputs to the problem. I need to be able to >>> kill all processes once 1 of n processes has come up with the solution. >>> There will only be one answer. >>> >>> I have tried: >>> >>> sys.exit(0) #this causes the program to hang >>> pool.close() >>> pool.terminate >>> >> >> Did you actually mean "pool.terminate", or is that a typo for >> >> "pool.terminate()"? >> >>> These still allow further processing before the program terminates. What >>> else can I try? I am not able to share the exact code at this time. I can >>> provide more detail if I am unclear. Thank you >>> > > I am not sure to be honest, however it turns out that I can't use > pool.terminate() because pool is defined in main and not accessible under my > def in which I check for the correct answer. > -- > https://mail.python.org/mailman/listinfo/python-list So, the simplest solution to that situation is to have whichever subprocess that finds the correct answer set a flag which the calling process can check. Depending on your OS, that flag can be anything from setting a lock to something as simple as creating a file which the calling process periodically wakes up and looks for, maybe just a file in which the subprocess has written the answer. Bill -- https://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing: child process race to answer
On Friday, November 1, 2013 10:52:40 PM UTC-4, MRAB wrote: > On 02/11/2013 02:35, smhall05 wrote: > > > I am using a basic multiprocessing snippet I found: > > > > > > #- > > > from multiprocessing import Pool > > > > > > def f(x): > > > return x*x > > > > > > if __name__ == '__main__': > > > pool = Pool(processes=4) # start 4 worker processes > > > result = pool.apply_async(f, [10])# evaluate "f(10)" asynchronously > > > print result.get(timeout=1) > > > print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" > > > #- > > > > > > I am using this code to have each process go off and solve the same > > problem, just with different inputs to the problem. I need to be able to > > kill all processes once 1 of n processes has come up with the solution. > > There will only be one answer. > > > > > > I have tried: > > > > > > sys.exit(0) #this causes the program to hang > > > pool.close() > > > pool.terminate > > > > > Did you actually mean "pool.terminate", or is that a typo for > > "pool.terminate()"? > > > > > These still allow further processing before the program terminates. What > > else can I try? I am not able to share the exact code at this time. I can > > provide more detail if I am unclear. Thank you > > > I am not sure to be honest, however it turns out that I can't use pool.terminate() because pool is defined in main and not accessible under my def in which I check for the correct answer. -- https://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing: child process race to answer
On 02/11/2013 02:35, smhall05 wrote: I am using a basic multiprocessing snippet I found: #- from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': pool = Pool(processes=4) # start 4 worker processes result = pool.apply_async(f, [10])# evaluate "f(10)" asynchronously print result.get(timeout=1) print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" #- I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer. I have tried: sys.exit(0) #this causes the program to hang pool.close() pool.terminate Did you actually mean "pool.terminate", or is that a typo for "pool.terminate()"? These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you -- https://mail.python.org/mailman/listinfo/python-list
multiprocessing: child process race to answer
I am using a basic multiprocessing snippet I found: #- from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': pool = Pool(processes=4) # start 4 worker processes result = pool.apply_async(f, [10])# evaluate "f(10)" asynchronously print result.get(timeout=1) print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]" #- I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer. I have tried: sys.exit(0) #this causes the program to hang pool.close() pool.terminate These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you -- https://mail.python.org/mailman/listinfo/python-list
