Re: How to initialize each multithreading Pool worker with an individual value?
In article mailman.61.1291152972.2649.python-l...@python.org, Valery Khamenya khame...@gmail.com wrote: However it doesn't look possible to use it to initialize each Pool's worker with some individual value (I'd wish to be wrong here) So, how to initialize each multithreading Pool worker with the individual values? The typical use case might be a connection pool, say, of 3 workers, where each of 3 workers has its own TCP/IP port. from multiprocessing.pool import Pool def port_initializer(_port): global port port = _port def use_connection(some_packet): global _port print sending data over port # %s % port if __name__ == __main__: ports=((4001,4002, 4003), ) p = Pool(3, port_initializer, ports) # oops... :-) You probably can't use initargs here. Your port_initializer needs to be some kind of class instance that works with multiprocessing and emits one port number when its __call__() method gets invoked. (There may be other ways to accomplish the same effect, but that's what springs to mind.) -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ Think of it as evolution in action. --Tony Rand -- http://mail.python.org/mailman/listinfo/python-list
Re: How to initialize each multithreading Pool worker with an individual value?
On Thu, 2010-12-30 at 08:01 -0800, Aahz wrote: In article mailman.61.1291152972.2649.python-l...@python.org, Valery Khamenya khame...@gmail.com wrote: However it doesn't look possible to use it to initialize each Pool's worker with some individual value (I'd wish to be wrong here) So, how to initialize each multithreading Pool worker with the individual values? The typical use case might be a connection pool, say, of 3 workers, where each of 3 workers has its own TCP/IP port. from multiprocessing.pool import Pool def port_initializer(_port): global port port = _port def use_connection(some_packet): global _port print sending data over port # %s % port if __name__ == __main__: ports=((4001,4002, 4003), ) p = Pool(3, port_initializer, ports) # oops... :-) You probably can't use initargs here. Your port_initializer needs to be some kind of class instance that works with multiprocessing and emits one port number when its __call__() method gets invoked. (There may be other ways to accomplish the same effect, but that's what springs to mind.) Maybe this is obvious; but it is possible to create a pool of workers all listening on the same socket. An idle worker will pick-up the connection. Just open the socket in the initial process and then fork your workers - they will inherit the file handle and can accept() on it. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to initialize each multithreading Pool worker with an individual value?
On Dec 1, 3:24 am, James Mills prolo...@shortcircuit.net.au wrote: I assume you are talking about multiprocessing despite you mentioning multithreading in the mix. yes, sorry. Have a look at the source code for multiprocessing.pool and how the Pool object works and what it does with the initializer argument. I'm not entirely sure it does what you expect and yes documentation on this is lacking... I see I found my way to seed each member of Pool with own data. I do it right after after initialization: port = None def port_seeder(port_val) from time import sleep sleep(1) # or less... global port port = port_val if __name__ == '__main__': pool = Pool(3) pool.map(port_seeder, range(3), chunksize=1) # now child processes are initialized with individual values. Another (a bit more heavier) approach would be via shared resource. P.S. sorry, I found your answer only now. reagrds -- Valery -- http://mail.python.org/mailman/listinfo/python-list
Re: How to initialize each multithreading Pool worker with an individual value?
Hi Dan, If you create in the parent a queue in shared memory (multiprocessing facilitates this nicely), and fill that queue with the values in your ports tuple, then you could have each child in the worker pool extract a single value from this queue so each worker can have its own, unique port value. this port number is supposed to be used once, namely during initialization. Quite usual situation with conections is so, that it is a bit expensive to initiate it each time the connection is about to be used. So, it is often initialized once and dropped only when all communication is done. In contrast, your case looks for me that you rather propose to initiate the connection each time a new job comes from queue for an execution. Regards, Valery -- http://mail.python.org/mailman/listinfo/python-list
How to initialize each multithreading Pool worker with an individual value?
Hi, multithreading.pool Pool has a promissing initializer argument in its constructor. However it doesn't look possible to use it to initialize each Pool's worker with some individual value (I'd wish to be wrong here) So, how to initialize each multithreading Pool worker with the individual values? The typical use case might be a connection pool, say, of 3 workers, where each of 3 workers has its own TCP/IP port. from multiprocessing.pool import Pool def port_initializer(_port): global port port = _port def use_connection(some_packet): global _port print sending data over port # %s % port if __name__ == __main__: ports=((4001,4002, 4003), ) p = Pool(3, port_initializer, ports) # oops... :-) some_data_to_send = range(20) p.map(use_connection, some_data_to_send) best regards -- Valery A.Khamenya -- http://mail.python.org/mailman/listinfo/python-list
Re: How to initialize each multithreading Pool worker with an individual value?
On Tue, Nov 30, 2010 at 1:35 PM, Valery Khamenya khame...@gmail.com wrote: Hi, multithreading.pool Pool has a promissing initializer argument in its constructor. However it doesn't look possible to use it to initialize each Pool's worker with some individual value (I'd wish to be wrong here) So, how to initialize each multithreading Pool worker with the individual values? The typical use case might be a connection pool, say, of 3 workers, where each of 3 workers has its own TCP/IP port. from multiprocessing.pool import Pool def port_initializer(_port): global port port = _port def use_connection(some_packet): global _port print sending data over port # %s % port if __name__ == __main__: ports=((4001,4002, 4003), ) p = Pool(3, port_initializer, ports) # oops... :-) some_data_to_send = range(20) p.map(use_connection, some_data_to_send) Using an initializer with multiprocessing is something I've never tried. I have used queues with multiprocessing though, and I believe you could use them, at least as a fallback plan if you prefer to get the initializer to work. If you create in the parent a queue in shared memory (multiprocessing facilitates this nicely), and fill that queue with the values in your ports tuple, then you could have each child in the worker pool extract a single value from this queue so each worker can have its own, unique port value. HTH -- http://mail.python.org/mailman/listinfo/python-list
Re: How to initialize each multithreading Pool worker with an individual value?
On Wed, Dec 1, 2010 at 7:35 AM, Valery Khamenya khame...@gmail.com wrote: multithreading.pool Pool has a promissing initializer argument in its constructor. However it doesn't look possible to use it to initialize each Pool's worker with some individual value (I'd wish to be wrong here) So, how to initialize each multithreading Pool worker with the individual values? The typical use case might be a connection pool, say, of 3 workers, where each of 3 workers has its own TCP/IP port. from multiprocessing.pool import Pool def port_initializer(_port): global port port = _port def use_connection(some_packet): global _port print sending data over port # %s % port if __name__ == __main__: ports=((4001,4002, 4003), ) p = Pool(3, port_initializer, ports) # oops... :-) some_data_to_send = range(20) p.map(use_connection, some_data_to_send) I assume you are talking about multiprocessing despite you mentioning multithreading in the mix. Have a look at the source code for multiprocessing.pool and how the Pool object works and what it does with the initializer argument. I'm not entirely sure it does what you expect and yes documentation on this is lacking... cheers James -- -- James Mills -- -- Problems are solved by method -- http://mail.python.org/mailman/listinfo/python-list
