Davin Potts added the comment: My understanding is that example uses a queue.Queue() to demonstrate how to create a custom, remote service from scratch. The implementation in this simple example lacks the sophistication of multiprocessing.Queue() for handling situations such as the one raised by the OP. The example was not attempting to demonstrate a comprehensive replacement for multiprocessing.Queue(), rather it was attempting to demonstrate the mechanism for creating and consuming a callable service hosted by a remote manager. The documentation currently does not introduce this example well nor describe the above motivation.
As to why this simplistic implementation of a distributed queue appears to lose an item when the client is killed, it works in the following way: 1. Let's say a server is started to hold a queue.Queue() which is populated with 1 item. 2. A client requests an item from the server. 3. The server receives the request and performs a blocking q.get() (where q is the queue.Queue() object held by the server). 4. When the q.get() releases and returns an item, q has had one item removed leaving a queue size of 0 in our scenario, and then that item is sent from the server to the client. 5. A client requests another item from the server. 6. The server receives the request and performs a blocking q.get() on the queue. Because there's nothing left to grab from the queue, the server blocks and waits for something to magically appear in the queue. We'll have a "producer" put something into the queue in a moment but for the time being the server is stuck waiting on the q.get() and likewise the client is waiting on a response from the server. 7. That client is killed in an unexpected, horrible death because someone accidentally hits it with a Cntrl-C. 8. A "producer" comes along and puts a new item into the server's queue. 9. The server's blocking q.get() call releases, q has had one item removed leaving a queue size of 0 again, and then that item is sent from the server to the client only the client is dead and the transmission fails. 10. A "producer" comes along and puts another new item into the server's queue. 11. The someone who accidentally, horribly killed the client now frantically restarts the client; the client requests an item from the server and the server responds with a new item. However, this is the item introduced in step 10 and not the item from step 8. Hence the item from step 8 appears lost. Note that in our simplistic example from the docs, there is no functionality to repopulate the queue object when communication of the item fails to complete. In general, a multiprocessing.manager has no idea what a manager will contain and has no insight on what to do when a connection to a client is severed. Augmenting the example in the docs to cover situations like this would significantly complicate the example but there are many others to consider on the way to building a comprehensive solution -- instead a person should choose multiprocessing.Queue() unless they have something particular in mind. I think the example should be better introduced (the intro is terse) to explain its purpose and warn that it does not offer a comprehensive replacement for multiprocessing.Queue(). It does not need to go into all of the above explanation. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29454> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
