[
https://issues.apache.org/jira/browse/QPID-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Bouterse updated QPID-5637:
---------------------------------
Attachment: celery_worker_output.txt
Example worker output showing the task is failing.
> qpid.messaging Issues With Forking
> ----------------------------------
>
> Key: QPID-5637
> URL: https://issues.apache.org/jira/browse/QPID-5637
> Project: Qpid
> Issue Type: Bug
> Components: Python Client
> Affects Versions: 0.24
> Reporter: Brian Bouterse
> Fix For: 0.18, 0.22, 0.24, 0.26, 0.27
>
> Attachments: celery_worker_output.txt, pid_aware_selector.patch
>
>
> qpid.messaging has an issue with forking in the following situation.
> 1. A parent Python process imports and uses qpid.messaging to connect to a a
> Qpid broker
> 2. The parent process forks a child process
> 3. The child process imports qpid.messaging and tries to connect to a Qpid
> broker.
> I expected to see the child process use qpid.messaging normally as it would
> if it weren't forked in the way described above. Instead, the server
> receives the opening of a TCP socket, but the client never sends the AMQP
> protocol announcement.
> [Forking bring child descriptors with
> it|http://man7.org/linux/man-pages/man2/fork.2.html] Because file
> descriptors on the parent and the child are the same, I expect qpid.messaging
> to have the parent and child process share the file descriptors and socket
> and work without any modification.
> There is at least one place where I do understand how this can be avoided.
> One of the issues is that the file descriptors registered by the Selector
> object inside of qpid.messaging are stale after the fork. The Selector
> object uses a singleton pattern to provide a reference to the same Selector
> object no matter how many times you call it. This selector object already
> has registered file descriptors with the filesystem, which allow the selector
> to read/write data in an I/O efficient manner. See the attached patch for an
> example of this.
> This is not the only issue though, even with this patch in place,
> communication flows, but queue creation, and deletion seem to fail. Here is
> the overall way to reproduce this issue:
> 1. clone our fork of kombu: `git clone g...@github.com:pulp/kombu.git`
> 2. Change into the kombu folder `cd kombu`
> 3. Switch to the branch containing the qpid code: `git checkout
> pulp-dep-3.0.15-with-qpid`
> 4. Install kombu onto your system or virtualenv (I do it systemwide using
> sudo): `sudo python setup.py develop`
> 5. install celery version 3.1.11. I do it using pip. `sudo pip install
> celery==3.1.11`
> 6. Install qpid.messaging and qpidtoollibs. One way I do it is systemwide
> using pip. `sudo pip install qpid-tools qpid-python`
> 7. Start up qpidd (We've been testing with 0.24 and auth off). `sudo -u
> qpidd qpidd --auth=no`
> 8. Put the attached file tasks.py into a directory
> 9. Open two terminals and change their working directory to be the same as
> step 8.
> 10. In one one terminal start the celery worker `celery worker -A
> tasks --loglevel=INFO -c1`
> 11. In the other terminal dispatch 10 tasks `python tasks.py`
> You should see exceptions raised similar to those in the attached file
> [celery_worker_output.txt]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
