Re: AMQP listening and user-facing daemon
Justin Chiu c.justi...@gmail.com writes: What is the best approach to writing a concurrent daemon that can execute callbacks for different types of events (AMQP messages, parsed output of a subprocess, HTTP requests)? I fear your question is too specific (which means you must investigate yourself). First of all, it is always very difficult to answer questions like the best way -- many many details are involved there. You provided details but to appreciate them one has to know AMQP, Openstack Notifications, twisted, greenlets, ... There will not be a lot of people with this knowledge (I do not belong to them). I believe that you will be able to realize a solution based on (Python) threads. It gives you the more freedom than twisted or greenlets. But a different approach might be better (in some respect). -- http://mail.python.org/mailman/listinfo/python-list
AMQP listening and user-facing daemon
Hi all, What is the best approach to writing a concurrent daemon that can execute callbacks for different types of events (AMQP messages, parsed output of a subprocess, HTTP requests)? I am considering [twisted][1], the built-in [threading][2] module, and [greenlet][3]. I must admit that I am very unfamiliar with concurrent programming and Python programming in general (formerly a data analysis driven procedural programmer). Any resources on threaded/concurrent programming (specifically daemons...not just multi-threading a single task) would be much appreciated. Thank you. Details: 1) Listens into AMQP messaging queues and executes callbacks when messages arrive. Example: Immediately after startup, the daemon continuously listens to the [Openstack Notifications messaging queue][4]. When a virtual machine is launched, a notification is generated by Openstack with the hostname, IP address, etc. The daemon should read this message and write some info to a log (or POST the info to a server, or notify the user...something simple). 2) Parse the output of a subprocess and execute callbacks based on the output. Example: Every 30 seconds, a system command [qstat][5] is run to query a job resource manager (e.g. TORQUE). Similar callbacks to 1). 3) Receive requests from a user and process them. I think this will be via WSGI HTTP. Example: User submits an XML template with virtual machine templates. The daemon does some simple XML parsing and writes a job script for the job resource manager. The job is submitted to the resource manager and the daemon continually checks for the status of the job with qstat and for messages from AMQP. It should return live feedback to the user and write to a log. [1]: https://twistedmatrix.com/trac/wiki/Documentation [2]: http://docs.python.org/2/library/threading.html [3]: http://greenlet.readthedocs.org/en/latest/ [4]: https://wiki.openstack.org/wiki/NotificationEventExamples#Immediate_Notifications: [5]: http://www.clusterresources.com/torquedocs21/commands/qstat.shtml Justin ChiuTRIUMF -- http://mail.python.org/mailman/listinfo/python-list
AMQP listening and user-facing daemon
Hi all, What is the best approach to writing a concurrent daemon that can execute callbacks for different types of events (AMQP messages, parsed output of a subprocess, HTTP requests)? I am considering [twisted][1], the built-in [threading][2] module, and [greenlet][3]. I must admit that I am very unfamiliar with concurrent programming and Python programming in general (formerly a data analysis driven procedural programmer). Any resources on threaded/concurrent programming (specifically daemons...not just multi-threading a single task) would be much appreciated. Thank you. Details: 1) Listens into AMQP messaging queues and executes callbacks when messages arrive. Example: Immediately after startup, the daemon continuously listens to the [Openstack Notifications messaging queue][4]. When a virtual machine is launched, a notification is generated by Openstack with the hostname, IP address, etc. The daemon should read this message and write some info to a log (or POST the info to a server, or notify the user...something simple). 2) Parse the output of a subprocess and execute callbacks based on the output. Example: Every 30 seconds, a system command [qstat][5] is run to query a job resource manager (e.g. TORQUE). Similar callbacks to 1). 3) Receive requests from a user and process them. I think this will be via WSGI HTTP. Example: User submits an XML template with virtual machine templates. The daemon does some simple XML parsing and writes a job script for the job resource manager. The job is submitted to the resource manager and the daemon continually checks for the status of the job with qstat and for messages from AMQP. It should return live feedback to the user and write to a log. [1]: https://twistedmatrix.com/trac/wiki/Documentation [2]: http://docs.python.org/2/library/threading.html [3]: http://greenlet.readthedocs.org/en/latest/ [4]: https://wiki.openstack.org/wiki/NotificationEventExamples#Immediate_Notifications: [5]: http://www.clusterresources.com/torquedocs21/commands/qstat.shtml Justin ChiuTRIUMF -- http://mail.python.org/mailman/listinfo/python-list
