[email protected] wrote: > We need schduler to do certain tasks basically downloading and parsing > html files & RSS from many other sites. > > We would like to have a job scheduler with db backend to add and > remove the tasks using pylons. > > Have you used any specific job queue with pylons? Could you please > recommend some?
I would recommend taking a look at Apache ActiveMQ [1] as a message broker. It's not written in Python (more on that below), but it's easy to use and you can interface with it in Python. We use the STOMP protocol w/ ActiveMQ from Pylons to push messages (usually JSON-structured) onto worker queues or broadcast messages to topics that our GUI clients subscribe to (to be notified of changes). STOMP protocol has a few client implementations in python. There's stomp.py [2] which we're using from Pylons right now (though we simplified it to remove the "listener" parts). There's also stomper [3] which seems to be better architected, in that the protocol is separate from the transport. We ended up using Stomper with Twisted for our worker daemons (which subscribe to the queues), since stomp.py does some stuff under the hood with spawning off another thread that doesn't necessarily play nice with other frameworks. Anyway, there are other message brokers out there, but ActiveMQ was *extremely* easy to just get up & running and supported the concept of topics and queues. In default mode it will persist queues to files, but can be configured to use a database. There's also MorbidQ (written in Python, uses Twisted), but it was both difficult to get working and also seems to hack around with message bodies such that it is not possible to send binary data, etc. I also don't think it supported persisting the messages to a database. In short, it seemed to be a somewhat experimental implementation and didn't feel at all like something I'd want to trust with anything important. I'd be interested in learning the best way to manage socket connections from within a pylons app. In our case, we're setting up & tearing down the sockets with every request. This probably isn't optimum, but we only open those connections somewhat rarely; also I'm honestly not sure where the *right* place to put that would be. app_globals.py? (Globals() is a stacked object proxy, right?) Anyway, good luck in your search! (And let us know what you end up using.) Cheers, Hans [1] http://activemq.apache.org/ [2] http://code.google.com/p/stomppy/ [3] http://code.google.com/p/stomper/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~----------~----~----~----~------~----~------~--~---
