Hi Joe, That sounds fantastic - I think we might look at integrating it into the application as it's pretty vital (imagine having to wait for a 50 or 100mb repository to clone!).
If your interested, I'd be happy to provide some code back if we can make any improvements. For example, we're using a forked version of django-dbsettings and it would be very handy to put the settings to control the application here, stuff like throttling, limits, etc. Does it have it's own settings file if it's using a SQLLite backend? Would it be much work, or even worth it to integrate it into the database within our app? Of course our app supports using SQLLite as a DB option (especially good for our app running off a pen drive :) On Jan 30, 2008 5:38 PM, Joseph Heck <[EMAIL PROTECTED]> wrote: > > Hi Tane, > > Yes - it can help. It's a good tool to basically take long running > tasks and move them to an asynchronous mode. The trick, as always, is > getting the results back to the user. There are basically two ways to > use DQS with your app to make this work for you. > > The way it was originally cobbled together (and the current trunk > release) is set up to run standalone. No impact on your primary > database or even CPU. It's set up right out of the project with SQLite > as a backing store. That's reasonably efficient and probably fine for > any light uses. If you've got something sending gazillions of > messages, or the persistence is of critical important, I'll leave it > to you to put up a clustered database underneath it. DQS can be scaled > like any other django app. > > When it's running standalone, you communicate with it via http > requests. There are setup calls that you'd likely use once (or just > put the data into the fixtures) to create a queue. Once you had a > queue created, the time to insert something into the queue is the > latency of making that http request and getting a response. Inserting > into the queue (current release) is an HTTP Post, with the data being > a string. If you want to toss around slightly more complex data > structures, I'd recommend using JSON. > > Something like the following: > > import httplib2 > client = httplib2.Http() > body_string = "message=%s" % (message_to_be_inserted_in_queue,) > (response,content) = > client.request(uri='http://dqshost:8000/q/queue_name_here/put', > method='POST', body=body_string) > if response['status'] != 200: > print "Queue didn't accept the input" > > The DQS architecture is based on Amazon's SQS, so removing a item from > a queue is actually a two step process. You ask for the item at the > top of the queue, and just asking for it makes it "invisible" to > others for a certain timeout (our default is 5 minutes). If you don't > come back and invoke an explicit "delete" on that item, it will become > visible again for another process to pop it off the queue. > > The component that drives that "every 5 minutes clear things up" isn't > automatic, so in practice asking for something off the queue will make > it invisible forever unless you implement the clearing of expirations > as well. In practice, something like the following in a cron job > running every 5 minutes works fine: > curl -i http://dqshost:8000/q/queue_name_here/clearexpire/ > > While we've been working on the unstable branch and making this thing > more REST like (as opposed to my bastard REST/RPC initial > implementation), I put together a benchmark script to see how fast > things were going - basically to make sure we didn't bork anything in > the process of development. That script > (http://django-queue-service.googlecode.com/svn/branches/unstable/benchmark.py) > has a bunch of reasonably well encapsulated example calls to the DQS > using urllib2 like the example above. Check it out for examples of > getting something from the queue and deleting that item from the queue > - the code should be pretty self-evident. > > The other option is to embed the DQS application into your Django app. > In that case, we've kept things pretty nicely encapsulated into "qs" - > you add it to your application much like you add the Admin > application. You'll need to include: > (r'^', include('qs.queue.urls')) > into your urls.py, and add 'qs.queue' to INSTALL_APPS in settings.py. > At this point you can insert items into the queue and remove items > from a queue using django ORM calls. We really don't have the code set > up for that, but it's not hard - I'd recommend looking at the view > code that we use > (http://django-queue-service.googlecode.com/svn/trunk/qs/queue/views.py) > for examples of how to yank something off the queue. > > Ironically, that's how this idea actually started (for me, anyway). I > had some long running processes that I wanted to serialize for access > and effects, so I created a sub-application in a larger django app > that effectively used a database as a big queue by sorting results on > timestamps. As I investigated more and learned about other "service > bus" concepts, it turns out they're all variations and glorified > versions of this concept, with various add-ons to deal with other > interesting and specific problems. The core, though, it pretty simple. > I wanted a stand-alone version that I could run only having Python 2.5 > on a box (windows, linux, mac, whatever) - so that's what became the > seed for DQS at OSCON 2007. > > -joe > > > > On Jan 27, 2008 1:30 PM, Tane Piper <[EMAIL PROTECTED]> wrote: > > > > Hrm, looking at it, this looks very interesting for my project too! > > > > My project has a bottleneck, and if I describe it, possibly you could > > confirm or deny if your app will be useful. > > > > Our application is for managing mercurial repositories. Part of the > > functionality is to be able to create new repositories by either > > creating a empty directory or by cloning an existing one. When > > cloning a repo, after saving, a post_save signal calls mercurial's > > hg.clone function, and the screen waits until this is done, which can > > be a while for larger repos, or one on slower servers. > > > > I'd like to be able, once the user submits the form, validate the save > > information and create the database entry for it then hand off to a > > background process to get the repo, so acts more asynchronously. > > > > Any information would be greatly appreciated. > > > > On Jan 27, 2008 9:14 PM, Joseph Heck <[EMAIL PROTECTED]> wrote: > > > > > > Sorry for the lack of docs - I think we've all been focused elsewhere, > > > and the project is being somewhat quiet right now. > > > > > > Do you want to integrate it into your application directly, or use it > > > as a standalone critter? What kind of information on using it would be > > > most useful to you? > > > > > > I've got some ideas, but since I helped create it they may be well > > > skewed. I know the internals very well - and it all seems somewhat > > > obvious to me, although I know that's not an outside view at all. Your > > > suggestions for what you need would be very welcome and useful! > > > > > > Are you looking for an example of how to write a client and push and > > > pull messages from a queue? How to integrate it into your application > > > and what code to use to to push messages in? > > > > > > -joe > > > > > > > > > On Jan 26, 2008 10:48 PM, shabda <[EMAIL PROTECTED]> wrote: > > > > I want to use django-queue-service to run long running processess in > > > > background, but I can not find any documentation about this. Their > > > > website http://code.google.com/p/django-queue-service/ does not > > > > contain any docs. Does any one have any links, about how to use it? > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Tane Piper > > Blog - http://digitalspaghetti.me.uk > > Wii: 4734 3486 7149 1830 > > > > This email is: [ ] blogable [ x ] ask first [ ] private > > > > > > > > > > > > > > -- Tane Piper Blog - http://digitalspaghetti.me.uk Wii: 4734 3486 7149 1830 This email is: [ ] blogable [ x ] ask first [ ] private --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
