RabbitMQ with Celery is a distributed asynchronous code execution service. Keyword Distributed. The best thing to do in run several instances per machine of djcelery behind a supervisord. When you start encountering performance hits start by adding more services on the machine with supervisord. You will hit a limit at some point where you wont see any increase in performance from this. It is at this point that i suggest you scale down the amount of services on that machine and add another machine to the rabbit stack with the same configuration. RabbitMQ should handle the load balancing itself. However it is at this point i believe that logging will become more difficult. So be sure to have a solution in place for handling that.
You should also look into batching your processes among tasks, if you are going to be running so many tasks, than cant a task hold the last 10 sms messages to be sent out and send all 10 of them. This will improve scalability as well. On Saturday, April 20, 2013 7:28:55 PM UTC-4, sparky wrote: > > Quick question, just so I can compare, I would really like to hear other > devs experience. > > *How many workers do you run on one machine using django celery?* > > I'm running 3 workers on an EC2 small instance. It takes around 5 seconds > to complete 1 task running all 3 workers, does this sound right to you? > > My issue is I could have 100,000 tasks very soon... scale wise I'm unsure > what I'm going to need to do this. Bigger CPU, RAM and X workers, 5 seconds > is far too long for me. All the task is doing is sending a SMS message HTTP > thats it. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
