Re: How many workers do you run on one machine using django celery?

Do also check out uWSGI spooler - i haven't tested with heavy loads yet, but looks to do the work for a reasonably small site. -V @venkasub On Sun, Apr 21, 2013 at 8:37 PM, sparky wrote: > wow, some good advice here thanks. I tested with RabbitMQ and its fast, > all

Re: How many workers do you run on one machine using django celery?

wow, some good advice here thanks. I tested with RabbitMQ and its fast, all I can say is it seems to be SQS. My advice don't use it! -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from i

Re: How many workers do you run on one machine using django celery?

You can't test a system like this by sending one message: you're just testing the latency, not throughput. Latency is the end-to-end time it takes for a single message to make its way through the system. Throughput is the number of total messages per second that can make their way through. As l

Re: How many workers do you run on one machine using django celery?

you should try to not use Task Dispatching. Rather if you are doing this once a day, have a periodic task run every 24 hours. In the same module as the thread try doing something like what is at the bottom of this page http://docs.python.org/2/library/queue.html . It is a queue thread daemon ar

Re: How many workers do you run on one machine using django celery?

Sorry darn phone changed daemon to farming On Apr 21, 2013 9:39 AM, "Michael Hernandez" < michael.hernandez1...@gmail.com> wrote: > I have been working with heavy reporting and analytics lately. I would > suggest use thread queuing with many farming threads. I use python thread > but Greenlets it

Re: How many workers do you run on one machine using django celery?

I have been working with heavy reporting and analytics lately. I would suggest use thread queuing with many farming threads. I use python thread but Greenlets it some else should be better On Apr 21, 2013 5:48 AM, "sparky" wrote: > One last thing to add, the task it's self does not seems to be th

Re: How many workers do you run on one machine using django celery?

Maybe SQS is the issue here have a read: http://dataexcursions.wordpress.com/2011/12/15/amazon-sqs-vs-rabbitmq/ -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to

Re: How many workers do you run on one machine using django celery?

One last thing to add, the task it's self does not seems to be the issue, 'got message from broker' is the 3-4 second wait I can see. -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from

Re: How many workers do you run on one machine using django celery?

Thanks for the responses it's very helpful. You are right, I won't have 100,000 tasks every seconds it's just a huge batch I have to send which at the moment would be 100,000 tasks. But just to be clear: Loop each contact in my DB: TASK: SEND 1 SMS FOR CONTACT I'm use Amazon SQS for the bro

Re: How many workers do you run on one machine using django celery?

Both Michael and Shawn are spot-on in terms of scaling and using the queuing. However, I'd like to add that 5 seconds to complete a single task like this seems way too slow to me. I don't have much experience with sending SMS but if you're using an external SMS API, it should be extremely quick. I

Re: How many workers do you run on one machine using django celery?

In addition to Michael's good comments: I suspect you won't have 100,000 tasks coming in every second of every day. If you have to send out SMS messages and some of them take a few minutes to go out, that should be fine for most purposes. In addition, some SMS services have some limit per minute/h

Re: How many workers do you run on one machine using django celery?

messages to be sent out and send all 10 of them. This will improve scalability as well. On Saturday, April 20, 2013 7:28:55 PM UTC-4, sparky wrote: > > Quick question, just so I can compare, I would really like to hear other > devs experience. > > *How many workers do you run

How many workers do you run on one machine using django celery?

Quick question, just so I can compare, I would really like to hear other devs experience. *How many workers do you run on one machine using django celery?* I'm running 3 workers on an EC2 small instance. It takes around 5 seconds to complete 1 task running all 3 workers, does this sound