Re: Django Celery FIFO ORDER

[Nikolas Stevenson-Molnar]

No problem; best of luck! :)

_Nik

On 12/17/2012 10:47 AM, psychok7 wrote:
> ok i see your point.. Well i am going to use my redis db already used
> by celerey to do the sorting then. hope its fast enough.
>
> Thanks for your time to help me out ;)
>
> On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json
> messages). One written in Django and the other in Php. I have an
> exact database replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved
> on the current app database, and start a cron job with
> celery/redis to update the remote database for the other app using
> rest.
>
> /*My question is, how do i attribute the same worker to my tasks
> in order to keep a FIFO order?*/
>
> I need my data to be consistent and FIFO is really important.
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-users/-/bvwtkC89v2kJ.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[psychok7]

ok i see your point.. Well i am going to use my redis db already used by 
celerey to do the sorting then. hope its fast enough.

Thanks for your time to help me out ;)

On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json messages). 
> One written in Django and the other in Php. I have an exact database 
> replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved on the 
> current app database, and start a cron job with celery/redis to update the 
> remote database for the other app using rest.
>
> *My question is, how do i attribute the same worker to my tasks in order 
> to keep a FIFO order?*
>
> I need my data to be consistent and FIFO is really important.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/bvwtkC89v2kJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[Nikolas Stevenson-Molnar]

If you're processing 10k requests serially, then the database won't be
your bottleneck. Since you'd be sorting by primary key, and that gets
automatically indexed, your database won't even flinch at selecting the
top row from 10k records. The real bottleneck is going to be the fact
that you're processing one at a time, and that you have to wait to send
the REST request and for it to return. And there's going to be no way
(that I can see) to scale that until you rethink the architecture.

_Nik

On 12/15/2012 6:27 AM, psychok7 wrote:
> hey Nikolas, it looks like a solution that would work in fact, but
> there are some limitations in terms of performance (i need my app to
> be scalable). If for each request i sort my database it will
> eventually become slow for 10.000 requests, so i was actually looking
> for a more scalable solution.
>
> For know i can work with your idea, but wouldn't want it to be long
> term. If you have any more ideas please share.
>
> On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json
> messages). One written in Django and the other in Php. I have an
> exact database replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved
> on the current app database, and start a cron job with
> celery/redis to update the remote database for the other app using
> rest.
>
> /*My question is, how do i attribute the same worker to my tasks
> in order to keep a FIFO order?*/
>
> I need my data to be consistent and FIFO is really important.
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-users/-/dmZ0utouw48J.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[psychok7]

hey Nikolas, it looks like a solution that would work in fact, but there 
are some limitations in terms of performance (i need my app to be 
scalable). If for each request i sort my database it will eventually become 
slow for 10.000 requests, so i was actually looking for a more scalable 
solution.

For know i can work with your idea, but wouldn't want it to be long term. 
If you have any more ideas please share.

On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json messages). 
> One written in Django and the other in Php. I have an exact database 
> replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved on the 
> current app database, and start a cron job with celery/redis to update the 
> remote database for the other app using rest.
>
> *My question is, how do i attribute the same worker to my tasks in order 
> to keep a FIFO order?*
>
> I need my data to be consistent and FIFO is really important.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/dmZ0utouw48J.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[Nikolas Stevenson-Molnar]

Ok, how about this: instead of sending the POST data as part of your
Celery task, serialize the data to your DB with an auto-increment id
(which is default for Django model ids), then have he celery task grab
the data from the database, sorted by asc id. Then you'll always be
processing the data the order it was received.

_Nik

On 12/14/2012 2:05 AM, psychok7 wrote:
> Ok i am going to detail what i want to do a little further:
>
> So i have this django app, and when i press submit after i fill in the
> form my celery worker wakes up and takes care of taking that submitted
> data and posting to a remote server. This i can do without problems.
>
> Now, imagine that my internet goes down at that exact time, my celery
> worker keeps retrying to send until it is successful  But imagine i do
> another submit before my previous data is submitted, my data wont be
> consistent on the other remote server.
>
> Now that is my problem. I am not able to make this requests FIFO with
> the retry option given by celery so i that's were i need some help
> figuring that out.
>
>
> On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json
> messages). One written in Django and the other in Php. I have an
> exact database replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved
> on the current app database, and start a cron job with
> celery/redis to update the remote database for the other app using
> rest.
>
> /*My question is, how do i attribute the same worker to my tasks
> in order to keep a FIFO order?*/
>
> I need my data to be consistent and FIFO is really important.
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-users/-/9uZtJG0v85UJ.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[psychok7]

Ok i am going to detail what i want to do a little further:

So i have this django app, and when i press submit after i fill in the form 
my celery worker wakes up and takes care of taking that submitted data and 
posting to a remote server. This i can do without problems.

Now, imagine that my internet goes down at that exact time, my celery 
worker keeps retrying to send until it is successful  But imagine i do 
another submit before my previous data is submitted, my data wont be 
consistent on the other remote server.

Now that is my problem. I am not able to make this requests FIFO with the 
retry option given by celery so i that's were i need some help figuring 
that out.


On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json messages). 
> One written in Django and the other in Php. I have an exact database 
> replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved on the 
> current app database, and start a cron job with celery/redis to update the 
> remote database for the other app using rest.
>
> *My question is, how do i attribute the same worker to my tasks in order 
> to keep a FIFO order?*
>
> I need my data to be consistent and FIFO is really important.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/9uZtJG0v85UJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[Nikolas Stevenson-Molnar]

Sorry, I'm not sure I understand the full problem correctly. If you're
running these tasks serially, why bother with a queue at all? Just run
each task one after the other in the same process.

_Nik

On 12/11/2012 8:01 AM, psychok7 wrote:
> I got the named queues working with 1 worker, but i am not able to
> make it FIFO.. Its only fifo for the tasks that were rescheduled, but
> my latest task runs and i only want that to happen after the other
> ones get executed..
>
> any more ideas?
>
> On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json
> messages). One written in Django and the other in Php. I have an
> exact database replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved
> on the current app database, and start a cron job with
> celery/redis to update the remote database for the other app using
> rest.
>
> /*My question is, how do i attribute the same worker to my tasks
> in order to keep a FIFO order?*/
>
> I need my data to be consistent and FIFO is really important.
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-users/-/1Ud3yLDgjicJ.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[psychok7]

I got the named queues working with 1 worker, but i am not able to make it 
FIFO.. Its only fifo for the tasks that were rescheduled, but my latest 
task runs and i only want that to happen after the other ones get executed..

any more ideas?

On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json messages). 
> One written in Django and the other in Php. I have an exact database 
> replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved on the 
> current app database, and start a cron job with celery/redis to update the 
> remote database for the other app using rest.
>
> *My question is, how do i attribute the same worker to my tasks in order 
> to keep a FIFO order?*
>
> I need my data to be consistent and FIFO is really important.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/1Ud3yLDgjicJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[Tom Evans]

On Tue, Dec 11, 2012 at 12:32 PM, psychok7  wrote:
> i need to get my queue back from the worker when i need to add a new task to
> the queue (if it isn't yet executed) in order to keep a FIFO order, i read
> somewhere that i cant do that and that its not a good idea.
>
> do you have a solution? my i am just thinking about this the wrong way,
> please let me know.
>

Hi Nuno

Nik gave you the appropriate solution:

Use named queues with celery:
http://docs.celeryproject.org/en/latest/userguide/workers.html#queues

Start a worker process with a single worker:
http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html#starting-the-worker-process

Set this worker to consume from the appropriate queue:
http://docs.celeryproject.org/en/latest/userguide/workers.html#queues-adding-consumers

There is no simple example showing exactly what you want, because it's
a little strange - normally you use celery to scale out, this
purposefully restricts the scale - so you will have to do some
investigation and trial and error.

Cheers

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[psychok7]

thanks guys you pointed me in the write path now ill try to figure it out 
by myself.


cheers

On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json messages). 
> One written in Django and the other in Php. I have an exact database 
> replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved on the 
> current app database, and start a cron job with celery/redis to update the 
> remote database for the other app using rest.
>
> *My question is, how do i attribute the same worker to my tasks in order 
> to keep a FIFO order?*
>
> I need my data to be consistent and FIFO is really important.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/bcrBdWk1ARkJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[psychok7]

i need to get my queue back from the worker when i need to add a new task 
to the queue (if it isn't yet executed) in order to keep a FIFO order, i 
read somewhere that i cant do that and that its not a good idea.

do you have a solution? my i am just thinking about this the wrong way, 
please let me know.

On Monday, December 10, 2012 4:20:52 PM UTC, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json messages). 
> One written in Django and the other in Php. I have an exact database 
> replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved on the 
> current app database, and start a cron job with celery/redis to update the 
> remote database for the other app using rest.
>
> *My question is, how do i attribute the same worker to my tasks in order 
> to keep a FIFO order?*
>
> I need my data to be consistent and FIFO is really important.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/QlYhBqPjSvUJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[Nikolas Stevenson-Molnar]

Use the manage.py command described here:
http://pypi.python.org/pypi/django-celery to start the worker (I use
supervisord to manage the process). If you only start the worker on a
single machine, then you'll only have one worker. This can be the same
machine that runs your webserver.

_Nik

On 12/10/2012 11:39 AM, Nuno Khan wrote:
> my problem is exactly setting up 1 worker.. can u provide some code
> please?
>
>
> On Mon, Dec 10, 2012 at 6:48 PM, Nikolas Stevenson-Molnar
> > wrote:
>
> You could use a separate queue for these tasks, and set up a
> single worker for that queue.
>
> _Nik
>
> On 12/10/2012 8:20 AM, psychok7 wrote:
>>
>> So I have this 2 applications connected with a REST API (json
>> messages). One written in Django and the other in Php. I have an
>> exact database replica on both sides (using mysql).
>>
>> When i press "submit" on one of them, i want that data to be
>> saved on the current app database, and start a cron job with
>> celery/redis to update the remote database for the other app
>> using rest.
>>
>> /*My question is, how do i attribute the same worker to my tasks
>> in order to keep a FIFO order?*/
>>
>> I need my data to be consistent and FIFO is really important.
>>
>> -- 
>> You received this message because you are subscribed to the
>> Google Groups "Django users" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/django-users/-/1JR6HYArsgQJ.
>> To post to this group, send email to
>> django-users@googlegroups.com .
>> To unsubscribe from this group, send email to
>> django-users+unsubscr...@googlegroups.com
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/django-users?hl=en.
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To post to this group, send email to django-users@googlegroups.com
> .
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>
>
>
> -- 
> Nuno Khan
> Software Engineer
> Cellphone : (+351) 962826983
> Email: nun...@gmail.com 
> Url: http://www.linkedin.com/pub/nuno-khan/29/132/6b2
>
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[Nuno Khan]

my problem is exactly setting up 1 worker.. can u provide some code please?


On Mon, Dec 10, 2012 at 6:48 PM, Nikolas Stevenson-Molnar <
nik.mol...@consbio.org> wrote:

>  You could use a separate queue for these tasks, and set up a single
> worker for that queue.
>
> _Nik
>
> On 12/10/2012 8:20 AM, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json messages).
> One written in Django and the other in Php. I have an exact database
> replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved on the
> current app database, and start a cron job with celery/redis to update the
> remote database for the other app using rest.
>
> *My question is, how do i attribute the same worker to my tasks in order
> to keep a FIFO order?*
>
> I need my data to be consistent and FIFO is really important.
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-users/-/1JR6HYArsgQJ.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>



-- 
Nuno Khan
Software Engineer
Cellphone : (+351) 962826983
Email: nun...@gmail.com
Url: http://www.linkedin.com/pub/nuno-khan/29/132/6b2

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django Celery FIFO ORDER

[Nikolas Stevenson-Molnar]

You could use a separate queue for these tasks, and set up a single
worker for that queue.

_Nik

On 12/10/2012 8:20 AM, psychok7 wrote:
>
> So I have this 2 applications connected with a REST API (json
> messages). One written in Django and the other in Php. I have an exact
> database replica on both sides (using mysql).
>
> When i press "submit" on one of them, i want that data to be saved on
> the current app database, and start a cron job with celery/redis to
> update the remote database for the other app using rest.
>
> /*My question is, how do i attribute the same worker to my tasks in
> order to keep a FIFO order?*/
>
> I need my data to be consistent and FIFO is really important.
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-users/-/1JR6HYArsgQJ.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Django Celery FIFO ORDER

[psychok7]

So I have this 2 applications connected with a REST API (json messages). 
One written in Django and the other in Php. I have an exact database 
replica on both sides (using mysql).

When i press "submit" on one of them, i want that data to be saved on the 
current app database, and start a cron job with celery/redis to update the 
remote database for the other app using rest.

*My question is, how do i attribute the same worker to my tasks in order to 
keep a FIFO order?*

I need my data to be consistent and FIFO is really important.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-users/-/1JR6HYArsgQJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.