Re: Alternative to celery

2017-05-11 Thread marcin . j . nowak 
RQ is simple, stable and reliable. Redis is also a persistent storage, but 
you should read about this and use the best solution for your case: 
https://redis.io/topics/persistence   
You may also consider deploying a fault-tolerant Redis cluster (min. 3 
nodes AFAIR)

RQ has different architecture than Celery. It is based on fork(), which is 
just slower. So you should ask yourself how many tasks per sec you need to 
handle. RQ may not be enough for all use cases.
Some things aren't OOTB, like repeating failed tasks (you should handle it 
manually, i.e. by using own worker class), nor  tasks scheduling (you 
should use rq-scheduler). 

RQ uses pickle, but there is a feature request about handling json and 
other serializers - https://github.com/nvie/rq/issues/369 


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/d2fc1e0f-f147-4ccd-991f-1b12e61c8964%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative to celery

2017-04-20 Thread Avraham Serour 
try googling 'python task queue'

see https://www.fullstackpython.com/task-queues.html

On Thu, Apr 20, 2017 at 12:25 PM, Andréas Kühne 
wrote:

> I don't know anything about the python-rq stuff - however redis can be
> configured to save to disk (see https://redis.io/topics/persistence) - so
> you wouldn't loose anything on a power-failure (if configured correctly).
> Redis is so much more than a cache nowadays.
>
> Regards,
>
> Andréas
>
> 2017-04-20 11:16 GMT+02:00 guettli :
>
>> Some years ago I used celery for a project.
>>
>> Today I face roughly the same use case again.
>>
>> The last time I used it, celery felt to heavy weighted, too complicated.
>>
>> I found this alternative: http://python-rq.org/
>>
>> There are two things which speak agains python-rq:
>>
>>  - redis is a cache. I need a persistent queue. If a power-failure
>> happens, no job must get lost.
>>
>>  - AFAIK python-rq uses python pickle to serialize data. I want a
>> language neutral data exchange format.
>>
>> At the moment my favourite protocol is gRPC (protocol-buffers). I never
>> used it up to now, but this would be a nice use case.
>>
>> Maybe I am missing the right terms. What does celery implement? According
>> to wikipedia "asynchronous task queue". But
>> my favorite search engine could not reveal a gRPC based server
>> implemented in Python 
>>
>> What do you think?
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-users.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/django-users/d43d2cef-f1a1-418d-9d62-ecce85961fbb%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/CAK4qSCefVFpgZEw4Ar4AGDUSdOH-
> A-7-kvgysqeHY9zET1DHZQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAFWa6t%2B7WTzt7o1SoOV45YrL%2B%3DaD_X30E1yNFj3-ki54ivNL2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Alternative to celery

2017-04-20 Thread Andréas Kühne 
I don't know anything about the python-rq stuff - however redis can be
configured to save to disk (see https://redis.io/topics/persistence) - so
you wouldn't loose anything on a power-failure (if configured correctly).
Redis is so much more than a cache nowadays.

Regards,

Andréas

2017-04-20 11:16 GMT+02:00 guettli :

> Some years ago I used celery for a project.
>
> Today I face roughly the same use case again.
>
> The last time I used it, celery felt to heavy weighted, too complicated.
>
> I found this alternative: http://python-rq.org/
>
> There are two things which speak agains python-rq:
>
>  - redis is a cache. I need a persistent queue. If a power-failure
> happens, no job must get lost.
>
>  - AFAIK python-rq uses python pickle to serialize data. I want a language
> neutral data exchange format.
>
> At the moment my favourite protocol is gRPC (protocol-buffers). I never
> used it up to now, but this would be a nice use case.
>
> Maybe I am missing the right terms. What does celery implement? According
> to wikipedia "asynchronous task queue". But
> my favorite search engine could not reveal a gRPC based server implemented
> in Python 
>
> What do you think?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/d43d2cef-f1a1-418d-9d62-ecce85961fbb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAK4qSCefVFpgZEw4Ar4AGDUSdOH-A-7-kvgysqeHY9zET1DHZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Alternative to celery

2017-04-20 Thread guettli 
Some years ago I used celery for a project.

Today I face roughly the same use case again.

The last time I used it, celery felt to heavy weighted, too complicated.

I found this alternative: http://python-rq.org/

There are two things which speak agains python-rq:

 - redis is a cache. I need a persistent queue. If a power-failure happens, 
no job must get lost.

 - AFAIK python-rq uses python pickle to serialize data. I want a language 
neutral data exchange format.

At the moment my favourite protocol is gRPC (protocol-buffers). I never 
used it up to now, but this would be a nice use case.

Maybe I am missing the right terms. What does celery implement? According 
to wikipedia "asynchronous task queue". But
my favorite search engine could not reveal a gRPC based server implemented 
in Python 

What do you think?


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/d43d2cef-f1a1-418d-9d62-ecce85961fbb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.