Between your initial layout of the problems with Celery and answers to the questions pointing out problems with RQ, I'm curious if we should consider alternatives to these two big names?
Have you looked into Kuyruk or Dramatiq which both still support RabbitMQ for instance? See a list here: https://www.fullstackpython.com/task-queues.html --Dana Dana Walker Associate Software Engineer Red Hat <https://www.redhat.com> <https://red.ht/sig> On Tue, Mar 20, 2018 at 4:18 PM, Brian Bouterse <bbout...@redhat.com> wrote: > Thank you for the questions. I put some responses inline. > > On Tue, Mar 20, 2018 at 4:03 PM, Austin Macdonald <aus...@redhat.com> > wrote: > >> Not being familiar with RQ, I have questions (but no opinion). >> >> Will we also be replacing RabbitMQ with Redis? >> > Yes this would replace RabbitMQ/Qpid with Redis. RQ only works with Redis. > > >> Does anyone on the team have experience with RQ? In production? >> > I do not have production experience. I've run it locally to see if it runs > as easy as they say it does. I also reviewed their code some. > > >> How well does RQ scale? >> > I started up several workers on my machine and they processed some dummy > tasks I dispatched well. Pulp has very low throughput needs, maybe a few > tasks per minute per worker. > > >> Is RQ's use of `pickle` a problem? https://pulp.plan.io/issues/23 >> > It's not preferred, but I think it's ok to go out with that for Pulp3 at > the start. At some point we could swap it for a json serializer which would > be safer. > > >> RQ doesn't work on Windows. Is that a problem? (jk) >> > Pulp could maybe run on Windows one day so this isn't ideal. > > >> >> >> On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse <bbout...@redhat.com> >> wrote: >> >>> Motivation: >>> 1. Celery causes many bugs and issues for Pulp2 and 3 users and there is >>> no end in sight. >>> >>> 2. The Pulp core team spends a lot of effort fixing Celery bugs. It's >>> often times just us doing it with little or no assistance from the upstream >>> communities. It's across 4 projects: celery, kombu, billiard, and pyamqp. >>> >>> 3. Celery will never allow a coverage report to be generated when >>> pulp-smash runs because Celery forked the multiprocessing library into >>> something called billiard. This will limit Pulp forever. >>> >> >>> 4. I don't want to work with Celery anymore and I think the other >>> maintainers (@dalley, @daviddavis) may feel the same. It's an endless >>> headache. Even basic things don't work in Celery regularly. >>> >>> Proposed change: Replace Pulp3's usage of Celery with RQ ( >>> http://python-rq.org/) >>> >>> We would keep the exact same design of a resource manager with n >>> workers, each worker pulling it's work exclusively from a dedicated queue. >>> I've looked into porting pulp3 to it and it's doable because all the same >>> concepts are there. There are a few details to work out, but I wanted to >>> start the "should we" discussion before we do all-out technical planning. >>> >>> When would we do this? I'm proposing soon. It doesn't need to block the >>> beta, but soon would be good. I don't think users will care much except for >>> their systemd files, but it is fundamental and important to pulp3 so we >>> want to get it testing sooner. >>> >>> Ideas, comments, questions are welcome! >>> >>> Thanks, >>> Brian >>> >>> _______________________________________________ >>> Pulp-dev mailing list >>> Pulp-dev@redhat.com >>> https://www.redhat.com/mailman/listinfo/pulp-dev >>> >>> >> > > _______________________________________________ > Pulp-dev mailing list > Pulp-dev@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-dev > >
_______________________________________________ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
