Thank you for the questions. I put some responses inline. On Tue, Mar 20, 2018 at 4:03 PM, Austin Macdonald <[email protected]> wrote:
> Not being familiar with RQ, I have questions (but no opinion). > > Will we also be replacing RabbitMQ with Redis? > Yes this would replace RabbitMQ/Qpid with Redis. RQ only works with Redis. > Does anyone on the team have experience with RQ? In production? > I do not have production experience. I've run it locally to see if it runs as easy as they say it does. I also reviewed their code some. > How well does RQ scale? > I started up several workers on my machine and they processed some dummy tasks I dispatched well. Pulp has very low throughput needs, maybe a few tasks per minute per worker. > Is RQ's use of `pickle` a problem? https://pulp.plan.io/issues/23 > It's not preferred, but I think it's ok to go out with that for Pulp3 at the start. At some point we could swap it for a json serializer which would be safer. > RQ doesn't work on Windows. Is that a problem? (jk) > Pulp could maybe run on Windows one day so this isn't ideal. > > > On Tue, Mar 20, 2018 at 3:35 PM, Brian Bouterse <[email protected]> > wrote: > >> Motivation: >> 1. Celery causes many bugs and issues for Pulp2 and 3 users and there is >> no end in sight. >> >> 2. The Pulp core team spends a lot of effort fixing Celery bugs. It's >> often times just us doing it with little or no assistance from the upstream >> communities. It's across 4 projects: celery, kombu, billiard, and pyamqp. >> >> 3. Celery will never allow a coverage report to be generated when >> pulp-smash runs because Celery forked the multiprocessing library into >> something called billiard. This will limit Pulp forever. >> > >> 4. I don't want to work with Celery anymore and I think the other >> maintainers (@dalley, @daviddavis) may feel the same. It's an endless >> headache. Even basic things don't work in Celery regularly. >> >> Proposed change: Replace Pulp3's usage of Celery with RQ ( >> http://python-rq.org/) >> >> We would keep the exact same design of a resource manager with n workers, >> each worker pulling it's work exclusively from a dedicated queue. I've >> looked into porting pulp3 to it and it's doable because all the same >> concepts are there. There are a few details to work out, but I wanted to >> start the "should we" discussion before we do all-out technical planning. >> >> When would we do this? I'm proposing soon. It doesn't need to block the >> beta, but soon would be good. I don't think users will care much except for >> their systemd files, but it is fundamental and important to pulp3 so we >> want to get it testing sooner. >> >> Ideas, comments, questions are welcome! >> >> Thanks, >> Brian >> >> _______________________________________________ >> Pulp-dev mailing list >> [email protected] >> https://www.redhat.com/mailman/listinfo/pulp-dev >> >> >
_______________________________________________ Pulp-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-dev
