Out of curiosity why ActiveMQ rather than RabbitMQ (except that it is an Apache project that is)?
J, On Thu, Mar 26, 2020 at 6:40 PM Alexandre Vermeerbergen < avermeerber...@gmail.com> wrote: > Hi, > > To be honest I found RQ while searching why Airflow isn't currently > supporting ActiveMQ instead of RabbitMQ. > > I found that RabbitMQ come from a dependency of Celery on Kombu; and I > found an incident in Kombu discussing about ActiveMQ support: > https://github.com/celery/kombu/issues/548 > > I asked if ActiveMQ support was still planned in Kombu, and to my > surprise I got the hint that some projects have moved from Celery to > RQ. > > Now I realize that RQ means using Redis as the queuing infrastructure, > which IHMO is not a so good idea when one realize that it takes 6 > nodes to make Redis HA. > > To sum up, I'm fine if Airflow sticks to Celery, and then I'll see if > there's a way to revive interest in adding ActiveMQ support to Kumbu. > After all, Airflow and ActiveMQ are Apache projects, so it seems to be > fair to look for ActiveMQ support for Airflow to get an Apache-based > stack, isn't it ? > > Kind regards, > Alexandre > > Le jeu. 26 mars 2020 à 18:29, Daniel Imberman > <daniel.imber...@gmail.com> a écrit : > > > > I mean… we’re not planning it (kind of an “if it’s not broke don’t fix > it” situation), but I don’t think we’re super set on Celery. Would you be > interested in making an AIP to discuss potential benefits? > > On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen < > avermeerber...@gmail.com>, wrote: > > > Hi there, > > > > > > Looks like couple of projects have moved from Celery to RQ: > > > * https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning > > > * https://frappe.io/blog/technology/why-we-moved-from-celery-to-rq > > > > > > or are planning such move: > > > * https://github.com/getredash/redash/issues/4092 > > > > > > Is such move considered for Airflow? > > > > > > Kind regards, > > > Alexandre > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
