We use a similar approach like Bolke mentioned with running multiple Airflow instances.
I haven't read the Pandora article yet, but we have an Astronomer Open Edition (fully open source) that bundles similar tools like Prometheus, Grafana, Celery, etc with Airflow and a Docker Compose file if you're looking to get a setup like that up and running quickly. https://github.com/astronomerio/astronomer/blob/master/examples/airflow-enterprise/docker-compose.yml https://github.com/astronomerio/astronomer *Taylor Edmiston* Blog <http://blog.tedmiston.com> | Stack Overflow CV <https://stackoverflow.com/story/taylor> | LinkedIn <https://www.linkedin.com/in/tedmiston/> | AngelList <https://angel.co/taylor> On Tue, Apr 24, 2018 at 3:30 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Related blog post about multi-tenant Airflow deployment out of Pandora: > https://engineering.pandora.com/apache-airflow-at-pandora-1d7a844d68ee > > On Tue, Apr 24, 2018 at 10:20 AM, Bolke de Bruin <bdbr...@gmail.com> > wrote: > > > My suggestion would be to deploy airflow per project. You could even use > > airflow to manage your ci/cd pipeline. > > > > B. > > > > Sent from my iPhone > > > > > On 24 Apr 2018, at 18:33, Maxime Beauchemin < > maximebeauche...@gmail.com> > > wrote: > > > > > > People have been talking about namespacing DAGs in the past. I'd > > recommend > > > using tags (many to many) instead of categories/projects (one to many). > > > > > > It should be fairly easy to add this feature. One question is whether > > tags > > > are defined as code or in the UI/db only. > > > > > > Max > > > > > >> On Tue, Apr 24, 2018 at 1:48 AM, Song Liu <song...@outlook.com> > wrote: > > >> > > >> Hi, > > >> > > >> Basically the DAGs are created for a project purpose, so if I have > many > > >> different projects, will the Airflow support the Project concept and > > >> organize them separately ? > > >> > > >> Is this a known requirement or any plan for this already ? > > >> > > >> Thanks, > > >> Song > > >> > > >
