And just to call out I have been working on this for quite some time, and now along with Vincent, we are making good progress. We can start giving status updates in the Airflow Dev calls to distribute status on that.
Cheers, Niko ________________________________ From: Amogh Desai <[email protected]> Sent: Sunday, November 16, 2025 11:01:13 PM To: [email protected] Subject: RE: [EXT] [DISCUSS] Airflow multi-tenancy CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Good idea. Yeah Natanel, we have not been able to prioritise it due to other priorities with Airflow 3.x. It would be of great help if you can read the AIP and help in any way possible. Thanks & Regards, Amogh Desai On Wed, Nov 12, 2025 at 3:09 AM Jens Scheffler <[email protected]> wrote: > Hi Natanel! > > Yes, the setup of a multi-tenant deployment is a long lasting demand, > now probably top 1 after we implemented Dag versioning in Airflow 3. > > There had been numerous concepts and scope being discussed, currently > the scope is named "Multi Team" because tenancy is usually really a full > client separation in a provider context. > > I propose to take a reading in > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components > alongside linked AIP documents (AIP=Airflow Improvement Proposals) and > if you like (1) you can contribute and (2) also comment to concepts. It > will still take some time but compared to original scope the current > state in Airflow 3 is not really far away. > > Alternatives are also discussed in the paper. > > Jens > > On 11/11/25 22:30, Natanel wrote: > > Hello, lately in my organisation we have started to experience issues due > > to having to use airflow as a multi-tenant environment, due to having too > > many airflow environments to manage. > > > > I have seen the same issue in other organisations as well, where one team > > had to deploy, monitor and upgrade dozens of airflow instances, which > > causes a lot of issues and complexity. > > > > After some thought, I came to the idea of supporting multi-tenant airflow > > clusters, as I know that now it is not supported and not recommended, > > however, In my opinion and from what I have seen, it would benefit many > > airflow clusters, and improve the usability and ease of maintenance > > operations. > > > > We, in our organisation, have a couple of possible propositions to allow > > airflow cluster multi-tenancy, which include: > > *1) In the airflow chart & code:* > > > > - Define the pools in the chart instead of the DB. > > - In the chart, set a new yaml array to define a tenant, whom > consists > > of a list of pools. > > - For each said tenant, deploy a scheduler and a trigerrer (if > needed). > > - Each deployed component only processes the related pools of the > tenant. > > - Each connection or variable is changed to be accessed by a specific > > tenant (taken from the owner of the dag or any other way) > > > > > > *2) In the code only:* > > > > - Create a tenant table in the database. > > - Create the ralation for tenant and pool. > > - Make the connections and variables accessed from the tenant table, > > thus achieving isolation. > > - For each tenant, create at least #schedulers / 2 instances of the > > scheduler and triggerrer job (on the same pod). > > - Change the code of the scheduler and trigerrer so that every job > only > > queries on the pools of the related tenants. > > > > > > These 2 issues would solve most of our issues that we have, such as > > starvation and noisy neighbours, keep in mind that these 2 are very very > > rough drafts, and are not the full spec of the idea, as I want to first > > keep this mail as a discussion rather than a proposal, in hopes that it > > will help me understand if an AIP can be opened on the thread. > > > > I would love to hear what the airflow community thinks about the topic, > in > > addition to propositions or ideas on what can or should be done, and > > whether it may solve any issues that the community is experiencing. > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
