Great summary, Jarek. Thank you! Amazing work, everyone who contributed to making this possible! Special kudos to Vincent for leading this effort so well. His work has truly shaped the result 🎉
It is also exciting to see the possibility of more Auth Managers joining the ecosystem 🤞 I would be happy to help with any such effort. On Wed, Dec 3, 2025 at 8:23 PM Vincent Beck <[email protected]> wrote: > Thanks A LOT Jarek for this :) I am extremely pleased to see other auth > managers coming from the community. This is definitely the end result we > were aiming for ... 2 years ago. It was a long road but I agree, I really > like the end result. I would not have made it without you so thank you as > well for having enabled this :) > > As you mentioned, that'd be great, after some time, to incorporate this > LDAP auth manager into LDAP provider so that it makes it easier for users > to use/install it. > > This is also a very good example that implementing an auth manager is not > so hard :) so if you would like to implement an auth manager based on your > favorite authN/authZ tool, please do so and I would love to help you out if > needed. Who knows, maybe it will so cool it will be the next default auth > manager in Airflow when Fab auth manager will be gone in the future? > > On 2025/12/03 11:20:24 Jarek Potiuk wrote: > > Hello here, > > > > New day - new prefix of the email. > > > > I wanted to actually BOAST about the Extensible User Management AIP-56 ( > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management > > - we completed the AIP a long time ago, but I think the actual > completion > > of the work is really now. > > > > Big shoutout to Vincent leading it and everyone else who helped with it. > > > > The soundness of the approach we took has just been finally confirmed by > a > > 3rd-party implementation of LDAP Auth Manager - by Emre Can (@emredjan) - > > https://github.com/emredjan/airflow-ldap-auth-manager just published and > > made available at our Ecosystem page. > > > > I find it super exciting, and I feel somewhat proud.y f - mostl, > personally > > even if I was just mostly gently pulling and pushing things here and > there > > and came up with the initial direction. Not everyone followed it, but I > > wanted to share the small success story of some of the more "hairy" > things > > we have done. Or rather mostly Vincent did mostly with a lot of people > > helping. > > > > I think that one is a really a good example to follow of a long-term > > feature that was done the right way with a big-hairy long term goal that > > has been relentlessly followed - that was done really in parallel of > > Airflow 3 work and had started a long time ago (2 years almost to the > date). > > > > We are doing it now with Amogh leadership now started by Ash - with > > isolation of Task.sdk - even more "hairy" thing - and even more > impactful. > > And I hope we can take it as a good example for things like state > > management - where we should take time and deliberate effort to make some > > longer-term vision and decisions and relentlessly follow it to > completion. > > > > The story is pretty long, but I wanted to describe it here - even if not > > for you to read today, but to capture some of the things that made it > > happen, so that we can refer to it in the future. I thought about the day > > that I will write the email for a looong time. And I knew what I wanted > to > > write - and the day finally came. > > > > Brace yourselves, those how will venture into reading it. > > > > ---- > > > > It's a long journey since we had just mostly *whined* about FAB being > such > > a pain. In Airflow 1 FAB RBAC was already a step-up compared to the > earlier > > approach (everyone can do everything approach) - yes that was a default I > > still remember from Airflow 1. > > > > This one somehow disappears in memories even of those who were here 6 > years > > ago. But it was a great improvement back then to have RBAC and user > > management. And one could think that we could only get it even more > > feature-full as something that Airflow has "built-in" - but it turned out > > to be a big dependency hog, FAB for a long time was not really evolving > as > > fast as Airflow was with its enterprise integration needs. Groups were > only > > recently added to FAB (and we are not even using groups even if we > switched > > to FAB that supports it). It was a pain for people to configure it, they > > had to switch between Airflow and FAB docs, it was confusing whether we > > have Airflow or FAB issue, we also at some point of time vendored-in > > Security Manager partially in Airflow (and since then we need to manually > > synchronise changes implemented in Fab to the manager we have in > Airflow). > > A lot of security issues came through FA and often we had to wait for FAB > > or even contribute back the fixes. We worked closely with Daniel, that is > > great cooperation, we even had calls when some security issues were > raised > > that affected FAB, Airflow and Superset and we discussed our responses > > together and synced the remediations. But still - this was a lot, a lot, > a > > lot of pain, and we had this recurring whining *"Oh we wish we did not > have > > to depend on FAB".* > > > > When we initially discussed it with Vincent we had many back-forth on how > > to design the Auth Manager. We all came up with the idea to do it > > differently - and rather than complicate the way how RBAC is done with > > groups or other features, we decided that it's not really an Airflow job > to > > do Authentication and Authorisation management - and to delegate it to > > those who do it better. > > > > We made some choices that I think stand the pass of time with simple, yet > > powerful API design where instead of thinking what "more" we can do in > > Airflow, is what we can do to make Airflow more of an "extensible > platform" > > - that will be super-easy to extend. We believe that people will come and > > do their own Auth Manager implementation some day. This day happened. > > > > Then AWS team with AWS Auth Manager - mostly Vincent, Nick and AWS team > > dogfooded and polished the rough edges in - initially experimental AWS > Auth > > Manager. That was still way back in Airflow 2. Great POC of the idea. One > > that was absolutely necessary to move forward. > > > > At the same time we (Vincent again!) implemented FAB Auth Manager > interface > > and moved all functionality to the provider and step-by-step we moved out > > all FAB-y things to the provider, piece-by-piece, release by release. > > Keeping the long-term vision in mind and getting ever closer to it. That > > was **not an easy task** and it had MANY bumps, but with Vinc > relentlessly > > leading it and a number of people helping, (I had a very little part in > it > > - mostly lurking from behind Vincent's back - but I was in fact following > > very closely, ready to step-in and help any time with some difficult > > choices and decisions. There were many things around with dependencies, > > issues, back-compatibility, etc. etc. And the work continues there still > to > > complete some issues. > > > > In the meantime of course Airflow 3 happened with all the FastApi > > conversion (which turns out to be a great decision) and new UI where we > got > > rid of the Fab ties from the UI part - Widgets, Connections, Plugins some > > internal tooling we got this out as huge part of Airflow 3 migration. > Jens, > > Pierre. Karthikeyan and many others did that. Then we got > SimpleAuthManager > > as default - which is "good" as demo, development and shows the basic > > capabilities of Auth Managers - with different types of (hard-coded) > users > > - which shows what Auth Managers can do, without even pretending, it can > be > > used in production (actually even actively shouting you in the face it > > should not - stll some people wants to do it of course). > > > > Eventually - we are extremely closely to cut the final last ties of FAB > and > > make the provider truly, truly optional and with its dependencies not > > holding us back - there are literally few more APIs to convert to Fast > API > > for Fab Auth management left (and Yun-Ting Chiu - @chiuinggum has done > and > > continues doing a great job there). Some last remaining bridge between UI > > and API for custom - old AuthBackends from Airflow 2 is to be added. Once > > we do it, the day where we whine "Oh I wish we did not have to rely on > FAB" > > will be finally gone. It was great choice when we started, but it turned > > out too much of a "gravity center" with a number of choices that held us > > back, > > > > Then KeyCloak implementation that was really "far fetched idea" - (also > > Vincent's doing) - started. Tt is really nice and simple and extremely > > powerful. Not everyone realizes but thanks to the way we designed Auth > > Manager, the enterprise users of our can do a LOOOOOOT more than the > > "constrained" FAB RBAC allows. You can configure all the complexity that > > KeyCloak provides - mix and match authentication policies and rules - and > > you are only limited to what KeyCloak provides. One of the examples is > that > > for example y)ou could configure that a given Dag or Dag group can be > > triggered by a group of users between 9am and 5pm only - and at other > times > > it will be rejected. And generally any complexity of deciding what users > > should be capable of can be implemented in a simple way by defining and > > combining javascript policies of KeyCloak. I really like how KeyCloak > does > > one thing BEST -> ID management integrated with enterprise ID systems. > And > > we now have a really nice (and simple, yet powerful) provider which > people > > can use. If you already used Keycloak - you could integrate it via FAB, > but > > with KeyCloak provider, it's so much more powerful having native > > integration. > > > > We already managed to implement nice batch optimizations and evolved tha > > Auth Manager API. We can have a single call for all Dags you are > > attempting to display in dag view (avoiding n+1 issue) and see only those > > that you can have access to. We have that in KeyCloak (or maybe is about > to > > be completed - but we know how to do it). And any implementation of Auth > > Manager can use it as well. And we are discussing some discovery of > > permission API that will allow for example to know if you should gray out > > the "Trigger Run" button before you attempt it in the new UI if you do > not > > have permissions to get better UX. > > > > For many of our users KeyCloak is too heavy to rely on. What if you have > > LDAP and you don't care about all the features of KeyCloak and do not > want > > to bank on it? Fret not - as of a few days ago there is this 3rd-party > LDAP > > provider - so you can use it instead, without having to run and install > the > > KeyCloak for your company. The day might come if Emre would like that and > > it proves to be useful we might get it contributed and we might be even > > super happy to take maintenance of it - following our - just voted - new > > provider acceptance policy. And it's the first fully reusable, generic > Auth > > Manager by "someone else" that is functional and published. > > > > And ... you can also roll your own - and Emrecan has promised to review > and > > update our docs to make it even easier to do your own Auth Manager, so I > > hope we will soon have some other "simpler-purpose" Auth Managers. > > > > Fingers crossed. > > > > That was a journey I wanted to share. > > > > Vinc, you rock > > > > J. > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Bugra Ozturk
