Good work Vincent and everyone involved! It's super impressive to see our ecosystem expanding based on the foundation that was set up.
Jarek - I wonder how long it took you to type that! Worth the read :) Thanks & Regards, Amogh Desai On Thu, Dec 4, 2025 at 2:33 AM Jens Scheffler <[email protected]> wrote: > Need to chime-in - very cool story! Big big Kudos to Vincent and all > other contributors to make this! > > On 12/3/25 20:36, Buğra Öztürk wrote: > > Great summary, Jarek. Thank you! > > > > Amazing work, everyone who contributed to making this possible! Special > > kudos to Vincent for leading this effort so well. His work has truly > shaped > > the result 🎉 > > > > It is also exciting to see the possibility of more Auth Managers joining > > the ecosystem 🤞 I would be happy to help with any such effort. > > > > > > On Wed, Dec 3, 2025 at 8:23 PM Vincent Beck <[email protected]> wrote: > > > >> Thanks A LOT Jarek for this :) I am extremely pleased to see other auth > >> managers coming from the community. This is definitely the end result we > >> were aiming for ... 2 years ago. It was a long road but I agree, I > really > >> like the end result. I would not have made it without you so thank you > as > >> well for having enabled this :) > >> > >> As you mentioned, that'd be great, after some time, to incorporate this > >> LDAP auth manager into LDAP provider so that it makes it easier for > users > >> to use/install it. > >> > >> This is also a very good example that implementing an auth manager is > not > >> so hard :) so if you would like to implement an auth manager based on > your > >> favorite authN/authZ tool, please do so and I would love to help you > out if > >> needed. Who knows, maybe it will so cool it will be the next default > auth > >> manager in Airflow when Fab auth manager will be gone in the future? > >> > >> On 2025/12/03 11:20:24 Jarek Potiuk wrote: > >>> Hello here, > >>> > >>> New day - new prefix of the email. > >>> > >>> I wanted to actually BOAST about the Extensible User Management AIP-56 > ( > >>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management > >>> - we completed the AIP a long time ago, but I think the actual > >> completion > >>> of the work is really now. > >>> > >>> Big shoutout to Vincent leading it and everyone else who helped with > it. > >>> > >>> The soundness of the approach we took has just been finally confirmed > by > >> a > >>> 3rd-party implementation of LDAP Auth Manager - by Emre Can > (@emredjan) - > >>> https://github.com/emredjan/airflow-ldap-auth-manager just published > and > >>> made available at our Ecosystem page. > >>> > >>> I find it super exciting, and I feel somewhat proud.y f - mostl, > >> personally > >>> even if I was just mostly gently pulling and pushing things here and > >> there > >>> and came up with the initial direction. Not everyone followed it, but I > >>> wanted to share the small success story of some of the more "hairy" > >> things > >>> we have done. Or rather mostly Vincent did mostly with a lot of people > >>> helping. > >>> > >>> I think that one is a really a good example to follow of a long-term > >>> feature that was done the right way with a big-hairy long term goal > that > >>> has been relentlessly followed - that was done really in parallel of > >>> Airflow 3 work and had started a long time ago (2 years almost to the > >> date). > >>> We are doing it now with Amogh leadership now started by Ash - with > >>> isolation of Task.sdk - even more "hairy" thing - and even more > >> impactful. > >>> And I hope we can take it as a good example for things like state > >>> management - where we should take time and deliberate effort to make > some > >>> longer-term vision and decisions and relentlessly follow it to > >> completion. > >>> The story is pretty long, but I wanted to describe it here - even if > not > >>> for you to read today, but to capture some of the things that made it > >>> happen, so that we can refer to it in the future. I thought about the > day > >>> that I will write the email for a looong time. And I knew what I wanted > >> to > >>> write - and the day finally came. > >>> > >>> Brace yourselves, those how will venture into reading it. > >>> > >>> ---- > >>> > >>> It's a long journey since we had just mostly *whined* about FAB being > >> such > >>> a pain. In Airflow 1 FAB RBAC was already a step-up compared to the > >> earlier > >>> approach (everyone can do everything approach) - yes that was a > default I > >>> still remember from Airflow 1. > >>> > >>> This one somehow disappears in memories even of those who were here 6 > >> years > >>> ago. But it was a great improvement back then to have RBAC and user > >>> management. And one could think that we could only get it even more > >>> feature-full as something that Airflow has "built-in" - but it turned > out > >>> to be a big dependency hog, FAB for a long time was not really evolving > >> as > >>> fast as Airflow was with its enterprise integration needs. Groups were > >> only > >>> recently added to FAB (and we are not even using groups even if we > >> switched > >>> to FAB that supports it). It was a pain for people to configure it, > they > >>> had to switch between Airflow and FAB docs, it was confusing whether we > >>> have Airflow or FAB issue, we also at some point of time vendored-in > >>> Security Manager partially in Airflow (and since then we need to > manually > >>> synchronise changes implemented in Fab to the manager we have in > >> Airflow). > >>> A lot of security issues came through FA and often we had to wait for > FAB > >>> or even contribute back the fixes. We worked closely with Daniel, that > is > >>> great cooperation, we even had calls when some security issues were > >> raised > >>> that affected FAB, Airflow and Superset and we discussed our responses > >>> together and synced the remediations. But still - this was a lot, a > lot, > >> a > >>> lot of pain, and we had this recurring whining *"Oh we wish we did not > >> have > >>> to depend on FAB".* > >>> > >>> When we initially discussed it with Vincent we had many back-forth on > how > >>> to design the Auth Manager. We all came up with the idea to do it > >>> differently - and rather than complicate the way how RBAC is done with > >>> groups or other features, we decided that it's not really an Airflow > job > >> to > >>> do Authentication and Authorisation management - and to delegate it to > >>> those who do it better. > >>> > >>> We made some choices that I think stand the pass of time with simple, > yet > >>> powerful API design where instead of thinking what "more" we can do in > >>> Airflow, is what we can do to make Airflow more of an "extensible > >> platform" > >>> - that will be super-easy to extend. We believe that people will come > and > >>> do their own Auth Manager implementation some day. This day happened. > >>> > >>> Then AWS team with AWS Auth Manager - mostly Vincent, Nick and AWS team > >>> dogfooded and polished the rough edges in - initially experimental AWS > >> Auth > >>> Manager. That was still way back in Airflow 2. Great POC of the idea. > One > >>> that was absolutely necessary to move forward. > >>> > >>> At the same time we (Vincent again!) implemented FAB Auth Manager > >> interface > >>> and moved all functionality to the provider and step-by-step we moved > out > >>> all FAB-y things to the provider, piece-by-piece, release by release. > >>> Keeping the long-term vision in mind and getting ever closer to it. > That > >>> was **not an easy task** and it had MANY bumps, but with Vinc > >> relentlessly > >>> leading it and a number of people helping, (I had a very little part in > >> it > >>> - mostly lurking from behind Vincent's back - but I was in fact > following > >>> very closely, ready to step-in and help any time with some difficult > >>> choices and decisions. There were many things around with dependencies, > >>> issues, back-compatibility, etc. etc. And the work continues there > still > >> to > >>> complete some issues. > >>> > >>> In the meantime of course Airflow 3 happened with all the FastApi > >>> conversion (which turns out to be a great decision) and new UI where we > >> got > >>> rid of the Fab ties from the UI part - Widgets, Connections, Plugins > some > >>> internal tooling we got this out as huge part of Airflow 3 migration. > >> Jens, > >>> Pierre. Karthikeyan and many others did that. Then we got > >> SimpleAuthManager > >>> as default - which is "good" as demo, development and shows the basic > >>> capabilities of Auth Managers - with different types of (hard-coded) > >> users > >>> - which shows what Auth Managers can do, without even pretending, it > can > >> be > >>> used in production (actually even actively shouting you in the face it > >>> should not - stll some people wants to do it of course). > >>> > >>> Eventually - we are extremely closely to cut the final last ties of FAB > >> and > >>> make the provider truly, truly optional and with its dependencies not > >>> holding us back - there are literally few more APIs to convert to Fast > >> API > >>> for Fab Auth management left (and Yun-Ting Chiu - @chiuinggum has done > >> and > >>> continues doing a great job there). Some last remaining bridge between > UI > >>> and API for custom - old AuthBackends from Airflow 2 is to be added. > Once > >>> we do it, the day where we whine "Oh I wish we did not have to rely on > >> FAB" > >>> will be finally gone. It was great choice when we started, but it > turned > >>> out too much of a "gravity center" with a number of choices that held > us > >>> back, > >>> > >>> Then KeyCloak implementation that was really "far fetched idea" - (also > >>> Vincent's doing) - started. Tt is really nice and simple and extremely > >>> powerful. Not everyone realizes but thanks to the way we designed Auth > >>> Manager, the enterprise users of our can do a LOOOOOOT more than the > >>> "constrained" FAB RBAC allows. You can configure all the complexity > that > >>> KeyCloak provides - mix and match authentication policies and rules - > and > >>> you are only limited to what KeyCloak provides. One of the examples is > >> that > >>> for example y)ou could configure that a given Dag or Dag group can be > >>> triggered by a group of users between 9am and 5pm only - and at other > >> times > >>> it will be rejected. And generally any complexity of deciding what > users > >>> should be capable of can be implemented in a simple way by defining and > >>> combining javascript policies of KeyCloak. I really like how KeyCloak > >> does > >>> one thing BEST -> ID management integrated with enterprise ID systems. > >> And > >>> we now have a really nice (and simple, yet powerful) provider which > >> people > >>> can use. If you already used Keycloak - you could integrate it via FAB, > >> but > >>> with KeyCloak provider, it's so much more powerful having native > >>> integration. > >>> > >>> We already managed to implement nice batch optimizations and evolved > tha > >>> Auth Manager API. We can have a single call for all Dags you are > >>> attempting to display in dag view (avoiding n+1 issue) and see only > those > >>> that you can have access to. We have that in KeyCloak (or maybe is > about > >> to > >>> be completed - but we know how to do it). And any implementation of > Auth > >>> Manager can use it as well. And we are discussing some discovery of > >>> permission API that will allow for example to know if you should gray > out > >>> the "Trigger Run" button before you attempt it in the new UI if you do > >> not > >>> have permissions to get better UX. > >>> > >>> For many of our users KeyCloak is too heavy to rely on. What if you > have > >>> LDAP and you don't care about all the features of KeyCloak and do not > >> want > >>> to bank on it? Fret not - as of a few days ago there is this 3rd-party > >> LDAP > >>> provider - so you can use it instead, without having to run and install > >> the > >>> KeyCloak for your company. The day might come if Emre would like that > and > >>> it proves to be useful we might get it contributed and we might be even > >>> super happy to take maintenance of it - following our - just voted - > new > >>> provider acceptance policy. And it's the first fully reusable, generic > >> Auth > >>> Manager by "someone else" that is functional and published. > >>> > >>> And ... you can also roll your own - and Emrecan has promised to review > >> and > >>> update our docs to make it even easier to do your own Auth Manager, so > I > >>> hope we will soon have some other "simpler-purpose" Auth Managers. > >>> > >>> Fingers crossed. > >>> > >>> That was a journey I wanted to share. > >>> > >>> Vinc, you rock > >>> > >>> J. > >>> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
