> Jarek - I wonder how long it took you to type that! Worth the read :) Did it on a train, while travelling from Warsaw to Krakow, (2h40 min) - and I did a few other things there. So less than that :). I type fast when I have the message already well thought out what and how I want to write. I also make mistakes (thank my ADHD for both) - sorry for that.
But you should rather ask me, how long I thought about it - rather than how long it took to type - and then my answer would be - almost 2 years - I started thinking about it when we started. And I also did a **few** things in parallel while doing so :) J. On Thu, Dec 4, 2025 at 7:06 AM Amogh Desai <[email protected]> wrote: > Good work Vincent and everyone involved! > > It's super impressive to see our ecosystem expanding based on the > foundation that was set up. > > Jarek - I wonder how long it took you to type that! Worth the read :) > > Thanks & Regards, > Amogh Desai > > > On Thu, Dec 4, 2025 at 2:33 AM Jens Scheffler <[email protected]> wrote: > > > Need to chime-in - very cool story! Big big Kudos to Vincent and all > > other contributors to make this! > > > > On 12/3/25 20:36, Buğra Öztürk wrote: > > > Great summary, Jarek. Thank you! > > > > > > Amazing work, everyone who contributed to making this possible! Special > > > kudos to Vincent for leading this effort so well. His work has truly > > shaped > > > the result 🎉 > > > > > > It is also exciting to see the possibility of more Auth Managers > joining > > > the ecosystem 🤞 I would be happy to help with any such effort. > > > > > > > > > On Wed, Dec 3, 2025 at 8:23 PM Vincent Beck <[email protected]> > wrote: > > > > > >> Thanks A LOT Jarek for this :) I am extremely pleased to see other > auth > > >> managers coming from the community. This is definitely the end result > we > > >> were aiming for ... 2 years ago. It was a long road but I agree, I > > really > > >> like the end result. I would not have made it without you so thank you > > as > > >> well for having enabled this :) > > >> > > >> As you mentioned, that'd be great, after some time, to incorporate > this > > >> LDAP auth manager into LDAP provider so that it makes it easier for > > users > > >> to use/install it. > > >> > > >> This is also a very good example that implementing an auth manager is > > not > > >> so hard :) so if you would like to implement an auth manager based on > > your > > >> favorite authN/authZ tool, please do so and I would love to help you > > out if > > >> needed. Who knows, maybe it will so cool it will be the next default > > auth > > >> manager in Airflow when Fab auth manager will be gone in the future? > > >> > > >> On 2025/12/03 11:20:24 Jarek Potiuk wrote: > > >>> Hello here, > > >>> > > >>> New day - new prefix of the email. > > >>> > > >>> I wanted to actually BOAST about the Extensible User Management > AIP-56 > > ( > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management > > >>> - we completed the AIP a long time ago, but I think the actual > > >> completion > > >>> of the work is really now. > > >>> > > >>> Big shoutout to Vincent leading it and everyone else who helped with > > it. > > >>> > > >>> The soundness of the approach we took has just been finally confirmed > > by > > >> a > > >>> 3rd-party implementation of LDAP Auth Manager - by Emre Can > > (@emredjan) - > > >>> https://github.com/emredjan/airflow-ldap-auth-manager just published > > and > > >>> made available at our Ecosystem page. > > >>> > > >>> I find it super exciting, and I feel somewhat proud.y f - mostl, > > >> personally > > >>> even if I was just mostly gently pulling and pushing things here and > > >> there > > >>> and came up with the initial direction. Not everyone followed it, > but I > > >>> wanted to share the small success story of some of the more "hairy" > > >> things > > >>> we have done. Or rather mostly Vincent did mostly with a lot of > people > > >>> helping. > > >>> > > >>> I think that one is a really a good example to follow of a long-term > > >>> feature that was done the right way with a big-hairy long term goal > > that > > >>> has been relentlessly followed - that was done really in parallel of > > >>> Airflow 3 work and had started a long time ago (2 years almost to the > > >> date). > > >>> We are doing it now with Amogh leadership now started by Ash - with > > >>> isolation of Task.sdk - even more "hairy" thing - and even more > > >> impactful. > > >>> And I hope we can take it as a good example for things like state > > >>> management - where we should take time and deliberate effort to make > > some > > >>> longer-term vision and decisions and relentlessly follow it to > > >> completion. > > >>> The story is pretty long, but I wanted to describe it here - even if > > not > > >>> for you to read today, but to capture some of the things that made it > > >>> happen, so that we can refer to it in the future. I thought about the > > day > > >>> that I will write the email for a looong time. And I knew what I > wanted > > >> to > > >>> write - and the day finally came. > > >>> > > >>> Brace yourselves, those how will venture into reading it. > > >>> > > >>> ---- > > >>> > > >>> It's a long journey since we had just mostly *whined* about FAB being > > >> such > > >>> a pain. In Airflow 1 FAB RBAC was already a step-up compared to the > > >> earlier > > >>> approach (everyone can do everything approach) - yes that was a > > default I > > >>> still remember from Airflow 1. > > >>> > > >>> This one somehow disappears in memories even of those who were here 6 > > >> years > > >>> ago. But it was a great improvement back then to have RBAC and user > > >>> management. And one could think that we could only get it even more > > >>> feature-full as something that Airflow has "built-in" - but it turned > > out > > >>> to be a big dependency hog, FAB for a long time was not really > evolving > > >> as > > >>> fast as Airflow was with its enterprise integration needs. Groups > were > > >> only > > >>> recently added to FAB (and we are not even using groups even if we > > >> switched > > >>> to FAB that supports it). It was a pain for people to configure it, > > they > > >>> had to switch between Airflow and FAB docs, it was confusing whether > we > > >>> have Airflow or FAB issue, we also at some point of time vendored-in > > >>> Security Manager partially in Airflow (and since then we need to > > manually > > >>> synchronise changes implemented in Fab to the manager we have in > > >> Airflow). > > >>> A lot of security issues came through FA and often we had to wait for > > FAB > > >>> or even contribute back the fixes. We worked closely with Daniel, > that > > is > > >>> great cooperation, we even had calls when some security issues were > > >> raised > > >>> that affected FAB, Airflow and Superset and we discussed our > responses > > >>> together and synced the remediations. But still - this was a lot, a > > lot, > > >> a > > >>> lot of pain, and we had this recurring whining *"Oh we wish we did > not > > >> have > > >>> to depend on FAB".* > > >>> > > >>> When we initially discussed it with Vincent we had many back-forth on > > how > > >>> to design the Auth Manager. We all came up with the idea to do it > > >>> differently - and rather than complicate the way how RBAC is done > with > > >>> groups or other features, we decided that it's not really an Airflow > > job > > >> to > > >>> do Authentication and Authorisation management - and to delegate it > to > > >>> those who do it better. > > >>> > > >>> We made some choices that I think stand the pass of time with simple, > > yet > > >>> powerful API design where instead of thinking what "more" we can do > in > > >>> Airflow, is what we can do to make Airflow more of an "extensible > > >> platform" > > >>> - that will be super-easy to extend. We believe that people will come > > and > > >>> do their own Auth Manager implementation some day. This day happened. > > >>> > > >>> Then AWS team with AWS Auth Manager - mostly Vincent, Nick and AWS > team > > >>> dogfooded and polished the rough edges in - initially experimental > AWS > > >> Auth > > >>> Manager. That was still way back in Airflow 2. Great POC of the idea. > > One > > >>> that was absolutely necessary to move forward. > > >>> > > >>> At the same time we (Vincent again!) implemented FAB Auth Manager > > >> interface > > >>> and moved all functionality to the provider and step-by-step we moved > > out > > >>> all FAB-y things to the provider, piece-by-piece, release by release. > > >>> Keeping the long-term vision in mind and getting ever closer to it. > > That > > >>> was **not an easy task** and it had MANY bumps, but with Vinc > > >> relentlessly > > >>> leading it and a number of people helping, (I had a very little part > in > > >> it > > >>> - mostly lurking from behind Vincent's back - but I was in fact > > following > > >>> very closely, ready to step-in and help any time with some difficult > > >>> choices and decisions. There were many things around with > dependencies, > > >>> issues, back-compatibility, etc. etc. And the work continues there > > still > > >> to > > >>> complete some issues. > > >>> > > >>> In the meantime of course Airflow 3 happened with all the FastApi > > >>> conversion (which turns out to be a great decision) and new UI where > we > > >> got > > >>> rid of the Fab ties from the UI part - Widgets, Connections, Plugins > > some > > >>> internal tooling we got this out as huge part of Airflow 3 migration. > > >> Jens, > > >>> Pierre. Karthikeyan and many others did that. Then we got > > >> SimpleAuthManager > > >>> as default - which is "good" as demo, development and shows the basic > > >>> capabilities of Auth Managers - with different types of (hard-coded) > > >> users > > >>> - which shows what Auth Managers can do, without even pretending, it > > can > > >> be > > >>> used in production (actually even actively shouting you in the face > it > > >>> should not - stll some people wants to do it of course). > > >>> > > >>> Eventually - we are extremely closely to cut the final last ties of > FAB > > >> and > > >>> make the provider truly, truly optional and with its dependencies not > > >>> holding us back - there are literally few more APIs to convert to > Fast > > >> API > > >>> for Fab Auth management left (and Yun-Ting Chiu - @chiuinggum has > done > > >> and > > >>> continues doing a great job there). Some last remaining bridge > between > > UI > > >>> and API for custom - old AuthBackends from Airflow 2 is to be added. > > Once > > >>> we do it, the day where we whine "Oh I wish we did not have to rely > on > > >> FAB" > > >>> will be finally gone. It was great choice when we started, but it > > turned > > >>> out too much of a "gravity center" with a number of choices that held > > us > > >>> back, > > >>> > > >>> Then KeyCloak implementation that was really "far fetched idea" - > (also > > >>> Vincent's doing) - started. Tt is really nice and simple and > extremely > > >>> powerful. Not everyone realizes but thanks to the way we designed > Auth > > >>> Manager, the enterprise users of our can do a LOOOOOOT more than the > > >>> "constrained" FAB RBAC allows. You can configure all the complexity > > that > > >>> KeyCloak provides - mix and match authentication policies and rules - > > and > > >>> you are only limited to what KeyCloak provides. One of the examples > is > > >> that > > >>> for example y)ou could configure that a given Dag or Dag group can be > > >>> triggered by a group of users between 9am and 5pm only - and at other > > >> times > > >>> it will be rejected. And generally any complexity of deciding what > > users > > >>> should be capable of can be implemented in a simple way by defining > and > > >>> combining javascript policies of KeyCloak. I really like how KeyCloak > > >> does > > >>> one thing BEST -> ID management integrated with enterprise ID > systems. > > >> And > > >>> we now have a really nice (and simple, yet powerful) provider which > > >> people > > >>> can use. If you already used Keycloak - you could integrate it via > FAB, > > >> but > > >>> with KeyCloak provider, it's so much more powerful having native > > >>> integration. > > >>> > > >>> We already managed to implement nice batch optimizations and evolved > > tha > > >>> Auth Manager API. We can have a single call for all Dags you are > > >>> attempting to display in dag view (avoiding n+1 issue) and see only > > those > > >>> that you can have access to. We have that in KeyCloak (or maybe is > > about > > >> to > > >>> be completed - but we know how to do it). And any implementation of > > Auth > > >>> Manager can use it as well. And we are discussing some discovery of > > >>> permission API that will allow for example to know if you should gray > > out > > >>> the "Trigger Run" button before you attempt it in the new UI if you > do > > >> not > > >>> have permissions to get better UX. > > >>> > > >>> For many of our users KeyCloak is too heavy to rely on. What if you > > have > > >>> LDAP and you don't care about all the features of KeyCloak and do not > > >> want > > >>> to bank on it? Fret not - as of a few days ago there is this > 3rd-party > > >> LDAP > > >>> provider - so you can use it instead, without having to run and > install > > >> the > > >>> KeyCloak for your company. The day might come if Emre would like that > > and > > >>> it proves to be useful we might get it contributed and we might be > even > > >>> super happy to take maintenance of it - following our - just voted - > > new > > >>> provider acceptance policy. And it's the first fully reusable, > generic > > >> Auth > > >>> Manager by "someone else" that is functional and published. > > >>> > > >>> And ... you can also roll your own - and Emrecan has promised to > review > > >> and > > >>> update our docs to make it even easier to do your own Auth Manager, > so > > I > > >>> hope we will soon have some other "simpler-purpose" Auth Managers. > > >>> > > >>> Fingers crossed. > > >>> > > >>> That was a journey I wanted to share. > > >>> > > >>> Vinc, you rock > > >>> > > >>> J. > > >>> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: [email protected] > > >> For additional commands, e-mail: [email protected] > > >> > > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
