+1 to the long-term vision. The `airflow.core.*` namespace makes sense for clarity. Given the immediate goals discussed by Jarek in the thread; task-sdk separation from airflow-core and providers/task-sdk separation from airflow-core, we should consider a phased approach:
- 3.2: Focus on the immediate decoupling goals without migrating the namespace - 3.3+: Introduce airflow.core.* once dependency boundaries are clear Regards, Ankit Chaurasia *Ankit Chaurasia* HomePage <https://ankitchaurasia.info/> | LinkedIn <https://www.linkedin.com/in/sunank200/> | +91-9987351649 On Tue, Jan 6, 2026 at 12:13 PM Amogh Desai <[email protected]> wrote: > Thanks for your inputs. > > > If we are moving all server components anyway, would it make sense to > then define namespaces per server component below airflow.core. directly > such that a future separation into API server/scheduler/dag parser can > keep a moved structure? > > Certainly. We should be aiming to do that with some points Jarek mentioned > as well. > > > Just a technicality, in the api_fastapi folder we already have `core_api` > (public + UI) and `execution_api`. This sub structure wouldn't make sense > anymore, `airflow.core.api_fastapi.core_api`. (all 3 are core server > component I guess) > > Good call Pierre. Yes, all of them are server components, we could have > a structure like this if it makes sense: > > airflow.core.api > -- ui > -- public > -- execution > > Or something similar. > > > * separating dag-processor, api-server, scheduler, trigger - possibly > each > with different distribution > > Agreed. > > > * separating executors, providers, api-server plugins, scheduler plugins > into different kind of airflow "extensions" - separate distributions with > separate dependencies > > We should also probably consider separate provider distribution for > provider code vs executors in providers. Provider code can solely depend > on task sdk but executors will need airflow core(mostly). > > Did you mean api-server plugins vs worker/task-sdk plugins? > > > * improving import airflow.* behaviour and rather than implicit > initialization of internal components, do explicit initialization in > well-thought and defined sequence (avoiding circular deps and removing lazy > initialization hacks we have scattered across all of our code) > > Most certainly. We do some things very eagerly which can be improved > with the injection + explicit init pattern we have been following with > shared libraries lately. > > > Thanks & Regards, > Amogh Desai > > > On Mon, Jan 5, 2026 at 5:27 PM Jarek Potiuk <[email protected]> wrote: > > > +1 in general - but also +1 on what Kaxil wrote. We should hold our > horses > > a bit (though they are definitely galloping in my head as well). > > > > We can almost infinitely continue refactoring - I have quite a few things > > on my plate on how we should split it even further: > > > > * separating dag-processor, api-server, scheduler, trigger - possibly > each > > with different distribution > > * separating executors, providers, api-server plugins, scheduler plugins > > into different kind of airflow "extensions" - separate distributions with > > separate dependencies > > * improving import airflow.* behaviour and rather than implicit > > initialization of internal components, do explicit initialization in > > well-thought and defined sequence (avoiding circular deps and removing > lazy > > initialization hacks we have scattered across all of our code) > > > > But I think we should absolutely (for 3.2) focus on two goals: > > > > 1) task-sdk is not needed for airflow-core > > 2) airflow-core is not needed by providers and task-sdk > > > > Once we get there, we can focus on the next steps. > > > > We should definitely **keep** all the long terms goals in our heads when > we > > implement 1) and 2) - and when we make decisions necessary for 1) and 2) > we > > should implement them in the right way so that it's easier to implement > the > > long term goals (for example explicit initialization of shared components > > we discussed with Amogh is already needed, solves some of the 1) and 2) > > problems nicely - even if we do not do explicit initialization of > > everything and our imports are still doing s*tload of things in somewhat > > uncontrolled sequence. And that's ok for now. A good example is the > > initialisation sequence of core/configuration/providers and basically > > injection of provider configuration instead of configuration "knowing" > > about providers > > https://github.com/apache/airflow/pull/60074#issuecomment-3710123173 -> > > all > > that is needed to complete the isolation work - and is a step in the > right > > direction IMHO, but we are not yet implement all of it, just what we find > > is necessary why we complete 1) and 2). > > > > J. > > > > > > > > On Mon, Jan 5, 2026 at 11:46 AM Kaxil Naik <[email protected]> wrote: > > > > > Well — this year 😂 its 2026 already haha. > > > > > > Happy New Year > > > > > > On Mon, 5 Jan 2026 at 10:36, Kaxil Naik <[email protected]> wrote: > > > > > > > We should wait to complete the entire decoupling story imo including > > > > AIP-92. Until then there won’t be a any net benefit. So 3.3 would be > > the > > > > earliest for this — mid next year. > > > > > > > > On Mon, 5 Jan 2026 at 10:01, Pierre Jeambrun <[email protected]> > > > > wrote: > > > > > > > >> Sounds good to me. > > > >> > > > >> Just a technicality, in the api_fastapi folder we already have > > > `core_api` > > > >> (public + UI) and `execution_api`. This sub structure wouldn't make > > > sense > > > >> anymore, `airflow.core.api_fastapi.core_api`. (all 3 are core server > > > >> component I guess) > > > >> > > > >> On Mon, Jan 5, 2026 at 10:49 AM Aritra Basu < > [email protected] > > > > > > >> wrote: > > > >> > > > >> > +1 from me for the thought, Agree with Jens on this as well, if > > we're > > > >> > cleaning up let's also take the opportunity to move them under > > > >> components > > > >> > too! > > > >> > > > > >> > -- > > > >> > Regards, > > > >> > Aritra Basu > > > >> > > > > >> > On Mon, 5 Jan 2026, 3:04 pm Rahul Vats, <[email protected]> > > > wrote: > > > >> > > > > >> > > +1 to this. Clear namespace boundaries will definitely help > folks > > > >> > > understand what depends on what. Agree with Jens on organizing > by > > > >> server > > > >> > > component too. > > > >> > > > > > >> > > Regards, > > > >> > > Rahul Vats > > > >> > > > > > >> > > On Mon, 5 Jan 2026 at 14:02, Jens Scheffler < > [email protected]> > > > >> wrote: > > > >> > > > > > >> > > > Hallo Amogh, > > > >> > > > > > > >> > > > that totally makes sense to get the "house clean" and > separate. > > > >> > > > > > > >> > > > If we are moving all server components anyway, would it make > > sense > > > >> to > > > >> > > > then define namespaces per server component below > airflow.core. > > > >> > directly > > > >> > > > such that a future separation into API server/scheduler/dag > > parser > > > >> can > > > >> > > > keep a moved structure? > > > >> > > > > > > >> > > > e.g. moving all API server specific stuff to airflow.core.api. > > and > > > >> all > > > >> > > > scheduler specific to airflow.core.scheduler. > > > >> > > > > > > >> > > > Jens > > > >> > > > > > > >> > > > On 1/5/26 09:17, Amogh Desai wrote: > > > >> > > > > Hello All, > > > >> > > > > > > > >> > > > > Wishing you all a happy new year and hope you spent good > time > > > with > > > >> > your > > > >> > > > > loved ones. > > > >> > > > > > > > >> > > > > With the ongoing efforts on Airflow Client Server > separation, > > I > > > >> > figured > > > >> > > > > that this would be a > > > >> > > > > good time to discuss a proposal in a similar vein to those > > > >> efforts. > > > >> > > > > > > > >> > > > > I'd like to propose establishing an `airflow.core.*` module > > > >> namespace > > > >> > > for > > > >> > > > > server-only components > > > >> > > > > as part of our AIP-72 client-server separation work. > > > >> > > > > > > > >> > > > > *What:* > > > >> > > > > Move server-specific modules(API, scheduler, migrations, > ORMs, > > > and > > > >> > > more) > > > >> > > > > that are specifically > > > >> > > > > used by the server components, to a new `airflow.core.*` > > > >> namespace. > > > >> > > > > > > > >> > > > > For example: > > > >> > > > > - airflow.models.* → airflow.core.models.* > > > >> > > > > - airflow.jobs.* → airflow.core.jobs.* > > > >> > > > > - airflow.api_fastapi.* → airflow.core.api.* (or retain > > > >> > `api_fastapi`) > > > >> > > > > > > > >> > > > > Backward compatibility would be maintained for the older > paths > > > >> with > > > >> > > some > > > >> > > > > tooling that we already > > > >> > > > > have > > > >> > > > > < > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/deprecation_tools.py > > > >> > > > > > > > >> > > > > by > > > >> > > > > issuing deprecation warnings and providing users with a > clear > > > >> > > > > migration path. > > > >> > > > > > > > >> > > > > *Why:* > > > >> > > > > AIP-72 separates airflow into server and client components, > > but > > > >> today > > > >> > > > there > > > >> > > > > is no way to tell from > > > >> > > > > an import path whether the code is intended to be server > only > > or > > > >> > client > > > >> > > > > only, or shared(shared is > > > >> > > > > still better). This makes it easy to accidentally couple > > > >> components > > > >> > > that > > > >> > > > > should be independent. > > > >> > > > > We have some prek hooks in place to avoid this as much as > > > >> possible, > > > >> > but > > > >> > > > > there is a limit to how much > > > >> > > > > we can restrict. > > > >> > > > > > > > >> > > > > Moving the server code to `airflow.core.*` would make the > > > boundary > > > >> > much > > > >> > > > > clearer. The end goal would > > > >> > > > > look like: > > > >> > > > > - airflow.core.* = server-only > > > >> > > > > - airflow.sdk.* = client-side > > > >> > > > > - airflow._shared.* = shared between both > > > >> > > > > > > > >> > > > > It would also bring in some added benefits: > > > >> > > > > 1. Self documenting architecture: the import paths would now > > > >> reveal > > > >> > > > > dependencies between > > > >> > > > > components > > > >> > > > > 2. Reduced accidental coupling > > > >> > > > > > > > >> > > > > We already have two issues in place to track this: > > > >> > > > > 1. https://github.com/apache/airflow/issues/51554 > > > >> > > > > 2. https://github.com/apache/airflow/issues/51555 > > > >> > > > > > > > >> > > > > I have deliberately not listed the estimate of efforts here > to > > > >> agree > > > >> > on > > > >> > > > the > > > >> > > > > approach first. > > > >> > > > > I would love to hear what folks think about this approach > > until > > > >> > > Saturday: > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://www.timeanddate.com/worldclock/fixedtime.html?msg=Voting+ends&iso=20260110T0830&p1=1440 > > > >> > > > > and will start a lazy consensus after that. > > > >> > > > > > > > >> > > > > Thanks & Regards, > > > >> > > > > Amogh Desai > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > --------------------------------------------------------------------- > > > >> > > > To unsubscribe, e-mail: [email protected] > > > >> > > > For additional commands, e-mail: [email protected] > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
