Hi Jason, > For the location where DagImporter lives, would it be better to keep it in Task-SDK (`sdk/importers`)?
Yes, as of now the AbstractDagImporter is implemented under `/sdk` and I will ensure the modifications to it do not include Core modules. > Registering the importer in provider metadata Makes sense. I actually already tested this scenario locally (using import_string()); thank you for informing me about the community consensus. > need to include providing the "source code" to be displayed in "Code" tab Adding get_source_code(self, file_path) -> str directly to the AbstractDagImporter interface makes total sense. It cleanly delegates the responsibility of source-text retrieval to the importer itself. This ensures that whether it's a native YAML tree or source code managed by a language SDK, the Web UI has a standardized, non-Python way to fetch and display it. Thank you! Overall, when implementing the proof of concept locally, the YAML definition was already showing without an additional method because I was setting the `dag.fileloc` to the `file_path`. However, the above method seems like a cleaner and more functional way to implement it. On Sat, Jun 6, 2026 at 6:01 PM Zhe-You(Jason) Liu <[email protected]> wrote: > Hi all, > > For the location where DagImporter lives, would it be better to keep it in > Task-SDK (`sdk/importers`)? In the long-term of Dag-processing, the > "parsing" should be treated on the Task-SDK side. Additionally, the > mentioned "unblocking the authoring of DAGs written in non-Python > languages" means that Coordinator will build on top of DagImporter. The > Coordinator lives in Task-SDK now, so the DagImporter _needs_ to live in > Task-SDK as well to make this happen. (We shouldn't import any module from > Core in Task-SDK) > > > Registering the importer in provider metadata > > Would it be better to just dynamically import the class-path specified in > the Airflow config instead of overwhelming the ProvidersManager interface? > Same as the recent discussion on community consensus regarding the > Coordinator interface feature. Since we're able to dynamically import the > class-path with `import_string` anyway, we can free the ProviderManager > entirely. > > > > This would need to include providing the "source code" to be displayed in > "Code" tab - an improvement > > From Jens' comment on the Confluence page > > Yes, it's a necessary functional requirement for the "defining whole Dag in > lang-sdk" that we defer in the multi-lang feature. The coordinator > interface back then supported `get_code_from_file(file_path)` [1] In other > words, we implement the concept of DagImporter as a shallow interfaces on > the coordinator then. > > IMHO, we should support the `get_source_code(self, file_path) -> str` > method for the `AbstractDagImporter` interface. What I imagine for the > further Coordinator interface is to add a class attribute that specifies > the DagImporter (e.g. `JavaCoordinator.dag_importer = JavaDagImporter`). > > [1] > > https://github.com/jason810496/airflow/blob/task-sdk/feature/coordinator-interface-backup-with-pure-java-dag-feature/sdk/coordinators/java/src/airflow/sdk/coordinators/java/coordinator.py#L75-L88 > > Thanks. > > Best regards, > Jason > > On Sat, Jun 6, 2026 at 2:04 AM Igor Kholopov via dev < > [email protected]> > wrote: > > > > What’s the added benefit of implementing this pattern? Is the main > value > > prop that DAGs can be defined in new ways and parsed with a specific > > interface? Because Airflow already supports parsing YAML files with > > dag-factory. > > > > One primary goal of this AIP is to make support for solutions like DAG > > Factory Airflow-native. One user pain point (which we've observed > > first-hand on multiple occasions with our Managed Airflow) when using DAG > > Factory & similar solutions is the necessity to maintain a Python > "bridge" > > between the declarative format and Airflow. That's because Airflow's DAG > > processor is purely Python-file oriented, and most optimizations are > > tailored only for those files. So, as number of DAGs in a single Airflow > > deployment grows, users must manage both dimensions: > > - DAG processor(s) itself (which is already quite complex) > > - Individual files for populating DAGs based on the declarative format. > As > > the number of declarative DAGs grows, the naive approach of loading all > > configs from a single Python file quickly stops working well (especially > in > > more dynamic development Airflow deployments). So, users must either > > complicate their CI by populating additional 1-to-1 Python files for > > declarative configs or devise elaborate sharding schemas. > > I believe Data Engineers (or their coding agents :) ) shouldn't have to > > care about the latter dimension in the first place. With the DAG importer > > they should be able to use off-the-shelf solutions and fully benefit from > > existing DAG processor optimizations as it will ingest DAGs from > individual > > YAML files (or maybe not even files) without needing to introduce any > > additional Python code in-between. > > > > Another benefit is that it will introduce a clear entry point for > > unblocking the authoring of DAGs written in non-Python languages by > > providing the interface to abstract away the individual language runtime > > complexity (see also discussion in comments of > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-108+Language+Task+SDK+and+the+Language+Coordinator+Layer > > ). > > > > On Fri, Jun 5, 2026 at 5:23 PM Vikram Koka <[email protected]> wrote: > > > > > I just left a few comments on the Confluence page as well, but in the > > > interest of visibility, sharing them here as well. > > > > > > What are the actual deliverables of this AIP (in it's revised form)? > > > > > > Is this purely an internal code refactor with no immediate user facing > > > benefits? > > > I ask because this refactored AIP also refers to an already merged PR > > 60127 > > > > > > If this for performance reasons - mentioned above, but not yet > precisely > > > defined with respect to before / after. > > > > > > Are there any user facing benefits as a result of this AIP? > > > > > > On Fri, Jun 5, 2026 at 6:27 AM Jake McGrath via dev < > > > [email protected]> wrote: > > > > > >> Quick question from my side; what’s the added benefit of implementing > > >> this > > >> pattern? Is the main value prop that DAGs can be defined in new ways > and > > >> parsed with a specific interface? Because Airflow already supports > > parsing > > >> YAML files with dag-factory, and things like JSON with homegrown DAG > > >> generators. Is there a performance component? > > >> > > >> > > >> On Jun 5, 2026 at 7:43:11 AM, Jarek Potiuk <[email protected]> wrote: > > >> > > >> > Generally lack of comments or even 'looks good' means that no-one > had > > >> time > > >> > to look at it - and not that they have no objections. No objections > > >> > assumption is reserved for LAZY CONSENSUS when consensus seems to > be > > >> > achieved > > >> > > > >> > On Fri, Jun 5, 2026, 13:40 Jarek Potiuk <[email protected]> wrote: > > >> > > > >> > To be honest - I do not think there was enough of a discussion on it > > to > > >> > > > >> > start vote thread. And certainly not recently. No-one had any > comments > > >> on > > >> > > > >> > it after scoping down and placing it in Confluence > > >> > > > >> > > > >> > J. > > >> > > > >> > > > >> > On Fri, Jun 5, 2026, 13:23 Dilnaz Amanzholova via dev < > > >> > > > >> > [email protected]> wrote: > > >> > > > >> > > > >> > > Hi all, > > >> > > > >> > > > > >> > > > >> > > I’m calling vote on AIP-85: DAG importer > > >> > > > >> > > https://cwiki.apache.org/confluence/x/_Q7OEg > > >> > > > >> > > > > >> > > > >> > > You can also review the design document here > > >> > > > >> > > < > > >> > > > >> > > > > >> > > > >> > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 > > >> > > > >> > > > > > >> > > > >> > > . > > >> > > > >> > > > > >> > > > >> > > Current discussion thread (after scoping down): > > >> > > > >> > > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt > > >> > > > >> > > Original discussion thread (with wider "extendable DAG parsing > > >> controls" > > >> > > > >> > > title): > > >> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl > > >> > > > >> > > > > >> > > > >> > > The vote will run for 5 days, closing on Wednesday, 10th June > 2026, > > at > > >> > > > >> > > 10:00 UTC. > > >> > > > >> > > > > >> > > > >> > > Everyone is encouraged to vote, but only PMC members and > Committers' > > >> > votes > > >> > > > >> > > are considered binding. Please vote accordingly. > > >> > > > >> > > > > >> > > > >> > > [ ] +1 Approve > > >> > > > >> > > [ ] +0 no opinion > > >> > > > >> > > [ ] -1 disapprove with the reason > > >> > > > >> > > > > >> > > > >> > > Kind regards, > > >> > > > >> > > Dilnaz Amanzholova > > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > > > > >
