Hi all, For the location where DagImporter lives, would it be better to keep it in Task-SDK (`sdk/importers`)? In the long-term of Dag-processing, the "parsing" should be treated on the Task-SDK side. Additionally, the mentioned "unblocking the authoring of DAGs written in non-Python languages" means that Coordinator will build on top of DagImporter. The Coordinator lives in Task-SDK now, so the DagImporter _needs_ to live in Task-SDK as well to make this happen. (We shouldn't import any module from Core in Task-SDK)
> Registering the importer in provider metadata Would it be better to just dynamically import the class-path specified in the Airflow config instead of overwhelming the ProvidersManager interface? Same as the recent discussion on community consensus regarding the Coordinator interface feature. Since we're able to dynamically import the class-path with `import_string` anyway, we can free the ProviderManager entirely. > This would need to include providing the "source code" to be displayed in "Code" tab - an improvement > From Jens' comment on the Confluence page Yes, it's a necessary functional requirement for the "defining whole Dag in lang-sdk" that we defer in the multi-lang feature. The coordinator interface back then supported `get_code_from_file(file_path)` [1] In other words, we implement the concept of DagImporter as a shallow interfaces on the coordinator then. IMHO, we should support the `get_source_code(self, file_path) -> str` method for the `AbstractDagImporter` interface. What I imagine for the further Coordinator interface is to add a class attribute that specifies the DagImporter (e.g. `JavaCoordinator.dag_importer = JavaDagImporter`). [1] https://github.com/jason810496/airflow/blob/task-sdk/feature/coordinator-interface-backup-with-pure-java-dag-feature/sdk/coordinators/java/src/airflow/sdk/coordinators/java/coordinator.py#L75-L88 Thanks. Best regards, Jason On Sat, Jun 6, 2026 at 2:04 AM Igor Kholopov via dev <[email protected]> wrote: > > What’s the added benefit of implementing this pattern? Is the main value > prop that DAGs can be defined in new ways and parsed with a specific > interface? Because Airflow already supports parsing YAML files with > dag-factory. > > One primary goal of this AIP is to make support for solutions like DAG > Factory Airflow-native. One user pain point (which we've observed > first-hand on multiple occasions with our Managed Airflow) when using DAG > Factory & similar solutions is the necessity to maintain a Python "bridge" > between the declarative format and Airflow. That's because Airflow's DAG > processor is purely Python-file oriented, and most optimizations are > tailored only for those files. So, as number of DAGs in a single Airflow > deployment grows, users must manage both dimensions: > - DAG processor(s) itself (which is already quite complex) > - Individual files for populating DAGs based on the declarative format. As > the number of declarative DAGs grows, the naive approach of loading all > configs from a single Python file quickly stops working well (especially in > more dynamic development Airflow deployments). So, users must either > complicate their CI by populating additional 1-to-1 Python files for > declarative configs or devise elaborate sharding schemas. > I believe Data Engineers (or their coding agents :) ) shouldn't have to > care about the latter dimension in the first place. With the DAG importer > they should be able to use off-the-shelf solutions and fully benefit from > existing DAG processor optimizations as it will ingest DAGs from individual > YAML files (or maybe not even files) without needing to introduce any > additional Python code in-between. > > Another benefit is that it will introduce a clear entry point for > unblocking the authoring of DAGs written in non-Python languages by > providing the interface to abstract away the individual language runtime > complexity (see also discussion in comments of > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-108+Language+Task+SDK+and+the+Language+Coordinator+Layer > ). > > On Fri, Jun 5, 2026 at 5:23 PM Vikram Koka <[email protected]> wrote: > > > I just left a few comments on the Confluence page as well, but in the > > interest of visibility, sharing them here as well. > > > > What are the actual deliverables of this AIP (in it's revised form)? > > > > Is this purely an internal code refactor with no immediate user facing > > benefits? > > I ask because this refactored AIP also refers to an already merged PR > 60127 > > > > If this for performance reasons - mentioned above, but not yet precisely > > defined with respect to before / after. > > > > Are there any user facing benefits as a result of this AIP? > > > > On Fri, Jun 5, 2026 at 6:27 AM Jake McGrath via dev < > > [email protected]> wrote: > > > >> Quick question from my side; what’s the added benefit of implementing > >> this > >> pattern? Is the main value prop that DAGs can be defined in new ways and > >> parsed with a specific interface? Because Airflow already supports > parsing > >> YAML files with dag-factory, and things like JSON with homegrown DAG > >> generators. Is there a performance component? > >> > >> > >> On Jun 5, 2026 at 7:43:11 AM, Jarek Potiuk <[email protected]> wrote: > >> > >> > Generally lack of comments or even 'looks good' means that no-one had > >> time > >> > to look at it - and not that they have no objections. No objections > >> > assumption is reserved for LAZY CONSENSUS when consensus seems to be > >> > achieved > >> > > >> > On Fri, Jun 5, 2026, 13:40 Jarek Potiuk <[email protected]> wrote: > >> > > >> > To be honest - I do not think there was enough of a discussion on it > to > >> > > >> > start vote thread. And certainly not recently. No-one had any comments > >> on > >> > > >> > it after scoping down and placing it in Confluence > >> > > >> > > >> > J. > >> > > >> > > >> > On Fri, Jun 5, 2026, 13:23 Dilnaz Amanzholova via dev < > >> > > >> > [email protected]> wrote: > >> > > >> > > >> > > Hi all, > >> > > >> > > > >> > > >> > > I’m calling vote on AIP-85: DAG importer > >> > > >> > > https://cwiki.apache.org/confluence/x/_Q7OEg > >> > > >> > > > >> > > >> > > You can also review the design document here > >> > > >> > > < > >> > > >> > > > >> > > >> > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 > >> > > >> > > > > >> > > >> > > . > >> > > >> > > > >> > > >> > > Current discussion thread (after scoping down): > >> > > >> > > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt > >> > > >> > > Original discussion thread (with wider "extendable DAG parsing > >> controls" > >> > > >> > > title): > >> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl > >> > > >> > > > >> > > >> > > The vote will run for 5 days, closing on Wednesday, 10th June 2026, > at > >> > > >> > > 10:00 UTC. > >> > > >> > > > >> > > >> > > Everyone is encouraged to vote, but only PMC members and Committers' > >> > votes > >> > > >> > > are considered binding. Please vote accordingly. > >> > > >> > > > >> > > >> > > [ ] +1 Approve > >> > > >> > > [ ] +0 no opinion > >> > > >> > > [ ] -1 disapprove with the reason > >> > > >> > > > >> > > >> > > Kind regards, > >> > > >> > > Dilnaz Amanzholova > >> > > >> > > > >> > > >> > > >> > > >> > > >
