Hi Dilnaz, and indeed - thanks for pushing this forward. The scoped-down "DAG importer" framing is much clearer, and native non-Python DAGs are a genuinely good direction. +1 on the goal.
I am happy to make comments and suggestions or even fill-in some blanks (security especially) in the AIP proposal if you are fine with me doing it - see some of the comments below. Before I can vote on the design, I'd like to raise one structural concern plus a couple of smaller ones. My main point is that the foundation already merged in https://github.com/apache/airflow/pull/60127 contradicts the design the AIP describes, and I think we should reconcile the two before the vote closes rather than after. Some of these points are already explicitly stated, either in the AIP or opened questions raised by Tatiana - I just want to reiterate them here and see if we can reach consensus on the shape of those changes. Concretely, in airflow-core/src/airflow/dag_processing/importers/: 1. Registry scope — global singleton vs. per-bundle. The AIP describes per-bundle importer mapping with a bundle -> global -> default resolution order (and Tatiana's Q4/Q7 ask for the same). But the merged DagImporterRegistry is a process-wide singleton keyed by extension, "last registered wins, one importer per extension." A global extension->importer map structurally can't give two bundles different importers for the same extension, so as it stands the foundation can't support the resolution model the AIP promises. If per-bundle is the intended end state, I think we want that in the interface now, because swapping a singleton out later is expensive. 2. Zip handling. The AIP describes a generic ZipImporter that delegates to inner-format importers. The merged PythonDagImporter instead hard-owns both .py and .zip, with a docstring saying YAML-in-zip would require extending it or writing a composite importer — the opposite of the AIP. I think we should decide on one or the other direction, I like the composite importer more - even if it is more complex, it allows for greater flexibility if we have composite importers. But I would love to hear what others think. 3. DagImportResult.file_dependencies is in the AIP dataclass but not in the merged one — and this is more than cosmetic. Declarative DAGs routinely share a defaults.yaml; the DFP decides what to re-parse from per-file mtime, so without importer-declared dependencies a change to a shared file won't re-parse its dependents (a staleness bug). This is Tatiana's Q5, and I'd treat it as a correctness requirement, not a nice-to-have. Worth spelling out how it interacts with AIP-66's git-sha bundle versioning too. On security, I'd like the AIP to state explicitly that a declarative DAG file is fully code-equivalent trust: a YAML importer resolving operators by classpath has the same power as arbitrary Python. People tend to treat YAML as "data" and .py as "code," and I want to make sure nobody builds a "review-free YAML bundle" path on the back of this. I'd also like it stated as a hard constraint that importers run only in the DAG processor, in-process, under the same parse timeout, and never touch the metadata DB — consistent with the AIP-66/AIP-72 boundary. And a quick confirmation that the old "push DAGs over API / ingester" idea stays out of scope, since that one genuinely changes the threat model. Again, for security, if you are happy with it, I would love to be able to fill all those security blanks - this can be described on high level in the AIP, and when implementation progresses, it would turn into security model updates (I'm also happy to help there). Two smaller things: - The safe_mode/might_contain_dag heuristic ("file contains the strings airflow + dag") currently lives in the abstract list_dag_files, but it's Python-specific and will mis-handle YAML — it should be delegated per-importer (Tatiana's Q3). - It'd be good to confirm the registry indirection adds no measurable regression on the plain .py path - some performance tests with basic big-numbers-of-dags scenarios would be great to do as part of the implementation. None of this is a blocker on the idea — I'm supportive. But given the open design questions and that the vote has been quiet, It reaffirms we treat this as still in [DISCUSS], reconcile the AIP with the merged foundation (especially point 1), and re-run the vote once the interface reflects the intended end state. Happy to help review the per-bundle registry shape. Best, Jarek On Tue, Jul 7, 2026 at 4:45 PM Tatiana Al-Chueyr Martins < [email protected]> wrote: > Hi Dilnaz, > > Thanks a lot for pushing this forward, and for the updates after the > earlier feedback - AIP 85 looks much clearer than before. > > I'm one of the maintainers of DAG Factory ( > https://github.com/astronomer/dag-factory), and I like the direction the > AIP is taking. I agree with the pain points Igor raised earlier; it would > be great to support non-Python DAGs natively in Airflow, rather than > converting them as we currently do in DAG Factory. > > I've reviewed the AIP page and left some inline comments & questions. Since > the authors are already responding on the AIP page, happy to continue the > detailed discussion there - sharing the summary here mainly for visibility: > > 1. *Location of schemas and importers*: Where do we see the schemas and > actual importers defined - and how do we plan to govern them? It would be > important to align, for instance, on how non-primitive operator arguments > should be defined in YAML (e.g., Asset, V1Pod, arbitrary classes) and on > the security aspects of loading arbitrary classes from YAML. > > 2. *Validation contract*: If users define unknown or unsupported keys and > configurations, should we fail loudly with DagImportErrors referencing > files/lines like the Python DAG import errors? It would be great if we > didn't just skip them. > > 3.* Decision on which files to parse: *As raised by TP on > < > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > > the > Google > < > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > > Doc > < > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > > , I feel that we should have additional criteria to decide which files to > treat as a DAG - and not rely only on the file extension. As an example, > bundles normally have non-DAG YAML - dbt_project.yml, CI configs, K8s > manifests. Would it make sense to have something similar to the Python > content-level heuristic ("airflow + dag" string check)? > > 4. *Different importers for the same file extension*: Can/should we support > different importers for the same file extension? It would be great if the > architecture could support overriding the importer per file. > > 5.* Cross-file dependency tracking:* Declarative DAGs are rarely > single-file. They can share default definitions, for example: > https://astronomer.github.io/dag-factory/1.1.0/configuration/defaults/. > Could importers declare file dependencies in DagImportResult, so the > processor can use them? > > 6. *Code tab UI*: On get_source_code(), could we have a parameter related > to the file language, which could be used by the Code tab (it is currently > hard-coded with Python)? How are we planning to resolve ZipImporter and > other transformed sources in the UI? > > 7. *Precedence*: What should be the precedence (per-bundle vs global vs > default importers)? > > For the performance questions raised earlier in the thread, we could use > DAG Factory workloads as a baseline, comparing the non-built-in-Airflow > Python layer against a native YAML importer. > > Our team is very keen to contribute to the YAML contract definition. > > Kind regards, > > Tatiana > > > On Tue, 7 Jul 2026 at 13:48, Dilnaz Amanzholova via dev < > [email protected]> wrote: > > > Hi all, I am following up to request your feedback on AIP-85: DAG > importer. > > I attempted to present this at the last Airflow dev call, but the agenda > > was full, and today’s meeting was unfortunately cancelled. As it has been > > some time since this AIP was submitted for review, I would greatly > > appreciate it if you could share your thoughts or any remaining concerns > > here. I understand there are many active discussions, but your input > would > > be invaluable in helping move this proposal forward. Thanks, Dilnaz > > Amanzholova > > > > On Thu, Jun 18, 2026 at 2:08 PM Dilnaz Amanzholova <[email protected]> > > wrote: > > > > > Hey Kaxil, > > > > > > Thanks for the clarification. I have decided to attend, as the next > > > meeting is not for another two weeks. I have added the AIP to the > agenda. > > > > > > Best regards, > > > Dilnaz Amanzholova > > > > > > > > > On Wed, Jun 17, 2026 at 6:37 PM Kaxil Naik <[email protected]> > wrote: > > > > > >> Hey not required at all. > > >> > > >> This is just in case you wanted to solicit feedback live on the call > > with > > >> other Airflow devs. Sometimes it is easier that way -- but not > > mandatory. > > >> > > >> More details on the dev calls: > > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Call > if > > >> you decide about presenting it next time. > > >> > > >> Regards, > > >> Kaxil > > >> > > >> On Wed, 17 Jun 2026 at 17:32, Dilnaz Amanzholova <[email protected]> > > >> wrote: > > >> > > >>> Hi Kaxil, > > >>> > > >>> I was not aware there was a regular call. Is it required to present > the > > >>> AIP before voting? > > >>> > > >>> Kind regards, > > >>> Dilnaz Amanzholova > > >>> > > >>> > > >>> On Wed, Jun 17, 2026 at 6:26 PM Kaxil Naik <[email protected]> > > wrote: > > >>> > > >>>> Hi @dilnaz -- Are you up for presenting it tomorrow or on the next > dev > > >>>> call? > > >>>> > > >>>> > > >>>> > > >>>> On Tue, 16 Jun 2026 at 15:15, Dilnaz Amanzholova via dev < > > >>>> [email protected]> wrote: > > >>>> > > >>>>> Hi all, > > >>>>> > > >>>>> I am following up on AIP-85: DAG importer. All previous concerns > have > > >>>>> been > > >>>>> addressed, and the content has been updated on the AIP Confluence > > page. > > >>>>> > > >>>>> Please review the latest updates at your earliest convenience. I > > would > > >>>>> appreciate your feedback and am available to answer any further > > >>>>> questions > > >>>>> you may have. > > >>>>> > > >>>>> Best regards, > > >>>>> Dilnaz Amanzholova > > >>>>> > > >>>>> > > >>>>> On Fri, Jun 5, 2026 at 1:23 PM Dilnaz Amanzholova < > > [email protected]> > > >>>>> wrote: > > >>>>> > > >>>>> > Hi all, > > >>>>> > > > >>>>> > I’m calling vote on AIP-85: DAG importer > > >>>>> > https://cwiki.apache.org/confluence/x/_Q7OEg > > >>>>> > > > >>>>> > You can also review the design document here > > >>>>> > < > > >>>>> > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 > > >>>>> > > > >>>>> > . > > >>>>> > > > >>>>> > Current discussion thread (after scoping down): > > >>>>> > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt > > >>>>> > Original discussion thread (with wider "extendable DAG parsing > > >>>>> controls" > > >>>>> > title): > > >>>>> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl > > >>>>> > > > >>>>> > The vote will run for 5 days, closing on Wednesday, 10th June > 2026, > > >>>>> at > > >>>>> > 10:00 UTC. > > >>>>> > > > >>>>> > Everyone is encouraged to vote, but only PMC members and > > Committers' > > >>>>> votes > > >>>>> > are considered binding. Please vote accordingly. > > >>>>> > > > >>>>> > [ ] +1 Approve > > >>>>> > [ ] +0 no opinion > > >>>>> > [ ] -1 disapprove with the reason > > >>>>> > > > >>>>> > Kind regards, > > >>>>> > Dilnaz Amanzholova > > >>>>> > > > >>>>> > > >>>> > > > > > -- > Tatiana Al-Chueyr >
