100% on what Tatiana wrote :D On Wed, Jul 8, 2026 at 3:06 PM Tatiana Al-Chueyr Martins < [email protected]> wrote:
> Thanks a lot for being so responsive to the feedback on the AIP Confluent > page, Dilnaz. Since yesterday, you've added file_dependencies to > DagImportResult, introduced a DagSource class that carries the source > language for the Code tab, and made the importer precedence order explicit > (bundle explicit mapping > global config > defaults) - all great > improvements. > > I agree with Jarek on reconciling the merged foundation with the AIP > (particularly the per-bundle registry) before re-running the vote. And > yeah, explicitly stating that a declarative DAG file carries the same trust > level as the Python code is important - in our experience, users often > think YAML is just data, so we should make this explicit. > > On the zip question, composite importers feel like the right path forward. > Once we get non-Python formats treated as first-class citizens, it feels > like the next natural step is mixed content. Delegating to inner importers > handles that cleanly, whereas I feel that extending PythonDagImporter > wouldn't scale as well. > > It does feel we should wrap up the bigger questions in [DISCUSS] - this > will allow maintainers to VOTE on the proposed design with confidence, and > the AIP will be stronger with it. With the pace I've observed since > yesterday, it doesn't feel like we're far off - looking forward to seeing > it back into VOTE. > > Kind regards, > > Tatiana > > On Wed, 8 Jul 2026 at 13:24, Jarek Potiuk <[email protected]> wrote: > > > Hi Dilnaz, and indeed - thanks for pushing this forward. > > > > The scoped-down "DAG importer" framing is much clearer, and > > native non-Python DAGs are a genuinely good direction. +1 on the goal. > > > > I am happy to make comments and suggestions or even fill-in some blanks > > (security especially) in the AIP proposal if you are fine with me doing > it > > - see > > some of the comments below. > > > > Before I can vote on the design, I'd like to raise one structural concern > > plus a couple of smaller ones. My main point is that the foundation > already > > merged in https://github.com/apache/airflow/pull/60127 > > contradicts the design the AIP describes, and I think we > > should reconcile the two before the vote closes rather than after. > > > > Some of these points are already explicitly stated, either in the AIP or > > opened > > questions raised by Tatiana - I just want to reiterate them here and see > if > > we can > > reach consensus on the shape of those changes. > > > > Concretely, in airflow-core/src/airflow/dag_processing/importers/: > > > > 1. Registry scope — global singleton vs. per-bundle. The AIP describes > > per-bundle importer mapping with a bundle -> global -> default resolution > > order (and Tatiana's Q4/Q7 ask for the same). But the merged > > DagImporterRegistry is a process-wide singleton keyed by extension, "last > > registered wins, one importer per extension." A global > extension->importer > > map structurally can't give two bundles different importers for the same > > extension, so as it stands the foundation can't support the resolution > > model the AIP promises. If per-bundle is the intended end state, I think > we > > want that in the interface now, because swapping a singleton out later is > > expensive. > > > > 2. Zip handling. The AIP describes a generic ZipImporter that delegates > to > > inner-format importers. The merged PythonDagImporter instead hard-owns > both > > .py and .zip, with a docstring saying YAML-in-zip would require extending > > it > > or writing a composite importer — the opposite of the AIP. I think we > > should > > decide on one or the other direction, I like the composite importer more > - > > even > > if it is more complex, it allows for greater flexibility if we have > > composite > > importers. But I would love to hear what others think. > > > > 3. DagImportResult.file_dependencies is in the AIP dataclass but not in > the > > merged one — and this is more than cosmetic. Declarative DAGs routinely > > share a defaults.yaml; the DFP decides what to re-parse from per-file > > mtime, > > so without importer-declared dependencies a change to a shared file won't > > re-parse its dependents (a staleness bug). This is Tatiana's Q5, and I'd > > treat it as a correctness requirement, not a nice-to-have. Worth spelling > > out how it interacts with AIP-66's git-sha bundle versioning too. > > > > On security, I'd like the AIP to state explicitly that a declarative DAG > > file is fully code-equivalent trust: a YAML importer resolving operators > by > > classpath has the same power as arbitrary Python. People tend to treat > YAML > > as "data" and .py as "code," and I want to make sure nobody builds a > > "review-free YAML bundle" path on the back of this. I'd also like it > stated > > as a hard constraint that importers run only in the DAG processor, > > in-process, under the same parse timeout, and never touch the metadata > DB — > > consistent with the AIP-66/AIP-72 boundary. And a quick confirmation that > > the old "push DAGs over API / ingester" idea stays out of scope, since > that > > one genuinely changes the threat model. > > > > Again, for security, if you are happy with it, I would love to be able to > > fill > > all those security blanks - this can be described on high level in the > AIP, > > and when implementation progresses, it would turn into security model > > updates (I'm also happy to help there). > > > > Two smaller things: > > > > - The safe_mode/might_contain_dag heuristic ("file contains the strings > > airflow + dag") currently lives in the abstract list_dag_files, but it's > > Python-specific and will mis-handle YAML — it should be delegated > > per-importer (Tatiana's Q3). > > > > - It'd be good to confirm the registry indirection adds no measurable > > regression on the plain .py path - some performance tests with basic > > big-numbers-of-dags scenarios would be great to do as part of the > > implementation. > > > > None of this is a blocker on the idea — I'm supportive. But given the > open > > design questions and that the vote has been quiet, It reaffirms we treat > > this > > as still in [DISCUSS], reconcile the AIP with the merged foundation > > (especially point 1), and re-run the vote once the interface reflects the > > intended end state. > > > > Happy to help review the per-bundle registry shape. > > > > Best, > > Jarek > > > > On Tue, Jul 7, 2026 at 4:45 PM Tatiana Al-Chueyr Martins < > > [email protected]> wrote: > > > > > Hi Dilnaz, > > > > > > Thanks a lot for pushing this forward, and for the updates after the > > > earlier feedback - AIP 85 looks much clearer than before. > > > > > > I'm one of the maintainers of DAG Factory ( > > > https://github.com/astronomer/dag-factory), and I like the direction > the > > > AIP is taking. I agree with the pain points Igor raised earlier; it > would > > > be great to support non-Python DAGs natively in Airflow, rather than > > > converting them as we currently do in DAG Factory. > > > > > > I've reviewed the AIP page and left some inline comments & questions. > > Since > > > the authors are already responding on the AIP page, happy to continue > the > > > detailed discussion there - sharing the summary here mainly for > > visibility: > > > > > > 1. *Location of schemas and importers*: Where do we see the schemas and > > > actual importers defined - and how do we plan to govern them? It would > > be > > > important to align, for instance, on how non-primitive operator > arguments > > > should be defined in YAML (e.g., Asset, V1Pod, arbitrary classes) and > on > > > the security aspects of loading arbitrary classes from YAML. > > > > > > 2. *Validation contract*: If users define unknown or unsupported keys > and > > > configurations, should we fail loudly with DagImportErrors referencing > > > files/lines like the Python DAG import errors? It would be great if we > > > didn't just skip them. > > > > > > 3.* Decision on which files to parse: *As raised by TP on > > > < > > > > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > > > > > > the > > > Google > > > < > > > > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > > > > > > Doc > > > < > > > > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > > > > > > , I feel that we should have additional criteria to decide which files > to > > > treat as a DAG - and not rely only on the file extension. As an > example, > > > bundles normally have non-DAG YAML - dbt_project.yml, CI configs, K8s > > > manifests. Would it make sense to have something similar to the Python > > > content-level heuristic ("airflow + dag" string check)? > > > > > > 4. *Different importers for the same file extension*: Can/should we > > support > > > different importers for the same file extension? It would be great if > the > > > architecture could support overriding the importer per file. > > > > > > 5.* Cross-file dependency tracking:* Declarative DAGs are rarely > > > single-file. They can share default definitions, for example: > > > https://astronomer.github.io/dag-factory/1.1.0/configuration/defaults/ > . > > > Could importers declare file dependencies in DagImportResult, so the > > > processor can use them? > > > > > > 6. *Code tab UI*: On get_source_code(), could we have a parameter > related > > > to the file language, which could be used by the Code tab (it is > > currently > > > hard-coded with Python)? How are we planning to resolve ZipImporter and > > > other transformed sources in the UI? > > > > > > 7. *Precedence*: What should be the precedence (per-bundle vs global vs > > > default importers)? > > > > > > For the performance questions raised earlier in the thread, we could > use > > > DAG Factory workloads as a baseline, comparing the non-built-in-Airflow > > > Python layer against a native YAML importer. > > > > > > Our team is very keen to contribute to the YAML contract definition. > > > > > > Kind regards, > > > > > > Tatiana > > > > > > > > > On Tue, 7 Jul 2026 at 13:48, Dilnaz Amanzholova via dev < > > > [email protected]> wrote: > > > > > > > Hi all, I am following up to request your feedback on AIP-85: DAG > > > importer. > > > > I attempted to present this at the last Airflow dev call, but the > > agenda > > > > was full, and today’s meeting was unfortunately cancelled. As it has > > been > > > > some time since this AIP was submitted for review, I would greatly > > > > appreciate it if you could share your thoughts or any remaining > > concerns > > > > here. I understand there are many active discussions, but your input > > > would > > > > be invaluable in helping move this proposal forward. Thanks, Dilnaz > > > > Amanzholova > > > > > > > > On Thu, Jun 18, 2026 at 2:08 PM Dilnaz Amanzholova < > [email protected] > > > > > > > wrote: > > > > > > > > > Hey Kaxil, > > > > > > > > > > Thanks for the clarification. I have decided to attend, as the next > > > > > meeting is not for another two weeks. I have added the AIP to the > > > agenda. > > > > > > > > > > Best regards, > > > > > Dilnaz Amanzholova > > > > > > > > > > > > > > > On Wed, Jun 17, 2026 at 6:37 PM Kaxil Naik <[email protected]> > > > wrote: > > > > > > > > > >> Hey not required at all. > > > > >> > > > > >> This is just in case you wanted to solicit feedback live on the > call > > > > with > > > > >> other Airflow devs. Sometimes it is easier that way -- but not > > > > mandatory. > > > > >> > > > > >> More details on the dev calls: > > > > >> > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Call > > > if > > > > >> you decide about presenting it next time. > > > > >> > > > > >> Regards, > > > > >> Kaxil > > > > >> > > > > >> On Wed, 17 Jun 2026 at 17:32, Dilnaz Amanzholova < > > [email protected]> > > > > >> wrote: > > > > >> > > > > >>> Hi Kaxil, > > > > >>> > > > > >>> I was not aware there was a regular call. Is it required to > present > > > the > > > > >>> AIP before voting? > > > > >>> > > > > >>> Kind regards, > > > > >>> Dilnaz Amanzholova > > > > >>> > > > > >>> > > > > >>> On Wed, Jun 17, 2026 at 6:26 PM Kaxil Naik <[email protected]> > > > > wrote: > > > > >>> > > > > >>>> Hi @dilnaz -- Are you up for presenting it tomorrow or on the > next > > > dev > > > > >>>> call? > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> On Tue, 16 Jun 2026 at 15:15, Dilnaz Amanzholova via dev < > > > > >>>> [email protected]> wrote: > > > > >>>> > > > > >>>>> Hi all, > > > > >>>>> > > > > >>>>> I am following up on AIP-85: DAG importer. All previous > concerns > > > have > > > > >>>>> been > > > > >>>>> addressed, and the content has been updated on the AIP > Confluence > > > > page. > > > > >>>>> > > > > >>>>> Please review the latest updates at your earliest convenience. > I > > > > would > > > > >>>>> appreciate your feedback and am available to answer any further > > > > >>>>> questions > > > > >>>>> you may have. > > > > >>>>> > > > > >>>>> Best regards, > > > > >>>>> Dilnaz Amanzholova > > > > >>>>> > > > > >>>>> > > > > >>>>> On Fri, Jun 5, 2026 at 1:23 PM Dilnaz Amanzholova < > > > > [email protected]> > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>> > Hi all, > > > > >>>>> > > > > > >>>>> > I’m calling vote on AIP-85: DAG importer > > > > >>>>> > https://cwiki.apache.org/confluence/x/_Q7OEg > > > > >>>>> > > > > > >>>>> > You can also review the design document here > > > > >>>>> > < > > > > >>>>> > > > > > > > > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 > > > > >>>>> > > > > > >>>>> > . > > > > >>>>> > > > > > >>>>> > Current discussion thread (after scoping down): > > > > >>>>> > > > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt > > > > >>>>> > Original discussion thread (with wider "extendable DAG > parsing > > > > >>>>> controls" > > > > >>>>> > title): > > > > >>>>> > https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl > > > > >>>>> > > > > > >>>>> > The vote will run for 5 days, closing on Wednesday, 10th June > > > 2026, > > > > >>>>> at > > > > >>>>> > 10:00 UTC. > > > > >>>>> > > > > > >>>>> > Everyone is encouraged to vote, but only PMC members and > > > > Committers' > > > > >>>>> votes > > > > >>>>> > are considered binding. Please vote accordingly. > > > > >>>>> > > > > > >>>>> > [ ] +1 Approve > > > > >>>>> > [ ] +0 no opinion > > > > >>>>> > [ ] -1 disapprove with the reason > > > > >>>>> > > > > > >>>>> > Kind regards, > > > > >>>>> > Dilnaz Amanzholova > > > > >>>>> > > > > > >>>>> > > > > >>>> > > > > > > > > > > > > > -- > > > Tatiana Al-Chueyr > > > > > > > > -- > Tatiana Al-Chueyr >
