Good discussion here, few points from our perspective: - +1 for separating DAG submission into an API endpoint or SDK method. Not only it creates a clean interface boundary for a parsed DAG, but also enables other modes of using that endpoint, e.g. event-driven, which could be a prelude for on-demand DAG processing without the parsing loop, executing DAG parsing remotely, etc.
- With the interface for DAG submission, the DAG lifecycle of parsing / serialisation / execution could be decoupled, both technically and in terms of the AIPs & workstreams of Airflow 3. Having aligned on the boundary shape, the Google team would be happy to contribute to the DAG parsing / processing part (which would be a separate AIP, potentially feeding DAGs into AIP-72). - DAG parser/processor should assume identity and related accesses in a similar way to executor described in AIP-72. I don't think they should be the same component, but could reuse the same parts. The exact shape (and flow) of DAG parser/processor should be decoupled into a separate AIP (we're happy to drive this). - Moving callback execution to workers seems like the right move. Since some callbacks might have 'central' dependencies, we could have two types of callbacks: 1) regular callbacks, executed on worker, and 2) remote callbacks, executed wherever, based on a predefined interface. Let's discuss further in tomorrow's meeting! On Wed, Jul 10, 2024 at 6:56 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > Sound good? > > Absolutely. I agree with all that. Even more - I think it's OK that > the implementation part of the DAGFileProcessor changes might be > shifted to AIP-67. While AIP-72 design should take into account (and > define) DAG serialization and callback passing, implementation of it > could be done as part of AIP-67 - so that the team implementing AIP-67 > can build on the foundation of what AIP-72 implement, > > This has the added advantage that the AIP-72 "ways" will be known more > intimately by more people. > > J. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
