I added some comments here and I think there is one big thing that should be clarified when we get to "task isolation" - mainly dependance of it on AIP-44.
The Internal gRPC API (AIP-44) was only designed in the way it was designed to allow using the same codebase to be used with/without DB. It's based on the assumption that a limited set of changes will be needed (that was underestimated) in order to support both DB and GRPC ways of communication between workers/triggerers/DAG file processors at the same time. That was a basic assumption for AIP-44 - that we will want to keep both ways and maximum backwards compatibility (including "pull" model of worker getting connections, variables, and updating task state in the Airflow DB). We are still using "DB" as a way to communicate between those components and this does not change with AIP-44. But for Airflow 3 the whole context is changed. If we go with the assumption that Airflow 3 will only have isolated tasks and no DB "option", I personally think using AIP-44 for that is a mistake. AIP-44 is merely a wrapper over existing DB calls designed to be kept updated together with the DB code, and the whole synchronisation of state, heartbeats, variables and connection access still uses the same "DB communication" model and there is basically no way we can get it more scalable this way. We will still have the same limitations on the DB - where a number of DB connections will be replaced with a number of GRPC connections, Essentially - more scalability and performance has never been the goal of AIP-44- all the assumptions are that it only brings isolation but nothing more will change. So I think it does not address some of the fundamental problems stated in this "isolation" document. Essentially AIP-44 merely exposes a small-ish number of methods (bigger than initially anticipated) but it only wraps around the existing DB mechanism. Essentially from the performance and scalability point of view - we do not get much more than currently when using pgbouncer. This one essentially turns a big number of connections coming from workers into a smaller number of pooled connections that pgbounder manages internal and multiplexes the calls over. With the difference that unlike AIP-44 Internal API server, pgbouncer does not limit the operations you can do from the worker/triggerer/dag file processor - that's the main difference between using pgbouncer and using our own Internal-API server. IMHO - if we do not want to support DB access at all from workers, triggerrers and DAG file processors, we should replace the current "DB" bound interface with a new one specifically designed for this bi-directional direct communication Executor <-> Workers, more in line with what Jens described in AIP-69 (and for example WebSocket and asynchronous communication comes immediately to my mind if I did not have to use DB for that communication). This is also why I put the AIP-67 on hold because IF we go that direction that we have "new" interface between worker, triggerer , dag file processor - it might be way easier (and safer) to introduce multi-team in Airflow 3 rather than 2 (or we can implement it differently in Airflow 2 and differently in Airflow 3). On Tue, Jun 4, 2024 at 3:58 PM Vikram Koka <vik...@astronomer.io.invalid> wrote: > Fellow Airflowers, > > I am following up on some of the proposed changes in the Airflow 3 proposal > < > https://docs.google.com/document/d/1MTr53101EISZaYidCUKcR6mRKshXGzW6DZFXGzetG3E/ > >, > where more information was requested by the community, specifically around > the injection of Task Execution Secrets. This topic has been discussed at > various times with a variety of names, but here is a holistic proposal > around the whole task context mechanism. > > This is not yet a full fledged AIP, but is intended to facilitate a > structured discussion, which will then be followed up with a formal AIP > within the next two weeks. I have included most of the text here, but > please give detailed feedback in the attached document > < > https://docs.google.com/document/d/1BG8f4X2YdwNgHTtHoAyxA69SC_X0FFnn17PlzD65ljA/ > >, > so that we can have a contextual discussion around specific points which > may need more detail. > --- > Motivation > > Historically, Airflow’s task execution context has been oriented around > local execution within a relatively trusted networking cluster. > > This includes: > > - > > the interaction between the Executor and the process of launching a task > on Airflow Workers, > - > > the interaction between the Workers and the Airflow meta-database for > connection and environment information as part of initial task startup, > - > > the interaction between the Airflow Workers and the rest of Airflow for > heartbeat information, and so on. > > This has been accomplished by colocating all of the Airflow task execution > code with the user task code in the same container and process. > > > > For Airflow users at scale i.e. supporting multiple data teams, this has > posed many operational challenges: > > - > > Dependency conflicts for administrators supporting data teams using > different versions of providers, libraries, or python packages > - > > Security challenge in the running of customer-defined code (task code > within the DAGs) for multiple customers within the same operating > environment and service accounts > - > > Scalability of Airflow since one of the core Airflow scalability > limitations has been the number of concurrent database connections > supported by the underlying database instance. To alleviate this > problem, > we have consistently, as an Airflow community, recommended the use of > PgBouncer for connection pooling, as part of an Airflow deployment. > - > > Operational issues caused by unintentional reliance on internal Airflow > constructs within the DAG/Task code, which only and unexpectedly show > up as > part of Airflow production operations, coincidentally with, but not > limited > to upgrades and migrations. > - > > Operational management based on the above for Airflow platform teams at > scale, because different data teams naturally operate at different > velocities. Attempting to support these different teams with a common > Airflow environment is unnecessarily challenging. > > > > The internal API to reduce the need for interaction between the Airflow > Workers and the metadatabase is a big and necessary step forward. However, > it doesn’t fully address the above challenges. The proposal below builds on > the internal API proposal and goes significantly further to not only > address these challenges above, but also enable the following key use > cases: > > 1. > > Ensure that this interface reduces the interaction between the code > running within the Task and the rest of Airflow. This is to address > unintended ripple effects from core Airflow changes which has caused > numerous Airflow upgrade issues, because Task (i.e. DAG) code relied on > Core Airflow abstractions. This has been a common problem pointed out by > numerous Airflow users including early adopters. > 2. > > Enable quick, performant execution of tasks on local, trusted networks, > without requiring the Airflow workers / tasks to connect to the Airflow > database to obtain all the information required for task startup, > 3. > > Enable remote execution of Airflow tasks across network boundaries, by > establishing a clean interface for Airflow workers on remote networks > to be > able to connect back to a central Airflow service to access all > information > needed for task execution. This is foundational work for remote > execution. > 4. > > Enable a clean language agnostic interface for task execution, with > support for multiple language bindings, so that Airflow tasks can be > written in languages beyond Python. > > Proposal > > The proposal here has multiple parts as detailed below. > > 1. > > Formally split out the Task Execution Interface as the Airflow Task SDK > (possibly name it as the Airflow SDK), which would be the only > interface to > and from Airflow Task User code to the Airflow system components > including > the meta-database, Airflow Executor, etc. > 2. > > Disable all direct database interaction from the Airflow Workers > including Tasks being run on those Airflow Workers and the Airflow > meta-database. > 3. > > The Airflow Task SDK will include interfaces for: > - > > Access to needed Airflow Connections, Variables, and XCom values > - > > Report heartbeat > - > > Record logs > - > > Report metrics > 4. > > The Airflow Task SDK will support a Push mechanism for speedy local > execution in trusted environments. > 5. > > The Airflow Task SDK will also support a Pull mechanism for the remote > Task execution environments to access information from an Airflow > instance > over network boundaries. > 6. > > The Airflow Task SDK will be designed to support multiple language > bindings, with the first language binding of course being Python. > > > Assumption: The existing AIP for Internal API covers the interaction > between the Airflow workers and Airflow metadatabase for heartbeat > information, persisting XComs, and so on. > -- > > Best regards, > > Vikram Koka, Ash Berlin-Taylor, Kaxil Naik, and Constance Martineau >
