One concern I have about Step 3 is the testing fidelity. If users can install only the Task SDK and run a DAG locally, what level of confidence can they get that the DAG will behave the same way in a real Airflow deployment?
For example, how do we ensure that the DB-less runner and the production Scheduler share the same execution semantics, especially for trigger rules, task state transitions, and dynamic task mapping? I also think we should consider the debugging experience when a DAG passes SDK testing but fails in a real Airflow environment. Users should be able to identify whether the failure comes from their task logic, the SDK runtime, or Airflow-specific components. Otherwise, users may still need to install and run a full Airflow environment to gain confidence before deployment, which would significantly reduce the value of Step 3. How do you envision addressing this gap? On 2026/11/16 04:39:18 Andrew Chang wrote: > Hi Kaxil, > > Thanks for the detailed questions! They helped me to think about the scope > more clearly. Here I want to clarify each of them: > > 1. What's the concrete failure? > I reproduced some cases (Claude helped me investigate this). In each of > them, when users follow the docs to write the test, the test passes, but > the real run fails. Here are the detailed examples: > https://gist.github.com/Andrushika/4d15594533993c05b1c3ef8d21350ae6 > So for the problems you listed, the first and the second kind are what > users hit when they follow the docs. If they want a real run to avoid this, > they hit the third kind, because the only public way today is `dag.test()`, > and it needs a DB. So I think the three kinds are connected, and they can > be addressed by the same approach that I am planning to propose. > > 2. Who is the user? > Mainly Dag authors who test their own task logic, with or without agents. > The needs of provider authors look more like our tests_common, so I would > not focus on them here. > > 3. What is the unit under test? > I am planning to split this into 3 steps. I will describe them below. > > 4. Which private interfaces are users relying on? > TBH `ti.run()` is also the only one I know, plus one third-party plugin > that follows our private code. I may have overstated this point in my first > mail. > > 5. Why isn't `dag.test()` enough? > Most of its problems can be fixed, like the DB setup, the real retry delay, > and not giving the error back. So improving `dag.test()` is an option for a > whole Dag run. But two things are harder to fix. It always runs the > upstream tasks, so users cannot test only one task. And it needs > airflow-core and a DB, which #61803 (an issue opened by Amogh) says should > go away for the client-server separation. > > 6. What's out of scope? > Agreed. Calls to external systems are still the user's job to mock. I also > don't plan to touch anything across Dags, or executors. > > What is in my mind is that we can split this into three steps: > 1. Let users run a single task for real, completely without a DB. This is > the smallest step, and it already solves most of the cases in my examples. > 2. Make `dag.test()` better for writing tests. Today it requires a prepared > DB, waits for the real retry delay, and does not give the error back. > 3. Let users run a whole Dag without core and without a DB. I think we need > this anyway if Dag authors only install the SDK one day. This is the > biggest one, and I am not sure about the right design yet. > > I would also like to clarify one thing. I used the word "package" in the > title and in my first mail, which could be misleading. Sorry for the > confusion. > What is in my mind is that step 1 and step 3 could be a separate module in > the Task SDK, as a DB-less testing module. The design for step 3 might need > further discussion (e.g. how do we simulate the task state transitions? Do > we need to share that logic between the scheduler and the SDK?). That's why > I think it might need an AIP. > > Hope this makes the direction clear. Looking forward to your thoughts! > > Thanks, > Andrew > > Kaxil Naik <[email protected]> 於 2026年11月16日週一 上午7:41寫道: > > > Hi Andrew, > > > > Thanks for starting this. I think there's a real gap here, but before we > > get to an AIP I'd like to pin down the exact problem and the scope, because > > "a testing package" can mean very different things. > > > > 1. What's the concrete failure? Can you share one or two real cases: a Dag, > > the test the user wrote, and what broke? There are a few different problems > > hiding in your list and they have different fixes: > > > > - A test passes but the real run fails, because the user built the > > context by hand. For example, a test that renders `{{ ds }}` with > > `context={"ds": ...}` passes, but on Airflow 3 a run without a logical date > > has no `ds` in the context, so the template fails to render. > > - A test can't catch something only the task runner does, for example a > > TaskFlow return value that Airflow can't serialize into XCom. > > - A test can't be written at all without a metadata DB. > > > > Which of these is the one you keep hitting? > > > > 2. Who is the user? Dag authors testing their own task logic, with or > > without agents, or people writing custom operators and providers? Jarek's > > earlier thread was mostly about the second group, and their needs look a > > lot like our tests_common. > > > > 3. What is the unit under test: one task, a few tasks with XCom between > > them, or a whole Dag run (trigger rules, branching, mapped tasks)? Each of > > these is a very different amount of work. > > > > 4. Which private interfaces are users relying on today, and which ones > > broke on upgrade? `ti.run()` (removed in 3.2) is the one I know of. Are > > there others, like copies of `create_runtime_ti`? > > > > 5. Why isn't `dag.test()` enough? Is it that it needs a metadata DB, that > > it's slow, or that it gives you nothing to assert on at the task level? If > > it's only the DB or the speed, is improving `dag.test()` an option? > > > > 6. What's out of scope? I'd assume calls to external systems (S3, a > > warehouse) are still the user's job to mock, even in your proposal. > > > > Answers to these, with real examples, would make it much easier to judge > > whether this should be a new API, a change to `dag.test()`, or better docs. > > > > Thanks, > > Kaxil > > > > On Sun, 15 Nov 2026 at 19:45, Andrew Chang <[email protected]> wrote: > > > > > Hi all, > > > > > > I have been thinking about proposing a public testing package for Airflow > > > users. Before I spend more time on it, I would like to hear your > > opinions. > > > > > > The problem I see is that there is no good way to test a Dag with pytest > > > today. The docs [1] suggest calling the operator directly with a context > > > you build by yourself, or running the whole Dag in a way that was made > > for > > > debugging. > > > > > > TBH I don't think either one is good enough. > > > > > > Why I think we need this: > > > > > > - If users build the context and fake the inputs by hand, they are > > testing > > > their own setup, not Airflow. The test still passes even when Airflow's > > > behavior changes. > > > > > > - Users who want a real test have to use our private interfaces. They > > only > > > find out it is broken after they upgrade. > > > > > > - A real test needs a prepared metadata database today, which is slow and > > > hard to set up in CI. A test for a single task should not need one. (This > > > might be related to the recent discussion on a faster `import airflow` > > > [2].) > > > > > > - Many Dags are written with AI agents now. I think a fast and real test > > is > > > what users will need most. It gives agents a quick way to verify their > > > work. > > > > > > I found an earlier discussion started by Jarek [3], where releasing our > > > “whole internal test utilities” to users was rejected because of the > > > compatibility burden. What I want to discuss here is different: a public > > > API designed for users, not our internal tools. And we already maintain > > the > > > hard part, because our own tests run tasks this way. > > > > > > I would like to hear your thoughts. If it sounds reasonable, I would be > > > happy to write an AIP for it! > > > > > > Thanks, > > > Andrew > > > > > > [1] > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#unit-tests > > > [2] https://lists.apache.org/thread/1nhdc8y1x7dclcpcck98q5chzh1tvodk > > > [3] https://lists.apache.org/thread/jonodvjnycv01qcsnlqczpvrjhfsx04g > > > > > >
