Hi Chao-Hung, Thanks for raising this. It's also the part I am least sure about. As I mentioned in my last mail, the design for step 3 might need further discussion (e.g. do we need to share the state transition logic between the scheduler and the SDK?). If the two behave differently, step 3 would not be very useful, as you said.
For now, I'd like to keep this thread on a simpler discussion: is this a real need for our users, and is it worth an AIP? If we decide it is, I will bring your points into the AIP so we can discuss how to handle them. Thanks, Andrew Chao-Hung Wan <[email protected]> 於 2026年11月17日週二 下午5:20寫道: > One concern I have about Step 3 is the testing fidelity. > > If users can install only the Task SDK and run a DAG locally, what level of > confidence can they get that the DAG will behave the same way in a real > Airflow deployment? > > For example, how do we ensure that the DB-less runner and the production > Scheduler share the same execution semantics, especially for trigger rules, > task state transitions, and dynamic task mapping? > > I also think we should consider the debugging experience when a DAG passes > SDK testing but fails in a real Airflow environment. Users should be able > to identify whether the failure comes from their task logic, the SDK > runtime, or Airflow-specific components. > > Otherwise, users may still need to install and run a full Airflow > environment to gain confidence before deployment, which would significantly > reduce the value of Step 3. > > How do you envision addressing this gap? > > > On 2026/11/16 04:39:18 Andrew Chang wrote: > > Hi Kaxil, > > > > Thanks for the detailed questions! They helped me to think about the > scope > > more clearly. Here I want to clarify each of them: > > > > 1. What's the concrete failure? > > I reproduced some cases (Claude helped me investigate this). In each of > > them, when users follow the docs to write the test, the test passes, but > > the real run fails. Here are the detailed examples: > > https://gist.github.com/Andrushika/4d15594533993c05b1c3ef8d21350ae6 > > So for the problems you listed, the first and the second kind are what > > users hit when they follow the docs. If they want a real run to avoid > this, > > they hit the third kind, because the only public way today is > `dag.test()`, > > and it needs a DB. So I think the three kinds are connected, and they can > > be addressed by the same approach that I am planning to propose. > > > > 2. Who is the user? > > Mainly Dag authors who test their own task logic, with or without agents. > > The needs of provider authors look more like our tests_common, so I would > > not focus on them here. > > > > 3. What is the unit under test? > > I am planning to split this into 3 steps. I will describe them below. > > > > 4. Which private interfaces are users relying on? > > TBH `ti.run()` is also the only one I know, plus one third-party plugin > > that follows our private code. I may have overstated this point in my > first > > mail. > > > > 5. Why isn't `dag.test()` enough? > > Most of its problems can be fixed, like the DB setup, the real retry > delay, > > and not giving the error back. So improving `dag.test()` is an option for > a > > whole Dag run. But two things are harder to fix. It always runs the > > upstream tasks, so users cannot test only one task. And it needs > > airflow-core and a DB, which #61803 (an issue opened by Amogh) says > should > > go away for the client-server separation. > > > > 6. What's out of scope? > > Agreed. Calls to external systems are still the user's job to mock. I > also > > don't plan to touch anything across Dags, or executors. > > > > What is in my mind is that we can split this into three steps: > > 1. Let users run a single task for real, completely without a DB. This is > > the smallest step, and it already solves most of the cases in my > examples. > > 2. Make `dag.test()` better for writing tests. Today it requires a > prepared > > DB, waits for the real retry delay, and does not give the error back. > > 3. Let users run a whole Dag without core and without a DB. I think we > need > > this anyway if Dag authors only install the SDK one day. This is the > > biggest one, and I am not sure about the right design yet. > > > > I would also like to clarify one thing. I used the word "package" in the > > title and in my first mail, which could be misleading. Sorry for the > > confusion. > > What is in my mind is that step 1 and step 3 could be a separate module > in > > the Task SDK, as a DB-less testing module. The design for step 3 might > need > > further discussion (e.g. how do we simulate the task state transitions? > Do > > we need to share that logic between the scheduler and the SDK?). That's > why > > I think it might need an AIP. > > > > Hope this makes the direction clear. Looking forward to your thoughts! > > > > Thanks, > > Andrew > > > > Kaxil Naik <[email protected]> 於 2026年11月16日週一 上午7:41寫道: > > > > > Hi Andrew, > > > > > > Thanks for starting this. I think there's a real gap here, but before > we > > > get to an AIP I'd like to pin down the exact problem and the scope, > because > > > "a testing package" can mean very different things. > > > > > > 1. What's the concrete failure? Can you share one or two real cases: a > Dag, > > > the test the user wrote, and what broke? There are a few different > problems > > > hiding in your list and they have different fixes: > > > > > > - A test passes but the real run fails, because the user built the > > > context by hand. For example, a test that renders `{{ ds }}` with > > > `context={"ds": ...}` passes, but on Airflow 3 a run without a logical > date > > > has no `ds` in the context, so the template fails to render. > > > - A test can't catch something only the task runner does, for > example a > > > TaskFlow return value that Airflow can't serialize into XCom. > > > - A test can't be written at all without a metadata DB. > > > > > > Which of these is the one you keep hitting? > > > > > > 2. Who is the user? Dag authors testing their own task logic, with or > > > without agents, or people writing custom operators and providers? > Jarek's > > > earlier thread was mostly about the second group, and their needs look > a > > > lot like our tests_common. > > > > > > 3. What is the unit under test: one task, a few tasks with XCom between > > > them, or a whole Dag run (trigger rules, branching, mapped tasks)? Each > of > > > these is a very different amount of work. > > > > > > 4. Which private interfaces are users relying on today, and which ones > > > broke on upgrade? `ti.run()` (removed in 3.2) is the one I know of. Are > > > there others, like copies of `create_runtime_ti`? > > > > > > 5. Why isn't `dag.test()` enough? Is it that it needs a metadata DB, > that > > > it's slow, or that it gives you nothing to assert on at the task level? > If > > > it's only the DB or the speed, is improving `dag.test()` an option? > > > > > > 6. What's out of scope? I'd assume calls to external systems (S3, a > > > warehouse) are still the user's job to mock, even in your proposal. > > > > > > Answers to these, with real examples, would make it much easier to > judge > > > whether this should be a new API, a change to `dag.test()`, or better > docs. > > > > > > Thanks, > > > Kaxil > > > > > > On Sun, 15 Nov 2026 at 19:45, Andrew Chang <[email protected]> wrote: > > > > > > > Hi all, > > > > > > > > I have been thinking about proposing a public testing package for > Airflow > > > > users. Before I spend more time on it, I would like to hear your > > > opinions. > > > > > > > > The problem I see is that there is no good way to test a Dag with > pytest > > > > today. The docs [1] suggest calling the operator directly with a > context > > > > you build by yourself, or running the whole Dag in a way that was > made > > > for > > > > debugging. > > > > > > > > TBH I don't think either one is good enough. > > > > > > > > Why I think we need this: > > > > > > > > - If users build the context and fake the inputs by hand, they are > > > testing > > > > their own setup, not Airflow. The test still passes even when > Airflow's > > > > behavior changes. > > > > > > > > - Users who want a real test have to use our private interfaces. They > > > only > > > > find out it is broken after they upgrade. > > > > > > > > - A real test needs a prepared metadata database today, which is slow > and > > > > hard to set up in CI. A test for a single task should not need one. > (This > > > > might be related to the recent discussion on a faster `import > airflow` > > > > [2].) > > > > > > > > - Many Dags are written with AI agents now. I think a fast and real > test > > > is > > > > what users will need most. It gives agents a quick way to verify > their > > > > work. > > > > > > > > I found an earlier discussion started by Jarek [3], where releasing > our > > > > “whole internal test utilities” to users was rejected because of the > > > > compatibility burden. What I want to discuss here is different: a > public > > > > API designed for users, not our internal tools. And we already > maintain > > > the > > > > hard part, because our own tests run tasks this way. > > > > > > > > I would like to hear your thoughts. If it sounds reasonable, I would > be > > > > happy to write an AIP for it! > > > > > > > > Thanks, > > > > Andrew > > > > > > > > [1] > > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#unit-tests > > > > [2] https://lists.apache.org/thread/1nhdc8y1x7dclcpcck98q5chzh1tvodk > > > > [3] https://lists.apache.org/thread/jonodvjnycv01qcsnlqczpvrjhfsx04g > > > > > > > > > >
