Hi Chao-Hung,

Thanks for raising this. It's also the part I am least sure about.
As I mentioned in my last mail, the design for step 3 might need further
discussion (e.g. do we need to share the state transition logic between the
scheduler and the SDK?). If the two behave differently, step 3 would not be
very useful, as you said.

For now, I'd like to keep this thread on a simpler discussion: is this a
real need for our users, and is it worth an AIP? If we decide it is, I will
bring your points into the AIP so we can discuss how to handle them.

Thanks,
Andrew

Chao-Hung Wan <[email protected]> 於 2026年11月17日週二 下午5:20寫道:

> One concern I have about Step 3 is the testing fidelity.
>
> If users can install only the Task SDK and run a DAG locally, what level of
> confidence can they get that the DAG will behave the same way in a real
> Airflow deployment?
>
> For example, how do we ensure that the DB-less runner and the production
> Scheduler share the same execution semantics, especially for trigger rules,
> task state transitions, and dynamic task mapping?
>
> I also think we should consider the debugging experience when a DAG passes
> SDK testing but fails in a real Airflow environment. Users should be able
> to identify whether the failure comes from their task logic, the SDK
> runtime, or Airflow-specific components.
>
> Otherwise, users may still need to install and run a full Airflow
> environment to gain confidence before deployment, which would significantly
> reduce the value of Step 3.
>
> How do you envision addressing this gap?
>
>
> On 2026/11/16 04:39:18 Andrew Chang wrote:
> > Hi Kaxil,
> >
> > Thanks for the detailed questions! They helped me to think about the
> scope
> > more clearly. Here I want to clarify each of them:
> >
> > 1. What's the concrete failure?
> > I reproduced some cases (Claude helped me investigate this). In each of
> > them, when users follow the docs to write the test, the test passes, but
> > the real run fails. Here are the detailed examples:
> > https://gist.github.com/Andrushika/4d15594533993c05b1c3ef8d21350ae6
> > So for the problems you listed, the first and the second kind are what
> > users hit when they follow the docs. If they want a real run to avoid
> this,
> > they hit the third kind, because the only public way today is
> `dag.test()`,
> > and it needs a DB. So I think the three kinds are connected, and they can
> > be addressed by the same approach that I am planning to propose.
> >
> > 2. Who is the user?
> > Mainly Dag authors who test their own task logic, with or without agents.
> > The needs of provider authors look more like our tests_common, so I would
> > not focus on them here.
> >
> > 3. What is the unit under test?
> > I am planning to split this into 3 steps. I will describe them below.
> >
> > 4. Which private interfaces are users relying on?
> > TBH `ti.run()` is also the only one I know, plus one third-party plugin
> > that follows our private code. I may have overstated this point in my
> first
> > mail.
> >
> > 5. Why isn't `dag.test()` enough?
> > Most of its problems can be fixed, like the DB setup, the real retry
> delay,
> > and not giving the error back. So improving `dag.test()` is an option for
> a
> > whole Dag run. But two things are harder to fix. It always runs the
> > upstream tasks, so users cannot test only one task. And it needs
> > airflow-core and a DB, which #61803 (an issue opened by Amogh) says
> should
> > go away for the client-server separation.
> >
> > 6. What's out of scope?
> > Agreed. Calls to external systems are still the user's job to mock. I
> also
> > don't plan to touch anything across Dags, or executors.
> >
> > What is in my mind is that we can split this into three steps:
> > 1. Let users run a single task for real, completely without a DB. This is
> > the smallest step, and it already solves most of the cases in my
> examples.
> > 2. Make `dag.test()` better for writing tests. Today it requires a
> prepared
> > DB, waits for the real retry delay, and does not give the error back.
> > 3. Let users run a whole Dag without core and without a DB. I think we
> need
> > this anyway if Dag authors only install the SDK one day. This is the
> > biggest one, and I am not sure about the right design yet.
> >
> > I would also like to clarify one thing. I used the word "package" in the
> > title and in my first mail, which could be misleading. Sorry for the
> > confusion.
> > What is in my mind is that step 1 and step 3 could be a separate module
> in
> > the Task SDK, as a DB-less testing module. The design for step 3 might
> need
> > further discussion (e.g. how do we simulate the task state transitions?
> Do
> > we need to share that logic between the scheduler and the SDK?). That's
> why
> > I think it might need an AIP.
> >
> > Hope this makes the direction clear. Looking forward to your thoughts!
> >
> > Thanks,
> > Andrew
> >
> > Kaxil Naik <[email protected]> 於 2026年11月16日週一 上午7:41寫道:
> >
> > > Hi Andrew,
> > >
> > > Thanks for starting this. I think there's a real gap here, but before
> we
> > > get to an AIP I'd like to pin down the exact problem and the scope,
> because
> > > "a testing package" can mean very different things.
> > >
> > > 1. What's the concrete failure? Can you share one or two real cases: a
> Dag,
> > > the test the user wrote, and what broke? There are a few different
> problems
> > > hiding in your list and they have different fixes:
> > >
> > >    - A test passes but the real run fails, because the user built the
> > > context by hand. For example, a test that renders `{{ ds }}` with
> > > `context={"ds": ...}` passes, but on Airflow 3 a run without a logical
> date
> > > has no `ds` in the context, so the template fails to render.
> > >    - A test can't catch something only the task runner does, for
> example a
> > > TaskFlow return value that Airflow can't serialize into XCom.
> > >    - A test can't be written at all without a metadata DB.
> > >
> > >    Which of these is the one you keep hitting?
> > >
> > > 2. Who is the user? Dag authors testing their own task logic, with or
> > > without agents, or people writing custom operators and providers?
> Jarek's
> > > earlier thread was mostly about the second group, and their needs look
> a
> > > lot like our tests_common.
> > >
> > > 3. What is the unit under test: one task, a few tasks with XCom between
> > > them, or a whole Dag run (trigger rules, branching, mapped tasks)? Each
> of
> > > these is a very different amount of work.
> > >
> > > 4. Which private interfaces are users relying on today, and which ones
> > > broke on upgrade? `ti.run()` (removed in 3.2) is the one I know of. Are
> > > there others, like copies of `create_runtime_ti`?
> > >
> > > 5. Why isn't `dag.test()` enough? Is it that it needs a metadata DB,
> that
> > > it's slow, or that it gives you nothing to assert on at the task level?
> If
> > > it's only the DB or the speed, is improving `dag.test()` an option?
> > >
> > > 6. What's out of scope? I'd assume calls to external systems (S3, a
> > > warehouse) are still the user's job to mock, even in your proposal.
> > >
> > > Answers to these, with real examples, would make it much easier to
> judge
> > > whether this should be a new API, a change to `dag.test()`, or better
> docs.
> > >
> > > Thanks,
> > > Kaxil
> > >
> > > On Sun, 15 Nov 2026 at 19:45, Andrew Chang <[email protected]> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have been thinking about proposing a public testing package for
> Airflow
> > > > users. Before I spend more time on it, I would like to hear your
> > > opinions.
> > > >
> > > > The problem I see is that there is no good way to test a Dag with
> pytest
> > > > today. The docs [1] suggest calling the operator directly with a
> context
> > > > you build by yourself, or running the whole Dag in a way that was
> made
> > > for
> > > > debugging.
> > > >
> > > > TBH I don't think either one is good enough.
> > > >
> > > > Why I think we need this:
> > > >
> > > > - If users build the context and fake the inputs by hand, they are
> > > testing
> > > > their own setup, not Airflow. The test still passes even when
> Airflow's
> > > > behavior changes.
> > > >
> > > > - Users who want a real test have to use our private interfaces. They
> > > only
> > > > find out it is broken after they upgrade.
> > > >
> > > > - A real test needs a prepared metadata database today, which is slow
> and
> > > > hard to set up in CI. A test for a single task should not need one.
> (This
> > > > might be related to the recent discussion on a faster `import
> airflow`
> > > > [2].)
> > > >
> > > > - Many Dags are written with AI agents now. I think a fast and real
> test
> > > is
> > > > what users will need most. It gives agents a quick way to verify
> their
> > > > work.
> > > >
> > > > I found an earlier discussion started by Jarek [3], where releasing
> our
> > > > “whole internal test utilities” to users was rejected because of the
> > > > compatibility burden. What I want to discuss here is different: a
> public
> > > > API designed for users, not our internal tools. And we already
> maintain
> > > the
> > > > hard part, because our own tests run tasks this way.
> > > >
> > > > I would like to hear your thoughts. If it sounds reasonable, I would
> be
> > > > happy to write an AIP for it!
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > [1]
> > > >
> > > >
> > >
>
> https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#unit-tests
> > > > [2] https://lists.apache.org/thread/1nhdc8y1x7dclcpcck98q5chzh1tvodk
> > > > [3] https://lists.apache.org/thread/jonodvjnycv01qcsnlqczpvrjhfsx04g
> > > >
> > >
> >
>

Reply via email to