Hi Andrew,
Thanks for starting this. I think there's a real gap here, but before we
get to an AIP I'd like to pin down the exact problem and the scope, because
"a testing package" can mean very different things.
1. What's the concrete failure? Can you share one or two real cases: a Dag,
the test the user wrote, and what broke? There are a few different problems
hiding in your list and they have different fixes:
- A test passes but the real run fails, because the user built the
context by hand. For example, a test that renders `{{ ds }}` with
`context={"ds": ...}` passes, but on Airflow 3 a run without a logical date
has no `ds` in the context, so the template fails to render.
- A test can't catch something only the task runner does, for example a
TaskFlow return value that Airflow can't serialize into XCom.
- A test can't be written at all without a metadata DB.
Which of these is the one you keep hitting?
2. Who is the user? Dag authors testing their own task logic, with or
without agents, or people writing custom operators and providers? Jarek's
earlier thread was mostly about the second group, and their needs look a
lot like our tests_common.
3. What is the unit under test: one task, a few tasks with XCom between
them, or a whole Dag run (trigger rules, branching, mapped tasks)? Each of
these is a very different amount of work.
4. Which private interfaces are users relying on today, and which ones
broke on upgrade? `ti.run()` (removed in 3.2) is the one I know of. Are
there others, like copies of `create_runtime_ti`?
5. Why isn't `dag.test()` enough? Is it that it needs a metadata DB, that
it's slow, or that it gives you nothing to assert on at the task level? If
it's only the DB or the speed, is improving `dag.test()` an option?
6. What's out of scope? I'd assume calls to external systems (S3, a
warehouse) are still the user's job to mock, even in your proposal.
Answers to these, with real examples, would make it much easier to judge
whether this should be a new API, a change to `dag.test()`, or better docs.
Thanks,
Kaxil
On Sun, 15 Nov 2026 at 19:45, Andrew Chang <[email protected]> wrote:
> Hi all,
>
> I have been thinking about proposing a public testing package for Airflow
> users. Before I spend more time on it, I would like to hear your opinions.
>
> The problem I see is that there is no good way to test a Dag with pytest
> today. The docs [1] suggest calling the operator directly with a context
> you build by yourself, or running the whole Dag in a way that was made for
> debugging.
>
> TBH I don't think either one is good enough.
>
> Why I think we need this:
>
> - If users build the context and fake the inputs by hand, they are testing
> their own setup, not Airflow. The test still passes even when Airflow's
> behavior changes.
>
> - Users who want a real test have to use our private interfaces. They only
> find out it is broken after they upgrade.
>
> - A real test needs a prepared metadata database today, which is slow and
> hard to set up in CI. A test for a single task should not need one. (This
> might be related to the recent discussion on a faster `import airflow`
> [2].)
>
> - Many Dags are written with AI agents now. I think a fast and real test is
> what users will need most. It gives agents a quick way to verify their
> work.
>
> I found an earlier discussion started by Jarek [3], where releasing our
> “whole internal test utilities” to users was rejected because of the
> compatibility burden. What I want to discuss here is different: a public
> API designed for users, not our internal tools. And we already maintain the
> hard part, because our own tests run tasks this way.
>
> I would like to hear your thoughts. If it sounds reasonable, I would be
> happy to write an AIP for it!
>
> Thanks,
> Andrew
>
> [1]
>
> https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#unit-tests
> [2] https://lists.apache.org/thread/1nhdc8y1x7dclcpcck98q5chzh1tvodk
> [3] https://lists.apache.org/thread/jonodvjnycv01qcsnlqczpvrjhfsx04g
>