+1 for the idea and what Shahar wrote.
> As part of this, you raised a topic that I had lost track of,
specifically:
> AIP-59.
> Is it actively deployed and being used now?
Not really. It is not used in CI or it was never finished. I think that
active development of it was couple years ago when e.g.
https://github.com/apache/airflow/tree/main/performance was created, but
it was never finished. So after so match time passing and the fact that
AIP-59 was created while Airflow 3 was not the topic, it would require
probably rewrite to adjusting to Airflow 3 architecture.
Looking at the AIP now -
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/278465469/ADIP-9+Performance+tests+framework
- it looks like someone is working on adjusting it.
On 13/10/2026 10:19, Shahar Epstein wrote:
1.
Do we want micro-benchmarks for hot
paths at all? If so, is this
*
an extension of AIP-59, or does it need its >
own AIP?
I think that it would be a great idea to be able to evaluate performance of
hot paths. For example, it would be very helpful for implementing AIP-106
(composable trigger rules), where concerns of performance had been raised.
I'd make these tests as part of AIP-59, and polish the rest of it on the
way.
2.
Where should they live and how should > they be named? I'd propose
airflow-core/tests/performance, I assume? :)
I think that it's better to focus the discussion on the essence, use cases
and high-level design (oh, and naming things!), so implementation parts
could be left for the PR.
Shahar
On Mon, Oct 12, 2026, 23:42 Blain David <[email protected]> wrote:
Hi all,
I'd like to restart the discussion on performance testing. This time
it's about a layer that AIP-59 doesn't cover: micro-benchmarks for
single hot-path methods, run automatically on pull requests.
Why now?
While addressing review comments on Task Iteration (#62922 [1]), a fix
to CommsDecoder.asend changed it from two thread hops per request to
one. Every async SDK call a task makes goes through it: Variables,
Connections, XComs, the task state store. Measured locally (median of
three runs, before -> after):
send, one sync round trip (unchanged) 11.5 us -> 11.4 us (none)
asend, one async round trip 108.7 us -> 58.0 us (-47%)
asend, 50 concurrent coroutines on a loop 18.6 ms -> 11.7 ms (-37%)
The reverse change, an extra thread hop added in passing, would have
cost the same amount, and nothing in CI would have noticed. Today we
only find such regressions when someone measures by hand (or by agent),
often after a release: #71721 [2] reports structured task logging in 3.2.0
multiplying high-volume task runtime by 2.64.
Micro-benchmarks could relate to AIP-59 (Performance tests framework [3]),
but would sit one layer below it, not replace it. They target single
methods on
hot paths and flag a regression in the PR that introduces it, in seconds
and
on every PR. AIP-59 answers the question micro-benchmarks can't: how a
whole release behaves under load.
Draft PR #74562 [8] adds:
*
task-sdk/benchmarks/, next to tests/ and outside the default test
*
paths, with three benchmarks of the supervisor comms written as
plain pytest functions using pytest-codspeed (MIT);
*
a "benchmarks" dependency group;
*
a workflow that runs the benchmarks through CodSpeed in simulation
*
That mode counts CPU instructions rather than measuring time,
so results don't depend on how busy the shared runner is, and
CodSpeed comments each PR with its change against main.
CodSpeedHQ/action is already on the ASF allow-list (v5.2.1, pinned in
apache/infrastructure-actions). It authenticates through OIDC, so no
secret is needed. Someone with admin rights would still have to
enable the CodSpeed app for apache/airflow.
In the meantime
Following a suggestion in the contributors Slack, draft PR #74579 [9]
adds a "Performance-sensitive components" section to the
airflow-publish-changes agent skill. It lists the components that run
for every task, SDK call, log line or scheduler loop, and asks agents
to measure the changed path against the base commit before pushing
and to put the numbers in the PR description. It needs no tooling,
and can point at the benchmarks/ folders once they exist.
Questions I like your opinion on:
1.
Do we want micro-benchmarks for hot paths at all? If so, is this
*
an extension of AIP-59, or does it need its own AIP?
2.
Where should they live and how should they be named? I'd propose
*
<distribution>/benchmarks/, with paths mirroring the source module
*
they cover:
src/airflow/sdk/execution_time/comms.py
-> benchmarks/execution_time/test_comms_benchmark.py
CI could then run only the benchmarks covering the files a PR
changes.
1.
Which tool? pytest-codspeed with the hosted comparison, or
*
something purely local like pytest-benchmark? The local option
*
compares against a saved baseline with a threshold, but wall-time
*
thresholds on shared runners are noisy.
2.
What should a regression do: report only (a PR comment), or fail
*
the check above a threshold?
3.
Which paths should come first? Beyond the comms, candidates are
*
XCom serialisation, template rendering, DAG parsing, and the
*
scheduler's critical-section queries.
[1] https://github.com/apache/airflow/pull/62922
[2] https://github.com/apache/airflow/issues/71721
[3]
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-59+Performance+tests+framework
[4] https://lists.apache.org/thread/0kyp33cp7q5m98s8q7c3pb6nobhprdyt
[5] https://github.com/apache/airflow/pull/41961
[6] https://github.com/apache/airflow/pull/41315
[7] https://lists.apache.org/thread/kg54mjdj1wg8r6jof28jrbjh7d3p7122
[8] https://github.com/apache/airflow/pull/74562
[9] https://github.com/apache/airflow/pull/74579
Thanks,
David
General (Internal Property)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
