Hey everyone,
Thank you for attending the dev call on the 10th. I updated
our meeting notes on the Airflow wiki and the link for those notes is here
<https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974137/2026-09-10+Dev+call+Minutes>
To everyone who attended the meeting, please check the summary and add
anything I may have missed. For those who could not join, please let us
know if you disagree with anything discussed and agreed upon in
the meeting. Also, please ask questions if something is unclear.
Our next meeting is scheduled for Thursday, the 24th of September at the
same time i.e. for 8 a.m. Pacific Time.
If you would like to add anything to the agenda or if I missed anything
from the last call, please let me know
Best regards,
Vikram
--
Below is the summary from the call:
- *Unplanned Topic: Fix Grid View for Bidirectional TaskGroup
Dependencies (Dheeraj Turaga)*
- Raised as small talk before the formal agenda, not a planned item.
- Dheeraj described a DAG where two task groups depend on each other
bidirectionally. There's no cycle at the individual task level, but the
grid view can't render this correctly at the collapsed group
level unless a
user double-clicks into the group.
- Ash attributed this to task groups being UI-only rather than part
of actual scheduling logic. A bidirectional dependency between
groups would
become a genuine cycle if task groups were scheduling-aware, especially
with task-group restarts or loops.
- Elad proposed raising a broken DAG error for this pattern instead.
Vikram agreed it shouldn't be allowed at all.
- The underlying case: a DAG author's pipeline runs 4,000 to 5,000
tasks across nested task groups, used to give project managers one
end-to-end view of a large, multi-team design process. The current
workaround is breaking the task groups apart and clearing old DAG runs so
the UI renders.
- Vikram asked why such large DAGs aren't split into smaller DAGs
linked by assets or a trigger DAG run. Dheeraj said users want
one unified
view without clicking across multiple runs, and there are no
real assets to
link since the tasks call the same external tool repeatedly rather than
producing distinct outputs.
- Ash tied this to his ongoing work on task loops and the dynamic
execution graph (AIP-111, AIP-113), and is considering whether
task groups
need to become more than UI-only as part of that.
- Dheeraj hadn't yet identified the specific 3.3 change causing the
issue and will follow up with details.
- *Release Updates: 3.3.2 (Rahul Vats)*
- RC1 targeted for tomorrow, pending a sync with Vincent on the
provider release wave. Most backports are done; a few milestone PRs are
being merged in, and Rahul expects to revert a few that aren't
sufficiently
targeted patches.
- Release expected next week, around the 16th or 17th, barring major
blockers in testing.
- 3.4 feature freeze: Rahul initially said third week of October, but
Vikram corrected this to end of September per the wiki. Historically, 3.2
and 3.3 both took a full three weeks from freeze to release; Elad expects
this cycle to be shorter given a lighter feature set, though Vikram noted
he'd made the same prediction ahead of 3.3, which still took the
full three
weeks.
- Jarek confirmed once Vincent pushes providers to RC, the next 3.3.2
RC build picks them up automatically, no manual workflow trigger needed.
- *Other Development Updates*
- *Road to Common AI 1.0 (Kaxil Naik, Vikram Koka)*
- Kaxil, back after missing Summit and the last three dev calls,
outlined five remaining items for production-ready 1.0:
1. Batch APIs: Anthropic and OpenAI batch support already
exists separately; wants a common layer so users can
switch model IDs
interchangeably. Resonates with users partly due to over
50% cost savings
on large embedding workloads.
2. Cost guardrails: a per-agent spending cap PR from Wei exists
today. Exploring a budget store, via the Task SDK and task
state store, for
cross-agent budgeting; needs a proof of concept.
3. Sandboxing: Docker SBX toolset merged. Next is a model
toolset for remote execution of individual tool calls, building on
Pydantic's code-mode sandbox. An executor-level sandboxing
prototype
(restricting network and usage) is already working; email
coming within the
week.
4. Memory: reusing the task state store, with retention
policies still being worked out at DAG scope and a new agent scope,
alongside the existing task scope.
5. Managed agent alignment: agent invocation PRs for Vertex AI,
AWS Bedrock, and Azure Foundry merged with inconsistent
interfaces relative
to each other, despite Vikram and Jed presenting an
aligned interface at
Summit. Vikram asked reviewers to watch for this going
forward, similar to
how Common SQL enforces a common interface.
- Kaxil wants 1.0 to hold for some time rather than needing a 2.0
within weeks, given how fast this space is moving.
- Shahar asked if 1.0 is release-bound. Kaxil confirmed it isn't:
point releases ship as needed, while 1.0 is a statement of intent and
interface stability, targeted internally for end of September to early
October.
- Vikram raised AIP-105 (Common AI's retry/recovery integration),
noting heavy interest at Summit around what the model or agent can do
during automated recovery, which isn't well documented today. Kaxil
clarified the current implementation is LLM-only, no tool
calls: the model
outputs a retry-timing schema in text, and Airflow's own code
executes the
retry. Jarek suggested this could evolve into a bounded
agentic model with
a defined tool set. Vikram called adding an agent into a
recovery path high
risk.
- *AIP-104: Task Iteration and AIP-103: Task State Store (David
Blain)*
- Following up on last call's concern (Vikram, Ash) that a worker
crash caused task iteration to reprocess all items from
scratch, David's
fix uses the task state store to track which items are
already processed,
so a restarted task only handles what's left.
- This required skipping Airflow's normal XCom purge on restart,
specifically for iterable or batched operators, since task
iteration uses
one task instance referencing multiple XComs (keyed as default_0,
default_1, etc.) rather than one XCom per task.
- Ash noted the task state store is pluggable like XCom. David
hadn't realized this; he'd avoided storing large payloads
there to keep
them out of the metadata database.
- Jarek and Amogh proposed a cleaner design: use the task state
store purely for intermediate per-iteration state, and write
to XCom only
on completion, since XCom is meant for final results, not
checkpointing,
and the task state store already has a comparable object
storage backend.
- David agreed this is workable, though slower since state must be
saved at each step. Jarek suggested batching state writes to
offset this,
and noted reusing one connection across iterations still
beats dynamic task
mapping on performance.
- Timeline concern: this feature has already slipped from 3.2 to
3.3 and now targets 3.4. Jarek and Amogh agreed it should
move to 3.5 if it
can't be completed correctly by 3.4.
- Amogh's framing: an incomplete or incorrectly designed feature
is as good as no feature.
- *Discussion Topics*
- *Docs Quality and AI-Generated Content (Elad Kalif)*
- Elad flagged a pattern of AI-generated docs being merged that
don't directly answer the user's question and add unnecessary length,
citing a 100-line troubleshooting doc that could have been
five lines. He
asked contributors to keep docs brief and avoid merging low-value
AI-generated content, given how much documentation already exists.
- Vikram agreed, noting a growing, increasingly confusing
misalignment between what the project does and what the docs describe.
- Jarek broadened this to a need for a conscious review,
synchronization, and consolidation pass, since much existing
content has
simply gone outdated or points to parts of the project that changed
direction.
- Ash connected it to a broader point: good-looking code or docs
alone aren't sufficient reason to merge a PR, and the project needs a
better way to build consensus among the 30 to 40 people
involved in reviews.
- *Semi-Automated AI-Assisted PR Triage and Security Issue Handling
(Jarek Potiuk)*
- Jarek's PR triage today can assess quality (tests, formatting,
per AGENTS.md <http://agents.md/> criteria) but not whether a PR
should merge. It runs from his personal CLI via Magpie, a
single point of
failure since nobody else runs it.
- Proposal: run PR triage and security issue triage as Airflow
DAGs on an AWS instance (credits available), so it runs
automatically with
human-in-the-loop approval. Magpie is one option for defining
workflows,
not the only one; DAGs could be written by hand. Jarek framed
this as an
opportunity to dogfood Airflow's own agentic capabilities.
- Vikram's initial reaction: write this up as an AIP first. Jarek
agreed an AIP makes sense, similar in spirit to AIP-6 covering a
CI/CD-related process. Shubham has already started drafting a
supporting
document.
- Ash flagged a trade-off: auto-triage risks making a contributor
jump through hoops only for a human to close the PR anyway. Jarek
acknowledged this as a live concern he's iterating on, and
reiterated the
goal is a process the community can run and modify together,
server-side,
without needing to run it locally.
- Vikram asked why the community couldn't just start by sharing
Magpie skills directly. Jarek said that's possible and well documented
today, but almost no one has done it in three months of him
proposing it,
since a skill still needs someone to run it regularly; it's
pull-based, not
push-based.
- Vikram raised a concern this could look like the PMC taking
responsibility for Magpie. Jarek clarified the workflows themselves
wouldn't be Magpie; Magpie would at most help prototype the
DAGs, which
could otherwise be written as standard, tool-agnostic DAGs.
- Vikram suggested the AIP present Magpie as one implementation
option, not the only path. Jarek agreed, then Vikram had to
drop from the
call, noting the group should wait for the AIP.
- Action Items
- Rahul to sync with Vincent on the provider release wave and target
RC1 for tomorrow.
- Dheeraj to identify what changed in 3.3 that caused the TaskGroup
grid view rendering issue, and share findings with the group.
- Kaxil to track interface alignment across Vertex AI, AWS Bedrock,
and Azure Foundry agent invocation PRs against the Common AI common
interface, and continue reporting progress on the Common AI 1.0
roadmap on
future calls.
- David to rework AIP-104 task iteration to use the task state store
for intermediate per-iteration state, writing to XCom only on completion,
targeting 3.4, with 3.5 as a fallback if it can't be completed
correctly in
time.
- Elad to share the AI-generated docs PR example with the community.
- Jarek to write an AIP for Airflow-based, semi-automated PR triage
and security issue handling, presenting Magpie as one
implementation option
rather than the only one.
Vikram Koka
Chief Strategy Officer
Email: [email protected]
<https://www.astronomer.io/>
