Dear Community Members, I am bringing this thread back to your attention for additional visibility. If no further issues are raised, I can proceed to a vote on Wednesday, 21st October.
Best regards, Shubham On Wed, 14 Oct 2026 at 03:07, Shubham Raj <[email protected]> wrote: > Thanks Jarek; I agree with all of this. I'll update the "Where to get > tokens?" section of the AIP along these lines: > > - Treat token sources as a pool (Bedrock, Azure credits, llm.apache.org, > donated Google/Amazon credits) rather than a single choice. Models are > reached only through an Airflow Connection, so switching is config, not > code. > - Deterministic-first as an explicit design principle: rules start > deterministic, and a model is consulted only for a bounded typed > classification or to tailor a templated message. Token usage per rule/Dag > becomes a pilot metric next to time-to-first-review and findings > accepted/corrected. > - Edge executors in Azure added as a dogfooding stretch goal; initial > pilot stays in one account to keep debugging simple. > > Specifically regarding llm.apache.org: for other community members, it's > a LiteLLM gateway with an OpenAI-compatible endpoint, so common.ai should > work with just a connection change. It also supports script keys on request > ("Request a key" under My Keys), which the deployment would use rather than > anyone's personal PAT. Per-project budgets and usage reports are also > traceable. One thing to note: the models on llm.apache.org today are > three self-hosted open models: gemma4-26b, qwen3.8-27b, and qwen3-8b, I > think the Magpie baseline, was produced with a frontier model via the > Claude CLI. We need to assess or maybe divide the tasks based on which > model could be good for them. Also Jarek, do you know whether a PMC member > can self-serve one for the project workflow? > > Also, For more reference, https://github.com/apache/tooling-llmao > > Thanks, > Shubham > > On Fri, Oct 9, 2026 at 8:02 PM Jarek Potiuk <[email protected]> wrote: > >> Yeah. Subscription cannot be used as an API replacement "really" (at least >> long term and this is how subscriptions are being "targeted"). You really >> need to use them via the "Claude" or "Codex" CLI. While there are some >> workarounds (like `claude -p`), this method might stop working (it was >> scheduled for disabling - but Anthropic backed out temporarily). >> >> But yeah absolutely - we can use the Azure Credits, for that as well. >> Ideally this will be **very little** credit use. We could even use Edge >> executors for that (which is even more Dogfooding) - even if Airflow will >> be running in Amazon - the runners could be set-up in Azure for example. >> Or >> we could connect to the models Azure exposes. >> >> From my Magpie experiments about 80%-90% of such checks can be purely >> deterministic, and the vast majority of the AI usage falls into two cases: >> >> * generating responses (most of them can be templated - only small amount >> of tokens is spent to make messages "tailored" to the PR/ Security issue) >> * decision making on how to classify things (which can be way cheaper now >> with Jev - and similar solutions - Open AI just 3 days ago released >> "Decisions API" - and pretty much everyone will have similar services soon >> [1] >> >> And i think we should treat it as a starting point - there are and will be >> many sources of tokens we can use: >> >> * We have stakeholders - Google and Amazon—who can likely donate some of >> their credits >> * ASF is finalizing llm service [2] for projects to use (open models on >> rented GPUs, plus donated credits to ASF by big models). This is >> specifically foreseen as a shared resource that ASF projects can use, and >> using it for "around CI" work is one of the stronger use-cases. >> >> I think it's important that we monitor and use the capability of our >> subscriptions to iterate on ideas with AI - and turn them in **mostly** >> deterministic workflows with as-small-as-possible usage of tokens and >> monitor it. >> >> This is one of the things we have almost done in Magpie -> - we have a >> SKILL that optimizes other skills by simplifying language, refactoring and >> extracting pieces of it - and turning everything that can be turned into >> deterministict Python code. And I am plannning to use it on Airflow >> AGENTS.md and SKILLs (see the other discussion). >> >> [1] >> >> https://community.openai.com/t/decisions-api-is-now-available-in-public-beta/1403877 >> [2] https://llm.apache.org - LLM service from ASF that is about to be >> announced >> [3] Skill optimization effort in Magpie >> https://github.com/apache/magpie/issues/1342 >> >> J. >> >> >> >> On Fri, Oct 9, 2026 at 12:07 PM Shubham Raj <[email protected]> >> wrote: >> >> > Thanks Ash for the quick response and feedback, >> > >> > Initially, yes (while in shadow mode), I plan to use copy/paste or GH >> APIs. >> > But the final goal would be like using a service account creds, which >> will >> > automatically post the reviews on the PR when approved, Today when we >> use >> > the harness to post the comments, it ends with "Drafted-by: <harness> >> > (<model>); reviewed by <username> before posting", we will follow the >> same >> > thing, it's just the username would be the maintainer's name who >> approved >> > the HITL. >> > >> > Initially, I'm thinking of implementing this for committers+, and we can >> > decide on scaling later based on guardrails or rules implementation. >> > >> > Yes, those are good programs, but they are specific to users. I think >> that >> > would be really needed for contributors regarding their own PR creation >> and >> > related work. I don't think the program provides API keys for >> developers, >> > which we need for the setup. Good to know we have Azure credits; this >> adds >> > to the viable options. >> > >> > Thanks, >> > Shubham >> > >> > On Fri, Oct 9, 2026 at 4:18 PM Ash Berlin-Taylor <[email protected]> >> wrote: >> > >> > > Love the idea of using Airflow to manage the Airflow project. >> > > >> > > Some clarifying questions: >> > > >> > > > human-approved comments, labels, draft conversion, closure where >> > > permitted, and review dispositions; >> > > >> > > For the initial trail, and in the long term, who or what is making >> > posting >> > > comments? Initial version is “the human has to copy them etc” (and we >> > could >> > > make that easier by providing `gh` commands etc) or an “AirflowBot” GH >> > > user/app etc? (I think I’m fine either way, I’m just unsure) >> > > >> > > Who would have access to see the results of these runs? Who would be >> able >> > > to be the Human in the HITL to approve things? Were you thinking >> > everyone >> > > in Triage, Committers and PMC groups? I think from reading the AIP you >> > have >> > > it as Committers+? >> > > >> > > Another possible triage rule: history of opening PRs either too >> quickly, >> > > or to many and then never responding to comments? Might not be a real >> > case >> > > so might not be worth it. >> > > >> > > >> > > On question 4: Both Anthropic and OpenAI have “OSS” programs where >> they >> > > give (rolling, after re-applying) 6 month free things to contributors >> of >> > > projects like Airflow. Downside to that is it would be tied to an >> > > individual user I suspect. We also have a large number of Azure >> credits >> > > that have a (now 1year left) expiration on, so that might be a useful >> > thing >> > > to do with them. I know Shahar is also looking at using credits for CI >> > > runners. >> > > >> > > >> > > > On 9 Oct 2026, at 11:28, Shubham Raj <[email protected]> wrote: >> > > > >> > > > Hello everyone, >> > > > >> > > > As a follow-up to this thread, I've written up the proposal, Here >> is an >> > > AIP >> > > > < >> > > >> > >> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451979463/WIP+AIP-124+Airflow-Powered+Agentic+Workflows+for+Project+Operations >> > > > >> > > > , >> > > > >> > > > TL;DR: Move our PR triage and review assistance from a "push" model >> (a >> > > > maintainer running local agent sessions) to a "pull" model. Airflow >> > > > continuously discovers work, handles the deterministic parts, and >> > > surfaces >> > > > only the decisions that need a human in a shared HITL inbox. This >> > removes >> > > > the single point of failure and lets more maintainers share the >> load. >> > > > >> > > > *Key points:* >> > > > >> > > > - No new Airflow core subsystem. It reuses existing HITL, >> common.ai, >> > > > Assets, deferral, and the official Helm chart (requires Airflow >> > 3.3+). >> > > > - Runs in a project-controlled AWS account. Workflows, prompts, >> > > > policies, and tests live in a new airflow-agentic-workflows repo, >> so >> > > anyone >> > > > can propose changes via PRs. >> > > > - Bounded event loop, not an open-ended agent. Airflow owns the >> state >> > > > machine; the model returns typed results, and every write >> requiring >> > > > judgment is gated by a human. >> > > > - Explicit non-goals: no automatic merging, no unattended >> approvals >> > or >> > > > request-changes, no autonomous workflow approval/rerun. >> > > > - Rollout starts in shadow mode, comparing results against current >> > > > maintainer practice before enabling any GitHub writes. >> > > > >> > > > The initial scope is PR triage and review. The security issue >> workflow >> > > > would follow as a separate track. >> > > > >> > > > *Open questions where I'd especially value input:* >> > > > >> > > > 1. The initial triage rules (R1-R6). More inputs would help here; >> are >> > > > any rules missing or too aggressive? >> > > > 2. Hosting and operational ownership of the AWS deployment. >> > > > 3. The attribution model for reviews posted via a central GitHub >> > > account >> > > > on behalf of the approving maintainer. >> > > > 4. Token/model source: Bedrock, llm.apache.org, or something >> else? >> > > > 5. What quality gate should we require before leaving shadow mode? >> > > > >> > > > Please leave comments here or directly on the wiki page. Once the >> > > > discussion settles, I'll update the AIP and we can move toward a >> vote. >> > > > >> > > > Thanks, >> > > > Shubham >> > > > >> > > > On Fri, Sep 11, 2026 at 3:02 PM Jarek Potiuk <[email protected]> >> wrote: >> > > > >> > > >> Hi everyone, >> > > >> >> > > >> A quick follow-up: something I read today offers a slightly >> different >> > > >> perspective on the proposal to use Airflow, which might help >> clarify >> > my >> > > aim: >> > > >> >> > > >> Read that*: AWS open-sources Pizza Bot: *email-style inbox for >> > > background >> > > >> AI agents - The New Stack: >> > > >> https://thenewstack.io/aws-pizza-bot-agent-inbox/ >> > > >> >> > > >> I want our triage and security process to follow a similar model. >> > Agents >> > > >> would work mostly in the background, completing tasks autonomously >> and >> > > >> surfacing to maintainers (in an inbox/to-do list fashion) only when >> > > human >> > > >> decisions are required. We would have a shared `inbox` style places >> > > where >> > > >> maintainers can log in and perform Human-in-the-Loop (HITL) >> actions. >> > > >> >> > > >> Using Airflow to drive this allows us to tailor the experience >> closely >> > > to >> > > >> our needs. It also showcases Airflow for this type of workload and >> > > >> Dogfooding it offers numerous advantages over using a third-party >> > > solution >> > > >> like Pizza Bot. Specifically, we could implement: >> > > >> >> > > >> - A shared inbox for the security team to review issues, make >> triage >> > > >> decisions, trigger fix PRs, and handle release announcements. >> > > >> - Per-area inboxes for maintainers to make triage decisions on new >> > and >> > > >> outstanding PRs >> > > >> - Learning what Airflow still needs to support these kinds of >> > workflows >> > > >> >> > > >> This proposal heavily optimizes for minimizing human interaction by >> > > >> automatically handling straightforward tasks and batching >> decisions to >> > > >> surface only high-priority items with full context. It also >> eliminates >> > > the >> > > >> Single Point of Failure (SPOF)—currently myself—by enabling the >> > broader >> > > >> community to share the load (but only the "make decisions" part; >> > Airflow >> > > >> will handle the execution). >> > > >> >> > > >> Quick FAQ: >> > > >> >> > > >> *How much Magpie is involved?* >> > > >> >> > > >> On the runtime/execution side for Airflow: none. Magpie serves >> purely >> > > as a >> > > >> starting point and interactive testing ground. It already contains >> > > useful >> > > >> building blocks, mini-workflows, and integrations that we can learn >> > > from or >> > > >> adapt. We can learn from or adapt these integrations and try out >> new >> > > things >> > > >> However, embedding or weaving Magpie into Airflow DAGs is a >> non-goal. >> > We >> > > >> might re-use some code/SKILLs initially, but making this a >> long-term >> > > >> dependency is not the goal. Possibly Magpie could eventually >> implement >> > > an >> > > >> "export" feature to Airflow for any project wanting to automate its >> > > >> triaging/security issue handling with Airflow - following our >> > blueprint. >> > > >> However, this is not a goal - it's more of a side effect of using >> > > similar >> > > >> features while interactively testing things with Magpie and >> wanting to >> > > make >> > > >> them into a DAG. >> > > >> >> > > >> *Should we analyze stats on what works first?* >> > > >> >> > > >> Yes. Absolutely. We have, and we will continue analyzing stats >> while >> > > >> incrementally moving some workflows to run via Airflow. I have >> already >> > > >> gathered and shared these stats from my manual runs with Magpie >> and we >> > > can >> > > >> continue doing so. Look at my past messages where I shared stats, >> drew >> > > >> conclusions, and decided on next steps. Magpie provides the >> tooling to >> > > run >> > > >> experiments and evaluate results quickly. Transitioning this to a >> > > >> continuous, automated Airflow setup removes the single point of >> > failure >> > > >> (SPOF) and allows us to iterate and improve these workflows more >> > > >> collaboratively. Since the decisions we make will have a >> continuous, >> > > >> immediate effect once implemented and deployed in our "Airflow >> > driver," >> > > we >> > > >> will hopefully make them more consciously. >> > > >> >> > > >> Best regards, >> > > >> Jarek >> > > >> >> > > >> >> > > >> On Thu, Sep 10, 2026 at 7:02 PM Jarek Potiuk <[email protected]> >> > wrote: >> > > >> >> > > >>> Hello everyone, >> > > >>> >> > > >>> Following up on today's Dev call, Shubham will share a proposal >> > > document >> > > >>> soon detailing our discussion from yesterday. However, I wanted to >> > > outline >> > > >>> the idea here first to clear up a few misconceptions that arose >> > during >> > > >>> brief conversations on Slack and with PMC members. If agreed upon, >> > this >> > > >>> might eventually become an AIP. >> > > >>> >> > > >>> >> > > >>> *TL;DR: I propose experimenting with dogfooding Airflow >> > deterministic + >> > > >>> agentic workflows to handle our triaging and security issue >> handling* >> > > >>> >> > > >>> *Current State* >> > > >>> We currently use agentic automation via "Magpie" SKILLs ( >> > > >>> https://magpie.apache.org/) for both areas, with varying success: >> > > >>> >> > > >>> - Security issue handling: Working well, processing over 60 >> issues >> > > >>> monthly (a 10x increase over six months ago). >> > > >>> - PR triaging: Partially effective (e.g., filtering bad PRs, with >> > > >>> potential to automate closing large PRs from inexperienced >> > > contributors). >> > > >>> >> > > >>> *The Problem* >> > > >>> >> > > >>> Currently, these workflows require a "push" model where I (in >> Claude >> > > CLI) >> > > >>> initiate actions, confirm proposals, and approve decisions. >> > Regularly. >> > > I >> > > >>> initiate them regularly. Others might also initiate actions, but >> very >> > > few >> > > >>> people have done so. This creates a Single Point of Failure >> (SPOF). >> > > IMHO - >> > > >>> we need to shift to an "automated pull" model to eliminate this >> SPOF >> > > and >> > > >>> allow more community members to act as Human-In-The-Loop (HITL) >> > > reviewers >> > > >>> without needing manual initiation. This would also allow more >> people >> > to >> > > >>> contribute to and review those workflows. >> > > >>> >> > > >>> *The Proposal* >> > > >>> >> > > >>> Instead of relying on local CLI pushes, we can translate these >> > > learnings >> > > >>> into Airflow DAGs. Roughly 60%–80% of triage actions are >> > deterministic >> > > and >> > > >>> require no human intervention. For the rest, the automated >> workflow >> > > would >> > > >>> run in the background and trigger "pull" notifications (e.g., >> Slack >> > > alerts) >> > > >>> when bulk human decisions are needed. >> > > >>> >> > > >>> *Key Benefits & Features:* >> > > >>> >> > > >>> - Shared UI: A cloud-hosted Airflow UI allows multiple >> maintainers >> > to >> > > >>> review and approve decisions. >> > > >>> - Community-Driven Workflows: Workflows can live in a new Airflow >> > repo >> > > >>> (e.g., airflow-agentic-workflows), allowing anyone to modify DAGs >> via >> > > pull >> > > >>> requests. >> > > >>> - No External Dependencies: While Magpie SKILLs can still be used >> > for >> > > >>> local testing or syncs if preferred, using Magpie will not be >> > required >> > > for >> > > >>> PMC/Dev team members. >> > > >>> - They workflows **might** by somewhat synced with Magpie in >> either >> > > way >> > > >>> (and likely I will start by transplanting parts of the process >> from >> > > Magpie >> > > >>> to Airflow Dags (and implement some features for that in Magpie) . >> > But >> > > >>> there will be exactly "0" Magpie components in Airflow Agentic >> Dags. >> > > >>> - Dogfooding: Managing Airflow with our latest Helm chart >> provides >> > > >>> direct operational feedback for the community—feedback that we >> > > currently >> > > >>> miss. >> > > >>> - We could make this a "use case" demonstrating how Airflow >> Agentic >> > > >>> workflows helped solve our problem, which could confirm that >> Airflow >> > is >> > > >>> ideally suited for these kinds of workflows. >> > > >>> >> > > >>> *Note on Workflow Architecture (Addressing Vikram's feedback):* >> > > >>> >> > > >>> Regarding REACT vs. REPL: Triage and security handling fit the >> REACT >> > > >>> pattern (Reason -> Act -> Iterate) due to the continuous stream of >> > > incoming >> > > >>> PRs and issues. It's literally -> get issues, evaluate them, make >> > > actions >> > > >>> (with or without human) and go back to the beginning. >> > > >>> REPL relies more on immediate code-execution loops for task >> > > >>> implementation, whereas our needs align with REACT's iterative >> > > reasoning >> > > >>> over time. >> > > >>> >> > > >>> Let's discuss further in this thread. Looking forward to your >> > comments >> > > >>> and feedback. >> > > >>> >> > > >>> Best regards, >> > > >>> Jarek >> > > >>> >> > > >>> >> > > >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: [email protected] >> > > For additional commands, e-mail: [email protected] >> > > >> > > >> > >> >
