shahar1 commented on code in PR #69381: URL: https://github.com/apache/airflow/pull/69381#discussion_r3525274466
########## dev/breeze/doc/adr/0018-internal-mcp-server-for-breeze-development.md: ########## @@ -0,0 +1,222 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [18. Run an internal MCP server for the Airflow REST API as a Breeze service](#18-run-an-internal-mcp-server-for-the-airflow-rest-api-as-a-breeze-service) + - [Status](#status) + - [Context](#context) + - [Decision](#decision) + - [Consequences](#consequences) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +# 18. Run an internal MCP server for the Airflow REST API as a Breeze service + +Date: 2026-07-04 + +## Status + +Accepted + +## Context + +Contributors increasingly drive their Breeze development environment with coding agents +(Claude Code, Cursor, and any other client of the vendor-neutral +[Model Context Protocol](https://modelcontextprotocol.io/)). The recurring loop in local +development is not writing code — it is *debugging a running Airflow*: is my Dag parsed, did +the run succeed, why did a task fail, what do the logs say, re-run it and check again. Today an +agent can only answer those questions by shelling out: hand-rolling a JWT against +`POST /auth/token`, then stringing together `curl` calls to the REST API, parsing JSON out of +each one, and keeping track of run ids and map indexes by hand. That is brittle, easy to get +wrong, and burns the agent's steps on plumbing instead of on the actual problem. + +To be precise about the target: what we want to expose is **Airflow's own REST API** — the data +and state behind the running Airflow instance (Dags, Dag runs, task instances, logs, import +errors) — **not Breeze**. Breeze has no data of its own to serve; it is only *how* the Airflow +under test (and this proxy) is launched. This is therefore not a wrapper around Breeze or its +CLI, and it does not expose Breeze commands. Encoding *how to drive Breeze itself* — build, +run the right static checks, reproduce CI, host-vs-container awareness — is the job of agent +skills (e.g. the GSoC Breeze Agent Skills effort); this MCP is the complementary layer that +inspects the *running Airflow* those workflows produce. The two operate at different layers and +compose (a verification skill can drive Breeze and then use these tools to check the outcome). + +Separately, [AIP-91](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/364349979) +proposes a *user-facing* Airflow MCP server: a stateless proxy over the public REST API, +read-only in its first phase, with per-user RBAC enforced by passing the caller's token through +to Airflow's own auth, deployable via the official Helm chart and Breeze, opt-in and +non-breaking. That is a production feature with production concerns (RBAC, secret masking on the +log path, session refresh, rate limiting, audit logging) and its own review/approval track. + +There is a gap between "an agent debugging my local Breeze" and "a hardened multi-user +production server". The debugging need is real and immediate; the production server is neither. +We want to serve the debugging need now, cheaply, without waiting on — or prejudging — the +larger AIP-91 design, while deliberately following the same architecture so the two do not +diverge. + +## Decision + +We ship a small, **internal, development-only** MCP server that lives in the repository at +`dev/mcp_server` and is wired into Breeze as a first-class service. + +- **It is a stateless proxy for the Airflow REST API** (not for Breeze). Every tool call is + translated into a request against the running *Airflow* instance's public REST API + (`/api/v2/...`) — the instance Breeze happens to launch. It never touches the metadata database + directly and adds no capability to Airflow itself — it only repackages the REST API behind a + curated, agent-oriented tool surface (`list_dags`, `list_task_instances`, `get_task_log`, a + composite `diagnose_dag_run`, `trigger_dag_run`, `clear_task_instances`, …), plus two escape + hatches (`airflow_api_get` / `airflow_api_call`) so no endpoint is out of reach. The surface is + overwhelmingly read-only runtime *data*; no tool shells out to Breeze or the CLI. +- **It runs as a Breeze service.** `breeze start-airflow --mcp-server` starts it inside the same + container as the rest of Airflow (host port `28081` → container `8081`, analogous to `28080` + for the API server). It is launched straight from the mounted sources via + `uvx --from /opt/airflow/dev/mcp_server` (consistent with [ADR 0017](0017-use-uvx-to-run-breeze-from-local-sources.md)), + so it always runs the current worktree's code and never installs anything into the checkout. + The server's lifetime is exactly the Breeze session's: **when Breeze is off, the endpoint is + gone.** There is nothing to stop, clean up, or leave running. +- **Adding it to an agent is minimal client configuration.** The contributor points their MCP + client at the endpoint — for HTTP, a few lines: + + ```json + { "mcpServers": { "airflow-dev": { "type": "http", "url": "http://localhost:28081/mcp"; } } } + ``` + + (a `uvx`-based stdio entry is also supported). No Airflow-side change, no account setup. +- **It is safe by construction for a single-user disposable environment.** Read-only by default; + writes are opt-in behind `AIRFLOW_MCP_ALLOW_WRITES` (Breeze sets it on by default because the + environment is disposable); `DELETE` is gated more strictly still, behind its own + `AIRFLOW_MCP_ALLOW_DELETES` flag that stays off even when writes are enabled (deletions are + irreversible) and whose tool is annotated destructive so MCP clients ask for explicit per-call + confirmation. Deliberately, there is **no dedicated `delete_*` tool**: deletion is reachable + only through the generic `airflow_api_call` escape hatch, so an agent must construct the method + and path explicitly rather than invoke a one-purpose delete tool — keeping deletion a conscious, + visible action and avoiding a standing, easily-triggered destructive verb in the tool list. Review Comment: 1. It's not an extra tool use - it's a hard rejection at the request layer, enforced as a barrier on every call to the Airflow REST API. 2. I agree that bypassing the MCP is possible by design, and there's nothing to do about it except for scoping the credentials via RBAC - which is out of scope for this PR. However, we should eventually implement the defense mechanism in two layers: a. Skill level - prevents bypass attempts via explicit-consent wording in the SKILL (I'd prefer to leave that implementation to a follow-up PR, since this one is already large). b. MCP level - prevents accidents from agents that already got past the skill-level guard. 3. During development of the MCP, I tried to aim towards the most common tools in use for internal development of Airflow (@jason810496 - airflowctl currently does not support some useful features such as task instances). I think that the current list is good enough to start with, we could always change it later as part of the evaluation or on the go. If and when we reach the scale that justifies usage of CodeMode - we could consider using it as well (considering it will also become stable at that time). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
