Dev-iL commented on code in PR #69381: URL: https://github.com/apache/airflow/pull/69381#discussion_r3524302106
########## dev/breeze/doc/adr/0018-internal-mcp-server-for-breeze-development.md: ########## @@ -0,0 +1,222 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [18. Run an internal MCP server for the Airflow REST API as a Breeze service](#18-run-an-internal-mcp-server-for-the-airflow-rest-api-as-a-breeze-service) + - [Status](#status) + - [Context](#context) + - [Decision](#decision) + - [Consequences](#consequences) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +# 18. Run an internal MCP server for the Airflow REST API as a Breeze service + +Date: 2026-07-04 + +## Status + +Accepted + +## Context + +Contributors increasingly drive their Breeze development environment with coding agents +(Claude Code, Cursor, and any other client of the vendor-neutral +[Model Context Protocol](https://modelcontextprotocol.io/)). The recurring loop in local +development is not writing code — it is *debugging a running Airflow*: is my Dag parsed, did +the run succeed, why did a task fail, what do the logs say, re-run it and check again. Today an +agent can only answer those questions by shelling out: hand-rolling a JWT against +`POST /auth/token`, then stringing together `curl` calls to the REST API, parsing JSON out of +each one, and keeping track of run ids and map indexes by hand. That is brittle, easy to get +wrong, and burns the agent's steps on plumbing instead of on the actual problem. + +To be precise about the target: what we want to expose is **Airflow's own REST API** — the data +and state behind the running Airflow instance (Dags, Dag runs, task instances, logs, import +errors) — **not Breeze**. Breeze has no data of its own to serve; it is only *how* the Airflow +under test (and this proxy) is launched. This is therefore not a wrapper around Breeze or its +CLI, and it does not expose Breeze commands. Encoding *how to drive Breeze itself* — build, +run the right static checks, reproduce CI, host-vs-container awareness — is the job of agent +skills (e.g. the GSoC Breeze Agent Skills effort); this MCP is the complementary layer that +inspects the *running Airflow* those workflows produce. The two operate at different layers and +compose (a verification skill can drive Breeze and then use these tools to check the outcome). + +Separately, [AIP-91](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/364349979) +proposes a *user-facing* Airflow MCP server: a stateless proxy over the public REST API, +read-only in its first phase, with per-user RBAC enforced by passing the caller's token through +to Airflow's own auth, deployable via the official Helm chart and Breeze, opt-in and +non-breaking. That is a production feature with production concerns (RBAC, secret masking on the +log path, session refresh, rate limiting, audit logging) and its own review/approval track. + +There is a gap between "an agent debugging my local Breeze" and "a hardened multi-user +production server". The debugging need is real and immediate; the production server is neither. +We want to serve the debugging need now, cheaply, without waiting on — or prejudging — the +larger AIP-91 design, while deliberately following the same architecture so the two do not +diverge. + +## Decision + +We ship a small, **internal, development-only** MCP server that lives in the repository at +`dev/mcp_server` and is wired into Breeze as a first-class service. + +- **It is a stateless proxy for the Airflow REST API** (not for Breeze). Every tool call is + translated into a request against the running *Airflow* instance's public REST API + (`/api/v2/...`) — the instance Breeze happens to launch. It never touches the metadata database + directly and adds no capability to Airflow itself — it only repackages the REST API behind a + curated, agent-oriented tool surface (`list_dags`, `list_task_instances`, `get_task_log`, a + composite `diagnose_dag_run`, `trigger_dag_run`, `clear_task_instances`, …), plus two escape + hatches (`airflow_api_get` / `airflow_api_call`) so no endpoint is out of reach. The surface is + overwhelmingly read-only runtime *data*; no tool shells out to Breeze or the CLI. +- **It runs as a Breeze service.** `breeze start-airflow --mcp-server` starts it inside the same + container as the rest of Airflow (host port `28081` → container `8081`, analogous to `28080` + for the API server). It is launched straight from the mounted sources via + `uvx --from /opt/airflow/dev/mcp_server` (consistent with [ADR 0017](0017-use-uvx-to-run-breeze-from-local-sources.md)), + so it always runs the current worktree's code and never installs anything into the checkout. + The server's lifetime is exactly the Breeze session's: **when Breeze is off, the endpoint is + gone.** There is nothing to stop, clean up, or leave running. +- **Adding it to an agent is minimal client configuration.** The contributor points their MCP + client at the endpoint — for HTTP, a few lines: + + ```json + { "mcpServers": { "airflow-dev": { "type": "http", "url": "http://localhost:28081/mcp"; } } } + ``` + + (a `uvx`-based stdio entry is also supported). No Airflow-side change, no account setup. +- **It is safe by construction for a single-user disposable environment.** Read-only by default; + writes are opt-in behind `AIRFLOW_MCP_ALLOW_WRITES` (Breeze sets it on by default because the + environment is disposable); `DELETE` is gated more strictly still, behind its own + `AIRFLOW_MCP_ALLOW_DELETES` flag that stays off even when writes are enabled (deletions are + irreversible) and whose tool is annotated destructive so MCP clients ask for explicit per-call + confirmation. Deliberately, there is **no dedicated `delete_*` tool**: deletion is reachable + only through the generic `airflow_api_call` escape hatch, so an agent must construct the method + and path explicitly rather than invoke a one-purpose delete tool — keeping deletion a conscious, + visible action and avoiding a standing, easily-triggered destructive verb in the tool list. Review Comment: I have some reservations about this approach: 1. Either allow deletion normally, or forbid it completely. At the moment it is "hidden" behind an extra tool use, so the only real effect is making it harder (i.e. require more steps), **but not impossible**, for an agent to run a risky action. What you'll get is agents that are worse at tool calling will be unable to run it, and agents that are good at it won't care. 2. When an MCP available action is "risky", the right place to flag it is in the SKILL accompanying the MCP - it should use wording like "requires explicit user consent before calling" etc. In the absence of such instructions, there's little stopping the agent from sidestepping the MCP and calling the API directly Additionally: 1. I don't understand why have a "semi [Code Mode](https://gofastmcp.com/servers/transforms/code-mode)" implementation - the split seems arbitrary, and a main benefit of the code mode transform (having an unlimited number of mcp endpoints w/o confusing agents) gets diluted. 2. ~How about having two flavors of the MCP, one with destructive actions enabled and one without?~ **Edit:** I noticed that this is what you already did with the env vars later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
