shahar1 opened a new pull request, #69381: URL: https://github.com/apache/airflow/pull/69381
## Human Summary To prepare the ground for production-scale user-facing MCP as proposed by AIP-91 (still in draft), I came to the conclusion that it might be better to start in lower scope with an MCP server for internal development purposes of the open-source (specifically the API parts), bundled with `breeze` as an opt-in feature. There's a breeze ADR included as part of this PR (ADR-0018), I'll give tha main parts here: ### Why? It is intended mostly for using AI agents to tackle issues whose resolution requires **multiple** API calls Without MCP, you have to rely on API calls using the CLI (hoping that the agent doesn't make typos), and there's some authentication friction (generating JWT tokens). With MCP - the server is deployed as part of breeze and takes care of the authentication parts, and the methods are exposed to the client. The bigger payoff, though, is that exposing the API as typed, ready-to-call methods lets the agent cheaply inspect the live state of a running Airflow - so on issues that hinge on what's actually happening at runtime, it converges on a fix with far less trial-and-error instead of guessing. ### How do you know that it (may) work? I've empirically tested the resolving 2*** GitHub issues by subagents, where each issue is resolved independently by 2 subagents with the same model (A/B testing): - One without access to MCP. - One with MCP access. Both agents resolved the tested issues successfully, generating equivalent solutions. In the issue with less requirement for multiple API calls - the MCP arm was no faster (slightly slower, in fact) and consumed more tokens. However, in the other issue where there were multiple API calls - the MCP subagent took less than 50% of the running time and reached the fix with far fewer iterations (about half the turns and a third of the test runs). *** **Disclaimer**: I'm aware that the sample space of 2 issues doesn't say much about statistical significance, but I'll be happy to reevaluate it when it's in used by others! ### How do we use it? Deploying it as part of the breeze deployment is as easy as: - Runnning `breeze start-airflow --mcp-server` to activate it - Running an additional command for adding and exposing the MCP server to the AI client (Claude Code/Codex/Copilot/etc.) ### Important things - There are basic guardrail flags for Create/Update/Delete operations (write - enabled by default, delete - disabled by default). - Tests and docs are included. - I'd like to thank @aritra24 for the brainstorming, discussion, and implementing a prototype. ## AI Summary <details><summary>Click here</summary> Adds an **internal, development-only** [MCP](https://modelcontextprotocol.io/) server (`dev/mcp_server`) that lets any MCP-capable coding agent inspect and debug the Airflow instance running in Breeze through the public REST API — list Dags/runs/task instances, fetch tailed logs with tracebacks, surface import errors, diagnose a failed run in one call, and (opt-in) trigger/clear runs. It is **not** shipped to users and makes **no** airflow-core changes. This is the "internal Breeze tool" step of AIP-91: a stateless REST proxy (no direct metastore access, read-only by default) so later AIP-91/AIP-101 phases can grow out of it. Full rationale, alternatives, and the pros/cons are captured in a new ADR: `dev/breeze/doc/adr/0018-internal-mcp-server-for-breeze-development.md`. **Highlights** - **Deployed as a Breeze service:** `breeze start-airflow --mcp-server` (host port 28081), run from mounted sources via `uvx`; also runnable over stdio by any MCP client. Gone when Breeze is off. - **Safety model:** read-only by default; writes behind `AIRFLOW_MCP_ALLOW_WRITES`; `DELETE` behind a separate, stricter `AIRFLOW_MCP_ALLOW_DELETES` (off even when writes are on) and reachable only via the generic escape hatch — no dedicated delete tool. Secrets stripped from list responses. - **Curated tool surface** (~20 tools) plus two documented escape hatches, rather than one-tool-per-endpoint. - **Testing/CI:** unit tests run in CI as their own `Dev MCP server tests` job; a contract test asserts every endpoint the server calls exists (with its method) in Airflow's committed OpenAPI spec, with a second test guarding that endpoint list against drift; the live-Breeze and real-LLM-client integration tests are opt-in/manual for now. </details> --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.8) Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
