Thanks Shahar for driving this, and Vaquar for the concrete security review. Both useful.
TL;DR: my main concern on AIP-91 (MCP) and AIP-101 (AI Assistant) is maintenance. The recent maintenance issues with Airflow Helm Chart and Python Client for examples are the precedent: both good and useful packages for the ecosystem, but the ongoing cost was and still is real. We already have a persistent PR review backlog on the existing apache/airflow codebase; adding two new subprojects compounds that, and brings with it release-cycle complexity and a live security-boundary surface. That said: Astronomer is happy to donate the Airflow MCP server we maintain at astronomer/agents (the astro-airflow-mcp <https://github.com/astronomer/agents/tree/main/astro-airflow-mcp>subpackage) as a starting point for AIP-91, similar in spirit to the Helm Chart. It works with both Airflow 2 and Airflow 3 and is now used by 100s of companies: both Astronomer customers and OSS users. We'd have to think about the backwards-compat for the current users of that MCP -- I have some rough ideas already on that front already so that is not a blocker. Regards, Kaxil On Tue, 7 Apr 2026 at 05:33, vaquar khan <[email protected]> wrote: > Hi Shahar , > > Thank you for putting together this comprehensive proposal. The decision to > architect the MCP server as a stateless proxy that delegates all RBAC > evaluation to Airflow's native Auth Manager is a fantastic design choice. > It cleanly avoids the Confused Deputy problem and ensures per-user > permission enforcement without forcing us to maintain a parallel > authorization layer. > > I've been reviewing the AIP-91 design specifications against the current > codebase (specifically the v3-2-test branch), and I wanted to raise a few > concrete findings regarding the implementation. Because AI agents consume > data fundamentally differently than human operators do, there are a few > architectural gaps we might need to address: > > > *1. Task Logs Are Not Redacted on the Read Path*AIP-91 states: The MCP > Server MUST pipe all retrieved free-text artifacts (specifically Task Logs > and Rendered Templates) through Airflow's native SecretsMasker before > serializing the tool response. However, looking at the current API > execution path (get_log -> TaskLogReader.read_log_chunks -> > FileTaskHandler.read), there is no read-time invocation of > SecretsMasker.redact(). Because SecretsMasker is a logging.Filter, it only > masks secrets at write time. If a secret was leaked via a standard out > print() before being registered, or if the masker simply wasn't active, the > REST API will serve the leaked credential as-is. > > Question: Should we introduce a read-time redaction layer directly within > TaskLogReader, or should we explicitly make this the responsibility of the > MCP proxy? > > > *2. Lack of Heuristic Detection for Unregistered Secrets*The native > SecretsMasker is highly optimized for deterministic masking (key-name > matching against fields like password or api_key, and exact-string > matching). It does not use regex heuristics to detect high-entropy strings, > which makes sense given historical performance concerns with regex in the > main logging pipeline. However, for an AI assistant blindly consuming > arbitrary tracebacks, unregistered secrets (like an AWS AKIA... key or a > JWT eyJ... token) represent a severe exfiltration risk. > > Question: Should the MCP server implement a secondary, regex-based > heuristic scrubbing layer exclusively for LLM payloads before they leave > the network boundary? > > > *3. The get_log_tail Tool Relies on Non-Existent API Functionality*AIP-91 > specifies that truncated log payloads will instruct the LLM to use a > get_log_tail tool. Currently, the log API handles pagination via an opaque > continuation token (URLSafeSerializer). Because this is a forward-only > cryptographic cursor, it cannot be used for a bounded page backwards from > the end of the file navigation, which is exactly what an AI needs to > quickly read a stack trace. > > Question: Will get_log_tail require us to build a new byte-offset/tailing > endpoint into the Core API, or will the MCP server cache the full log > stream internally to virtualize this capability? > > > *4. JWT Token Refresh for Bearer-Authenticated Clients*The Core API's > JWTRefreshMiddleware is currently optimized for browser UI sessions > (cookies). When the MCP server authenticates via a Bearer token in the > Authorization header,the standard for machine-to-machine integration > expired tokens trigger an exception rather than a refresh. The Execution > API handles this beautifully with a JWTReissueMiddleware that > auto-refreshes tokens with <20% validity remaining. > > Question: For long-running AI debugging sessions, should we harmonize the > Core API to adopt the Execution API's reissue middleware, or is the MCP > server expected to handle token rotation out-of-band? > > > *5. Resource Saturation and Rate Limiting*AIP-91 rightly proposes > token-bucket rate limiting. It’s worth noting that the current Core API > does not enforce rate limits on resource-intensive endpoints (like NDJSON > log streaming). An LLM caught in a hallucination loop could easily > overwhelm the webserver's connection pool. > > Question: This seems to reinforce the idea that the MCP server must > implement its own distributed rate limiting (e.g., backed by Redis) rather > than relying on the Core API to throttle rogue traffic. Do we agree this is > a hard requirement for Phase 1? > > I am highly supportive of this AIP and the proxy-based approach. I would be > more than happy to help contribute to the implementation or assist with > testing these specific security boundaries. > > Best regards, > Viquar Khan > > On Fri, Apr 3, 2026 at 6:38 AM Shahar Epstein <[email protected]> wrote: > > > Hey everyone, > > > > A lot of water has passed under the AI bridge since we first discussed > > this, and I think it's a good time to get things moving. > > > > I have drafted AIP-91 for the official Airflow MCP Server: > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=364349979 > > > > tl;dr - In its 1st phase, the main use case of the MCP server will be to > > serve the planned Airflow AI Assistant (which will be discussed in a > > separate thread for AIP-101). To ensure strict security and per-user RBAC > > enforcement, both the MCP server and the assistant will be strictly in > > read-only mode for Phase 1. > > > > A quick note: While I'm proposing this as a separate AIP, I highly > > recommend reading it in the context of AIP-101 to see exactly how the two > > pieces fit together. > > > > I'd love to get your thoughts and feedback on the design. > > > > > > Shahar > > > > On 2025/05/28 17:25:06 Shahar Epstein wrote: > > > Dear community, > > > > > > Following the thread on Slack [1], initiated by Jason Sebastian Kusuma, > > I'd > > > like to start an effort to officially support MCP in Airflow's > codebase. > > > > > > *Some background * > > > Model Context Protocol (MCP) is an open standard, open-source framework > > > that standardizes the way AI models like LLM integrate and share data > > with > > > external tools, systems and data sources. Think of it as a "USB-C for > > AI" - > > > a universal connector that simplifies and standardizes AI > integrations. A > > > notable example of an MCP server is GitHub's official implementation > > [3], which > > > allows LLMs such as Claude, Copilot, and OpenAI (or: "MCP clients") to > > > fetch pull request details, analyze code changes, and generate review > > > summaries. > > > > > > *How could an MCP server be useful in Airflow?* > > > Imagine the possibilities when LLMs can seamlessly interact with > > Airflow’s > > > API: triggering DAGs using natural language, retrieving DAG run > history, > > > enabling smart debugging, and more. This kind of integration opens the > > door > > > to a more intuitive, conversational interface for workflow > orchestration. > > > > > > *Why do we need to support it officially?* > > > Quid pro quo - LLMs become an integral part of the modern development > > > experience, while Airflow evolves into the go-to for orchestrating AI > > > workflows. By officially supporting it, we’ll enable multiple users to > > > interact with Airflow through their LLMs, streamlining automation and > > > improving accessibility across diverse workflows. All of that is viable > > > with relatively small development effort (see next paragraph). > > > > > > *How should it be implemented?* > > > As of today, there have been several implementations of MCP servers for > > > Airflow API, the most visible one [4] made by Abhishek Bhakat from > > > Astronomer. > > > The efforts of implementing it and maintaining it in our codebase > > shouldn't > > > be too cumbersome (at least in theory), as we could utilize packages > like > > > fastmcp to auto-generate the server using the existing OpenAPI specs. > I'd > > > be very happy if Abhishek could share his experience in this thread. > > > > > > *Where else could we utilize MCP?* > > > Beyond the scope of the public API, I could also imagine using it to > > > communicate with Breeze. > > > > > > *How do we proceed from here?* > > > Feel free to share your thoughts here in this discussion. > > > If there are no objections, I'll be happy to start working on an AIP. > > > > > > > > > Sincerely, > > > Shahar Epstein > > > > > > > > > *References:* > > > [1] Slack discussion, > > > > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746121916951569 > > > [2] Introducing the model context protocol, > > > https://www.anthropic.com/news/model-context-protocol > > > [3] GitHub Official MCP server, > > https://github.com/github/github-mcp-server > > > [4] Unofficial MCP Server made by Abhishek Hakat, > > > https://github.com/abhishekbhakat/airflow-mcp-server > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
