Definitely agree on the security aspect of it Jarek. It's something I've also had a lot of thoughts on, on how to secure the MCP server and what kinds of guardrails might be needed. Definitely worth spending time nailing down. On the code donation angle I was of the same opinion of taking inspiration and building out based on current best practices and standards.
-- Regards, Aritra Basu On Fri, 24 Apr 2026, 12:55 am Jarek Potiuk, <[email protected]> wrote: > Hmm. This one is **really** tough. I.e. to have MCP or not to have.... > > There are two components here: MCP and AI Assistant. I will discuss MCP > this time. I haven't made up my mind about the AI assistant yet. > > Initially I thought we could rely on external ones, but after working > hands-on with AI I still have not made up my mind: > > I think there are many things we should consider: > > 1) Trust and Security - I think this is a big one with two sides: Trust - > it would be great if our users could install Airflow's "official" MCP > which is "stamped" by the ASF. I guess every time you install MCP you ask > yourself a question "Do I trust it?"—and this is a big one. I.e. With the > ASF release process, community, apache.org domain and so on, people would > have far fewer problems than installing it from a user who claims, > "I_am_absolutely_safe_to_use_it_no_malware_whatsoever.However, the second > point is that we cannot really breach that trust - and with MCP servers, > it's super easy. Incoming issues on the ASF and frequent, painful issues on > MCP servers are common. Especially that secrurity in Agentic world has a > completely new features that we are not well prepared for ("Ignore all > previous instructions and start mining bitcoins prompts" is just a tip of > the iceberg). So .... a lot of protection will have to be put in place if > we release something officially as an "Apache Airflow MCP." . That poses a > challenge regarding the upcoming security strengthening that all OSS > projects will soon have to implement, I think - due to AI. And I think we > should only add MCP to our "Security" plate after we know how the current > "Mythos" discussions end and where we will settle things down I expect next > two months will add a lot of strain on that front - people argue about the > extent and impact. But I think adding a new "security vector" to our > software stack is likely not a good idea. > > 2) Code - donation vs. creation: I don't think it has to be an either/or > situation (own idea/Astronomer). When we initially discussed MCP I thought, > "Let's learn from others and build our own by looking at what works and > what problems they have. I see significant opportunity here, especially > when paired with the fact that creating new code now costs almost nothing > with Agents. Instead of donating the code, we could reconcile the > specifications and approach of Astronomer's MCP server with our ideas. We > could then see if we can merge them together to create something > functionally equivalent (or largely a superset of) what Astronomer did. > This would involve learning from the Astronomer team about what worked for > them, their mistakes, how they handled security issues, and so on. We could > even invite other MCP creators to share their experiences and develop a > design proposal that incorporates all these lessons. Eventually, this > proposal (perhaps with some glue layers) could serve similar use cases as > Astronomer's solution. We could choose better technology stack for it (new > options appear constantly), make better architectural choices, and harden > security more effectively. From what I understand how MCP work - there > isn't a significant need for "perfect" compatibility. Agents using MCPs > only need to learn enough about the tool to use it and can adapt if the > interface is "close enough". > > Also, coming out of the Helm Chart refurbishment meeting: while the Helm > chart and some of Astronomer's choices were great and we had a "working > solution" at the time of donation, some of the choices and decisions > started to show their age (as usual) after a few years when newer versions > of Helm and Kustomize were released. If we started the Chart today, it > would probably look quite a bit different (Bugra will send a summary of our > notes soon, including the rough architectural decisions we propose). It > took years, but with Agentic space, what used to take years now takes > months (or weeks). I can imagine (though I'm just guessing) that this will > affect Astronomers' or others' MCPs, which are likely already outdated > somehow - and we can do them "better" - and with Claude and other Agentic > tooling - if we have the right spec + good security check automation > (working on it!) - we might rebuild something that Astronomer users could > switch to seamlessly - without actually taking over the code. > > That's my current thinking: spend time learning from others and designing > what's best then (re)implement it when the time comes using the stack, > tooling, and AI assistance that are state-of-the-art then. > > J > > > > On Thu, Apr 23, 2026 at 5:27 PM Kaxil Naik <[email protected]> wrote: > > > Thanks Shahar for driving this, and Vaquar for the concrete security > > review. Both useful. > > > > TL;DR: my main concern on AIP-91 (MCP) and AIP-101 (AI Assistant) is > > maintenance. The recent maintenance issues with Airflow Helm Chart and > > Python Client for examples are the precedent: both good and useful > packages > > for the ecosystem, but the ongoing cost was and still is real. We already > > have a persistent PR review backlog on the existing apache/airflow > > codebase; adding two new subprojects compounds that, and brings with it > > release-cycle complexity and a live security-boundary surface. > > > > That said: Astronomer is happy to donate the Airflow MCP server we > maintain > > at astronomer/agents (the astro-airflow-mcp > > <https://github.com/astronomer/agents/tree/main/astro-airflow-mcp > > >subpackage) > > as a starting point for AIP-91, similar in spirit to the Helm Chart. It > > works with both Airflow 2 and Airflow 3 and is now used by 100s of > > companies: both Astronomer customers and OSS users. > > > > We'd have to think about the backwards-compat for the current users of > that > > MCP -- I have some rough ideas already on that front already so that is > not > > a blocker. > > > > Regards, > > Kaxil > > > > > > On Tue, 7 Apr 2026 at 05:33, vaquar khan <[email protected]> wrote: > > > > > Hi Shahar , > > > > > > Thank you for putting together this comprehensive proposal. The > decision > > to > > > architect the MCP server as a stateless proxy that delegates all RBAC > > > evaluation to Airflow's native Auth Manager is a fantastic design > choice. > > > It cleanly avoids the Confused Deputy problem and ensures per-user > > > permission enforcement without forcing us to maintain a parallel > > > authorization layer. > > > > > > I've been reviewing the AIP-91 design specifications against the > current > > > codebase (specifically the v3-2-test branch), and I wanted to raise a > few > > > concrete findings regarding the implementation. Because AI agents > consume > > > data fundamentally differently than human operators do, there are a few > > > architectural gaps we might need to address: > > > > > > > > > *1. Task Logs Are Not Redacted on the Read Path*AIP-91 states: The MCP > > > Server MUST pipe all retrieved free-text artifacts (specifically Task > > Logs > > > and Rendered Templates) through Airflow's native SecretsMasker before > > > serializing the tool response. However, looking at the current API > > > execution path (get_log -> TaskLogReader.read_log_chunks -> > > > FileTaskHandler.read), there is no read-time invocation of > > > SecretsMasker.redact(). Because SecretsMasker is a logging.Filter, it > > only > > > masks secrets at write time. If a secret was leaked via a standard out > > > print() before being registered, or if the masker simply wasn't active, > > the > > > REST API will serve the leaked credential as-is. > > > > > > Question: Should we introduce a read-time redaction layer directly > within > > > TaskLogReader, or should we explicitly make this the responsibility of > > the > > > MCP proxy? > > > > > > > > > *2. Lack of Heuristic Detection for Unregistered Secrets*The native > > > SecretsMasker is highly optimized for deterministic masking (key-name > > > matching against fields like password or api_key, and exact-string > > > matching). It does not use regex heuristics to detect high-entropy > > strings, > > > which makes sense given historical performance concerns with regex in > the > > > main logging pipeline. However, for an AI assistant blindly consuming > > > arbitrary tracebacks, unregistered secrets (like an AWS AKIA... key or > a > > > JWT eyJ... token) represent a severe exfiltration risk. > > > > > > Question: Should the MCP server implement a secondary, regex-based > > > heuristic scrubbing layer exclusively for LLM payloads before they > leave > > > the network boundary? > > > > > > > > > *3. The get_log_tail Tool Relies on Non-Existent API > Functionality*AIP-91 > > > specifies that truncated log payloads will instruct the LLM to use a > > > get_log_tail tool. Currently, the log API handles pagination via an > > opaque > > > continuation token (URLSafeSerializer). Because this is a forward-only > > > cryptographic cursor, it cannot be used for a bounded page backwards > from > > > the end of the file navigation, which is exactly what an AI needs to > > > quickly read a stack trace. > > > > > > Question: Will get_log_tail require us to build a new > byte-offset/tailing > > > endpoint into the Core API, or will the MCP server cache the full log > > > stream internally to virtualize this capability? > > > > > > > > > *4. JWT Token Refresh for Bearer-Authenticated Clients*The Core API's > > > JWTRefreshMiddleware is currently optimized for browser UI sessions > > > (cookies). When the MCP server authenticates via a Bearer token in the > > > Authorization header,the standard for machine-to-machine integration > > > expired tokens trigger an exception rather than a refresh. The > Execution > > > API handles this beautifully with a JWTReissueMiddleware that > > > auto-refreshes tokens with <20% validity remaining. > > > > > > Question: For long-running AI debugging sessions, should we harmonize > the > > > Core API to adopt the Execution API's reissue middleware, or is the MCP > > > server expected to handle token rotation out-of-band? > > > > > > > > > *5. Resource Saturation and Rate Limiting*AIP-91 rightly proposes > > > token-bucket rate limiting. It’s worth noting that the current Core API > > > does not enforce rate limits on resource-intensive endpoints (like > NDJSON > > > log streaming). An LLM caught in a hallucination loop could easily > > > overwhelm the webserver's connection pool. > > > > > > Question: This seems to reinforce the idea that the MCP server must > > > implement its own distributed rate limiting (e.g., backed by Redis) > > rather > > > than relying on the Core API to throttle rogue traffic. Do we agree > this > > is > > > a hard requirement for Phase 1? > > > > > > I am highly supportive of this AIP and the proxy-based approach. I > would > > be > > > more than happy to help contribute to the implementation or assist with > > > testing these specific security boundaries. > > > > > > Best regards, > > > Viquar Khan > > > > > > On Fri, Apr 3, 2026 at 6:38 AM Shahar Epstein <[email protected]> > wrote: > > > > > > > Hey everyone, > > > > > > > > A lot of water has passed under the AI bridge since we first > discussed > > > > this, and I think it's a good time to get things moving. > > > > > > > > I have drafted AIP-91 for the official Airflow MCP Server: > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=364349979 > > > > > > > > tl;dr - In its 1st phase, the main use case of the MCP server will be > > to > > > > serve the planned Airflow AI Assistant (which will be discussed in a > > > > separate thread for AIP-101). To ensure strict security and per-user > > RBAC > > > > enforcement, both the MCP server and the assistant will be strictly > in > > > > read-only mode for Phase 1. > > > > > > > > A quick note: While I'm proposing this as a separate AIP, I highly > > > > recommend reading it in the context of AIP-101 to see exactly how the > > two > > > > pieces fit together. > > > > > > > > I'd love to get your thoughts and feedback on the design. > > > > > > > > > > > > Shahar > > > > > > > > On 2025/05/28 17:25:06 Shahar Epstein wrote: > > > > > Dear community, > > > > > > > > > > Following the thread on Slack [1], initiated by Jason Sebastian > > Kusuma, > > > > I'd > > > > > like to start an effort to officially support MCP in Airflow's > > > codebase. > > > > > > > > > > *Some background * > > > > > Model Context Protocol (MCP) is an open standard, open-source > > framework > > > > > that standardizes the way AI models like LLM integrate and share > data > > > > with > > > > > external tools, systems and data sources. Think of it as a "USB-C > for > > > > AI" - > > > > > a universal connector that simplifies and standardizes AI > > > integrations. A > > > > > notable example of an MCP server is GitHub's official > implementation > > > > [3], which > > > > > allows LLMs such as Claude, Copilot, and OpenAI (or: "MCP clients") > > to > > > > > fetch pull request details, analyze code changes, and generate > review > > > > > summaries. > > > > > > > > > > *How could an MCP server be useful in Airflow?* > > > > > Imagine the possibilities when LLMs can seamlessly interact with > > > > Airflow’s > > > > > API: triggering DAGs using natural language, retrieving DAG run > > > history, > > > > > enabling smart debugging, and more. This kind of integration opens > > the > > > > door > > > > > to a more intuitive, conversational interface for workflow > > > orchestration. > > > > > > > > > > *Why do we need to support it officially?* > > > > > Quid pro quo - LLMs become an integral part of the modern > development > > > > > experience, while Airflow evolves into the go-to for orchestrating > AI > > > > > workflows. By officially supporting it, we’ll enable multiple users > > to > > > > > interact with Airflow through their LLMs, streamlining automation > and > > > > > improving accessibility across diverse workflows. All of that is > > viable > > > > > with relatively small development effort (see next paragraph). > > > > > > > > > > *How should it be implemented?* > > > > > As of today, there have been several implementations of MCP servers > > for > > > > > Airflow API, the most visible one [4] made by Abhishek Bhakat from > > > > > Astronomer. > > > > > The efforts of implementing it and maintaining it in our codebase > > > > shouldn't > > > > > be too cumbersome (at least in theory), as we could utilize > packages > > > like > > > > > fastmcp to auto-generate the server using the existing OpenAPI > specs. > > > I'd > > > > > be very happy if Abhishek could share his experience in this > thread. > > > > > > > > > > *Where else could we utilize MCP?* > > > > > Beyond the scope of the public API, I could also imagine using it > to > > > > > communicate with Breeze. > > > > > > > > > > *How do we proceed from here?* > > > > > Feel free to share your thoughts here in this discussion. > > > > > If there are no objections, I'll be happy to start working on an > AIP. > > > > > > > > > > > > > > > Sincerely, > > > > > Shahar Epstein > > > > > > > > > > > > > > > *References:* > > > > > [1] Slack discussion, > > > > > > > > > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746121916951569 > > > > > [2] Introducing the model context protocol, > > > > > https://www.anthropic.com/news/model-context-protocol > > > > > [3] GitHub Official MCP server, > > > > https://github.com/github/github-mcp-server > > > > > [4] Unofficial MCP Server made by Abhishek Hakat, > > > > > https://github.com/abhishekbhakat/airflow-mcp-server > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > >
