@Shahar -- Absolutely, I think you are driving it with this email. So I think you can lead it from here and whoever wants to join can co-lead or join in development.
Please feel free to drive :) On Thu, 29 May 2025 at 17:07, Aaron Dantley <aarondant...@gmail.com> wrote: > Hey All! > > I’d be grateful to be included in the AIP discussions to help if possible > too! Like Shahar, I’ve never worked on any of these items so it’d be great > to see how work gets assigned and goes through a whole development cycle! > > Looking forward to it! > Aaron > > On Thu, May 29, 2025 at 7:32 AM Shahar Epstein <sha...@apache.org> wrote: > > > If it's ok, I would like to lead the AIP effort (or at least co-lead), as > > I've never written an AIP before. I could start drafting it during the > next > > week. > > Avi - please let me know if it works for you. > > > > > > Shahar > > > > > > On Thu, May 29, 2025, 13:09 Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > Yes separate repo, please and we would need someone to lead this effort > > on > > > the proposal & development too. Avi - you are probably well equipped to > > > lead it and I am sure more folks like Aaraon would be eager to work on > > its > > > development and on-going maintenance. > > > > > > Regards, > > > Kaxil > > > > > > On Thu, 29 May 2025 at 15:25, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > Yep. Having MCP is cool and drawing our implementation from > experiences > > > and > > > > usage of other MCP servers out there is even cooler (especially that > we > > > can > > > > have some insights how people already use them with Airflow) - if we > > can > > > > bring together a few of those, put some nice, relevant Airflow > prompts. > > > > Ideally we could have some examples of how MCP can be used taken from > > > those > > > > who are using airflow (the debugging example by Avi is cool) > > > > > > > > I am not sure implementing it as provider is really "the way" though > - > > I > > > > would rather see `apache-airflow-mcp" separate repo - it's so > different > > > and > > > > distinct from airflow it does not really require any of Airflow > > internals > > > > and code to be implemented - it makes very little sense to be the > part > > of > > > > airflow "workspace" where we would develop it together with airflow - > > > > because if it will talk over the REST api, all we need is the > `client` > > > that > > > > might be just a dependency. And there is even no reason for MCP and > > > airflow > > > > to be installed and developed together (that's the main reason why we > > > want > > > > providers to be kept in monorepo. > > > > > > > > J. > > > > > > > > > > > > On Thu, May 29, 2025 at 8:37 AM Amogh Desai < > amoghdesai....@gmail.com> > > > > wrote: > > > > > > > > > Seems like a promising area to invest in given the benefits it can > > > > provide > > > > > to > > > > > the users as mentioned by Shahar and Abhishek. > > > > > > > > > > Abhishek also has a promising talk submitted which i am looking > > forward > > > > to > > > > > this year at the summit. > > > > > > > > > > In any case, this seems to be one of the first of the very few > > > > > implementations of trying > > > > > to integrate Airflow officially / unofficially with an MCP server. > > > > > > > > > > Thanks & Regards, > > > > > Amogh Desai > > > > > > > > > > > > > > > On Thu, May 29, 2025 at 2:56 AM Aaron Dantley < > > aarondant...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hey! > > > > > > > > > > > > I also think this is a great idea! > > > > > > > > > > > > Would it be possible to be included in the development process? > > > > > > > > > > > > Sorry I’m new to this group, but would appreciate any suggestions > > on > > > > how > > > > > to > > > > > > contribute to the MCP server development! > > > > > > > > > > > > Regards! > > > > > > Aaron > > > > > > > > > > > > On Wed, May 28, 2025 at 2:57 PM Avi <a...@astronomer.io.invalid> > > > wrote: > > > > > > > > > > > > > Nice to see the idea to incorporate an official MCP server for > > > > > > > Airflow. It's been really magical to see what a simple LLM can > do > > > > with > > > > > an > > > > > > > Airflow MCP server built just from APIs. > > > > > > > > > > > > > > A few things that I noticed in my experience: > > > > > > > - The number of tools that the OpenAPI spec generates is quite > > > huge. > > > > > Most > > > > > > > tools (*Claude, VS Code with GitHub Copilot, Cursor, Windsurf*) > > > which > > > > > > uses > > > > > > > mcp-client limits it to a number of 100 tools. (*The read-only > > mode > > > > > > creates > > > > > > > less tools in comparison*.) > > > > > > > - MCP server are just not tools. There are other things as > well, > > > like > > > > > > > resources and prompts. Prompts are super helpful in case of > > > debugging > > > > > for > > > > > > > example. It is a way of teaching LLM about Airflow. Say I want > to > > > > have > > > > > a > > > > > > > failing task investigated. A prompt can be helpful in letting > LLM > > > > know > > > > > a > > > > > > > step-by-step process of carrying out the investigation. > > > > > > > - Where do you run the MCP server? I wouldn't want my laptop to > > do > > > > the > > > > > > > heavy processing, which would want us to go for the SSE instead > > of > > > > > stdio. > > > > > > > > > > > > > > This is why I chose two different path of using mcp server with > > > > > airflow, > > > > > > > which I intend to talk about at the summit. > > > > > > > > > > > > > > 1. AI-Augmented Airflow - This helped me add a chat interface > > > inside > > > > > > > Airflow using a plugin to talk to an Airflow instance (read > only > > > > mode). > > > > > > > > > > > > > > 2. Airflow-Powered AI - Experimenting with this has been > totally > > > > > magical, > > > > > > > how powerful AI can become when it has access to airflow. > Also, a > > > > > > directory > > > > > > > structure to maintain the DAGs, and it can write DAGs on the > > fly. I > > > > > > totally > > > > > > > see a need where LLMs eventually will need a scheduler, > although > > a > > > > > > complete > > > > > > > airflow just for an LLM might seem a bit overkill to the rest > of > > > the > > > > > > > community. > > > > > > > > > > > > > > I chose to build this on top of open API is because that was > the > > > only > > > > > way > > > > > > > to get proper RBAC enabled. > > > > > > > > > > > > > > I have so many points to discuss. Would love to hear from the > > > > community > > > > > > and > > > > > > > then take it forward. > > > > > > > > > > > > > > Thanks, > > > > > > > Avi > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 28, 2025 at 6:32 PM Aritra Basu < > > > > aritrabasu1...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > I definitely think there's potential to interact with an > > airflow > > > > MCP > > > > > > > > server. Though I think I'd be interested to see how many and > > how > > > > > > > frequently > > > > > > > > people are making use of MCP servers in the wild before > > investing > > > > > > effort > > > > > > > in > > > > > > > > building and maintaining one for airflow. I'm sure the data > is > > > > > > available > > > > > > > > out there, just needs finding. > > > > > > > > -- > > > > > > > > Regards, > > > > > > > > Aritra Basu > > > > > > > > > > > > > > > > On Wed, 28 May 2025, 11:18 pm Julian LaNeve, > > > > > > > <jul...@astronomer.io.invalid > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I think this would be interesting now that the Streamable > > HTTP > > > > > spec < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://modelcontextprotocol.io/specification/2025-03-26/basic/transports> > > > > > > > > > is out. I think in theory we could publish this first as an > > > > Airflow > > > > > > > > > provider that installs a plugin to expose an MCP endpoint, > > as a > > > > > PoC - > > > > > > > > this > > > > > > > > > becomes a much nicer experience than a local stdio one. > > > > > > > > > -- > > > > > > > > > Julian LaNeve > > > > > > > > > CTO > > > > > > > > > > > > > > > > > > Email: jul...@astronomer.io > > > > > > > > > <mailto:jul...@astronomer.io>Mobile: 330 509 5792 > > > > > > > > > > > > > > > > > > > On May 28, 2025, at 1:25 PM, Shahar Epstein < > > > sha...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Dear community, > > > > > > > > > > > > > > > > > > > > Following the thread on Slack [1], initiated by Jason > > > Sebastian > > > > > > > Kusuma, > > > > > > > > > I'd > > > > > > > > > > like to start an effort to officially support MCP in > > > Airflow's > > > > > > > > codebase. > > > > > > > > > > > > > > > > > > > > *Some background * > > > > > > > > > > Model Context Protocol (MCP) is an open standard, > > open-source > > > > > > > framework > > > > > > > > > > that standardizes the way AI models like LLM integrate > and > > > > share > > > > > > data > > > > > > > > > with > > > > > > > > > > external tools, systems and data sources. Think of it as > a > > > > "USB-C > > > > > > for > > > > > > > > > AI" - > > > > > > > > > > a universal connector that simplifies and standardizes AI > > > > > > > > integrations. A > > > > > > > > > > notable example of an MCP server is GitHub's official > > > > > > implementation > > > > > > > > > [3], which > > > > > > > > > > allows LLMs such as Claude, Copilot, and OpenAI (or: "MCP > > > > > clients") > > > > > > > to > > > > > > > > > > fetch pull request details, analyze code changes, and > > > generate > > > > > > review > > > > > > > > > > summaries. > > > > > > > > > > > > > > > > > > > > *How could an MCP server be useful in Airflow?* > > > > > > > > > > Imagine the possibilities when LLMs can seamlessly > interact > > > > with > > > > > > > > > Airflow’s > > > > > > > > > > API: triggering DAGs using natural language, retrieving > DAG > > > run > > > > > > > > history, > > > > > > > > > > enabling smart debugging, and more. This kind of > > integration > > > > > opens > > > > > > > the > > > > > > > > > door > > > > > > > > > > to a more intuitive, conversational interface for > workflow > > > > > > > > orchestration. > > > > > > > > > > > > > > > > > > > > *Why do we need to support it officially?* > > > > > > > > > > Quid pro quo - LLMs become an integral part of the modern > > > > > > development > > > > > > > > > > experience, while Airflow evolves into the go-to for > > > > > orchestrating > > > > > > AI > > > > > > > > > > workflows. By officially supporting it, we’ll enable > > multiple > > > > > users > > > > > > > to > > > > > > > > > > interact with Airflow through their LLMs, streamlining > > > > automation > > > > > > and > > > > > > > > > > improving accessibility across diverse workflows. All of > > that > > > > is > > > > > > > viable > > > > > > > > > > with relatively small development effort (see next > > > paragraph). > > > > > > > > > > > > > > > > > > > > *How should it be implemented?* > > > > > > > > > > As of today, there have been several implementations of > MCP > > > > > servers > > > > > > > for > > > > > > > > > > Airflow API, the most visible one [4] made by Abhishek > > Bhakat > > > > > from > > > > > > > > > > Astronomer. > > > > > > > > > > The efforts of implementing it and maintaining it in our > > > > codebase > > > > > > > > > shouldn't > > > > > > > > > > be too cumbersome (at least in theory), as we could > utilize > > > > > > packages > > > > > > > > like > > > > > > > > > > fastmcp to auto-generate the server using the existing > > > OpenAPI > > > > > > specs. > > > > > > > > I'd > > > > > > > > > > be very happy if Abhishek could share his experience in > > this > > > > > > thread. > > > > > > > > > > > > > > > > > > > > *Where else could we utilize MCP?* > > > > > > > > > > Beyond the scope of the public API, I could also imagine > > > using > > > > it > > > > > > to > > > > > > > > > > communicate with Breeze. > > > > > > > > > > > > > > > > > > > > *How do we proceed from here?* > > > > > > > > > > Feel free to share your thoughts here in this discussion. > > > > > > > > > > If there are no objections, I'll be happy to start > working > > on > > > > an > > > > > > AIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sincerely, > > > > > > > > > > Shahar Epstein > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *References:* > > > > > > > > > > [1] Slack discussion, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746121916951569 > > > > > > > > > > [2] Introducing the model context protocol, > > > > > > > > > > https://www.anthropic.com/news/model-context-protocol > > > > > > > > > > [3] GitHub Official MCP server, > > > > > > > > > https://github.com/github/github-mcp-server > > > > > > > > > > [4] Unofficial MCP Server made by Abhishek Hakat, > > > > > > > > > > https://github.com/abhishekbhakat/airflow-mcp-server > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
