I agree that a use case driven approach is the way to go too. When we go full-blown, it is easy to lose track of the intention we started with.
Some of the use cases related to debuggability improvements is something we already had a north star for in:https://github.com/apache/airflow/issues/40975, with a good amount of analysis already done. Maybe you can get some data here that will be useful. Thanks & Regards, Amogh Desai On Fri, May 30, 2025 at 9:27 PM Avi <a...@astronomer.io.invalid> wrote: > Hi All, > > Had a chat with @Kaxil. And we discussed on use-case first approach rather > than opening up to all possibilities, on building and maintaining the MCP > server. > > A few use-cases which I had in mind were: > - Debugging task failures > - Schedule sparse across all dags for better resource utilization. > - Recommend provider usage in place of native python codes. > - Cross-DAG Dependency Analysis > - Migration/Refactor Planning > > We would need an ongoing effort on this, as every other day something new > comes up. Could be more optimization, or newer use-cases. We also need to > maintain the security aspect of if. Like, where do we recommend running, > auth and transport methods, etc. > > I made a new release today with FastMCP and discovery of tools based on > category. After seeing today that FastMCP mount and sending refresh the > tools list notification is working (*which it wasn't 3 months back*). > > Things I have on my roadmap: > - HTTP based transport specifically for OAuth currently it is stdio only > for it to be installed in a plugin (*airflow-wingman*) as a dependency. > - Waiting for search tools to be implemented. > - Decide to bake prompts with the server, perhaps. > - Very minimal Docs as a resource which tells about recent changes in > Airflow behavior (Example: *every LLM till date still tries to write > schedule_interval instead of schedule*) > > @Shahar feel free to take a lead and see if there are things you would like > to cherry-pick from it. > > Thanks, > Avi > > On Fri, May 30, 2025 at 9:57 AM Avi <a...@astronomer.io> wrote: > > > 💯 agreed > > > > On Fri, May 30, 2025 at 9:52 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > >> Agreed > >> > >> On Fri, 30 May 2025 at 15:18, Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >> > Yeah, I think none of us want to just publish the code and be "done" > >> with > >> > it ... There is a real work to be done to make MCP much more useful > and > >> "AI > >> > friendly" and the examples you gave Brian are cool, because I think we > >> need > >> > to (and this is mostly the ask from maintainers for the users to come > >> and > >> > participate in the design phase of what the MCP can do). > >> > It looks like that "backbone" of the MCP and the glue between the REST > >> API > >> > and MCP can be done in a simple (and easy to maintain) way. But the > >> added > >> > value is indeed in figuring out what would be useful missing things - > >> > starting from what broad use cases we want to address - and whether > >> some > >> > help for the agent can be described as prompts, better description of > >> the > >> > API and examples or maybe we need more aggregated, new APIs (maybe > >> simply > >> > new REST API calls we need) that will allow the agents to reason > better > >> and > >> > faster. All of that is possible. > >> > > >> > J > >> > > >> > On Fri, May 30, 2025 at 11:29 AM Kaxil Naik <kaxiln...@gmail.com> > >> wrote: > >> > > >> > > You can easily add as many tools you want: > >> > > https://gofastmcp.com/servers/tools > >> > > > >> > > I would be surprised if there is a thing you can't do with FastMCP > >> 2.0+ > >> > > that you can do with the MCP Python SDK. > >> > > > >> > > Like I said earlier: This is a simplistic example :) but the gist is > >> we > >> > > should be using the newer abstractions which I am happy to comment > >> during > >> > > the development phase too. Like everything else we need to ensure > >> > > maintainability is worth the value we create. > >> > > > >> > > > >> > > > >> > > On Fri, 30 May 2025 at 14:48, Kaxil Naik <kaxiln...@gmail.com> > wrote: > >> > > > >> > > > Btw we don’t need to use FastMCP just for create MCP from OpenApi > >> spec. > >> > > > Many of you mighht already be aware - FastMCP 1.0 was adopted in > the > >> > > > official mcp python sdk since 1.2 and is recommended high-level > >> server > >> > > > framework. > >> > > > > >> > > > Check: > >> > > > > >> https://github.com/modelcontextprotocol/python-sdk/releases/tag/v1.2.0 > >> > > > > >> > > > @Bryan Coder: I will be surprised if you can’t do the use-case you > >> > > > mentioned with FastMCP - either the one donated to MCP Python SDK > or > >> > > > FastMCP 2.0 - have you tried that? It isn’t just a wrapper! > >> > > > > >> > > > On Fri, 30 May 2025 at 13:16, Avi <a...@astronomer.io.invalid> > >> wrote: > >> > > > > >> > > >> Yeah FastMCP is nice, I didn't select fast mcp for this specific > >> > reason: > >> > > >> - The sheer number of tools that are created using OpenAPI spec > >> > doesn't > >> > > >> need to be passed to AI every single message. > >> > > >> - Instead, we can do a hierarchical tool discovery based on > >> > categories. > >> > > >> And > >> > > >> let AI select a particular category and then get tools only for > >> that > >> > > >> category. > >> > > >> > >> > > >> python3 -c " > >> > > >> > import json > >> > > >> > with open('path/to/openapi.json') as f: > >> > > >> > spec = json.load(f) > >> > > >> > > >> > > >> > tags = {} > >> > > >> > for path, methods in spec['paths'].items(): > >> > > >> > for method, details in methods.items(): > >> > > >> > if 'tags' in details: > >> > > >> > for tag in details['tags']: > >> > > >> > tags[tag] = tags.get(tag, 0) + 1 > >> > > >> > > >> > > >> > print('Tags and their counts:') > >> > > >> > for tag, count in sorted(tags.items(), key=lambda x: x[1], > >> > > >> reverse=True): > >> > > >> > print(f'{tag}: {count}') > >> > > >> > " > >> > > >> Tags and their counts: > >> > > >> Task Instance: 19 > >> > > >> Asset: 13 > >> > > >> Connection: 8 > >> > > >> DagRun: 8 > >> > > >> Backfill: 7 > >> > > >> DAG: 7 > >> > > >> Pool: 6 > >> > > >> Variable: 6 > >> > > >> XCom: 4 > >> > > >> Config: 2 > >> > > >> Event Log: 2 > >> > > >> Import Error: 2 > >> > > >> Plugin: 2 > >> > > >> Task: 2 > >> > > >> DagVersion: 2 > >> > > >> Login: 2 > >> > > >> DagSource: 1 > >> > > >> DagStats: 1 > >> > > >> DagReport: 1 > >> > > >> DagWarning: 1 > >> > > >> Extra Links: 1 > >> > > >> Job: 1 > >> > > >> Provider: 1 > >> > > >> DAG Parsing: 1 > >> > > >> Monitor: 1 > >> > > >> Version: 1 > >> > > >> > >> > > >> My last attempt to do a hierarchical discovery with FastMCP > didn't > >> go > >> > as > >> > > >> expected. > >> > > >> But this could be short term. There is something cooking in the > >> model > >> > > >> context protocol repo for search of a tool. Ref: > >> > > >> > >> https://github.com/modelcontextprotocol/modelcontextprotocol/pull/322 > >> > > >> > >> > > >> I'll give this a try with FastMCP to see if I can get the > >> > > >> hierarchical discovery working. > >> > > >> > >> > > >> - Avi > >> > > >> > >> > > >> On Fri, May 30, 2025 at 1:33 AM Bryan Corder < > >> bryancor...@gmail.com> > >> > > >> wrote: > >> > > >> > >> > > >> > In order to bring value, we might want to think beyond just > >> wrapping > >> > > the > >> > > >> > API. As Kaxil just showed, it's easy to create something with > 10 > >> > lines > >> > > >> of > >> > > >> > code and FastMCP. > >> > > >> > > >> > > >> > However, the Airflow API was made for Airflow operators' > >> > consumption, > >> > > >> not > >> > > >> > necessarily for LLM consumption. When you have an endpoint > called > >> > > >> "Delete > >> > > >> > DAG" with a description "Delete a specific DAG" that's very > easy > >> for > >> > > any > >> > > >> > user who has already navigated to the Airflow API spec to > >> > understand, > >> > > >> but > >> > > >> > maybe not the best tool description for an LLM. I think we'd > >> want to > >> > > >> either > >> > > >> > exclude that or add additional context for the LLM to know it's > >> > > >> > destructive. > >> > > >> > > >> > > >> > In addition, LLMs can struggle with tool selection when you > give > >> it > >> > 80 > >> > > >> > tools to work with. Things in the middle sometimes get lost in > >> the > >> > > >> context. > >> > > >> > There are ways to customize the FastMCP ( > >> > > >> > https://gofastmcp.com/servers/openapi#custom-route-maps) to > cut > >> > down > >> > > >> the > >> > > >> > list of options, should you choose. > >> > > >> > > >> > > >> > However, it may be better to create something more tailored to > >> LLMs. > >> > > >> > Thinking about the use case of getting LLM assistance with > >> > debugging a > >> > > >> > failed run, one of the things my teams do is put the "run book" > >> for > >> > > prod > >> > > >> > support in the doc_md notes right with the DAG, so if a file > >> never > >> > > >> shows up > >> > > >> > they know exactly what to do in that situation (potentially, do > >> > > >> nothing). > >> > > >> > We also include other information like, "xx task can be flaky. > If > >> > you > >> > > >> get > >> > > >> > this error, rerunning it will usually resolve it." The goal is > >> for > >> > any > >> > > >> > engineer armed with the stack trace and the run book to be able > >> to > >> > > solve > >> > > >> > any error. My team has all that information right in the UI. To > >> get > >> > > that > >> > > >> > information, the LLM would need to know to hit the DAG Details > >> > > endpoint > >> > > >> for > >> > > >> > one minor attribute amongst several for the doc_md and get the > >> > correct > >> > > >> dag > >> > > >> > id, run id, task id and try number to grab the stack trace from > >> the > >> > > >> failed > >> > > >> > run. It would then need to go elsewhere to find the DAG code to > >> > > debug. I > >> > > >> > think it would be better to just create a "debug_failed_task" > >> tool > >> > an > >> > > >> LLM > >> > > >> > could call from an MCP server that would string those calls > >> together > >> > > and > >> > > >> > serve them up to the LLM on a silver platter. The LLM could > focus > >> > all > >> > > >> its > >> > > >> > "reasoning" efforts on solving the problem instead of figuring > >> out > >> > how > >> > > >> to > >> > > >> > get the information it needs to even begin. > >> > > >> > > >> > > >> > Again, if we just want to wrap the API in FastMCP, we can share > >> > > Kaxil's > >> > > >> 10 > >> > > >> > lines of code in a Medium article and be done. I think the real > >> > value > >> > > >> is in > >> > > >> > providing an implementation of a limited set of more complex > base > >> > > tools > >> > > >> > like debug_failed_task (described above), pause_all_active_DAGs > >> > > (because > >> > > >> > I'm about to upgrade!), describe_DAG (grabs only the > description, > >> > > >> > dependencies, converts cron schedule to human readable if > >> > applicable, > >> > > >> etc) > >> > > >> > and giving people a way to extend the server. > >> > > >> > > >> > > >> > The above is tool focused. As Avi pointed out, there are also > >> > > resources > >> > > >> and > >> > > >> > prompts, but I've only personally worked with tools and have > >> nothing > >> > > to > >> > > >> add > >> > > >> > there. > >> > > >> > > >> > > >> > With all the LLM tools quickly advancing on the development > side > >> > (e.g. > >> > > >> code > >> > > >> > generation/review), it's great to see the community working on > >> > > building > >> > > >> > tools to help with the operational side. > >> > > >> > > >> > > >> > Bryan > >> > > >> > > >> > > >> > > >> > > >> > On Thu, May 29, 2025, 4:50 PM Kaxil Naik <kaxiln...@gmail.com> > >> > wrote: > >> > > >> > > >> > > >> > > One more comment: MCP SDKs have advanced quite a bit and I > was > >> > able > >> > > to > >> > > >> > get > >> > > >> > > an Airflow MCP Server working with just the following code > >> block. > >> > I > >> > > >> was > >> > > >> > > successfully able to pause/unpause a DAG from Claude and > other > >> MCP > >> > > >> client > >> > > >> > > as an example. So as much as possible we should utilize > higher > >> > level > >> > > >> > > abstraction like FastMCP which allows creating client from > >> OpenAPI > >> > > >> spec > >> > > >> > > <https://gofastmcp.com/servers/openapi#openapi-integration>: > >> > > >> > > > >> > > >> > > import os > >> > > >> > > > >> > > >> > > import httpx > >> > > >> > > from fastmcp import FastMCP > >> > > >> > > > >> > > >> > > token = os.environ.get("AF_ACCESS_TOKEN") > >> > > >> > > client = httpx.AsyncClient( > >> > > >> > > base_url="http://localhost:28080";, > >> > > >> > > headers={"Authorization": f"Bearer {token}"}, > >> > > >> > > ) > >> > > >> > > > >> > > >> > > openapi_spec = httpx.get(" > >> http://localhost:28080/openapi.json > >> > > >> > ").json() > >> > > >> > > > >> > > >> > > mcp = FastMCP.from_openapi( > >> > > >> > > openapi_spec=openapi_spec, > >> > > >> > > client=client, > >> > > >> > > name="Airflow 3.0 API Server" > >> > > >> > > ) > >> > > >> > > > >> > > >> > > if __name__ == "__main__": > >> > > >> > > mcp.run() > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> > > On Thu, 29 May 2025 at 20:32, Avi <a...@astronomer.io.invalid > > > >> > > wrote: > >> > > >> > > > >> > > >> > > > @Shahar -- Yes. Definitely. Feel free to reachout if you > need > >> > > >> anything. > >> > > >> > > > > >> > > >> > > > I totally agree, it to live as a separate repo. > >> > > >> > > > > >> > > >> > > > - Avi > >> > > >> > > > > >> > > >> > > > On Thu, May 29, 2025 at 12:50 PM Kaxil Naik < > >> > kaxiln...@gmail.com> > >> > > >> > wrote: > >> > > >> > > > > >> > > >> > > > > @Shahar -- Absolutely, I think you are driving it with > this > >> > > email. > >> > > >> > So I > >> > > >> > > > > think you can lead it from here and whoever wants to join > >> can > >> > > >> co-lead > >> > > >> > > or > >> > > >> > > > > join in development. > >> > > >> > > > > > >> > > >> > > > > Please feel free to drive :) > >> > > >> > > > > > >> > > >> > > > > On Thu, 29 May 2025 at 17:07, Aaron Dantley < > >> > > >> aarondant...@gmail.com> > >> > > >> > > > > wrote: > >> > > >> > > > > > >> > > >> > > > > > Hey All! > >> > > >> > > > > > > >> > > >> > > > > > I’d be grateful to be included in the AIP discussions > to > >> > help > >> > > if > >> > > >> > > > possible > >> > > >> > > > > > too! Like Shahar, I’ve never worked on any of these > >> items so > >> > > >> it’d > >> > > >> > be > >> > > >> > > > > great > >> > > >> > > > > > to see how work gets assigned and goes through a whole > >> > > >> development > >> > > >> > > > cycle! > >> > > >> > > > > > > >> > > >> > > > > > Looking forward to it! > >> > > >> > > > > > Aaron > >> > > >> > > > > > > >> > > >> > > > > > On Thu, May 29, 2025 at 7:32 AM Shahar Epstein < > >> > > >> sha...@apache.org> > >> > > >> > > > > wrote: > >> > > >> > > > > > > >> > > >> > > > > > > If it's ok, I would like to lead the AIP effort (or > at > >> > least > >> > > >> > > > co-lead), > >> > > >> > > > > as > >> > > >> > > > > > > I've never written an AIP before. I could start > >> drafting > >> > it > >> > > >> > during > >> > > >> > > > the > >> > > >> > > > > > next > >> > > >> > > > > > > week. > >> > > >> > > > > > > Avi - please let me know if it works for you. > >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > Shahar > >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > On Thu, May 29, 2025, 13:09 Kaxil Naik < > >> > kaxiln...@gmail.com > >> > > > > >> > > >> > > wrote: > >> > > >> > > > > > > > >> > > >> > > > > > > > Yes separate repo, please and we would need someone > >> to > >> > > lead > >> > > >> > this > >> > > >> > > > > effort > >> > > >> > > > > > > on > >> > > >> > > > > > > > the proposal & development too. Avi - you are > >> probably > >> > > well > >> > > >> > > > equipped > >> > > >> > > > > to > >> > > >> > > > > > > > lead it and I am sure more folks like Aaraon would > be > >> > > eager > >> > > >> to > >> > > >> > > work > >> > > >> > > > > on > >> > > >> > > > > > > its > >> > > >> > > > > > > > development and on-going maintenance. > >> > > >> > > > > > > > > >> > > >> > > > > > > > Regards, > >> > > >> > > > > > > > Kaxil > >> > > >> > > > > > > > > >> > > >> > > > > > > > On Thu, 29 May 2025 at 15:25, Jarek Potiuk < > >> > > >> ja...@potiuk.com> > >> > > >> > > > wrote: > >> > > >> > > > > > > > > >> > > >> > > > > > > > > Yep. Having MCP is cool and drawing our > >> implementation > >> > > >> from > >> > > >> > > > > > experiences > >> > > >> > > > > > > > and > >> > > >> > > > > > > > > usage of other MCP servers out there is even > cooler > >> > > >> > (especially > >> > > >> > > > > that > >> > > >> > > > > > we > >> > > >> > > > > > > > can > >> > > >> > > > > > > > > have some insights how people already use them > with > >> > > >> Airflow) > >> > > >> > - > >> > > >> > > if > >> > > >> > > > > we > >> > > >> > > > > > > can > >> > > >> > > > > > > > > bring together a few of those, put some nice, > >> relevant > >> > > >> > Airflow > >> > > >> > > > > > prompts. > >> > > >> > > > > > > > > Ideally we could have some examples of how MCP > can > >> be > >> > > used > >> > > >> > > taken > >> > > >> > > > > from > >> > > >> > > > > > > > those > >> > > >> > > > > > > > > who are using airflow (the debugging example by > >> Avi is > >> > > >> cool) > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > I am not sure implementing it as provider is > really > >> > "the > >> > > >> way" > >> > > >> > > > > though > >> > > >> > > > > > - > >> > > >> > > > > > > I > >> > > >> > > > > > > > > would rather see `apache-airflow-mcp" separate > >> repo - > >> > > >> it's so > >> > > >> > > > > > different > >> > > >> > > > > > > > and > >> > > >> > > > > > > > > distinct from airflow it does not really require > >> any > >> > of > >> > > >> > Airflow > >> > > >> > > > > > > internals > >> > > >> > > > > > > > > and code to be implemented - it makes very little > >> > sense > >> > > >> to be > >> > > >> > > the > >> > > >> > > > > > part > >> > > >> > > > > > > of > >> > > >> > > > > > > > > airflow "workspace" where we would develop it > >> together > >> > > >> with > >> > > >> > > > > airflow - > >> > > >> > > > > > > > > because if it will talk over the REST api, all we > >> need > >> > > is > >> > > >> the > >> > > >> > > > > > `client` > >> > > >> > > > > > > > that > >> > > >> > > > > > > > > might be just a dependency. And there is even no > >> > reason > >> > > >> for > >> > > >> > MCP > >> > > >> > > > and > >> > > >> > > > > > > > airflow > >> > > >> > > > > > > > > to be installed and developed together (that's > the > >> > main > >> > > >> > reason > >> > > >> > > > why > >> > > >> > > > > we > >> > > >> > > > > > > > want > >> > > >> > > > > > > > > providers to be kept in monorepo. > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > J. > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > On Thu, May 29, 2025 at 8:37 AM Amogh Desai < > >> > > >> > > > > > amoghdesai....@gmail.com> > >> > > >> > > > > > > > > wrote: > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > Seems like a promising area to invest in given > >> the > >> > > >> benefits > >> > > >> > > it > >> > > >> > > > > can > >> > > >> > > > > > > > > provide > >> > > >> > > > > > > > > > to > >> > > >> > > > > > > > > > the users as mentioned by Shahar and Abhishek. > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > Abhishek also has a promising talk submitted > >> which i > >> > > am > >> > > >> > > looking > >> > > >> > > > > > > forward > >> > > >> > > > > > > > > to > >> > > >> > > > > > > > > > this year at the summit. > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > In any case, this seems to be one of the first > of > >> > the > >> > > >> very > >> > > >> > > few > >> > > >> > > > > > > > > > implementations of trying > >> > > >> > > > > > > > > > to integrate Airflow officially / unofficially > >> with > >> > an > >> > > >> MCP > >> > > >> > > > > server. > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > Thanks & Regards, > >> > > >> > > > > > > > > > Amogh Desai > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > On Thu, May 29, 2025 at 2:56 AM Aaron Dantley < > >> > > >> > > > > > > aarondant...@gmail.com> > >> > > >> > > > > > > > > > wrote: > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > Hey! > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > I also think this is a great idea! > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > Would it be possible to be included in the > >> > > development > >> > > >> > > > process? > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > Sorry I’m new to this group, but would > >> appreciate > >> > > any > >> > > >> > > > > suggestions > >> > > >> > > > > > > on > >> > > >> > > > > > > > > how > >> > > >> > > > > > > > > > to > >> > > >> > > > > > > > > > > contribute to the MCP server development! > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > Regards! > >> > > >> > > > > > > > > > > Aaron > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > On Wed, May 28, 2025 at 2:57 PM Avi > >> > > >> > > > <a...@astronomer.io.invalid > >> > > >> > > > > > > >> > > >> > > > > > > > wrote: > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > Nice to see the idea to incorporate an > >> official > >> > > MCP > >> > > >> > > server > >> > > >> > > > > for > >> > > >> > > > > > > > > > > > Airflow. It's been really magical to see > >> what a > >> > > >> simple > >> > > >> > > LLM > >> > > >> > > > > can > >> > > >> > > > > > do > >> > > >> > > > > > > > > with > >> > > >> > > > > > > > > > an > >> > > >> > > > > > > > > > > > Airflow MCP server built just from APIs. > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > A few things that I noticed in my > experience: > >> > > >> > > > > > > > > > > > - The number of tools that the OpenAPI spec > >> > > >> generates > >> > > >> > is > >> > > >> > > > > quite > >> > > >> > > > > > > > huge. > >> > > >> > > > > > > > > > Most > >> > > >> > > > > > > > > > > > tools (*Claude, VS Code with GitHub > Copilot, > >> > > Cursor, > >> > > >> > > > > Windsurf*) > >> > > >> > > > > > > > which > >> > > >> > > > > > > > > > > uses > >> > > >> > > > > > > > > > > > mcp-client limits it to a number of 100 > >> tools. > >> > > (*The > >> > > >> > > > > read-only > >> > > >> > > > > > > mode > >> > > >> > > > > > > > > > > creates > >> > > >> > > > > > > > > > > > less tools in comparison*.) > >> > > >> > > > > > > > > > > > - MCP server are just not tools. There are > >> other > >> > > >> things > >> > > >> > > as > >> > > >> > > > > > well, > >> > > >> > > > > > > > like > >> > > >> > > > > > > > > > > > resources and prompts. Prompts are super > >> helpful > >> > > in > >> > > >> > case > >> > > >> > > of > >> > > >> > > > > > > > debugging > >> > > >> > > > > > > > > > for > >> > > >> > > > > > > > > > > > example. It is a way of teaching LLM about > >> > > Airflow. > >> > > >> > Say I > >> > > >> > > > > want > >> > > >> > > > > > to > >> > > >> > > > > > > > > have > >> > > >> > > > > > > > > > a > >> > > >> > > > > > > > > > > > failing task investigated. A prompt can be > >> > helpful > >> > > >> in > >> > > >> > > > letting > >> > > >> > > > > > LLM > >> > > >> > > > > > > > > know > >> > > >> > > > > > > > > > a > >> > > >> > > > > > > > > > > > step-by-step process of carrying out the > >> > > >> investigation. > >> > > >> > > > > > > > > > > > - Where do you run the MCP server? I > wouldn't > >> > want > >> > > >> my > >> > > >> > > > laptop > >> > > >> > > > > to > >> > > >> > > > > > > do > >> > > >> > > > > > > > > the > >> > > >> > > > > > > > > > > > heavy processing, which would want us to go > >> for > >> > > the > >> > > >> SSE > >> > > >> > > > > instead > >> > > >> > > > > > > of > >> > > >> > > > > > > > > > stdio. > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > This is why I chose two different path of > >> using > >> > > mcp > >> > > >> > > server > >> > > >> > > > > with > >> > > >> > > > > > > > > > airflow, > >> > > >> > > > > > > > > > > > which I intend to talk about at the summit. > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > 1. AI-Augmented Airflow - This helped me > add > >> a > >> > > chat > >> > > >> > > > interface > >> > > >> > > > > > > > inside > >> > > >> > > > > > > > > > > > Airflow using a plugin to talk to an > Airflow > >> > > >> instance > >> > > >> > > (read > >> > > >> > > > > > only > >> > > >> > > > > > > > > mode). > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > 2. Airflow-Powered AI - Experimenting with > >> this > >> > > has > >> > > >> > been > >> > > >> > > > > > totally > >> > > >> > > > > > > > > > magical, > >> > > >> > > > > > > > > > > > how powerful AI can become when it has > >> access to > >> > > >> > airflow. > >> > > >> > > > > > Also, a > >> > > >> > > > > > > > > > > directory > >> > > >> > > > > > > > > > > > structure to maintain the DAGs, and it can > >> write > >> > > >> DAGs > >> > > >> > on > >> > > >> > > > the > >> > > >> > > > > > > fly. I > >> > > >> > > > > > > > > > > totally > >> > > >> > > > > > > > > > > > see a need where LLMs eventually will need > a > >> > > >> scheduler, > >> > > >> > > > > > although > >> > > >> > > > > > > a > >> > > >> > > > > > > > > > > complete > >> > > >> > > > > > > > > > > > airflow just for an LLM might seem a bit > >> > overkill > >> > > to > >> > > >> > the > >> > > >> > > > rest > >> > > >> > > > > > of > >> > > >> > > > > > > > the > >> > > >> > > > > > > > > > > > community. > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > I chose to build this on top of open API is > >> > > because > >> > > >> > that > >> > > >> > > > was > >> > > >> > > > > > the > >> > > >> > > > > > > > only > >> > > >> > > > > > > > > > way > >> > > >> > > > > > > > > > > > to get proper RBAC enabled. > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > I have so many points to discuss. Would > love > >> to > >> > > hear > >> > > >> > from > >> > > >> > > > the > >> > > >> > > > > > > > > community > >> > > >> > > > > > > > > > > and > >> > > >> > > > > > > > > > > > then take it forward. > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > Thanks, > >> > > >> > > > > > > > > > > > Avi > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > On Wed, May 28, 2025 at 6:32 PM Aritra > Basu < > >> > > >> > > > > > > > > aritrabasu1...@gmail.com> > >> > > >> > > > > > > > > > > > wrote: > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > I definitely think there's potential to > >> > interact > >> > > >> with > >> > > >> > > an > >> > > >> > > > > > > airflow > >> > > >> > > > > > > > > MCP > >> > > >> > > > > > > > > > > > > server. Though I think I'd be interested > to > >> > see > >> > > >> how > >> > > >> > > many > >> > > >> > > > > and > >> > > >> > > > > > > how > >> > > >> > > > > > > > > > > > frequently > >> > > >> > > > > > > > > > > > > people are making use of MCP servers in > the > >> > wild > >> > > >> > before > >> > > >> > > > > > > investing > >> > > >> > > > > > > > > > > effort > >> > > >> > > > > > > > > > > > in > >> > > >> > > > > > > > > > > > > building and maintaining one for airflow. > >> I'm > >> > > sure > >> > > >> > the > >> > > >> > > > data > >> > > >> > > > > > is > >> > > >> > > > > > > > > > > available > >> > > >> > > > > > > > > > > > > out there, just needs finding. > >> > > >> > > > > > > > > > > > > -- > >> > > >> > > > > > > > > > > > > Regards, > >> > > >> > > > > > > > > > > > > Aritra Basu > >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > On Wed, 28 May 2025, 11:18 pm Julian > >> LaNeve, > >> > > >> > > > > > > > > > > > <jul...@astronomer.io.invalid > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > wrote: > >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > I think this would be interesting now > >> that > >> > the > >> > > >> > > > Streamable > >> > > >> > > > > > > HTTP > >> > > >> > > > > > > > > > spec < > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > > >> > > >> > https://modelcontextprotocol.io/specification/2025-03-26/basic/transports > >> > > >> > > >> > > >> > > > > > > > > > > > > > is out. I think in theory we could > >> publish > >> > > this > >> > > >> > first > >> > > >> > > > as > >> > > >> > > > > an > >> > > >> > > > > > > > > Airflow > >> > > >> > > > > > > > > > > > > > provider that installs a plugin to > >> expose an > >> > > MCP > >> > > >> > > > > endpoint, > >> > > >> > > > > > > as a > >> > > >> > > > > > > > > > PoC - > >> > > >> > > > > > > > > > > > > this > >> > > >> > > > > > > > > > > > > > becomes a much nicer experience than a > >> local > >> > > >> stdio > >> > > >> > > one. > >> > > >> > > > > > > > > > > > > > -- > >> > > >> > > > > > > > > > > > > > Julian LaNeve > >> > > >> > > > > > > > > > > > > > CTO > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > Email: jul...@astronomer.io > >> > > >> > > > > > > > > > > > > > <mailto:jul...@astronomer.io>Mobile: > >> 330 > >> > 509 > >> > > >> 5792 > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > On May 28, 2025, at 1:25 PM, Shahar > >> > Epstein > >> > > < > >> > > >> > > > > > > > sha...@apache.org > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > > > wrote: > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > Dear community, > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > Following the thread on Slack [1], > >> > initiated > >> > > >> by > >> > > >> > > Jason > >> > > >> > > > > > > > Sebastian > >> > > >> > > > > > > > > > > > Kusuma, > >> > > >> > > > > > > > > > > > > > I'd > >> > > >> > > > > > > > > > > > > > > like to start an effort to officially > >> > > support > >> > > >> MCP > >> > > >> > > in > >> > > >> > > > > > > > Airflow's > >> > > >> > > > > > > > > > > > > codebase. > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *Some background * > >> > > >> > > > > > > > > > > > > > > Model Context Protocol (MCP) is an > open > >> > > >> standard, > >> > > >> > > > > > > open-source > >> > > >> > > > > > > > > > > > framework > >> > > >> > > > > > > > > > > > > > > that standardizes the way AI models > >> like > >> > LLM > >> > > >> > > > integrate > >> > > >> > > > > > and > >> > > >> > > > > > > > > share > >> > > >> > > > > > > > > > > data > >> > > >> > > > > > > > > > > > > > with > >> > > >> > > > > > > > > > > > > > > external tools, systems and data > >> sources. > >> > > >> Think > >> > > >> > of > >> > > >> > > it > >> > > >> > > > > as > >> > > >> > > > > > a > >> > > >> > > > > > > > > "USB-C > >> > > >> > > > > > > > > > > for > >> > > >> > > > > > > > > > > > > > AI" - > >> > > >> > > > > > > > > > > > > > > a universal connector that simplifies > >> and > >> > > >> > > > standardizes > >> > > >> > > > > AI > >> > > >> > > > > > > > > > > > > integrations. A > >> > > >> > > > > > > > > > > > > > > notable example of an MCP server is > >> > GitHub's > >> > > >> > > official > >> > > >> > > > > > > > > > > implementation > >> > > >> > > > > > > > > > > > > > [3], which > >> > > >> > > > > > > > > > > > > > > allows LLMs such as Claude, Copilot, > >> and > >> > > >> OpenAI > >> > > >> > > (or: > >> > > >> > > > > "MCP > >> > > >> > > > > > > > > > clients") > >> > > >> > > > > > > > > > > > to > >> > > >> > > > > > > > > > > > > > > fetch pull request details, analyze > >> code > >> > > >> changes, > >> > > >> > > and > >> > > >> > > > > > > > generate > >> > > >> > > > > > > > > > > review > >> > > >> > > > > > > > > > > > > > > summaries. > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *How could an MCP server be useful in > >> > > >> Airflow?* > >> > > >> > > > > > > > > > > > > > > Imagine the possibilities when LLMs > can > >> > > >> > seamlessly > >> > > >> > > > > > interact > >> > > >> > > > > > > > > with > >> > > >> > > > > > > > > > > > > > Airflow’s > >> > > >> > > > > > > > > > > > > > > API: triggering DAGs using natural > >> > language, > >> > > >> > > > retrieving > >> > > >> > > > > > DAG > >> > > >> > > > > > > > run > >> > > >> > > > > > > > > > > > > history, > >> > > >> > > > > > > > > > > > > > > enabling smart debugging, and more. > >> This > >> > > kind > >> > > >> of > >> > > >> > > > > > > integration > >> > > >> > > > > > > > > > opens > >> > > >> > > > > > > > > > > > the > >> > > >> > > > > > > > > > > > > > door > >> > > >> > > > > > > > > > > > > > > to a more intuitive, conversational > >> > > interface > >> > > >> for > >> > > >> > > > > > workflow > >> > > >> > > > > > > > > > > > > orchestration. > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *Why do we need to support it > >> officially?* > >> > > >> > > > > > > > > > > > > > > Quid pro quo - LLMs become an > integral > >> > part > >> > > of > >> > > >> > the > >> > > >> > > > > modern > >> > > >> > > > > > > > > > > development > >> > > >> > > > > > > > > > > > > > > experience, while Airflow evolves > into > >> the > >> > > >> go-to > >> > > >> > > for > >> > > >> > > > > > > > > > orchestrating > >> > > >> > > > > > > > > > > AI > >> > > >> > > > > > > > > > > > > > > workflows. By officially supporting > it, > >> > > we’ll > >> > > >> > > enable > >> > > >> > > > > > > multiple > >> > > >> > > > > > > > > > users > >> > > >> > > > > > > > > > > > to > >> > > >> > > > > > > > > > > > > > > interact with Airflow through their > >> LLMs, > >> > > >> > > > streamlining > >> > > >> > > > > > > > > automation > >> > > >> > > > > > > > > > > and > >> > > >> > > > > > > > > > > > > > > improving accessibility across > diverse > >> > > >> workflows. > >> > > >> > > All > >> > > >> > > > > of > >> > > >> > > > > > > that > >> > > >> > > > > > > > > is > >> > > >> > > > > > > > > > > > viable > >> > > >> > > > > > > > > > > > > > > with relatively small development > >> effort > >> > > (see > >> > > >> > next > >> > > >> > > > > > > > paragraph). > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *How should it be implemented?* > >> > > >> > > > > > > > > > > > > > > As of today, there have been several > >> > > >> > > implementations > >> > > >> > > > of > >> > > >> > > > > > MCP > >> > > >> > > > > > > > > > servers > >> > > >> > > > > > > > > > > > for > >> > > >> > > > > > > > > > > > > > > Airflow API, the most visible one [4] > >> made > >> > > by > >> > > >> > > > Abhishek > >> > > >> > > > > > > Bhakat > >> > > >> > > > > > > > > > from > >> > > >> > > > > > > > > > > > > > > Astronomer. > >> > > >> > > > > > > > > > > > > > > The efforts of implementing it and > >> > > >> maintaining it > >> > > >> > > in > >> > > >> > > > > our > >> > > >> > > > > > > > > codebase > >> > > >> > > > > > > > > > > > > > shouldn't > >> > > >> > > > > > > > > > > > > > > be too cumbersome (at least in > >> theory), as > >> > > we > >> > > >> > could > >> > > >> > > > > > utilize > >> > > >> > > > > > > > > > > packages > >> > > >> > > > > > > > > > > > > like > >> > > >> > > > > > > > > > > > > > > fastmcp to auto-generate the server > >> using > >> > > the > >> > > >> > > > existing > >> > > >> > > > > > > > OpenAPI > >> > > >> > > > > > > > > > > specs. > >> > > >> > > > > > > > > > > > > I'd > >> > > >> > > > > > > > > > > > > > > be very happy if Abhishek could share > >> his > >> > > >> > > experience > >> > > >> > > > in > >> > > >> > > > > > > this > >> > > >> > > > > > > > > > > thread. > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *Where else could we utilize MCP?* > >> > > >> > > > > > > > > > > > > > > Beyond the scope of the public API, I > >> > could > >> > > >> also > >> > > >> > > > > imagine > >> > > >> > > > > > > > using > >> > > >> > > > > > > > > it > >> > > >> > > > > > > > > > > to > >> > > >> > > > > > > > > > > > > > > communicate with Breeze. > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *How do we proceed from here?* > >> > > >> > > > > > > > > > > > > > > Feel free to share your thoughts here > >> in > >> > > this > >> > > >> > > > > discussion. > >> > > >> > > > > > > > > > > > > > > If there are no objections, I'll be > >> happy > >> > to > >> > > >> > start > >> > > >> > > > > > working > >> > > >> > > > > > > on > >> > > >> > > > > > > > > an > >> > > >> > > > > > > > > > > AIP. > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > Sincerely, > >> > > >> > > > > > > > > > > > > > > Shahar Epstein > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > *References:* > >> > > >> > > > > > > > > > > > > > > [1] Slack discussion, > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > >> > > >> > > > >> > > >> > >> > > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746121916951569 > >> > > >> > > > > > > > > > > > > > > [2] Introducing the model context > >> > protocol, > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > > https://www.anthropic.com/news/model-context-protocol > >> > > >> > > > > > > > > > > > > > > [3] GitHub Official MCP server, > >> > > >> > > > > > > > > > > > > > > >> https://github.com/github/github-mcp-server > >> > > >> > > > > > > > > > > > > > > [4] Unofficial MCP Server made by > >> Abhishek > >> > > >> Hakat, > >> > > >> > > > > > > > > > > > > > > > >> > > >> > > https://github.com/abhishekbhakat/airflow-mcp-server > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > > > >> > > > >> > > >> > > >
