https://github.com/apache/airflow/pull/66319 -> has exactly this: -7500 lines of skills +1400 lines of "adoption" bootstrap that has been added automatically at adoption time.
On Sun, May 3, 2026 at 7:37 PM Jarek Potiuk <[email protected]> wrote: > Ok. Adoption should be super simple now: > > 1) Project does not need to have **anything** > 2) Tell your agent: "Adopt apache/airflow-steward in my repo". > 3) The agent does everything. You answer questions during configuration. > At the end you have a "setup-steward" skill in your repo. Commit the PR > 4) Those who wish to use steward in their copy should ask their agent to > "set up steward". That's it. All skills chosen for a steward at adoption > time will be installed > 5) At any point, you can run `setup-steward upgrade` to upgrade to the > latest steward, or `setup-steward verify` to verify if it is set up > correctly > 6) We also have basic scaffolding to modify steward workflows with local > "variants" -> "setup-steward override SKILL" (which will be > intention-driven). > > https://github.com/apache/airflow-steward#how-adoption-works > > Working on the next step -wiring the skills and PR to add it to airflow - > using the approach above - I will test it this way :) > > On Sun, May 3, 2026 at 11:47 AM Jarek Potiuk <[email protected]> wrote: > >> Actually... After your message, Shahar, I woke up today with a much >> better idea :D. >> >> I figured out that with agentic CLIs - we don't have to do subrepo or >> submodule..... We can have literally 1 SKILL to copy to "airiflow'` SKILL >> directory: `Install and manage "Apache Steward" :). No subrepot, No >> submodule, simply instructions on how to locally copy the "steward" from >> either released SVN or Github repo (tag or branch), adding .gitignore. >> >> This will be even better, and likely much more acceptable for projects >> that do not want to use submodules. When you want to modify "steward" >> behaviour, we will have protection so that you cannot modify the gitignored >> steward. Instead you will add local "modification" of the flow >> locally (which will not be .gitignored), and when you would like to >> contribute it back you will have to ask agent to turn your local >> modification into a proper PR to the repo. It's really magical with agents >> when you do not have to limit yourself to "common tools" (like submodule) - >> but you can simply describe your intentions and let the agent execute those >> :) >> >> I will close the PR and will follow that path :) >> >> On Sat, May 2, 2026 at 11:02 PM Shahar Epstein <[email protected]> wrote: >> >>> Yeah well - I forgot that there's the option of not installing the >>> submodule if you don't use it, and then there's no problem for most cases >>> (we could indeed handle the rest, with some AI skill and guidance in the >>> docs). >>> In any case it should be better than subrepo, according to your >>> description. >>> Let's give it a shot then this way. >>> >>> >>> Shahar >>> >>> On Sat, May 2, 2026, 22:43 Jarek Potiuk <[email protected]> wrote: >>> >>> > Subrepo has another disadvantage - namely that any "sync" of the >>> subrepo is >>> > a full "snapshot" of all changed there - and the files end up as >>> regular >>> > files in the target repo. Those files are physically committed files in >>> > Airflow repo, and subrepo is not "official" git feature - it's an >>> external >>> > package https://github.com/ingydotnet/git-subrepo. I think subrepo is >>> grep >>> > if you want to "vendor-in" part of an external "dependent" >>> repository—one >>> > that you plan to migrate to the next version, where a submodule is much >>> > better for "WIP," especially if none of your code actually depends on >>> > whether the submodule is checked out or not. >>> > >>> > Our only dependency on "steward" now is that symlinks to "skills" will >>> be >>> > missing if the submodule is not updated. >>> > >>> > So ... skills in Claude won't work in this case—that's the "default" >>> stage >>> > for people who do not use skills. No big deal (because they don't use >>> them) >>> > >>> > On the other hand if some wants to start using skills this is a >>> question of >>> > two commands (explained in setup): >>> > >>> > >>> > git submodule update --recursive >>> > >>> > Followed by (in Agent) >>> > >>> > /setup-apache-steward >>> > >>> > And the latter command will add a "post-merge" hook that removes the >>> need >>> > of manually updating the submodule.Once setup is complete, it should be >>> > totally transparent. >>> > >>> > Of course - this is theory, and with a little practice (it seems to >>> work), >>> > there will be some edge cases. But I would rather learn about those >>> edge >>> > cases and fix them :) >>> > >>> > J >>> > >>> > >>> > >>> > >>> > >>> > >>> > On Sat, May 2, 2026 at 9:25 PM Shahar Epstein <[email protected]> >>> wrote: >>> > >>> > > Thanks Jarek for working on it! >>> > > Confirming a smooth experience with "Steward" - some setup >>> > > shenanigans, but we figured it out eventually :) >>> > > >>> > > Overall the PR looks solid. However, I'm concerned regarding the >>> usage >>> > > of submodules in Airflow's main repository. Maintaining a submodule >>> > > over time, as well as removing a submodule from the repo, are not >>> > > trivial for the average contributor and people could easily get >>> > > troubled with it. AI can surely solve everything related with a >>> single >>> > > prompt - but still, I have to be the devil's advocate here. >>> > > Is it possible to make the transition to subrepo/automation before >>> > merging? >>> > > >>> > > >>> > > Shahar >>> > > >>> > > On Sat, May 2, 2026 at 8:39 PM Jarek Potiuk <[email protected]> >>> wrote: >>> > > > >>> > > > Hi everyone, >>> > > > >>> > > > I am moving Airflow's "maintainer skill" rules to the shared >>> "steward" >>> > > > project, which will serve as the base for the future ASF-wide >>> > initiative. >>> > > > For those curious, I have already ported the skills for security >>> > > workflows, >>> > > > PR triage, and review to the steward repository: >>> > > > https://github.com/apache/airflow-steward/tree/main/.claude/skills >>> . >>> > > > >>> > > > I propose wiring these skills into Airflow via a Git submodule. >>> While >>> > we >>> > > > have had past difficulties with submodules, this approach is >>> justified >>> > > here: >>> > > > >>> > > > >>> > > > - Users who do not utilize agentic skills will be unaffected. >>> > > > - We developed a skill to automate submodule updates, addressing >>> > > > previous pain points. >>> > > > - This removes all skill-related code and almost all >>> documentation >>> > > > (except bootstrapping) from the main Airflow repository. Once we >>> > > switch, >>> > > > and make sure all "airflow-specific" things are in Airflow, >>> further >>> > > > "steward" work should happen outside of Airflow repo - no more >>> 1000+ >>> > > > English markdown PRs in Airflow :). >>> > > > >>> > > > >>> > > > We have a similar setup for our security project, "airflow-s." I am >>> > > > applying feedback from Shahar; we previously collaborated on some >>> > > > security-management related work using it, and I recall it being a >>> > smooth >>> > > > experience (Shahar, please confirm). >>> > > > >>> > > > I would appreciate (positive :)) reviews on this prerequisite PR: >>> > > > https://github.com/apache/airflow/pull/66283. Once merged, I will >>> > > follow up >>> > > > by replacing the current skills with symbolic links to the >>> submodule. >>> > We >>> > > > can transition to a subrepo or automated extraction once the >>> "steward >>> > (to >>> > > > be renamed)" project stabilizes. >>> > > > >>> > > > Best, >>> > > > Jarek >>> > > > >>> > > > On Wed, Apr 29, 2026 at 11:37 AM Jarek Potiuk <[email protected]> >>> > wrote: >>> > > > >>> > > > > Hi everyone, >>> > > > > >>> > > > > I would like to provide an update regarding the recent >>> > > "airflow-steward" >>> > > > > and "airflow-s" messages you may have seen. I mistakenly >>> configured a >>> > > new >>> > > > > repository to send notifications to the dev@ list, >>> unintentionally >>> > > > > sharing some early-stage details a bit sooner than I thought. >>> > > > > >>> > > > > I have been working at an accelerated pace (pretty much 150% of >>> my >>> > time >>> > > > > over the last one-two months) due to an emerging security >>> "crisis" in >>> > > the >>> > > > > industry. New AI models, such as Anthropic’s Mythos (which >>> resulted >>> > in >>> > > the >>> > > > > recently announced cross-industry Project Glasswing [1]), have >>> become >>> > > > > exceptionally fast at identifying and chaining security >>> > > vulnerabilities. >>> > > > > This has forced a shift in how we handle security, moving from >>> > reactive >>> > > > > responses to proactive scanning. We expect an exponential growth >>> in >>> > > > > high-quality security reports, and we are preparing to catch >>> these >>> > > issues >>> > > > > during the development process—before code is even released. >>> > > > > >>> > > > > Last month, the ASF announced the Responsible AI Initiative [2], >>> > > supported >>> > > > > by Anthropic and Alpha-Omega. I am currently leading a proposal >>> to >>> > > > > transition our recent work into a new "Apache Steward" PMC, >>> which is >>> > > > > expected to be discussed at the next ASF board meeting on May >>> 20th. >>> > > > > >>> > > > > Because of the urgency, I have had to make some rapid decisions >>> and >>> > > move >>> > > > > faster than our usual PMC pace. My goal is to ensure Apache >>> Steward >>> > > helps >>> > > > > the open-source community navigate this transition successfully. >>> For >>> > > > > Airflow (and other projects), this starts with self-cleaning of >>> > > security >>> > > > > issues first. It very likely means a renewed focus on the >>> "Airflow >>> > > Beach >>> > > > > Cleaning" [3] initiative to prune historical baggage and manage >>> > > > > dependencies more effectively using new agentic tooling (it had >>> > > previously >>> > > > > hit some scaling roadblocks - that are now unblocked with Agentic >>> > AI). >>> > > > > >>> > > > > We now have Airflow Steward [4] as the first stage of the >>> "extracting >>> > > > > tooling" - this will likely become "Apache Steward" - if all >>> stars >>> > > align. >>> > > > > >>> > > > > I apologize for the hectic communication. I will continue to keep >>> > > everyone >>> > > > > updated as things progress. If you have any specific questions, >>> > please >>> > > feel >>> > > > > free to ask here. We can keep that thread to keep everyone >>> updated. >>> > > > > >>> > > > > Best regards, >>> > > > > >>> > > > > Jarek Potiuk >>> > > > > >>> > > > > [1] Project Glasswing https://www.anthropic.com/glasswing >>> > > > > [2] Responsible AI Initiative of ASF - >>> > > > > >>> > > >>> > >>> https://news.apache.org/foundation/entry/the-apache-software-foundation-launches-10m-responsible-ai-initiative-with-initial-1-75m-donation >>> > > > > [3] Airflow Beach Cleaning Keynote from Airflow Summit 2024 - >>> > > > > https://www.youtube.com/watch?v=f6gfoVJXWEE >>> > > > > [4] Airflow Steward - https://github.com/apache/airflow-steward >>> > > > > >>> > > > > >>> > > >>> > > --------------------------------------------------------------------- >>> > > To unsubscribe, e-mail: [email protected] >>> > > For additional commands, e-mail: [email protected] >>> > > >>> > > >>> > >>> >>
