> would it be also an option to add the missing features for instances into breeze start-airflow directly?
*TL;DR; My view: I personally think that the current split where we have "abm" as an "outside tool" to manage different repos/branches and "breeze" - is perfect and combining them is not a good idea. At least for now - until we can get rid of some of the release-management and build code that is currently (out of necessity) outside of build-backends (hatch/flit) that we use to prepare packages.* Bear with me ... It's long - but my head is full of learnings from the Reproducible Builds Summit that I was at two days ago in Vienna, and I need to get it out :). That would be a bit difficult because "breeze" (deliberately) is the same commit as the checked out version of airflow sources. This is *deliberate* decision - because this allows to not care about compatibility with all historical commits of airflow that you can switch to - mainly but also when you build released in CI or manually **ALL** that you need is checking out airflow commit from our monorepo and you have almost a guarantee that what you build will be the same and that it will "work". We could *potentially* split out breeze to a separate repo from our monorepo (and we discussed this is as a potential idea when discussing if we want to split out providers) - but .. the more I discuss about it in the context of security and rebuildable packages - it's way better to keep breeze in mono-repo. And if we do that - the *abm* kind of features could be merged-in then. But not before I think. This actually follows - very closely - something I ***just*** discussed at "Reproducible Builds Summit". We really need 3rd-parties to be able to build airflow and verify reproducibility of it - not only PMC members. Currently if **3rd-party** wants to build Airflow from sources in a reproducible build they can do this: * get *-sources.tar.gz *that we officially release as the "legal act of the foundation" or * checkout the *git repo* + *git commit hash* that you should check-out from our repo - that will check-out not only the source of Airflow but also all the build scripts, breeze and Docker images that are needed to build the packages. Because of security supply chain problems we all have now (ALL - without exceptions!). There are a number of projects currently that are attempting to provide attestations about "rebuild" status of projects. An example of it is "OSS Rebuild" by Google: https://oss-rebuild.dev/ - I spent a good deal of Tuesday and Wednesday talking about it with the maintainers of OSS rebuild project and other similar projects. Currently such projects are using a combination of deterministic tools and AI to attempt to rebuild and verify reproducibility of a number of open-source projects and the idea that you need: * several repos * not recorded and not discoverable connection between the version of Airflow and other repos (airflow-breeze) that was used to build airflow Is a terrible one. When I explained to them how the Airflow reproducible build works and that we are even **considering** splitting the repos - their eyes visibly widened and their face skin got visibly more gray. And I am not exaggerating - maybe a bit. Because they are not able to easily figure out (even with AI) what should be done to make a deterministic and reproducible build of the packages - now and in the near future. We can probably solve it **eventually** but if we really want to separate "breeze build code" from "airflow build backend/code" that could allow deterministic builds, we really need a much more sophisticated "reproducibility" support from tools like hatch and flit. Not impossible and this is one of our long term goals to convince the authors of those build backends to add more reproducibility support. J. On Fri, Oct 31, 2025 at 8:13 PM Jens Scheffler <[email protected]> wrote: > Hi Kaxil, > > very cool! Need to try soon! That would boost also my productivity. > > I already use worktrees a lot but mainly suffer from that I can not > start multiple breeze instances - so w/o having "ABM" as an additional > tool, would it be also an option to add the missing features for > instances into breeze start-airflow directly? > > Jens > > On 10/31/25 19:37, Jarek Potiuk wrote: > > BTW. One of the things I want to fix - I know how to easily remove that > > limitation: > > > > Postgres: > > ⚠️ Projects share the same database container (e.g., one postgres > container > > for all) > > > > Breeze has a somewhat prepared feature for that - that we use to run our > > tests in CI in parallel for multiple test types - there we have several > > "full breeze docker-compose instances" running on the same machine > > (including Postgres or MySQL databases). We can reuse that one. > > > > J. > > > > > > On Fri, Oct 31, 2025 at 7:33 PM Jarek Potiuk <[email protected]> wrote: > > > >> Excellent tool Kaxil !!!! > >> > >> I love it. I already installed it and I am giving it a go (and I might > >> have some PRs soon - I already see one thing I would like to change :D) > >> > >> I do manage a lot of those manually - but it is certainly worth trying > to > >> automate some of that especially if you work on many PRs in parallel. I > >> have a feeling that this is going to be my new best friend. > >> > >> > >> On Fri, Oct 31, 2025 at 2:12 PM Kaxil Naik <[email protected]> wrote: > >> > >>> Hello fellow Airflow developers, > >>> > >>> Before I go on paternity leave, I wanted to share a tool I've been > using > >>> that might help fellow contributors who work on multiple PRs > >>> simultaneously. > >>> > >>> ## What is ABM? > >>> > >>> *Airflow Breeze Manager (ABM)* is a CLI tool for managing multiple > Airflow > >>> development environments with isolated breeze instances. It uses git > >>> worktrees and port isolation so you can work on multiple branches at > the > >>> same time without conflicts. > >>> > >>> GitHub: https://github.com/kaxil/abm > >>> PyPI: `uv tool install airflow-breeze-manager` > >>> > >>> ## The Problem It Solves > >>> > >>> When working on multiple Airflow PRs/features simultaneously, we face: > >>> - Port conflicts when running multiple breeze instances > >>> - Container name collisions (all use the same Docker Compose project > name) > >>> - Lost context when switching branches > >>> - Waiting for containers to restart every time you switch > >>> > >>> ## How It Works > >>> > >>> ABM creates isolated environments for each branch: > >>> - ✅ Git worktrees (multiple branches checked out simultaneously) > >>> - ✅ Unique port assignments for all services (webserver, flower, > postgres, > >>> mysql, redis, ssh) > >>> - ✅ Isolated Docker containers (via unique COMPOSE_PROJECT_NAME) > >>> - ✅ Branch-specific documentation that persists (PROJECT.md, CLAUDE.md) > >>> - ✅ GitHub PR tracking > >>> - ✅ Disk space management (freeze/thaw node_modules) > >>> > >>> ## Quick Example > >>> > >>> ```bash > >>> # One-time setup (run from your Airflow repo) > >>> cd ~/airflow > >>> abm init --airflow-repo . --worktree-base ../airflow-wt > >>> > >>> # Create projects for different features > >>> abm add feature-async --create-branch > >>> abm add bugfix-scheduler --create-branch > >>> > >>> # Work on both simultaneously > >>> abm start-airflow feature-async # localhost:28180 > >>> abm start-airflow bugfix-scheduler # localhost:28181 (different > port!) > >>> > >>> # Each has its own isolated environment - no conflicts! > >>> abm list # See all your projects > >>> ``` > >>> > >>> ## Links > >>> > >>> - GitHub: https://github.com/kaxil/abm > >>> - Quick Start: https://github.com/kaxil/abm/blob/main/QUICKSTART.md > >>> > >>> I hope this helps some of you be more productive with parallel > >>> development! > >>> > >>> Regards, > >>> Kaxil > >>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
