Hello, The lazy consensus has been reached.
I have a PR [1] that I hope to make green soon - it moves a number of (mostly tests) files around and creates few more internal distributions - all dependencies are move to corresponding pyproject.toml files and a lot of custom code to create environments to run our tests is going to be simplified, essentially making it into a pattern of: cd DISTRIBUTION (say `cd kubernetes-tests`) uv run pytest I am also moving the `docs` code to `devel-common` with the ultimate goal of having specific configuration per providers, airflow-core, helm chart and others to be "isolated". We are not yet there - but I did not want to make too many changes at once, but the next step will be to separate those projects and with the power of `uv` build the docs for each package locally - without having to do it all in the image and from within the "distribution". I hope to be able to complete it next week, it does require some unentangling of interconnected test and doc harness, but maybe (let's see I will finally be able to isolate it enough to be able to iterate on the docs and rebuild it mostly on-the flight while you are working on them. That's a bit of side effect of all the uv/standard distribution approach - and this splitting is a necessary step to make it happen. My personal goal is to make it isolated enough and simple enough to move distributions (i.e. folders) around that once we release airflow 3 or even before we will be able to finally have the discussion here "Ok we have now split and isolated everything and maybe we want to move things around to make it even better-structured and - yes - maybe we will - together figure out some better consistency patterns. Maybe then repainting the interiors will be - indeed - an afterthought and something that we will be able to do with a flick of a finger. We are not yet there, but I am working to get us there. https://github.com/apache/airflow/pull/48223 J. On Mon, Mar 24, 2025 at 12:36 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Also: The lazy consensus will be reached if no objections are raised in 72 > HRs: 1pm CET, 23 March, 2025 > https://www.timeanddate.com/countdown/generic?iso=20250327T13 > > On Mon, Mar 24, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Hello here, >> >> *TL;DR; Following the >> discussion https://lists.apache.org/thread/88yj3qxqdmc4ony7k8nvp292m28df31c >> <https://lists.apache.org/thread/88yj3qxqdmc4ony7k8nvp292m28df31c> I am >> calling for a lazy consensus on using only `uv` for local development of >> Airlfow. * >> >> This adds a bit of reliance on a VC-backed astral-owned (but fully >> open-sourced) `uv` which has some risks, but we think the risks are small. >> It's only for airflow development - we will leave constraint mechanism for >> our users, so that their workflows will not require to change the frontend >> they use to install airflow (both `pip` and `uv` supporting our >> constraints, `poetry` and others installing airflow without constraints and >> having to manage their own limits - that does not change) >> >> A bit more context and summary of the discussion follows. >> >> I also have a half-ready change that performs all the removals and I have >> a bit more info on what's going to happen when we go `uv only. >> >> --------- >> >> More context: >> >> Here is what we will get if go `uv` only: >> >> Good: >> >> * currently we support `uv` and `pip` (or more generally "any >> PEP-standard frontend) workflows >> * however, some uv features (particularly `workspaces` and `uv.lock`) >> that are not yet standard, make the development workflows of ours very easy >> to follow (AKA uv provides much better "Contributor journey optimisation") >> * by focusing on "one way to do things" with uv we will be able to >> simplify a lot of contributor experience but also a lot of our packaging >> code and configuration: >> - we will be able to get rid of the dynamic pyproject.toml >> dependencies around preinstalled providers >> - we will be able to get rid of hatch_build.py in the main repo >> completely >> - we will only leave `hatch_build.py` in the "airflow-core" for custom >> build step compiling assets and performing git version injection (but this >> is purely optional and used only when releases are prepared) >> - we will be able to get rid of generated_provider_dependencies.json >> prepared by pre-commit - it was needed for hatch_build.py mainly and we can >> replace its use completely by directly reading pyproject.toml files in >> pre-commit and build scripts. >> - that in turn will enable us (eventually - that is another change) to >> switch to dependabot-only driven process of upgrading our dependencies and >> use `uv.lock` for local development and avoiding "works for me" syndrome - >> this will make it more controllable when things are going to break main and >> our "responses" will based on Dependabot PRs rather than notification of >> canary build breaking, which will make it - I think much easier to handle. >> >> * uv is true open-source - i.e. OSI-compliant licences (uv is >> dual-licenced with Apache 2 and MIT) >> >> Somewhat concerning (but acceptable risk it seems): >> >> * Those features we rely on are `uv-only` for now. The uv team is >> committed to support standards and actively participate in all the PEP >> packaging discussions - including reproducible installation, dependency >> groups and others (workspaces in the future most likely) - which means that >> in the future other tools will catch up and we will be able to follow >> standards when they are approved and implemented. >> >> * unlike other packaging tools (under Python Software Foundation - >> Python Packaging Governance) - `uv` is privately owned, by VC-backed >> Astral. They are great open-source stakeholders and understand the value of >> `uv` being open-source and we personally know the team and they have very >> good intentions, no doubt about it. But as various events have shown in the >> past - change-of-control events might influence "openness", so we cannot be >> sure what the future brings. >> >> * However, being "true OSS" - `uv` can be forked by anyone if anything >> bad happens. Also it does not impact our users - just contributing >> workflow, so the risks are small and mitigable (like freezing tooling and >> working on a solution in the meantime). >> >> >> J. >> >> >>
