Hi all,
Spawned by the other thread about what version of images we should publish in
the 3.4 Docker images, I have a proposal for 3.5:
We remove any and all user-facing documentation about using `breeze prod-image
build`, and remove all features from that which are not directly used in our
own build or test pipelines. We should also remove as much of the complexity
from the Dockerfile itself as possible.
For downstream users, the supported interface becomes `FROM apache/airflow:…`,
not building their own image using our Dockerfile.
Right now `breeze prod-image build` has 58 different options, including things
like these:
```
│ --airflow-extras Extras to install by default. [default:
│
│ --additional-airflow-extras Additional extra package while installing
Airflow in the image. (TEXT) │
│ --additional-python-deps Additional python dependencies to use when
building the images. (TEXT) │
│ --dev-apt-deps Apt dev dependencies to use when building
the images. (TEXT) │
│ --additional-dev-apt-deps Additional apt dev dependencies to use when
building the images. (TEXT) │
│ --dev-apt-command Command executed before dev apt deps are
installed. (TEXT) │
│ --additional-dev-apt-command Additional command executed before dev apt
deps are installed. (TEXT) │
│ --additional-dev-apt-env Additional environment variables set when
adding dev dependencies. (TEXT) │
│ --runtime-apt-deps Apt runtime dependencies to use when
building the images. (TEXT) │
│ --additional-runtime-apt-deps Additional apt runtime dependencies to use
when building the images. (TEXT) │
│ --runtime-apt-command Command executed before runtime apt deps
are installed. (TEXT) │
│ --additional-runtime-apt-command Additional command executed before runtime
apt deps are installed. (TEXT) │
│ --additional-runtime-apt-env Additional environment variables set when
adding runtime dependencies. (TEXT) │
│ --installation-method Install Airflow from: sources
(.) or packages (apache-airflow). [default: .] (. │
│ --install-airflow-reference Install Airflow using GitHub tag
or branch. (TEXT) │
```
My reason for suggesting we simplify are:
- It is trying to fit all such use case, including letting users build their
own image. We should not support that; if you want to customise the image you
run: a Dockerfile is right there.
- that the entire thing is way too complicated, and has essentially grown the
kitchen sink of features.
- The Dockerfile is 2526 lines. Each one of those carries a cognitive burden
(either you as a human, or extra tokens for your LLM to understand)
This is all just too complicated and I have not been happy with this status quo
for a long time. Additionally and I have never seen any other project do this.
TL;DR: We should maintain the tooling needed to build, test and publish _our_
production images, not a general-purpose framework for building customised
Airflow images. That should let us substantially simplify both the Dockerfile
and Breeze, without losing anything we actually need for CI or releases.
My concrete proposals (not exhaustive:
1. Reduce the Dockerfile to essentially “add uv, install apache-airflow with
constraints”.
2. Remove the ability to add airflow extras, or python deps
These can be done just as easily by users by doing “FROM apache/airflow; RUN
uv pip install apache-airflow[my-extra] some-other-dep”.
3. Similarly, remove all apt envs/commands/deps, as the same FROM+RUN works
just as well there.
4. Only use UV for building the image.
5. Remove the `—python-versions` argument. We publish a fixed set of python
versions, if users want a different one, then they should build their own image
from scratch.
6. Explicitly remove the “Dockerfile must be standalone”, and stop inlining all
the scripts. This only has merit if as long as “Dockerfile is meant for users
to build the kind of image they want”. If we stop doing that, we remove the
need for it being standalone.
7. --installation-method is gone/simplified, and the only options are
“published pypi release”, or local dir of .whl files.
8. Everything --additional-* is just removed.
9. --disable-$db-* is removed. Don’t want it: build your own Dockerfile
10. Separate build-time and runtime constraints.
We should continue using full constraints to build and test our images, but
we shouldn't enforce those constraints when users extend the image, as that
would prevent them from upgrading dependencies to pick up security fixes.
Instead, we should provide a minimal constraints file that pins only the
Airflow distributions installed in the image (`apache-airflow`,
`apache-airflow-core`, `apache-airflow-task-sdk`) to their actual installed
versions. Users wanting to upgrade Airflow itself should upgrade their base
image instead. Everything else should be free to upgrade if the user needs or
wants to, subject to package compatibility and uv’s existing rules and checks.
11. Remove the build-time configurable UID (`ARG AIRFLOW_UID`), but keep the
runtime arbitrary UID support (needed for OpenShift and a few other cases etc)
This is not an exhaustive list, and it is not an all-or-nothing list either, if
someone can articulate a good case why we should keep a point.
This would remove the ability to use Airflow's Dockerfile as a supported
framework for producing size-optimised custom images. Users with those
requirements can maintain their own multi-stage Dockerfiles. I don't think
Apache Airflow needs to own that general-purpose build framework.
-ash
