Personally I'd prefer option A, with C being a close second. Transitioning to hardened images is commendable but I think is generally something that should be clearly communicated ahead of time. So to me the ideal timeline would be something like:
1. Starting with version 3.x.0 we're going to offer hardened images along with the default and slim images. Users should test them thoroughly, as we plan on making them the default, phasing-out the current default images. 2. Starting with version 3.x+2.0 hardened is the default. The previous default image format will be renamed to legacy and gets a deprecation warning. Users should switch their build environments to the hardened images. 3. Starting with version 3.x+4.0 we no longer offer official legacy builds. Users can still build legacy images on their own, but should be aware that this build chain will be removed from the Dockerfile in the future. 4. Starting with version 3.x+6.0 the legacy image can no longer be built with the official Dockerfile. The reason I'm suggesting this careful and admittedly lengthy rollout, is that I suspect that there is a portion of users for whom maintenance of the Airflow deployment is a bit of a side-show. Something that mostly works, chugs along in the background and only gets more attention if it breaks. I suspect this because I was in this camp for quite a while. This staged approach gets new users for the hardened image in batches. 1. Gets the early adopters. The ones that pay fairly close attention to Airflow development. 2. Gets the attention of those users that don't follow Airflow development without forcing them to act immediately. But having to switch to legacy means they have been made aware. 3. By this point most users should have switched, except for a few dawdlers. 4. If you haven't switched by now, you're on your own. The other reason I urge caution when changing the build system is that I perceive it as somewhat fragile to begin with. I've had builds of older versions fail for no immediately apparent reason. Just a mixture a apt package versions no longer being available, python packages not pinning dependencies and pulling incompatible versions, python packages pinning dependencies but ending up with a dependency version that doesn't match the version of a required native library available in the repos. I think ideally we'd have a reproducible build setup, where every version is pinned, be it native or python package, and no matter where or when you build an image for any Airflow version, the result is always the same. Obviously that means an image _won't_ automatically receive security patches just by rebuilding. But the alternative is what Jarek already mentioned; builds randomly failing through no fault of the user or the maintainers of the airflow image. Cheers, Philip On 2026/11/05 03:49:23 Jarek Potiuk wrote: > To be perfectly honest, if we—as a community - want to take the risk and > someone is ready to monitor issues related to it, B is also an acceptable > option - even though I agree with Przemek that A would be safer. > > Good thing is that we have the option now. We can **always** do it - and > even retro-actively regenerate and publish such images. And we can treat it > as a "reality check"—that is, how many people will start "shouting at us" > because we broke their deployment. > > As long as we vote on it as a community and collectively take > responsibility to monitor it, respond to people's issues, and react when we > see it causing problems for many of our users, we will accept the risk that > they walk out from Airflow rather than reporting issues to us. (We might > become victims of "survivorship bias" and simply not hear from those who > decided not to upgrade because it caused them a problem.). > > This is the question of the collective risk we are willing to accept. We > can **just** leave instructions in the code how to enable the legacy build, > how to rebuild it - so that - no matter if I or anyone else will be > available, when it turns on we **need** to bring the images quickly - one > of the maintainers will take on the task of re-publishing those. > > I remember what happened when MySQL broke their certificates (twice), which > caused all our images to stop working (including those that our users just > extend from)—we had a small tsunami of issues raised by people, and I spent > almost the whole weekend recovering it. I certainly do not want that to > happen again. > > And this is really what we are talking about: rather than building a > "fallback solution" now - and telling our users how to use it, we defer it > until the moment it happens (hoping it won't). It's a bit similar to > "curing the disease when it happens" versus "preventing the disease by > taking some precautions that cost us little - but certain time". We are > trading a "certain" small investment now.+ some maintenance overhead for a > "maybe" reactive approach that might cost much more (in terms of stress, > potentially lost users, and plain time spent on fixing it). I would argue > that, finally, maintaining even complex "supplementary" code - such as > Dockerfiles—is several orders of magnitude cheaper with AI than it used to > be. > > If those who vote "yes" accept that they might spend some hours responding > to similar issues and rebuilding/regenerating the images, I am fine with B. > Though my usual approach (A) is to "anticipate and prevent issues" - > because I know that reactive time in the future, even if not very probable, > costs a lot of my personal time that could otherwise be used for example to > meet with friends. If I know I will not be the one scrambling to get old > images back because of a sudden issue, I am happy. > > This is what +1 / -1 here really means. I am happy to call for a vote if we > think the discussion is finished. > > J. > > > On Wed, Nov 4, 2026 at 7:21 PM Vikram Koka via dev <[email protected]> > wrote: > > > This is interesting and the 14M from last month is significantly lower than > > I expected. > > > > I agree that a fair number of users have platform teams for Airflow. From > > what I seen, most of them build the docker images for their internal teams > > so that they have precise controls including scanning the images for > > security reasons. > > > > Especially with the introduction of multiple languages and the decoupled > > Airflow core and Task SDKs in multiple languages, I would expect this > > trend to accelerate. > > > > Based on these, I think we need to revise our approach to be more > > lightweight. > > Therefore, I lean towards option B. > > > > Vikram > > > > > > On Wed, Nov 4, 2026 at 12:20 PM Przemysław Mirowski via dev < > > [email protected]> wrote: > > > > > I would vote for option A. > > > > > > > Many users extend our base images with custom system and Python > > > > dependencies. Upgrading the base OS changes library versions and tool > > > > behavior, which may break some extended images. A -legacy tag allows > > > > users to upgrade Airflow first and the OS later. However, maintaining > > > > this requires building, testing, and releasing a second set of images > > > > for every 3.4.x release, with the risk that users stall migration > > > > until 3.5.0 (when bookworm support is dropped entirely). > > > > > > Users may stall with migration if we publish legacy images, but without > > > them, we take away from users the possibility to do gradual changes on > > > environments. First Airflow version to 3.4 and after that separate > > > modification to DHI-based image. I assume that many users can have > > > different issues, and it may be hard for us to predict all of them; > > making > > > that separation possible should: > > > 1. make upgrading to 3.4.0 for users easier, which should mean quicker > > > feedback (and adoption) > > > 2. easier separation and debugging of potential problems in the > > > environments thanks to separation of modifications introduced to the > > > environments > > > 3. users will have an easy path back in case of things not working after > > > environment upgrade (no Airflow version revert needed) > > > > > > I think that if we do not publish legacy images, we will see slower > > > adoption of the 3.4 version, which, I would say, is rather unwanted. > > > > > > The second part, which Jarek mentioned already, is stabilisation. This > > can > > > be on users' side and on ours too. We already encountered a couple of > > > issues with new images, and I would assume that we will encounter more. > > > Having both images published, we will have some safeguard if things go > > bad > > > with DHI. > > > > > > > - Vikram suggested checking usage data first. However, Docker Hub > > > > only provides total pull counts (~14M in September), not > > > > version-specific or usage breakdown, so community reports remain our > > > > primary data source. > > > > > > We have to remember that these data will not be reliable and are missing > > > context. Not reliable, because there are users who are using internal > > > registries to which images from DockerHub are copied. We do not, and will > > > not, have data on what is happening there. Missing context to what is > > > mentioned already (no version specific, etc.) > > > > > > > DHI is relatively new, so in theory it'll become more stable as time > > > goes on. > > > > > > I agree, but the question is how long it can potentially take. We don't > > > know the answer to that question. It can be a month or a year. > > > > > > There is also one thing. Some users who are running Airflow may have a > > > dedicated platform team for Airflow and some number of users. Having a > > > possible split of business/platform change (Airflow version upgrade) and > > > platform change (DHI) can make planning easier (not sure tho if we are > > > looking at things like that 🤔, but I would consider organisational > > > approach and people there too). > > > > > > I have nothing against dropping legacy with 3.5 release (at least now, it > > > is hard to predict what situation will be in a couple months). > > > > > > Best, > > > Przemek > > > > > > On 2026/11/02 12:13:02 Ash Berlin-Taylor via dev wrote: > > > > Brain fart, sorry, I meant “which problems are these” but I have since > > > seen your reply in the thread where I proposed simplification. > > > > > > > > > On 2 Nov 2026, at 11:45, Jarek Potiuk <[email protected]> wrote: > > > > > > > > > > The ones we build on? > > > > > > > > > > Docker Hardened Images (Python images). Generally wdely accepted as a > > > way > > > > > how to keep your images as secure as possible (they are patching > > their > > > > > images up to few times a day). > > > > > > > > > > They are freely available and free for commercial use - Apache 2.0 > > > licensed > > > > > (these are the community images we are using). > > > > > > > > > > They also have commercial FIPS compliant drop-in replacements, which > > > people > > > > > can use as base images when following our 'build your own image' > > > recipe. > > > > > > > > > > This is a bonus because our users can easily achieve FIPS compliance > > > and > > > > > users often asked for it. So we do not have to provide it - they can > > > pay > > > > > Docker for access to those FIPS compliant replacement base images. > > > > > > > > > > More info: > > > > > > > > > > Hardened Images | Docker https://share.google/J7g1i0JXzdUGHkW6G > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 2, 2026, 12:22 Ash Berlin-Taylor via dev < > > > [email protected]> > > > > > wrote: > > > > > > > > > >> What images are these? > > > > >> > > > > >>> On 1 Nov 2026, at 22:06, Jarek Potiuk <[email protected]> wrote: > > > > >>> > > > > >>> Actually - after seeing the kind of issues we faced even after my > > > last > > > > >>> email 2 days ago -> the issues (and complexity of solutions) might > > be > > > > >>> somewhat intimidating to our users > > > > >>> > > > > >>> I personally think we should provide a bridge for our users. > > > > >>> > > > > >>> While I **strongly** believe DHI approach is good and has strong > > > security > > > > >>> implications, some of my users for 3.4.* would prefer to use the > > > > >> "original" > > > > >>> setup. > > > > >>> > > > > >>> I am fine for 3.5 - to drop all kind of legacy images, but I think > > > that > > > > >>> might be to ""steep** change - and being able to just switch to > > `FROM > > > > >>> apache/aifrlow:legacy-3.11` is an easy way to address all kinds of > > > > >>> incompatibilities. > > > > >>> > > > > >>> If we got (and we did) 3 library compatibility issues that > > required a > > > > >>> change in how the image uses the base DHI image - our users will > > have > > > > >> even > > > > >>> more of that - and they will reach out to us. > > > > >>> > > > > >>> Are we ready to handle our user's problems? > > > > >>> > > > > >>> For me - the easiest approach would be: > > > > >>> > > > > >>> * here are some general ways and you might follow it > > > > >>> * but if you have no time/energy to follow it - the legacy image > > will > > > > >> keep > > > > >>> you running. > > > > >>> > > > > >>> I would always ask anyone who says "drop it": Are you ready to > > solve > > > > >>> problems of our users if they see problems? > > > > >>> > > > > >>> My answer is: Yes - try to fix it following our vague instructions, > > > if > > > > >> you > > > > >>> fail, use the legacy image for an easy fallback. > > > > >>> > > > > >>> What is the answer if we decide not to publish legacy image ? Ash, > > > Jed? > > > > >>> what would you answer to the users? (and I am not precluding any > > > answer - > > > > >>> we just have to remember that this should be the answer PMC is > > firmly > > > > >>> behind, no matter who is answering it) > > > > >>> > > > > >>> On Fri, Oct 30, 2026 at 7:02 PM Jed Cunningham < > > > [email protected] > > > > >>> > > > > >>> wrote: > > > > >>> > > > > >>>> I still lean toward B, myself. My take is that leaving C as an > > > option > > > > >>>> injects some uncertainty for both sides, and offering legacy > > flavors > > > > >> just > > > > >>>> kicks the can down the road. > > > > >>>> > > > > >>>> DHI is relatively new, so in theory it'll become more stable as > > time > > > > >> goes > > > > >>>> on. > > > > >>>> > > > > >> > > > > >> > > > > >> > > --------------------------------------------------------------------- > > > > >> To unsubscribe, e-mail: [email protected] > > > > >> For additional commands, e-mail: [email protected] > > > > >> > > > > >> > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -- <div style="font-size:10pt"><p><b>Philip Abernethy</b Web Services</p> <p>BELENUS LOB Informatic GmbH<br> Rüdesheimer Straße 23 | DE - 80686 München<br> Tel: +49 (0) 89 / 547 15 - xxx | Fax: +49 (0) 89 / 547 15 - xxx | E-Mail: [email protected]</p> <p>www.lexcom.de</p><div> <footer> <hr> <div style="font-size:8pt"><div style="color:Gainsboro"><p>Geschäftsführer: Dipl.-Kaufmann Johannes Lex<br> Handelsregister München HRB 120536</p> <p>Die in dieser E-Mail gegebene Information ist nicht rechtsverbindlich. Nicht autorisierte Nutzung, Veröffentlichung, Vervielfältigung oder Weitergabe des Inhalts ist nicht gestattet. Die Nachricht ist nur für den bezeichneten Adressaten bestimmt. Sollten Sie nicht der vorgesehene Adressat sein, setzen Sie sich bitte mit dem Absender der E-Mail unverzüglich in Verbindung.</p> <p>For legal and security reasons the information provided in this e-mail is not legally binding. Any form of unauthorised use, publication, reproduction, copying or disclosure of the content of this e-mail is not permitted. This message is exclusively for the person addressed or their representative. If you are not the intended recipient of this message and its contents, please notify the sender immediately.</p></div> <div style="color:ForestGreen"><p>Bitte prüfen Sie der Umwelt zuliebe, ob der Ausdruck dieser Mail erforderlich ist.</p> <p>Save ink and paper, print this message only if necessary.</p></div></div> This Email was scanned by proofpoint
