Still Even TP had doubts how "stable" it is - I think it's good if **we** make a decision and when users ask we can refer them to that discussion. This is a user facing feature we are taking away.
This is what I often do when users ask "why we don't have that option any more" - same for example when Postgres stopped working and we explain that they won't be able to use it any more - I refer them not to "because my PR" but "because we discussed it in community and decision has been made by the community - if only by not opposing". I think we should always do it when we tell our users they can do something and we are taking away that possibility (which is clearly the case). 2 years ago they had a "mysql" client by default, we changed it to MariaDB by default, but left them with the possibility of building "our" image with MySQL drivers and not only gave them instructions, but also ran a CI job testing it. Now we are taking away that option. I think it's worth having a clear record of it that we can refer to - including context and why we are making this decision - and this is what "devlist' is also about - keeping track and archiving of such decisions. J. On Fri, Oct 24, 2025 at 10:46 PM Ash Berlin-Taylor <[email protected]> wrote: > My point is that a PR is enough for this to my mind. > > > On 24 Oct 2025, at 21:43, Jarek Potiuk <[email protected]> wrote: > > > > Still I think we change things we explicitly told our users they can use > > ... if someone asks, it should not be a single person decision, it should > > be something we as community agreed to - if only no-one opposed. > > > > It's not "Ash said it's ok" or "Jarek said it's ok". As a community we > > agreed to it. > > > > This is what decision making in the ASF project looks like. Governance is > > important. > > > > J > > > > On Fri, Oct 24, 2025 at 10:35 PM Ash Berlin-Taylor <[email protected]> > wrote: > > > >> I don’t think even a lazy consensus is needed. > >> > >> If someone wants to use the Mysql client library specifically, let them > >> build their own docker image from scratch. We don’t have to, nor can > we, > >> cater for every possible whim in our docker images. > >> > >> The Astronomer Runtime images have been built using > >> https://packages.debian.org/trixie/default-libmysqlclient-dev which > >> gives us mariadb (and the same on bookworm etc) for years, and none of > our > >> users have noticed (and yes, it _is_ being used, I checked that) > >> > >> Cheers, > >> -ash > >> > >>> On 24 Oct 2025, at 18:41, Jarek Potiuk <[email protected]> wrote: > >>> > >>> If there are no comments or objections - I will start a lazy consensus > >>> about it on Monday. MySQL client is already removed in main (to unblock > >>> CI) - so if lazy consensus passes, it will stay like that. > >>> > >>> On Thu, Oct 23, 2025 at 12:56 PM Jarek Potiuk <[email protected]> > wrote: > >>> > >>>>> And if there are issues, would we be able to fix them in Airflow? > >>>> > >>>> No idea - we had no issues of that sort. Yet it's the third time we > >>>> have issues with Oracle drivers that we actually know we are not able > to > >>>> fix without Oracle fixing their side. > >>>> > >>>> On Thu, Oct 23, 2025 at 12:52 PM Jarek Potiuk <[email protected]> > wrote: > >>>> > >>>>> PR to remove mysql client is here - > >>>>> https://github.com/apache/airflow/pull/57146 . We should merge it > now > >> to > >>>>> fix our canary CI builds. We can revert it later if we will find ways > >> to > >>>>> make MySQL work properly (though I fail to see how it can be achieved > >> while > >>>>> Oracle does what it does). > >>>>> > >>>>> On Thu, Oct 23, 2025 at 12:45 PM Jarek Potiuk <[email protected]> > >> wrote: > >>>>> > >>>>>>> What’s the current compatibility situation between MySQL and > MariaDB? > >>>>>> Would the OSS client be able to connect to Oracle-provided MySQL > >> server > >>>>>> software? And if there are issues, would we be able to fix them in > >> Airflow? > >>>>>> > >>>>>> As explained above - for two years CI and PROD images of Airflow use > >>>>>> MariaDB client. There were zero issues reported about compatibility. > >>>>>> > >>>>>> On Thu, Oct 23, 2025 at 12:37 PM Tzu-ping Chung via dev < > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>>> What’s the current compatibility situation between MySQL and > MariaDB? > >>>>>>> Would the OSS client be able to connect to Oracle-provided MySQL > >> server > >>>>>>> software? And if there are issues, would we be able to fix them in > >> Airflow? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> On 23 Oct 2025, at 18:25, Jarek Potiuk <[email protected]> wrote: > >>>>>>>> > >>>>>>>> Hello Everyone, > >>>>>>>> > >>>>>>>> *TL;DR; I would like to propose complete dropping of MySQL "Oracle > >>>>>>>> published" client libraries from our container images.* > >>>>>>>> > >>>>>>>> Two years ago [1] - we switched to MariaDB clients by default > >> because > >>>>>>> of a > >>>>>>>> very convoluted (and plain wrong) approach of Oracle for their Apt > >>>>>>>> repositories and we are back to the situation we faced 2 years > ago. > >>>>>>>> > >>>>>>>> We protected (nicely) against total disaster (where I had to > >> manually > >>>>>>> build > >>>>>>>> and push 100 of our broken images during the weekend) and switched > >> to > >>>>>>>> MariaDB by default. We still left the option to build the image > with > >>>>>>> MySQL > >>>>>>>> client and we still run it in our CI - this is how we found > tonight > >>>>>>>> that the problem is back, Luckily all that is needed is we need to > >>>>>>> drop the > >>>>>>>> optional support we have for MySQL images described in [2]. It > >>>>>>> requires the > >>>>>>>> users who wish to use MySQL client to build the images using our > >>>>>>>> Dockerfiles with specific build arguments. We kept it for > >>>>>>> compatibility and > >>>>>>>> convenience of those who would have to use the clients. We never > >>>>>>> heard back > >>>>>>>> from anyone if they are using or not - it's very likely, it's used > >>>>>>>> extremely rarely (if at all). > >>>>>>>> > >>>>>>>> The problem is (and you can find many articles, stack overflow > >>>>>>> issues, blog > >>>>>>>> posts about it) that Oracle uses a very convoluted and wrong way > of > >>>>>>> making > >>>>>>>> their apt packages available - they sign their packages and repos > >> with > >>>>>>>> expiring keys. No other company I know is using this, this is > >> against > >>>>>>>> debian recommendations and every two years it causes the same > >> problem > >>>>>>> - the > >>>>>>>> old packages are not installable, images released in the past that > >>>>>>> have > >>>>>>>> their repos added are blocked from installing **anything** (i.e. > apt > >>>>>>>> install fails to install anything unless you remove oracle's repos > >> and > >>>>>>>> keys) > >>>>>>>> > >>>>>>>> Just to be clear - this is (so far) not a problem for the server > >> side > >>>>>>> of > >>>>>>>> MySQL. We are ok with our tests where MySQL is used as a server - > >>>>>>> because > >>>>>>>> we can use images they publish to run the servers) and MariaDB > >>>>>>> clients work > >>>>>>>> well with those. But for MySQL clients - every 2 years (it's > already > >>>>>>> the > >>>>>>>> 3rd time it happened) it makes our images and dockerfiles broken - > >>>>>>> and our > >>>>>>>> users who want to use the clients - scrambling to install those. > >>>>>>>> > >>>>>>>> To add to that - when it happens, Oracle is surprised. Always. No > >>>>>>>> exception. It happened tonight and as of tonight, you have no way > >>>>>>>> installing the packages at all (even if you somehow get hold of > the > >>>>>>> new > >>>>>>>> key) because their repos are signed with old, expired keys - 2 > years > >>>>>>> ago it > >>>>>>>> took them almost a week to fix it and the bug created [3] was > >> created > >>>>>>> where > >>>>>>>> I - among others explained them the problem they had and what > >>>>>>> solution they > >>>>>>>> can apply. > >>>>>>>> > >>>>>>>> They ignored it. Today the story repeated itself. MySQL clients > >>>>>>> stopped > >>>>>>>> installing - because they are signed by expired keys and their > repo > >>>>>>> is also > >>>>>>>> signed by the same expired key. They have learned nothing and did > >> not > >>>>>>> fix > >>>>>>>> the problem. They will have another sh**tstom coming and will > >>>>>>> scramble to > >>>>>>>> fix it again. > >>>>>>>> > >>>>>>>> But I do not wish our community (and me particularly) to be part > of > >>>>>>> it any > >>>>>>>> more - my proposal is to simply drop that option and let the > users > >>>>>>> be on > >>>>>>>> their own if they want to use MySQL client. > >>>>>>>> > >>>>>>>> I will proceed with removing it now in a PR (completely) so that > we > >>>>>>> fix our > >>>>>>>> failing canary builds and If no-one objects, I will call for LAZY > >>>>>>> CONSENSUS > >>>>>>>> and will not revert the change. > >>>>>>>> > >>>>>>>> J. > >>>>>>>> > >>>>>>>> [1] Lazy consensus from 2023 > >>>>>>>> https://lists.apache.org/thread/rxbyxg11jg7y35k8om0f8wgb2l9h459l > >>>>>>>> [2] Optional support for MySQL clients > >>>>>>>> > >>>>>>> > >> > https://airflow.apache.org/docs/docker-stack/build.html#building-images-with-mysql-client > >>>>>>>> [3] Bug from 2 years ago - > https://bugs.mysql.com/bug.php?id=113432 > >>>>>>> > >>>>>>> > >>>>>>> > --------------------------------------------------------------------- > >>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>> For additional commands, e-mail: [email protected] > >>>>>>> > >>>>>>> > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
