I don’t think even a lazy consensus is needed. If someone wants to use the Mysql client library specifically, let them build their own docker image from scratch. We don’t have to, nor can we, cater for every possible whim in our docker images.
The Astronomer Runtime images have been built using https://packages.debian.org/trixie/default-libmysqlclient-dev which gives us mariadb (and the same on bookworm etc) for years, and none of our users have noticed (and yes, it _is_ being used, I checked that) Cheers, -ash > On 24 Oct 2025, at 18:41, Jarek Potiuk <[email protected]> wrote: > > If there are no comments or objections - I will start a lazy consensus > about it on Monday. MySQL client is already removed in main (to unblock > CI) - so if lazy consensus passes, it will stay like that. > > On Thu, Oct 23, 2025 at 12:56 PM Jarek Potiuk <[email protected]> wrote: > >>> And if there are issues, would we be able to fix them in Airflow? >> >> No idea - we had no issues of that sort. Yet it's the third time we >> have issues with Oracle drivers that we actually know we are not able to >> fix without Oracle fixing their side. >> >> On Thu, Oct 23, 2025 at 12:52 PM Jarek Potiuk <[email protected]> wrote: >> >>> PR to remove mysql client is here - >>> https://github.com/apache/airflow/pull/57146 . We should merge it now to >>> fix our canary CI builds. We can revert it later if we will find ways to >>> make MySQL work properly (though I fail to see how it can be achieved while >>> Oracle does what it does). >>> >>> On Thu, Oct 23, 2025 at 12:45 PM Jarek Potiuk <[email protected]> wrote: >>> >>>>> What’s the current compatibility situation between MySQL and MariaDB? >>>> Would the OSS client be able to connect to Oracle-provided MySQL server >>>> software? And if there are issues, would we be able to fix them in Airflow? >>>> >>>> As explained above - for two years CI and PROD images of Airflow use >>>> MariaDB client. There were zero issues reported about compatibility. >>>> >>>> On Thu, Oct 23, 2025 at 12:37 PM Tzu-ping Chung via dev < >>>> [email protected]> wrote: >>>> >>>>> What’s the current compatibility situation between MySQL and MariaDB? >>>>> Would the OSS client be able to connect to Oracle-provided MySQL server >>>>> software? And if there are issues, would we be able to fix them in >>>>> Airflow? >>>>> >>>>> >>>>> >>>>>> On 23 Oct 2025, at 18:25, Jarek Potiuk <[email protected]> wrote: >>>>>> >>>>>> Hello Everyone, >>>>>> >>>>>> *TL;DR; I would like to propose complete dropping of MySQL "Oracle >>>>>> published" client libraries from our container images.* >>>>>> >>>>>> Two years ago [1] - we switched to MariaDB clients by default because >>>>> of a >>>>>> very convoluted (and plain wrong) approach of Oracle for their Apt >>>>>> repositories and we are back to the situation we faced 2 years ago. >>>>>> >>>>>> We protected (nicely) against total disaster (where I had to manually >>>>> build >>>>>> and push 100 of our broken images during the weekend) and switched to >>>>>> MariaDB by default. We still left the option to build the image with >>>>> MySQL >>>>>> client and we still run it in our CI - this is how we found tonight >>>>>> that the problem is back, Luckily all that is needed is we need to >>>>> drop the >>>>>> optional support we have for MySQL images described in [2]. It >>>>> requires the >>>>>> users who wish to use MySQL client to build the images using our >>>>>> Dockerfiles with specific build arguments. We kept it for >>>>> compatibility and >>>>>> convenience of those who would have to use the clients. We never >>>>> heard back >>>>>> from anyone if they are using or not - it's very likely, it's used >>>>>> extremely rarely (if at all). >>>>>> >>>>>> The problem is (and you can find many articles, stack overflow >>>>> issues, blog >>>>>> posts about it) that Oracle uses a very convoluted and wrong way of >>>>> making >>>>>> their apt packages available - they sign their packages and repos with >>>>>> expiring keys. No other company I know is using this, this is against >>>>>> debian recommendations and every two years it causes the same problem >>>>> - the >>>>>> old packages are not installable, images released in the past that >>>>> have >>>>>> their repos added are blocked from installing **anything** (i.e. apt >>>>>> install fails to install anything unless you remove oracle's repos and >>>>>> keys) >>>>>> >>>>>> Just to be clear - this is (so far) not a problem for the server side >>>>> of >>>>>> MySQL. We are ok with our tests where MySQL is used as a server - >>>>> because >>>>>> we can use images they publish to run the servers) and MariaDB >>>>> clients work >>>>>> well with those. But for MySQL clients - every 2 years (it's already >>>>> the >>>>>> 3rd time it happened) it makes our images and dockerfiles broken - >>>>> and our >>>>>> users who want to use the clients - scrambling to install those. >>>>>> >>>>>> To add to that - when it happens, Oracle is surprised. Always. No >>>>>> exception. It happened tonight and as of tonight, you have no way >>>>>> installing the packages at all (even if you somehow get hold of the >>>>> new >>>>>> key) because their repos are signed with old, expired keys - 2 years >>>>> ago it >>>>>> took them almost a week to fix it and the bug created [3] was created >>>>> where >>>>>> I - among others explained them the problem they had and what >>>>> solution they >>>>>> can apply. >>>>>> >>>>>> They ignored it. Today the story repeated itself. MySQL clients >>>>> stopped >>>>>> installing - because they are signed by expired keys and their repo >>>>> is also >>>>>> signed by the same expired key. They have learned nothing and did not >>>>> fix >>>>>> the problem. They will have another sh**tstom coming and will >>>>> scramble to >>>>>> fix it again. >>>>>> >>>>>> But I do not wish our community (and me particularly) to be part of >>>>> it any >>>>>> more - my proposal is to simply drop that option and let the users >>>>> be on >>>>>> their own if they want to use MySQL client. >>>>>> >>>>>> I will proceed with removing it now in a PR (completely) so that we >>>>> fix our >>>>>> failing canary builds and If no-one objects, I will call for LAZY >>>>> CONSENSUS >>>>>> and will not revert the change. >>>>>> >>>>>> J. >>>>>> >>>>>> [1] Lazy consensus from 2023 >>>>>> https://lists.apache.org/thread/rxbyxg11jg7y35k8om0f8wgb2l9h459l >>>>>> [2] Optional support for MySQL clients >>>>>> >>>>> https://airflow.apache.org/docs/docker-stack/build.html#building-images-with-mysql-client >>>>>> [3] Bug from 2 years ago - https://bugs.mysql.com/bug.php?id=113432 >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>>
