> And if there are issues, would we be able to fix them in Airflow? No idea - we had no issues of that sort. Yet it's the third time we have issues with Oracle drivers that we actually know we are not able to fix without Oracle fixing their side.
On Thu, Oct 23, 2025 at 12:52 PM Jarek Potiuk <[email protected]> wrote: > PR to remove mysql client is here - > https://github.com/apache/airflow/pull/57146 . We should merge it now to > fix our canary CI builds. We can revert it later if we will find ways to > make MySQL work properly (though I fail to see how it can be achieved while > Oracle does what it does). > > On Thu, Oct 23, 2025 at 12:45 PM Jarek Potiuk <[email protected]> wrote: > >> > What’s the current compatibility situation between MySQL and MariaDB? >> Would the OSS client be able to connect to Oracle-provided MySQL server >> software? And if there are issues, would we be able to fix them in Airflow? >> >> As explained above - for two years CI and PROD images of Airflow use >> MariaDB client. There were zero issues reported about compatibility. >> >> On Thu, Oct 23, 2025 at 12:37 PM Tzu-ping Chung via dev < >> [email protected]> wrote: >> >>> What’s the current compatibility situation between MySQL and MariaDB? >>> Would the OSS client be able to connect to Oracle-provided MySQL server >>> software? And if there are issues, would we be able to fix them in Airflow? >>> >>> >>> >>> > On 23 Oct 2025, at 18:25, Jarek Potiuk <[email protected]> wrote: >>> > >>> > Hello Everyone, >>> > >>> > *TL;DR; I would like to propose complete dropping of MySQL "Oracle >>> > published" client libraries from our container images.* >>> > >>> > Two years ago [1] - we switched to MariaDB clients by default because >>> of a >>> > very convoluted (and plain wrong) approach of Oracle for their Apt >>> > repositories and we are back to the situation we faced 2 years ago. >>> > >>> > We protected (nicely) against total disaster (where I had to manually >>> build >>> > and push 100 of our broken images during the weekend) and switched to >>> > MariaDB by default. We still left the option to build the image with >>> MySQL >>> > client and we still run it in our CI - this is how we found tonight >>> > that the problem is back, Luckily all that is needed is we need to >>> drop the >>> > optional support we have for MySQL images described in [2]. It >>> requires the >>> > users who wish to use MySQL client to build the images using our >>> > Dockerfiles with specific build arguments. We kept it for >>> compatibility and >>> > convenience of those who would have to use the clients. We never heard >>> back >>> > from anyone if they are using or not - it's very likely, it's used >>> > extremely rarely (if at all). >>> > >>> > The problem is (and you can find many articles, stack overflow issues, >>> blog >>> > posts about it) that Oracle uses a very convoluted and wrong way of >>> making >>> > their apt packages available - they sign their packages and repos with >>> > expiring keys. No other company I know is using this, this is against >>> > debian recommendations and every two years it causes the same problem >>> - the >>> > old packages are not installable, images released in the past that have >>> > their repos added are blocked from installing **anything** (i.e. apt >>> > install fails to install anything unless you remove oracle's repos and >>> > keys) >>> > >>> > Just to be clear - this is (so far) not a problem for the server side >>> of >>> > MySQL. We are ok with our tests where MySQL is used as a server - >>> because >>> > we can use images they publish to run the servers) and MariaDB clients >>> work >>> > well with those. But for MySQL clients - every 2 years (it's already >>> the >>> > 3rd time it happened) it makes our images and dockerfiles broken - and >>> our >>> > users who want to use the clients - scrambling to install those. >>> > >>> > To add to that - when it happens, Oracle is surprised. Always. No >>> > exception. It happened tonight and as of tonight, you have no way >>> > installing the packages at all (even if you somehow get hold of the new >>> > key) because their repos are signed with old, expired keys - 2 years >>> ago it >>> > took them almost a week to fix it and the bug created [3] was created >>> where >>> > I - among others explained them the problem they had and what solution >>> they >>> > can apply. >>> > >>> > They ignored it. Today the story repeated itself. MySQL clients stopped >>> > installing - because they are signed by expired keys and their repo is >>> also >>> > signed by the same expired key. They have learned nothing and did not >>> fix >>> > the problem. They will have another sh**tstom coming and will scramble >>> to >>> > fix it again. >>> > >>> > But I do not wish our community (and me particularly) to be part of it >>> any >>> > more - my proposal is to simply drop that option and let the users be >>> on >>> > their own if they want to use MySQL client. >>> > >>> > I will proceed with removing it now in a PR (completely) so that we >>> fix our >>> > failing canary builds and If no-one objects, I will call for LAZY >>> CONSENSUS >>> > and will not revert the change. >>> > >>> > J. >>> > >>> > [1] Lazy consensus from 2023 >>> > https://lists.apache.org/thread/rxbyxg11jg7y35k8om0f8wgb2l9h459l >>> > [2] Optional support for MySQL clients >>> > >>> https://airflow.apache.org/docs/docker-stack/build.html#building-images-with-mysql-client >>> > [3] Bug from 2 years ago - https://bugs.mysql.com/bug.php?id=113432 >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>>
