If there are no comments or objections - I will start a lazy consensus about it on Monday. MySQL client is already removed in main (to unblock CI) - so if lazy consensus passes, it will stay like that.
On Thu, Oct 23, 2025 at 12:56 PM Jarek Potiuk <[email protected]> wrote: > > And if there are issues, would we be able to fix them in Airflow? > > No idea - we had no issues of that sort. Yet it's the third time we > have issues with Oracle drivers that we actually know we are not able to > fix without Oracle fixing their side. > > On Thu, Oct 23, 2025 at 12:52 PM Jarek Potiuk <[email protected]> wrote: > >> PR to remove mysql client is here - >> https://github.com/apache/airflow/pull/57146 . We should merge it now to >> fix our canary CI builds. We can revert it later if we will find ways to >> make MySQL work properly (though I fail to see how it can be achieved while >> Oracle does what it does). >> >> On Thu, Oct 23, 2025 at 12:45 PM Jarek Potiuk <[email protected]> wrote: >> >>> > What’s the current compatibility situation between MySQL and MariaDB? >>> Would the OSS client be able to connect to Oracle-provided MySQL server >>> software? And if there are issues, would we be able to fix them in Airflow? >>> >>> As explained above - for two years CI and PROD images of Airflow use >>> MariaDB client. There were zero issues reported about compatibility. >>> >>> On Thu, Oct 23, 2025 at 12:37 PM Tzu-ping Chung via dev < >>> [email protected]> wrote: >>> >>>> What’s the current compatibility situation between MySQL and MariaDB? >>>> Would the OSS client be able to connect to Oracle-provided MySQL server >>>> software? And if there are issues, would we be able to fix them in Airflow? >>>> >>>> >>>> >>>> > On 23 Oct 2025, at 18:25, Jarek Potiuk <[email protected]> wrote: >>>> > >>>> > Hello Everyone, >>>> > >>>> > *TL;DR; I would like to propose complete dropping of MySQL "Oracle >>>> > published" client libraries from our container images.* >>>> > >>>> > Two years ago [1] - we switched to MariaDB clients by default because >>>> of a >>>> > very convoluted (and plain wrong) approach of Oracle for their Apt >>>> > repositories and we are back to the situation we faced 2 years ago. >>>> > >>>> > We protected (nicely) against total disaster (where I had to manually >>>> build >>>> > and push 100 of our broken images during the weekend) and switched to >>>> > MariaDB by default. We still left the option to build the image with >>>> MySQL >>>> > client and we still run it in our CI - this is how we found tonight >>>> > that the problem is back, Luckily all that is needed is we need to >>>> drop the >>>> > optional support we have for MySQL images described in [2]. It >>>> requires the >>>> > users who wish to use MySQL client to build the images using our >>>> > Dockerfiles with specific build arguments. We kept it for >>>> compatibility and >>>> > convenience of those who would have to use the clients. We never >>>> heard back >>>> > from anyone if they are using or not - it's very likely, it's used >>>> > extremely rarely (if at all). >>>> > >>>> > The problem is (and you can find many articles, stack overflow >>>> issues, blog >>>> > posts about it) that Oracle uses a very convoluted and wrong way of >>>> making >>>> > their apt packages available - they sign their packages and repos with >>>> > expiring keys. No other company I know is using this, this is against >>>> > debian recommendations and every two years it causes the same problem >>>> - the >>>> > old packages are not installable, images released in the past that >>>> have >>>> > their repos added are blocked from installing **anything** (i.e. apt >>>> > install fails to install anything unless you remove oracle's repos and >>>> > keys) >>>> > >>>> > Just to be clear - this is (so far) not a problem for the server side >>>> of >>>> > MySQL. We are ok with our tests where MySQL is used as a server - >>>> because >>>> > we can use images they publish to run the servers) and MariaDB >>>> clients work >>>> > well with those. But for MySQL clients - every 2 years (it's already >>>> the >>>> > 3rd time it happened) it makes our images and dockerfiles broken - >>>> and our >>>> > users who want to use the clients - scrambling to install those. >>>> > >>>> > To add to that - when it happens, Oracle is surprised. Always. No >>>> > exception. It happened tonight and as of tonight, you have no way >>>> > installing the packages at all (even if you somehow get hold of the >>>> new >>>> > key) because their repos are signed with old, expired keys - 2 years >>>> ago it >>>> > took them almost a week to fix it and the bug created [3] was created >>>> where >>>> > I - among others explained them the problem they had and what >>>> solution they >>>> > can apply. >>>> > >>>> > They ignored it. Today the story repeated itself. MySQL clients >>>> stopped >>>> > installing - because they are signed by expired keys and their repo >>>> is also >>>> > signed by the same expired key. They have learned nothing and did not >>>> fix >>>> > the problem. They will have another sh**tstom coming and will >>>> scramble to >>>> > fix it again. >>>> > >>>> > But I do not wish our community (and me particularly) to be part of >>>> it any >>>> > more - my proposal is to simply drop that option and let the users >>>> be on >>>> > their own if they want to use MySQL client. >>>> > >>>> > I will proceed with removing it now in a PR (completely) so that we >>>> fix our >>>> > failing canary builds and If no-one objects, I will call for LAZY >>>> CONSENSUS >>>> > and will not revert the change. >>>> > >>>> > J. >>>> > >>>> > [1] Lazy consensus from 2023 >>>> > https://lists.apache.org/thread/rxbyxg11jg7y35k8om0f8wgb2l9h459l >>>> > [2] Optional support for MySQL clients >>>> > >>>> https://airflow.apache.org/docs/docker-stack/build.html#building-images-with-mysql-client >>>> > [3] Bug from 2 years ago - https://bugs.mysql.com/bug.php?id=113432 >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>>
