Thanks Dongjoon’s reply and questions, > A. Adding a new Apache Spark community policy (contract) to guarantee MySQL > LTS Versions Support.
Yes, at least the latest MySQL LTS version. To reduce the maintenance efforts on the Spark side, I think we can only run CI with the latest LTS version but accept reasonable patches for compatibility with older LTS versions. For example, Spark on K8s is only verified with the latest minikube in CI, and also accepts reasonable patches for older K8s. > B. Dropping the support of non-LTS version support (MySQL 8.3/8.2/8.1) Those versions likely work well too. For example, Spark currently officially supports JDK 17 and 21, it likely works on JDK 20 too, but has not been verified by the community. > 1. For (A), do you mean MySQL LTS versions are not supported by Apache Spark > releases properly due to the improper test suite? Not yet. MySQL retains good backward compatibilities so far, I see a lot of users use MySQL 8.0 drivers to access both MySQL 5.7/8.0 servers through Spark JDBC datasource, everything goes well so far. > 2. For (B), why does Apache Spark need to drop non-LTS MySQL support? I think we can accept reasonable patches with careful review, but neither official support declaration nor CI verification is required, just like we do for JDK version support. > 3. What about MariaDB? Do we need to stick to some versions? I’m not familiar with MariaDB, but I would treat it as a MySQL-compatible product, in the same position as Amazon RDS for MySQL, neither official support declaration nor CI verification is required, but considering the adoption rate of those products, reasonable patches should be considered too. Thanks, Cheng Pan On 2024/03/25 06:47:10 Dongjoon Hyun wrote: > Hi, Cheng. > > Thank you for the suggestion. Your suggestion seems to have at least two > themes. > > A. Adding a new Apache Spark community policy (contract) to guarantee MySQL > LTS Versions Support. > B. Dropping the support of non-LTS version support (MySQL 8.3/8.2/8.1) > > And, it brings me three questions. > > 1. For (A), do you mean MySQL LTS versions are not supported by Apache > Spark releases properly due to the improper test suite? > 2. For (B), why does Apache Spark need to drop non-LTS MySQL support? > 3. What about MariaDB? Do we need to stick to some versions? > > To be clear, if needed, we can have daily GitHub Action CIs easily like > Python CI (Python 3.8/3.10/3.11/3.12). > > - > https://github.com/apache/spark/blob/master/.github/workflows/build_python.yml > > Thanks, > Dongjoon. > > > On Sun, Mar 24, 2024 at 10:29 PM Cheng Pan <pan3...@gmail.com> wrote: > > > Hi, Spark community, > > > > I noticed that the Spark JDBC connector MySQL dialect is testing against > > the 8.3.0[1] now, a non-LTS version. > > > > MySQL changed the version policy recently[2], which is now very similar to > > the Java version policy. In short, 5.5, 5.6, 5.7, 8.0 is the LTS version, > > 8.1, 8.2, 8.3 is non-LTS, and the next LTS version is 8.4. > > > > I would say that MySQL is one of the most important infrastructures today, > > I checked the AWS RDS MySQL[4] and Azure Database for MySQL[5] version > > support policy, and both only support 5.7 and 8.0. > > > > Also, Spark officially only supports LTS Java versions, like JDK 17 and > > 21, but not 22. I would recommend using MySQL 8.0 for testing until the > > next MySQL LTS version (8.4) is available. > > > > Additional discussion can be found at [3] > > > > [1] https://issues.apache.org/jira/browse/SPARK-47453 > > [2] > > https://dev.mysql.com/blog-archive/introducing-mysql-innovation-and-long-term-support-lts-versions/ > > [3] https://github.com/apache/spark/pull/45581 > > [4] https://aws.amazon.com/rds/mysql/ > > [5] https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy > > > > Thanks, > > Cheng Pan > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > > > > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
