In response, off topic from Jareks statement: is somebody (already) on the migration plan MsSQL->Postgres or do we need to find a volunteer not to delay 2.8 release? In a matter of preparing would be best if a "tool" is released wirh 2.7.3 else we need to habe at least a DB scheme upgrade supported for MsSQL in order to be able to migrade to a new DB engine in 2.8.0 w/o need of transformation of structures for people being stuck on 2.7.3 wirh MsSQL.
Sent from Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Wednesday, October 25, 2023 2:21:29 PM To: Amogh Desai <amoghdesai....@gmail.com> Cc: dev@airflow.apache.org <dev@airflow.apache.org> Subject: Re: Keep Mssql support > The alternatives suggested by @Jarek Potiuk <ja...@potiuk.com> are something which is doable. We need to realise that these plugins are for the community and we can only support it if "majority" of the community uses it and is willing to maintain it :) Just to clarify - it's not about plugins, it's about "core" metastore support for MSSQL. The MSSQL provider will remain as is, but Airflow will lose the ability of using MsSQL as backend for scheduling/tasks/UI/executors/communication with workers via DB. The overhead / extra maintenance there is to make sure that all the queries, performance, locking and anything else airflow uses to make "core" works with MsSQL. In Airflow 2.8 the only remaining backends will be MySQL 8 (as MySQL 5.7 reaches end of life in a week), and Postgres 12-16 (in 2 weeks Postgres 11 reaches EOL and we will drop it). Those will be the only choices for Airflow 2.8 - providing that we will remove MsSQL as planned (condition for that is that we describe a viable migration path for MsSQL users). J. On Wed, Oct 25, 2023 at 1:23 PM Amogh Desai <amoghdesai....@gmail.com> wrote: > I agree with the comments and where this discussion has led to. > > The alternatives suggested by @Jarek Potiuk <ja...@potiuk.com> are > something which is doable. We need to realise that these plugins are for > the community and we can only support it if > "majority" of the community uses it and is willing to maintain it :) > > > Thanks & Regards, > Amogh Desai > > On Wed, Oct 25, 2023 at 4:13 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > >> Yeah agreed, I don’t think it is worth keeping the support of MSSQL given >> the amount of usage vs the maintenance effort required. >> >> On Tue, 24 Oct 2023 at 23:07, Jarek Potiuk <ja...@potiuk.com> wrote: >> >> > Yes. Agree with Andrey. I think our experience from the last few years >> was >> > "very" bad. The number of mssql users is very small. And the time that >> > maintainers and community members lose on various problems with it is >> huge. >> > Quite often every time we added a new feature requiring some new db >> > functionality, quite a lot of overhead was spent by the one adding new >> > features to solve the problems coming just from MSSQL support. It's not >> > "existing" issues - it's that it generally slows us down with making >> > changes >> > >> > I think there are two options for you when you. It's not "open issues", >> > it's the maintenance >> > >> > * switch to another backend (recommended). And it's not as complex as >> you >> > think. You can also use managed DB with all that is needed >> > (backup/maintenance), you do not have to manage it yourself. There are >> some >> > excellent postgres options available. >> > >> > * have your own fork airflow and keep the tests running and make your >> copy >> > works for MSSQL if you insist on keeping it. Since you already seem to >> be >> > ready to spend your engineering time on it, that seems doable. >> > >> > The second option I think might even be a business opportunity - for >> your >> > company or for anyone who would like to do it. Someone could even offer >> it >> > as a service or as a version to support it for others and make a small >> > business out of it if you are really so committed to it I guess, >> including >> > support for any mssql problems. >> > >> > That would actually be awesome if someone does it. >> > >> > J. >> > >> > >> > On Tue, Oct 24, 2023 at 11:44 PM Andrey Anshin < >> andrey.ans...@taragol.is> >> > wrote: >> > >> > > I don’t think there is any possibility left to keep MS SQL Server as >> DB >> > > backend for Airflow. >> > > >> > > I add Elad's message from the original discussion: >> > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fr06j306hldg03g2my1pd4nyjxg78b3h4&data=05%7C01%7CJens.Scheffler%40de.bosch.com%7C6e16370a4474475d1b7908dbd554f6ea%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638338333124062597%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QdbRqzbHV0th6QqgrottUfhT%2FdQ1Pn8xOOlNytq4ojo%3D&reserved=0<https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4> >> > > Because it cleary describe what is happen with MS SQL as DB backend >> for >> > the >> > > last 1.5 years >> > > >> > > > During this time we hoped it would become stable and widely adopted. >> > > > To my taste MsSQL a backend has left a niche and is *not* worth the >> > >> > > maintenance >> > > of it in our CI. >> > > >> > > I also want to note the following points >> > > - MS SQL have unstable tests in Airflow CI, and some cases we even >> don't >> > > run them for the last couple months (or even longer) >> > > - In additional it taking 2x memory than any other backend >> > > - Lack of ARM support, this is also quite important because it prevent >> > > maintainers to check some sort of things in their M1/M2 laptops >> > > - Additional backend required extra effort for any contributors who >> want >> > to >> > > add new feature that touches DB >> > > >> > > >> > > This has always been an experimental feature which are described in >> > AIrflow >> > > Release Process: >> > > >> > > >> > >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23experimental-features&data=05%7C01%7CJens.Scheffler%40de.bosch.com%7C6e16370a4474475d1b7908dbd554f6ea%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638338333124218869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2B%2FEleat699dUWHjfzNmFxTh0F%2Bhtvi5BEX1gi6eANck%3D&reserved=0<https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features> >> > > , I would recommend your team focuses on Airflow on Postgres rather >> than >> > > hanging on to vague hope that MS SQL keeping in Airflow. >> > > >> > > Quite a few companies provide Managed Airflow, see: >> > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fecosystem%2F%23airflow-as-a-service&data=05%7C01%7CJens.Scheffler%40de.bosch.com%7C6e16370a4474475d1b7908dbd554f6ea%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638338333124218869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HMaVb8MTFdsJrSY6af4fgRRiHzkuuu2W0gJ4DUG%2FmLE%3D&reserved=0<https://airflow.apache.org/ecosystem/#airflow-as-a-service> >> > > (this is >> not >> > > complete list) and AFAIK none of them use any other backend rather >> than >> > > Postgres, maybe one exception with Google Composer v1 which seems use >> > > MySQL, even on Azure Data Factory Managed Airflow use Postgres as DB >> > > backend, see: >> > > >> > > >> > >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fconcept-managed-airflow%23architecture&data=05%7C01%7CJens.Scheffler%40de.bosch.com%7C6e16370a4474475d1b7908dbd554f6ea%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638338333124218869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=i5Iu4Sbg5f%2FYPrmoVun8d8Owk%2FWvxKxRwXr1oosuxrY%3D&reserved=0<https://learn.microsoft.com/en-us/azure/data-factory/concept-managed-airflow#architecture> >> > > >> > > ---- >> > > Best Wishes >> > > *Andrey Anshin* >> > > >> > > >> > > >> > > On Mon, 23 Oct 2023 at 23:01, agateaaa <agate...@gmail.com> wrote: >> > > >> > > > Hi All: >> > > > >> > > > Mssql support was voted to be dropped. >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fr06j306hldg03g2my1pd4nyjxg78b3h4&data=05%7C01%7CJens.Scheffler%40de.bosch.com%7C6e16370a4474475d1b7908dbd554f6ea%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638338333124218869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=l0eY8qkZcX6Vz%2FCkTC%2Bfq%2BDFlbvRRd2wC3hhpacYJ2c%3D&reserved=0<https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4> >> > > > >> > > > One of our product requirements is that we can only use the Mssql >> > > database. >> > > > The product that uses airflow is installed with a suite of 8-10 >> other >> > > > products that all use Mssql database as their database. Preferably >> we >> > do >> > > > not want our customers to install another database like postgres or >> > MySQL >> > > > since it would involve extra overhead on their DBA team to maintain >> > (with >> > > > backup/restore functions) yet another database >> > > > >> > > > This has been already discussed and voted on but is there any way we >> > can >> > > > keep experimental support if we pitch in to fix any mssql related >> > issues? >> > > > >> > > > List of current mssql issues are here >> > > > * >> > > > >> > > > >> > > >> > >> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fissues%3Fq%3Dis%253Aissue%2Blabel%253Abackend-mssql-experimintal%2Bis%253Aopen&data=05%7C01%7CJens.Scheffler%40de.bosch.com%7C6e16370a4474475d1b7908dbd554f6ea%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638338333124218869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3SYq3%2BU0i%2BBilTmEayShi1uR2OnwaUlYvNHQE69kfaQ%3D&reserved=0<https://github.com/apache/airflow/issues?q=is%3Aissue+label%3Abackend-mssql-experimintal+is%3Aopen> >> > > > >> > > > Are there any other outstanding issues or can you please let us >> know a >> > > way >> > > > to identify mssql related bugs/problems that need to be addressed? >> > > > >> > > > e.g. >> > > > * >> > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fdiscussions%2F35114&data=05%7C01%7CJens.Scheffler%40de.bosch.com%7C6e16370a4474475d1b7908dbd554f6ea%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638338333124218869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NwOMkZexqX28RtVmwFoioRuuTTv2IPuHWFRrpTAJucA%3D&reserved=0<https://github.com/apache/airflow/discussions/35114> >> > > > >> > > > >> > > > We are just trying to understand how much effort will be required to >> > keep >> > > > supporting Mssql and if we can help with that in any way. >> > > > >> > > > Thanks >> > > > Ashwin >> > > > >> > > >> > >> >
