Wrong choice of words, didn't mean "plugin" but was referring to a more generic term.
Thank you! On Wed, Oct 25, 2023 at 5:51 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > The alternatives suggested by @Jarek Potiuk <ja...@potiuk.com> are > something which is doable. We need to realise that these plugins are for > the community and we can only support it if > "majority" of the community uses it and is willing to maintain it :) > > Just to clarify - it's not about plugins, it's about "core" metastore > support for MSSQL. The MSSQL provider will remain as is, but Airflow will > lose the ability of using MsSQL as backend for > scheduling/tasks/UI/executors/communication with workers via DB. > > The overhead / extra maintenance there is to make sure that all the > queries, performance, locking and anything else airflow uses to make "core" > works with MsSQL. > > In Airflow 2.8 the only remaining backends will be MySQL 8 (as MySQL 5.7 > reaches end of life in a week), and Postgres 12-16 (in 2 weeks Postgres 11 > reaches EOL and we will drop it). > Those will be the only choices for Airflow 2.8 - providing that we will > remove MsSQL as planned (condition for that is that we describe a viable > migration path for MsSQL users). > > J. > > > > On Wed, Oct 25, 2023 at 1:23 PM Amogh Desai <amoghdesai....@gmail.com> > wrote: > >> I agree with the comments and where this discussion has led to. >> >> The alternatives suggested by @Jarek Potiuk <ja...@potiuk.com> are >> something which is doable. We need to realise that these plugins are for >> the community and we can only support it if >> "majority" of the community uses it and is willing to maintain it :) >> >> >> Thanks & Regards, >> Amogh Desai >> >> On Wed, Oct 25, 2023 at 4:13 PM Kaxil Naik <kaxiln...@gmail.com> wrote: >> >>> Yeah agreed, I don’t think it is worth keeping the support of MSSQL given >>> the amount of usage vs the maintenance effort required. >>> >>> On Tue, 24 Oct 2023 at 23:07, Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>> > Yes. Agree with Andrey. I think our experience from the last few >>> years was >>> > "very" bad. The number of mssql users is very small. And the time that >>> > maintainers and community members lose on various problems with it is >>> huge. >>> > Quite often every time we added a new feature requiring some new db >>> > functionality, quite a lot of overhead was spent by the one adding new >>> > features to solve the problems coming just from MSSQL support. It's not >>> > "existing" issues - it's that it generally slows us down with making >>> > changes >>> > >>> > I think there are two options for you when you. It's not "open issues", >>> > it's the maintenance >>> > >>> > * switch to another backend (recommended). And it's not as complex as >>> you >>> > think. You can also use managed DB with all that is needed >>> > (backup/maintenance), you do not have to manage it yourself. There are >>> some >>> > excellent postgres options available. >>> > >>> > * have your own fork airflow and keep the tests running and make your >>> copy >>> > works for MSSQL if you insist on keeping it. Since you already seem to >>> be >>> > ready to spend your engineering time on it, that seems doable. >>> > >>> > The second option I think might even be a business opportunity - for >>> your >>> > company or for anyone who would like to do it. Someone could even >>> offer it >>> > as a service or as a version to support it for others and make a small >>> > business out of it if you are really so committed to it I guess, >>> including >>> > support for any mssql problems. >>> > >>> > That would actually be awesome if someone does it. >>> > >>> > J. >>> > >>> > >>> > On Tue, Oct 24, 2023 at 11:44 PM Andrey Anshin < >>> andrey.ans...@taragol.is> >>> > wrote: >>> > >>> > > I don’t think there is any possibility left to keep MS SQL Server as >>> DB >>> > > backend for Airflow. >>> > > >>> > > I add Elad's message from the original discussion: >>> > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 >>> > > Because it cleary describe what is happen with MS SQL as DB backend >>> for >>> > the >>> > > last 1.5 years >>> > > >>> > > > During this time we hoped it would become stable and widely >>> adopted. >>> > > > To my taste MsSQL a backend has left a niche and is *not* worth >>> the > >>> > > maintenance >>> > > of it in our CI. >>> > > >>> > > I also want to note the following points >>> > > - MS SQL have unstable tests in Airflow CI, and some cases we even >>> don't >>> > > run them for the last couple months (or even longer) >>> > > - In additional it taking 2x memory than any other backend >>> > > - Lack of ARM support, this is also quite important because it >>> prevent >>> > > maintainers to check some sort of things in their M1/M2 laptops >>> > > - Additional backend required extra effort for any contributors who >>> want >>> > to >>> > > add new feature that touches DB >>> > > >>> > > >>> > > This has always been an experimental feature which are described in >>> > AIrflow >>> > > Release Process: >>> > > >>> > > >>> > >>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features >>> > > , I would recommend your team focuses on Airflow on Postgres rather >>> than >>> > > hanging on to vague hope that MS SQL keeping in Airflow. >>> > > >>> > > Quite a few companies provide Managed Airflow, see: >>> > > https://airflow.apache.org/ecosystem/#airflow-as-a-service (this is >>> not >>> > > complete list) and AFAIK none of them use any other backend rather >>> than >>> > > Postgres, maybe one exception with Google Composer v1 which seems use >>> > > MySQL, even on Azure Data Factory Managed Airflow use Postgres as DB >>> > > backend, see: >>> > > >>> > > >>> > >>> https://learn.microsoft.com/en-us/azure/data-factory/concept-managed-airflow#architecture >>> > > >>> > > ---- >>> > > Best Wishes >>> > > *Andrey Anshin* >>> > > >>> > > >>> > > >>> > > On Mon, 23 Oct 2023 at 23:01, agateaaa <agate...@gmail.com> wrote: >>> > > >>> > > > Hi All: >>> > > > >>> > > > Mssql support was voted to be dropped. >>> > > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 >>> > > > >>> > > > One of our product requirements is that we can only use the Mssql >>> > > database. >>> > > > The product that uses airflow is installed with a suite of 8-10 >>> other >>> > > > products that all use Mssql database as their database. Preferably >>> we >>> > do >>> > > > not want our customers to install another database like postgres or >>> > MySQL >>> > > > since it would involve extra overhead on their DBA team to maintain >>> > (with >>> > > > backup/restore functions) yet another database >>> > > > >>> > > > This has been already discussed and voted on but is there any way >>> we >>> > can >>> > > > keep experimental support if we pitch in to fix any mssql related >>> > issues? >>> > > > >>> > > > List of current mssql issues are here >>> > > > * >>> > > > >>> > > > >>> > > >>> > >>> https://github.com/apache/airflow/issues?q=is%3Aissue+label%3Abackend-mssql-experimintal+is%3Aopen >>> > > > >>> > > > Are there any other outstanding issues or can you please let us >>> know a >>> > > way >>> > > > to identify mssql related bugs/problems that need to be addressed? >>> > > > >>> > > > e.g. >>> > > > * https://github.com/apache/airflow/discussions/35114 >>> > > > >>> > > > >>> > > > We are just trying to understand how much effort will be required >>> to >>> > keep >>> > > > supporting Mssql and if we can help with that in any way. >>> > > > >>> > > > Thanks >>> > > > Ashwin >>> > > > >>> > > >>> > >>> >>