Thanks for all the responses. We also have an on premise offering which means we will have to bundle an extra database, postgres along with mssql which adds to the complexity that the customer has to deal with which is not ideal from the product point of view. I understand your inputs. I will take that back to our team and see what we can do.
Thank you! On Wed, Oct 25, 2023 at 8:43 AM Amogh Desai <amoghdesai....@gmail.com> wrote: > Wrong choice of words, didn't mean "plugin" but was referring to a more > generic term. > > Thank you! > > On Wed, Oct 25, 2023 at 5:51 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > The alternatives suggested by @Jarek Potiuk <ja...@potiuk.com> are > > something which is doable. We need to realise that these plugins are for > > the community and we can only support it if > > "majority" of the community uses it and is willing to maintain it :) > > > > Just to clarify - it's not about plugins, it's about "core" metastore > > support for MSSQL. The MSSQL provider will remain as is, but Airflow will > > lose the ability of using MsSQL as backend for > > scheduling/tasks/UI/executors/communication with workers via DB. > > > > The overhead / extra maintenance there is to make sure that all the > > queries, performance, locking and anything else airflow uses to make > "core" > > works with MsSQL. > > > > In Airflow 2.8 the only remaining backends will be MySQL 8 (as MySQL 5.7 > > reaches end of life in a week), and Postgres 12-16 (in 2 weeks Postgres > 11 > > reaches EOL and we will drop it). > > Those will be the only choices for Airflow 2.8 - providing that we will > > remove MsSQL as planned (condition for that is that we describe a viable > > migration path for MsSQL users). > > > > J. > > > > > > > > On Wed, Oct 25, 2023 at 1:23 PM Amogh Desai <amoghdesai....@gmail.com> > > wrote: > > > >> I agree with the comments and where this discussion has led to. > >> > >> The alternatives suggested by @Jarek Potiuk <ja...@potiuk.com> are > >> something which is doable. We need to realise that these plugins are for > >> the community and we can only support it if > >> "majority" of the community uses it and is willing to maintain it :) > >> > >> > >> Thanks & Regards, > >> Amogh Desai > >> > >> On Wed, Oct 25, 2023 at 4:13 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > >> > >>> Yeah agreed, I don’t think it is worth keeping the support of MSSQL > given > >>> the amount of usage vs the maintenance effort required. > >>> > >>> On Tue, 24 Oct 2023 at 23:07, Jarek Potiuk <ja...@potiuk.com> wrote: > >>> > >>> > Yes. Agree with Andrey. I think our experience from the last few > >>> years was > >>> > "very" bad. The number of mssql users is very small. And the time > that > >>> > maintainers and community members lose on various problems with it is > >>> huge. > >>> > Quite often every time we added a new feature requiring some new db > >>> > functionality, quite a lot of overhead was spent by the one adding > new > >>> > features to solve the problems coming just from MSSQL support. It's > not > >>> > "existing" issues - it's that it generally slows us down with making > >>> > changes > >>> > > >>> > I think there are two options for you when you. It's not "open > issues", > >>> > it's the maintenance > >>> > > >>> > * switch to another backend (recommended). And it's not as complex as > >>> you > >>> > think. You can also use managed DB with all that is needed > >>> > (backup/maintenance), you do not have to manage it yourself. There > are > >>> some > >>> > excellent postgres options available. > >>> > > >>> > * have your own fork airflow and keep the tests running and make your > >>> copy > >>> > works for MSSQL if you insist on keeping it. Since you already seem > to > >>> be > >>> > ready to spend your engineering time on it, that seems doable. > >>> > > >>> > The second option I think might even be a business opportunity - for > >>> your > >>> > company or for anyone who would like to do it. Someone could even > >>> offer it > >>> > as a service or as a version to support it for others and make a > small > >>> > business out of it if you are really so committed to it I guess, > >>> including > >>> > support for any mssql problems. > >>> > > >>> > That would actually be awesome if someone does it. > >>> > > >>> > J. > >>> > > >>> > > >>> > On Tue, Oct 24, 2023 at 11:44 PM Andrey Anshin < > >>> andrey.ans...@taragol.is> > >>> > wrote: > >>> > > >>> > > I don’t think there is any possibility left to keep MS SQL Server > as > >>> DB > >>> > > backend for Airflow. > >>> > > > >>> > > I add Elad's message from the original discussion: > >>> > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 > >>> > > Because it cleary describe what is happen with MS SQL as DB backend > >>> for > >>> > the > >>> > > last 1.5 years > >>> > > > >>> > > > During this time we hoped it would become stable and widely > >>> adopted. > >>> > > > To my taste MsSQL a backend has left a niche and is *not* worth > >>> the > > >>> > > maintenance > >>> > > of it in our CI. > >>> > > > >>> > > I also want to note the following points > >>> > > - MS SQL have unstable tests in Airflow CI, and some cases we even > >>> don't > >>> > > run them for the last couple months (or even longer) > >>> > > - In additional it taking 2x memory than any other backend > >>> > > - Lack of ARM support, this is also quite important because it > >>> prevent > >>> > > maintainers to check some sort of things in their M1/M2 laptops > >>> > > - Additional backend required extra effort for any contributors who > >>> want > >>> > to > >>> > > add new feature that touches DB > >>> > > > >>> > > > >>> > > This has always been an experimental feature which are described in > >>> > AIrflow > >>> > > Release Process: > >>> > > > >>> > > > >>> > > >>> > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features > >>> > > , I would recommend your team focuses on Airflow on Postgres rather > >>> than > >>> > > hanging on to vague hope that MS SQL keeping in Airflow. > >>> > > > >>> > > Quite a few companies provide Managed Airflow, see: > >>> > > https://airflow.apache.org/ecosystem/#airflow-as-a-service (this > is > >>> not > >>> > > complete list) and AFAIK none of them use any other backend rather > >>> than > >>> > > Postgres, maybe one exception with Google Composer v1 which seems > use > >>> > > MySQL, even on Azure Data Factory Managed Airflow use Postgres as > DB > >>> > > backend, see: > >>> > > > >>> > > > >>> > > >>> > https://learn.microsoft.com/en-us/azure/data-factory/concept-managed-airflow#architecture > >>> > > > >>> > > ---- > >>> > > Best Wishes > >>> > > *Andrey Anshin* > >>> > > > >>> > > > >>> > > > >>> > > On Mon, 23 Oct 2023 at 23:01, agateaaa <agate...@gmail.com> wrote: > >>> > > > >>> > > > Hi All: > >>> > > > > >>> > > > Mssql support was voted to be dropped. > >>> > > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 > >>> > > > > >>> > > > One of our product requirements is that we can only use the Mssql > >>> > > database. > >>> > > > The product that uses airflow is installed with a suite of 8-10 > >>> other > >>> > > > products that all use Mssql database as their database. > Preferably > >>> we > >>> > do > >>> > > > not want our customers to install another database like postgres > or > >>> > MySQL > >>> > > > since it would involve extra overhead on their DBA team to > maintain > >>> > (with > >>> > > > backup/restore functions) yet another database > >>> > > > > >>> > > > This has been already discussed and voted on but is there any way > >>> we > >>> > can > >>> > > > keep experimental support if we pitch in to fix any mssql related > >>> > issues? > >>> > > > > >>> > > > List of current mssql issues are here > >>> > > > * > >>> > > > > >>> > > > > >>> > > > >>> > > >>> > https://github.com/apache/airflow/issues?q=is%3Aissue+label%3Abackend-mssql-experimintal+is%3Aopen > >>> > > > > >>> > > > Are there any other outstanding issues or can you please let us > >>> know a > >>> > > way > >>> > > > to identify mssql related bugs/problems that need to be > addressed? > >>> > > > > >>> > > > e.g. > >>> > > > * https://github.com/apache/airflow/discussions/35114 > >>> > > > > >>> > > > > >>> > > > We are just trying to understand how much effort will be required > >>> to > >>> > keep > >>> > > > supporting Mssql and if we can help with that in any way. > >>> > > > > >>> > > > Thanks > >>> > > > Ashwin > >>> > > > > >>> > > > >>> > > >>> > >> >
