I agree with the comments and where this discussion has led to. The alternatives suggested by @Jarek Potiuk <ja...@potiuk.com> are something which is doable. We need to realise that these plugins are for the community and we can only support it if "majority" of the community uses it and is willing to maintain it :)
Thanks & Regards, Amogh Desai On Wed, Oct 25, 2023 at 4:13 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > Yeah agreed, I don’t think it is worth keeping the support of MSSQL given > the amount of usage vs the maintenance effort required. > > On Tue, 24 Oct 2023 at 23:07, Jarek Potiuk <ja...@potiuk.com> wrote: > > > Yes. Agree with Andrey. I think our experience from the last few years > was > > "very" bad. The number of mssql users is very small. And the time that > > maintainers and community members lose on various problems with it is > huge. > > Quite often every time we added a new feature requiring some new db > > functionality, quite a lot of overhead was spent by the one adding new > > features to solve the problems coming just from MSSQL support. It's not > > "existing" issues - it's that it generally slows us down with making > > changes > > > > I think there are two options for you when you. It's not "open issues", > > it's the maintenance > > > > * switch to another backend (recommended). And it's not as complex as you > > think. You can also use managed DB with all that is needed > > (backup/maintenance), you do not have to manage it yourself. There are > some > > excellent postgres options available. > > > > * have your own fork airflow and keep the tests running and make your > copy > > works for MSSQL if you insist on keeping it. Since you already seem to be > > ready to spend your engineering time on it, that seems doable. > > > > The second option I think might even be a business opportunity - for your > > company or for anyone who would like to do it. Someone could even offer > it > > as a service or as a version to support it for others and make a small > > business out of it if you are really so committed to it I guess, > including > > support for any mssql problems. > > > > That would actually be awesome if someone does it. > > > > J. > > > > > > On Tue, Oct 24, 2023 at 11:44 PM Andrey Anshin <andrey.ans...@taragol.is > > > > wrote: > > > > > I don’t think there is any possibility left to keep MS SQL Server as DB > > > backend for Airflow. > > > > > > I add Elad's message from the original discussion: > > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 > > > Because it cleary describe what is happen with MS SQL as DB backend for > > the > > > last 1.5 years > > > > > > > During this time we hoped it would become stable and widely adopted. > > > > To my taste MsSQL a backend has left a niche and is *not* worth the > > > > maintenance > > > of it in our CI. > > > > > > I also want to note the following points > > > - MS SQL have unstable tests in Airflow CI, and some cases we even > don't > > > run them for the last couple months (or even longer) > > > - In additional it taking 2x memory than any other backend > > > - Lack of ARM support, this is also quite important because it prevent > > > maintainers to check some sort of things in their M1/M2 laptops > > > - Additional backend required extra effort for any contributors who > want > > to > > > add new feature that touches DB > > > > > > > > > This has always been an experimental feature which are described in > > AIrflow > > > Release Process: > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features > > > , I would recommend your team focuses on Airflow on Postgres rather > than > > > hanging on to vague hope that MS SQL keeping in Airflow. > > > > > > Quite a few companies provide Managed Airflow, see: > > > https://airflow.apache.org/ecosystem/#airflow-as-a-service (this is > not > > > complete list) and AFAIK none of them use any other backend rather than > > > Postgres, maybe one exception with Google Composer v1 which seems use > > > MySQL, even on Azure Data Factory Managed Airflow use Postgres as DB > > > backend, see: > > > > > > > > > https://learn.microsoft.com/en-us/azure/data-factory/concept-managed-airflow#architecture > > > > > > ---- > > > Best Wishes > > > *Andrey Anshin* > > > > > > > > > > > > On Mon, 23 Oct 2023 at 23:01, agateaaa <agate...@gmail.com> wrote: > > > > > > > Hi All: > > > > > > > > Mssql support was voted to be dropped. > > > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 > > > > > > > > One of our product requirements is that we can only use the Mssql > > > database. > > > > The product that uses airflow is installed with a suite of 8-10 other > > > > products that all use Mssql database as their database. Preferably we > > do > > > > not want our customers to install another database like postgres or > > MySQL > > > > since it would involve extra overhead on their DBA team to maintain > > (with > > > > backup/restore functions) yet another database > > > > > > > > This has been already discussed and voted on but is there any way we > > can > > > > keep experimental support if we pitch in to fix any mssql related > > issues? > > > > > > > > List of current mssql issues are here > > > > * > > > > > > > > > > > > > > https://github.com/apache/airflow/issues?q=is%3Aissue+label%3Abackend-mssql-experimintal+is%3Aopen > > > > > > > > Are there any other outstanding issues or can you please let us know > a > > > way > > > > to identify mssql related bugs/problems that need to be addressed? > > > > > > > > e.g. > > > > * https://github.com/apache/airflow/discussions/35114 > > > > > > > > > > > > We are just trying to understand how much effort will be required to > > keep > > > > supporting Mssql and if we can help with that in any way. > > > > > > > > Thanks > > > > Ashwin > > > > > > > > > >
