Yes. Agree with Andrey. I think our experience from the last few years was "very" bad. The number of mssql users is very small. And the time that maintainers and community members lose on various problems with it is huge. Quite often every time we added a new feature requiring some new db functionality, quite a lot of overhead was spent by the one adding new features to solve the problems coming just from MSSQL support. It's not "existing" issues - it's that it generally slows us down with making changes
I think there are two options for you when you. It's not "open issues", it's the maintenance * switch to another backend (recommended). And it's not as complex as you think. You can also use managed DB with all that is needed (backup/maintenance), you do not have to manage it yourself. There are some excellent postgres options available. * have your own fork airflow and keep the tests running and make your copy works for MSSQL if you insist on keeping it. Since you already seem to be ready to spend your engineering time on it, that seems doable. The second option I think might even be a business opportunity - for your company or for anyone who would like to do it. Someone could even offer it as a service or as a version to support it for others and make a small business out of it if you are really so committed to it I guess, including support for any mssql problems. That would actually be awesome if someone does it. J. On Tue, Oct 24, 2023 at 11:44 PM Andrey Anshin <andrey.ans...@taragol.is> wrote: > I don’t think there is any possibility left to keep MS SQL Server as DB > backend for Airflow. > > I add Elad's message from the original discussion: > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 > Because it cleary describe what is happen with MS SQL as DB backend for the > last 1.5 years > > > During this time we hoped it would become stable and widely adopted. > > To my taste MsSQL a backend has left a niche and is *not* worth the > > maintenance > of it in our CI. > > I also want to note the following points > - MS SQL have unstable tests in Airflow CI, and some cases we even don't > run them for the last couple months (or even longer) > - In additional it taking 2x memory than any other backend > - Lack of ARM support, this is also quite important because it prevent > maintainers to check some sort of things in their M1/M2 laptops > - Additional backend required extra effort for any contributors who want to > add new feature that touches DB > > > This has always been an experimental feature which are described in AIrflow > Release Process: > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features > , I would recommend your team focuses on Airflow on Postgres rather than > hanging on to vague hope that MS SQL keeping in Airflow. > > Quite a few companies provide Managed Airflow, see: > https://airflow.apache.org/ecosystem/#airflow-as-a-service (this is not > complete list) and AFAIK none of them use any other backend rather than > Postgres, maybe one exception with Google Composer v1 which seems use > MySQL, even on Azure Data Factory Managed Airflow use Postgres as DB > backend, see: > > https://learn.microsoft.com/en-us/azure/data-factory/concept-managed-airflow#architecture > > ---- > Best Wishes > *Andrey Anshin* > > > > On Mon, 23 Oct 2023 at 23:01, agateaaa <agate...@gmail.com> wrote: > > > Hi All: > > > > Mssql support was voted to be dropped. > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 > > > > One of our product requirements is that we can only use the Mssql > database. > > The product that uses airflow is installed with a suite of 8-10 other > > products that all use Mssql database as their database. Preferably we do > > not want our customers to install another database like postgres or MySQL > > since it would involve extra overhead on their DBA team to maintain (with > > backup/restore functions) yet another database > > > > This has been already discussed and voted on but is there any way we can > > keep experimental support if we pitch in to fix any mssql related issues? > > > > List of current mssql issues are here > > * > > > > > https://github.com/apache/airflow/issues?q=is%3Aissue+label%3Abackend-mssql-experimintal+is%3Aopen > > > > Are there any other outstanding issues or can you please let us know a > way > > to identify mssql related bugs/problems that need to be addressed? > > > > e.g. > > * https://github.com/apache/airflow/discussions/35114 > > > > > > We are just trying to understand how much effort will be required to keep > > supporting Mssql and if we can help with that in any way. > > > > Thanks > > Ashwin > > >
