Many of the MSSQL customers using Airflow with MSSQL as the backend are unlikely to participate in those types of surveys, unfortunately, so I fear the numbers are biased. We have had direct feedback from multiple very large MSSQL customers who see the removal of this support as a large blocker to using Airflow.
Although yes, Microsoft does support PostgreSQL (and MySQL), MSSQL is an extremely widely used and popular Database platform across different segments whether Enterprise, Government, Major or SMC. Various Oracle, IBM and OSS customers are diversifying their Database platform with SQL and it is important for Airflow-type products to support SQL. We also understand and are ready to address the concerns stated in the vote about support and resolving CI issues. From: Jarek Potiuk <ja...@potiuk.com> Date: Thursday, May 30, 2024 at 3:47 PM To: dev@airflow.apache.org <dev@airflow.apache.org> Subject: Re: [DISCUSS] Restore the SQL server backend Agree with all comments above. Also I think bringing MySQL back is going to make it way more complex to implement some of the improvements we thought about - mostly async DB operations (only recently - November 2023 async support has been added to MSSQL and we know from the history that MSSQL gave us a lot of headache while developing it and there is no reason to believe it will be different. And "helping in CI" is not going to cut it - we need every maintainer who wants to implement a new DB change to become expert on what is different in MSSQL. Honestly - if I'd lose 5% of users because their internal rules say MSSQL-only (and no Postgres, which as mentioned above is widely supported and popular including Azure) at the expense of better performance, less resource usage (as we expect with asyncio) delivered faster to remaining 95% users, then I know what my decision is. BTW. That's not really a criteria we use for such decisions about technology, but unlike Amazon and Google, Microsoft Azure Data Factory Airflow team is generally absent from any of those discussions we have here. Despite us reaching out in various ways they have never "Shown" here, never contributed anything (or at least we have no knowledge about it) - including contributions, improvements, system tests nor any other activities in the community. They are simply not giving back to the community., If they did and officially said (and had proven as the Amazon and Google team did multiple times for their integrations) that they are willing to support and maintain MSSQL DB, maybe we would reconsider - mostly because we could have counted on having them step in when needed (again - as it happened multiple times with Amazon and Google - when we reach out and need their help we know we can count on it). I don't see a particular reason why we should support their proprietary technology. J. On Fri, May 31, 2024 at 12:16 AM Damian Shaw <ds...@striketechnologies.com> wrote: > I would say that MSSQL was often marked as "experimental" ( > https://airflow.apache.org/docs/apache-airflow/2.6.0/howto/set-up-database.html), > so IMO I don't think the evidence of it only being used by 5% is > particularly convincing that it wouldn't eventually be popular. Users who > might want to primarily use MSSQL because of internal corporate > restrictions might have a large overlap with users who have restrictions on > anything that says "experimental". > > I think the more important fact is it was a real burden on development, > and there was no MSSQL champion in the Airflow maintainers. > > -----Original Message----- > From: Andrey Anshin <andrey.ans...@taragol.is> > Sent: Thursday, May 30, 2024 2:39 PM > To: dev@airflow.apache.org > Cc: james.du...@improving.com.invalid > Subject: Re: [DISCUSS] Restore the SQL server backend > > There was a proposal to keep it in the past [1] with a short explanation > why the maintainers did not want to keep it. > > > many Microsoft customers who are using Airflow > > Microsoft also supports and participates in the development of PostgreSQL, > there is one Core Team member and couple of Major Contributors working in > Microsoft [2] and in addition a couple years ago Microsoft acquired one of > the PostgreSQL vendors [3]. So I would like to believe that Microsoft also > could offer different services around PostgreSQL for their customers. > > > [1] Keep Mssql support: > https://lists.apache.org/thread/ot58ms069z4pyhj786j1m0dqds6lhjks > [2] PostgreSQL: Contributors Profiles: > https://www.postgresql.org/community/contributors/ > [3] Microsoft Acquires Citus Data: > https://www.citusdata.com/blog/2019/01/24/microsoft-acquires-citus-data/ > > On Thu, 30 May 2024 at 21:18, Pierre Jeambrun <pierrejb...@gmail.com> > wrote: > > > I share Jed feeling. The effort required to maintain those compare to > > the value it actually brings combined with the usage from the survey, > > it doesn’t seem worth it to me. > > > > On Thu 30 May 2024 at 19:16, Jed Cunningham <jedcunning...@apache.org> > > wrote: > > > > > Just for context, here were (roughly) the results from the 2023 > > > Airflow > > > survey: > > > > > > PostgreSQL: 75% > > > MySQL: 15% > > > MSSQL: 5% > > > > > > Also, there are already discussions about potentially dropping MySQL > > > support in Airflow 3. Given all that and the points from the past > > > vote, I don't think it makes much sense to bring MSSQL back. > > > > > > ________________________________ > Strike Technologies, LLC (“Strike”) is part of the GTS family of > companies. Strike is a technology solutions provider, and is not a broker > or dealer and does not transact any securities related business directly > whatsoever. This communication is the property of Strike and its > affiliates, and does not constitute an offer to sell or the solicitation of > an offer to buy any security in any jurisdiction. It is intended only for > the person to whom it is addressed and may contain information that is > privileged, confidential, or otherwise protected from disclosure. > Distribution or copying of this communication, or the information contained > herein, by anyone other than the intended recipient is prohibited. If you > have received this communication in error, please immediately notify Strike > at i...@striketechnologies.com, and delete and destroy any copies hereof. > ________________________________ > > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments > are intended solely for the addressee. This transmission is covered by the > Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The > information contained in this transmission is confidential in nature and > protected from further use or disclosure under U.S. Pub. L. 106-102, 113 > U.S. Stat. 1338 (1999), and may be subject to attorney-client or other > legal privilege. Your use or disclosure of this information for any purpose > other than that intended by its transmittal is strictly prohibited, and may > subject you to fines and/or penalties under federal and state law. If you > are not the intended recipient of this transmission, please DESTROY ALL > COPIES RECEIVED and confirm destruction to the sender via return > transmittal. >
