> We also understand and are ready to address the concerns stated in the vote about support and resolving CI issues
Hello James, Could you please explain how exactly are you planning to help a number of maintainers who are working on developing new feature to make sure they know and realise unobvious consequences of some of the DB changes they might have when some of the features of MYSQL are causing - for example heavy slowdown of inserts because of rebalancing B-TREES on UUID index for databases (that unlike Postgres and MariaDB) lack native UUID support (see . How would you help with discovering similar type of issues see here https://lists.apache.org/thread/7235o1bc3w4694sw8q9m4p58g3tdcjj7 Could you please explain how many people, effort and dedicated resources (i.e. continuous testing of stability and performance you are going to spend on fixing those)? IMHO. If you see a LOT of users that want MsSQL support - you are absolutely free to spend those money, effort and resources on making a fork of Airflow with MsSQL support and charge a premium for that (and a large one). That seems like a very good business model to make if you see a lot of interest there. This is all perfectly fine according to our licence and community would be really thankful for someone who would take the burden of maintaining MSSQL while also making it possible for MSSQL users. Maybe that's the way to go for you? J, On Fri, May 31, 2024 at 8:32 AM James Duong <james.du...@improving.com.invalid> wrote: > Many of the MSSQL customers using Airflow with MSSQL as the backend are > unlikely to participate in those types of surveys, unfortunately, so I fear > the numbers are biased. We have had direct feedback from multiple very > large MSSQL customers who see the removal of this support as a large > blocker to using Airflow. > > Although yes, Microsoft does support PostgreSQL (and MySQL), MSSQL is an > extremely widely used and popular Database platform across different > segments whether Enterprise, Government, Major or SMC. Various Oracle, IBM > and OSS customers are diversifying their Database platform with SQL and it > is important for Airflow-type products to support SQL. > > We also understand and are ready to address the concerns stated in the > vote about support and resolving CI issues. > > From: Jarek Potiuk <ja...@potiuk.com> > Date: Thursday, May 30, 2024 at 3:47 PM > To: dev@airflow.apache.org <dev@airflow.apache.org> > Subject: Re: [DISCUSS] Restore the SQL server backend > Agree with all comments above. Also I think bringing MySQL back is going to > make it way more complex to implement some of the improvements we thought > about - mostly async DB operations (only recently - November 2023 async > support has been added to MSSQL and we know from the history that MSSQL > gave us a lot of headache while developing it and there is no reason to > believe it will be different. And "helping in CI" is not going to cut it - > we need every maintainer who wants to implement a new DB change to become > expert on what is different in MSSQL. > > Honestly - if I'd lose 5% of users because their internal rules say > MSSQL-only (and no Postgres, which as mentioned above is widely supported > and popular including Azure) at the expense of better performance, less > resource usage (as we expect with asyncio) delivered faster to remaining > 95% users, then I know what my decision is. > > BTW. That's not really a criteria we use for such decisions about > technology, but unlike Amazon and Google, Microsoft Azure Data Factory > Airflow team is generally absent from any of those discussions we have > here. Despite us reaching out in various ways they have never "Shown" here, > never contributed anything (or at least we have no knowledge about it) - > including contributions, improvements, system tests nor any other > activities in the community. They are simply not giving back to the > community., > > If they did and officially said (and had proven as the Amazon and Google > team did multiple times for their integrations) that they are willing to > support and maintain MSSQL DB, maybe we would reconsider - mostly because > we could have counted on having them step in when needed (again - as it > happened multiple times with Amazon and Google - when we reach out and need > their help we know we can count on it). I don't see a particular reason why > we should support their proprietary technology. > > J. > > > On Fri, May 31, 2024 at 12:16 AM Damian Shaw <ds...@striketechnologies.com > > > wrote: > > > I would say that MSSQL was often marked as "experimental" ( > > > https://airflow.apache.org/docs/apache-airflow/2.6.0/howto/set-up-database.html > ), > > so IMO I don't think the evidence of it only being used by 5% is > > particularly convincing that it wouldn't eventually be popular. Users who > > might want to primarily use MSSQL because of internal corporate > > restrictions might have a large overlap with users who have restrictions > on > > anything that says "experimental". > > > > I think the more important fact is it was a real burden on development, > > and there was no MSSQL champion in the Airflow maintainers. > > > > -----Original Message----- > > From: Andrey Anshin <andrey.ans...@taragol.is> > > Sent: Thursday, May 30, 2024 2:39 PM > > To: dev@airflow.apache.org > > Cc: james.du...@improving.com.invalid > > Subject: Re: [DISCUSS] Restore the SQL server backend > > > > There was a proposal to keep it in the past [1] with a short explanation > > why the maintainers did not want to keep it. > > > > > many Microsoft customers who are using Airflow > > > > Microsoft also supports and participates in the development of > PostgreSQL, > > there is one Core Team member and couple of Major Contributors working in > > Microsoft [2] and in addition a couple years ago Microsoft acquired one > of > > the PostgreSQL vendors [3]. So I would like to believe that Microsoft > also > > could offer different services around PostgreSQL for their customers. > > > > > > [1] Keep Mssql support: > > https://lists.apache.org/thread/ot58ms069z4pyhj786j1m0dqds6lhjks > > [2] PostgreSQL: Contributors Profiles: > > https://www.postgresql.org/community/contributors/ > > [3] Microsoft Acquires Citus Data: > > https://www.citusdata.com/blog/2019/01/24/microsoft-acquires-citus-data/ > > > > On Thu, 30 May 2024 at 21:18, Pierre Jeambrun <pierrejb...@gmail.com> > > wrote: > > > > > I share Jed feeling. The effort required to maintain those compare to > > > the value it actually brings combined with the usage from the survey, > > > it doesn’t seem worth it to me. > > > > > > On Thu 30 May 2024 at 19:16, Jed Cunningham <jedcunning...@apache.org> > > > wrote: > > > > > > > Just for context, here were (roughly) the results from the 2023 > > > > Airflow > > > > survey: > > > > > > > > PostgreSQL: 75% > > > > MySQL: 15% > > > > MSSQL: 5% > > > > > > > > Also, there are already discussions about potentially dropping MySQL > > > > support in Airflow 3. Given all that and the points from the past > > > > vote, I don't think it makes much sense to bring MSSQL back. > > > > > > > > > ________________________________ > > Strike Technologies, LLC (“Strike”) is part of the GTS family of > > companies. Strike is a technology solutions provider, and is not a broker > > or dealer and does not transact any securities related business directly > > whatsoever. This communication is the property of Strike and its > > affiliates, and does not constitute an offer to sell or the solicitation > of > > an offer to buy any security in any jurisdiction. It is intended only for > > the person to whom it is addressed and may contain information that is > > privileged, confidential, or otherwise protected from disclosure. > > Distribution or copying of this communication, or the information > contained > > herein, by anyone other than the intended recipient is prohibited. If you > > have received this communication in error, please immediately notify > Strike > > at i...@striketechnologies.com, and delete and destroy any copies > hereof. > > ________________________________ > > > > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments > > are intended solely for the addressee. This transmission is covered by > the > > Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The > > information contained in this transmission is confidential in nature and > > protected from further use or disclosure under U.S. Pub. L. 106-102, 113 > > U.S. Stat. 1338 (1999), and may be subject to attorney-client or other > > legal privilege. Your use or disclosure of this information for any > purpose > > other than that intended by its transmittal is strictly prohibited, and > may > > subject you to fines and/or penalties under federal and state law. If you > > are not the intended recipient of this transmission, please DESTROY ALL > > COPIES RECEIVED and confirm destruction to the sender via return > > transmittal. > > >
