Yep what Kaxil said and a separate repo does not change anything. Anything that we (Apache Airflow PMC) releases is something that we (Apache Airflow PMC maintain, contribute to and only committers can approve and merge the code) - so no matter if it is a separate repo, the cost is the same .
The community already decided that we do not want to bear the cost. That's done. If there is someone who wants to bear the cost (and then get it back by charging their customers for this extra service maybe), they are free to do so, and what you are proposing is that we again start bearing the cost. Which we already decided not to do (by voting). I think the discussion is long and exhaustive enough, so if someone wants to revert that decision - they would have to start a vote thread. Just following the regular process we have in Apache project: https://www.apache.org/foundation/voting#votes-on-code-modification - but taking into account quite clear "no" from a few people - committers - it's not going to pass. From the discussion above it looks like there will be a VETO from a few people and even one VETO is enough for it to not be accepted. J. On Thu, Jun 13, 2024 at 7:01 PM James Duong <james.du...@improving.com.invalid> wrote: > I haven’t looked that much into the Airflow code yet, but the backend code > must already be pluggable since it supports multiple databases. > > What if we were to maintain a separate repo and artifact for just an > extension for the SQL Server backend? This seems like it’d be low-impact on > the Airflow community from a maintenance perspective, but keeps MSSQL users > using the official Airflow artifacts. (Maybe just have the Airflow > documentation link to the extension). > > From: Kaxil Naik <kaxiln...@gmail.com> > Date: Thursday, June 13, 2024 at 9:53 AM > To: dev@airflow.apache.org <dev@airflow.apache.org> > Subject: Re: [DISCUSS] Restore the SQL server backend > The cost of maintaining it in Airflow repo (with CI/CI, GH issues etc) is, > unfortunately, just too much higher. > > On Thu, 13 Jun 2024 at 17:11, James Duong <james.du...@improving.com > .invalid> > wrote: > > > Hi Jarek, > > > > Thanks for your response. > > > > I would prefer to make this work part of the main Airflow repository. > > > > My previous experience with maintaining forks of Apache projects has > > always been that it fragments the community unnecessarily. Users of a > > company-specific fork might ask for features (not just relating to MSSQL) > > that could benefit the community at large. The larger community can miss > > out on insights from the users and maintainers of the fork. > > > > There is a greater cost both getting updates from the main repo and > > upstreaming improvements to the main repo as the codebase diverges > further. > > > > I could see confusion from end users if there are multiple sets of > > artifacts for different forks of Airflow about which one to get. > > > > What are your thoughts on this? > > > > From: Jarek Potiuk <ja...@potiuk.com> > > Date: Monday, June 3, 2024 at 9:32 PM > > To: dev@airflow.apache.org <dev@airflow.apache.org> > > Cc: james.du...@improving.com.invalid <james.du...@improving.com > .invalid> > > Subject: Re: [DISCUSS] Restore the SQL server backend > > I am not sure if you read what I wrote with full understanding. > > > > To be perfectly honest - If you secure enough resources, I think *STILL* > it > > will be better if you maintain your own fork and apply necessary changes > > and offer it commercially to anyone who needs it. This is way easier for > > the community, and better for you commercially - and if you are > **really** > > committed for a long term to do MSSQL, then you should have no problem in > > maintaining the fork. > > > > On Mon, Jun 3, 2024 at 11:15 PM James Duong > > <james.du...@improving.com.invalid> wrote: > > > > > Thanks for all of your feedback and discussion. > > > > > > The interest and usage from the enterprise MSSQL community is very > large > > - > > > it's unfortunate that numbers are difficult to gather. > > > > > > In terms of the support - I hear you that it should not be limited to > > only > > > CI improvements and PR support and a more active role needs to be > taken. > > I > > > am working on a plan that would provide the necessary involvement in > the > > > community. > > > > > > Please allow me some time to see what is possible. > > > > > > From: Wei Lee <weilee...@gmail.com> > > > Date: Friday, May 31, 2024 at 8:45 AM > > > To: dev@airflow.apache.org <dev@airflow.apache.org> > > > Cc: james.du...@improving.com.invalid <james.du...@improving.com > > .invalid> > > > Subject: Re: [DISCUSS] Restore the SQL server backend > > > I agree with Jed and the following comments. If my memory serves me > > right, > > > this topic has been discussed a few times in the past. 5% doesn't seem > > very > > > convincing. Even if it's biased, I'm still not persuaded that there > are a > > > large number of users that are worth the community's effort. And Jarek > > > pointed out a great solution for forking Airflow and adding MSSQL > support > > > to it. > > > > > > Best, > > > Wei > > > > > > > On May 31, 2024, at 7:50 PM, Elad Kalif <elad...@apache.org> wrote: > > > > > > > > I agree with Jarek > > > > > > > > I am a bit worried about the mental model of this proposal as you are > > > > offering to deliver a feature but you are not offering being a > > community > > > > member. > > > > I had a lot of frustration with the MsSQL backend tests, it really > > caused > > > > me pain as a contributor. According to your mental model - will you > > > > actively review community PRs, triage Airflow issues and offer > guidance > > > and > > > > help when needed about MsSQL or will the maintainers have to track > > these > > > > problems and actively tag you/your team for assistance? > > > > > > > > Let me give an example: User opens a Github issue about HA scheduler. > > > Will > > > > your team participate in the issue triage? Or do you expect the > > community > > > > to triage the issue and only after some discussion when it turns out > > that > > > > it's MsSQL specific issue then we need to notify you? > > > > > > > > On Fri, May 31, 2024 at 10:05 AM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > > > > > >>> We also understand and are ready to address the concerns stated in > > the > > > >> vote about support and resolving CI issues > > > >> > > > >> Hello James, > > > >> > > > >> Could you please explain how exactly are you planning to help a > number > > > of > > > >> maintainers who are working on developing new feature to make sure > > > >> they know and realise unobvious consequences of some of the DB > changes > > > they > > > >> might have when some of the features of MYSQL are causing - for > > example > > > >> heavy slowdown of inserts because of rebalancing B-TREES on UUID > > index > > > for > > > >> databases (that unlike Postgres and MariaDB) lack native UUID > support > > > (see > > > >> . How would you help with discovering similar type of issues see > here > > > >> https://lists.apache.org/thread/7235o1bc3w4694sw8q9m4p58g3tdcjj7 > > > >> > > > >> Could you please explain how many people, effort and dedicated > > resources > > > >> (i.e. continuous testing of stability and performance you are going > to > > > >> spend on fixing those)? > > > >> > > > >> IMHO. If you see a LOT of users that want MsSQL support - you are > > > >> absolutely free to spend those money, effort and resources on > making a > > > fork > > > >> of Airflow with MsSQL support and charge a premium for that (and a > > large > > > >> one). That seems like a very good business model to make if you see > a > > > lot > > > >> of interest there. > > > >> > > > >> This is all perfectly fine according to our licence and community > > would > > > be > > > >> really thankful for someone who would take the burden of maintaining > > > MSSQL > > > >> while also making it possible for MSSQL users. Maybe that's the way > to > > > go > > > >> for you? > > > >> > > > >> J, > > > >> > > > >> > > > >> > > > >> On Fri, May 31, 2024 at 8:32 AM James Duong > > > >> <james.du...@improving.com.invalid> wrote: > > > >> > > > >>> Many of the MSSQL customers using Airflow with MSSQL as the backend > > are > > > >>> unlikely to participate in those types of surveys, unfortunately, > so > > I > > > >> fear > > > >>> the numbers are biased. We have had direct feedback from multiple > > very > > > >>> large MSSQL customers who see the removal of this support as a > large > > > >>> blocker to using Airflow. > > > >>> > > > >>> Although yes, Microsoft does support PostgreSQL (and MySQL), MSSQL > is > > > an > > > >>> extremely widely used and popular Database platform across > different > > > >>> segments whether Enterprise, Government, Major or SMC. Various > > Oracle, > > > >> IBM > > > >>> and OSS customers are diversifying their Database platform with SQL > > and > > > >> it > > > >>> is important for Airflow-type products to support SQL. > > > >>> > > > >>> We also understand and are ready to address the concerns stated in > > the > > > >>> vote about support and resolving CI issues. > > > >>> > > > >>> From: Jarek Potiuk <ja...@potiuk.com> > > > >>> Date: Thursday, May 30, 2024 at 3:47 PM > > > >>> To: dev@airflow.apache.org <dev@airflow.apache.org> > > > >>> Subject: Re: [DISCUSS] Restore the SQL server backend > > > >>> Agree with all comments above. Also I think bringing MySQL back is > > > going > > > >> to > > > >>> make it way more complex to implement some of the improvements we > > > thought > > > >>> about - mostly async DB operations (only recently - November 2023 > > async > > > >>> support has been added to MSSQL and we know from the history that > > MSSQL > > > >>> gave us a lot of headache while developing it and there is no > reason > > to > > > >>> believe it will be different. And "helping in CI" is not going to > cut > > > it > > > >> - > > > >>> we need every maintainer who wants to implement a new DB change to > > > become > > > >>> expert on what is different in MSSQL. > > > >>> > > > >>> Honestly - if I'd lose 5% of users because their internal rules say > > > >>> MSSQL-only (and no Postgres, which as mentioned above is widely > > > supported > > > >>> and popular including Azure) at the expense of better performance, > > less > > > >>> resource usage (as we expect with asyncio) delivered faster to > > > remaining > > > >>> 95% users, then I know what my decision is. > > > >>> > > > >>> BTW. That's not really a criteria we use for such decisions about > > > >>> technology, but unlike Amazon and Google, Microsoft Azure Data > > Factory > > > >>> Airflow team is generally absent from any of those discussions we > > have > > > >>> here. Despite us reaching out in various ways they have never > "Shown" > > > >> here, > > > >>> never contributed anything (or at least we have no knowledge about > > it) > > > - > > > >>> including contributions, improvements, system tests nor any other > > > >>> activities in the community. They are simply not giving back to the > > > >>> community., > > > >>> > > > >>> If they did and officially said (and had proven as the Amazon and > > > Google > > > >>> team did multiple times for their integrations) that they are > willing > > > to > > > >>> support and maintain MSSQL DB, maybe we would reconsider - mostly > > > because > > > >>> we could have counted on having them step in when needed (again - > as > > it > > > >>> happened multiple times with Amazon and Google - when we reach out > > and > > > >> need > > > >>> their help we know we can count on it). I don't see a particular > > reason > > > >> why > > > >>> we should support their proprietary technology. > > > >>> > > > >>> J. > > > >>> > > > >>> > > > >>> On Fri, May 31, 2024 at 12:16 AM Damian Shaw < > > > >> ds...@striketechnologies.com > > > >>>> > > > >>> wrote: > > > >>> > > > >>>> I would say that MSSQL was often marked as "experimental" ( > > > >>>> > > > >>> > > > >> > > > > > > https://airflow.apache.org/docs/apache-airflow/2.6.0/howto/set-up-database.html > > > >>> ), > > > >>>> so IMO I don't think the evidence of it only being used by 5% is > > > >>>> particularly convincing that it wouldn't eventually be popular. > > Users > > > >> who > > > >>>> might want to primarily use MSSQL because of internal corporate > > > >>>> restrictions might have a large overlap with users who have > > > >> restrictions > > > >>> on > > > >>>> anything that says "experimental". > > > >>>> > > > >>>> I think the more important fact is it was a real burden on > > > development, > > > >>>> and there was no MSSQL champion in the Airflow maintainers. > > > >>>> > > > >>>> -----Original Message----- > > > >>>> From: Andrey Anshin <andrey.ans...@taragol.is> > > > >>>> Sent: Thursday, May 30, 2024 2:39 PM > > > >>>> To: dev@airflow.apache.org > > > >>>> Cc: james.du...@improving.com.invalid > > > >>>> Subject: Re: [DISCUSS] Restore the SQL server backend > > > >>>> > > > >>>> There was a proposal to keep it in the past [1] with a short > > > >> explanation > > > >>>> why the maintainers did not want to keep it. > > > >>>> > > > >>>>> many Microsoft customers who are using Airflow > > > >>>> > > > >>>> Microsoft also supports and participates in the development of > > > >>> PostgreSQL, > > > >>>> there is one Core Team member and couple of Major Contributors > > working > > > >> in > > > >>>> Microsoft [2] and in addition a couple years ago Microsoft > acquired > > > one > > > >>> of > > > >>>> the PostgreSQL vendors [3]. So I would like to believe that > > Microsoft > > > >>> also > > > >>>> could offer different services around PostgreSQL for their > > customers. > > > >>>> > > > >>>> > > > >>>> [1] Keep Mssql support: > > > >>>> https://lists.apache.org/thread/ot58ms069z4pyhj786j1m0dqds6lhjks > > > >>>> [2] PostgreSQL: Contributors Profiles: > > > >>>> https://www.postgresql.org/community/contributors/ > > > >>>> [3] Microsoft Acquires Citus Data: > > > >>>> > > > >> > > > > https://www.citusdata.com/blog/2019/01/24/microsoft-acquires-citus-data/ > > > >>>> > > > >>>> On Thu, 30 May 2024 at 21:18, Pierre Jeambrun < > > pierrejb...@gmail.com> > > > >>>> wrote: > > > >>>> > > > >>>>> I share Jed feeling. The effort required to maintain those > compare > > to > > > >>>>> the value it actually brings combined with the usage from the > > survey, > > > >>>>> it doesn’t seem worth it to me. > > > >>>>> > > > >>>>> On Thu 30 May 2024 at 19:16, Jed Cunningham < > > > >> jedcunning...@apache.org> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> Just for context, here were (roughly) the results from the 2023 > > > >>>>>> Airflow > > > >>>>>> survey: > > > >>>>>> > > > >>>>>> PostgreSQL: 75% > > > >>>>>> MySQL: 15% > > > >>>>>> MSSQL: 5% > > > >>>>>> > > > >>>>>> Also, there are already discussions about potentially dropping > > > >> MySQL > > > >>>>>> support in Airflow 3. Given all that and the points from the > past > > > >>>>>> vote, I don't think it makes much sense to bring MSSQL back. > > > >>>>>> > > > >>>>> > > > >>>> ________________________________ > > > >>>> Strike Technologies, LLC (“Strike”) is part of the GTS family of > > > >>>> companies. Strike is a technology solutions provider, and is not a > > > >> broker > > > >>>> or dealer and does not transact any securities related business > > > >> directly > > > >>>> whatsoever. This communication is the property of Strike and its > > > >>>> affiliates, and does not constitute an offer to sell or the > > > >> solicitation > > > >>> of > > > >>>> an offer to buy any security in any jurisdiction. It is intended > > only > > > >> for > > > >>>> the person to whom it is addressed and may contain information > that > > is > > > >>>> privileged, confidential, or otherwise protected from disclosure. > > > >>>> Distribution or copying of this communication, or the information > > > >>> contained > > > >>>> herein, by anyone other than the intended recipient is prohibited. > > If > > > >> you > > > >>>> have received this communication in error, please immediately > notify > > > >>> Strike > > > >>>> at i...@striketechnologies.com, and delete and destroy any copies > > > >>> hereof. > > > >>>> ________________________________ > > > >>>> > > > >>>> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any > > > >> attachments > > > >>>> are intended solely for the addressee. This transmission is > covered > > by > > > >>> the > > > >>>> Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The > > > >>>> information contained in this transmission is confidential in > nature > > > >> and > > > >>>> protected from further use or disclosure under U.S. Pub. L. > 106-102, > > > >> 113 > > > >>>> U.S. Stat. 1338 (1999), and may be subject to attorney-client or > > other > > > >>>> legal privilege. Your use or disclosure of this information for > any > > > >>> purpose > > > >>>> other than that intended by its transmittal is strictly > prohibited, > > > and > > > >>> may > > > >>>> subject you to fines and/or penalties under federal and state law. > > If > > > >> you > > > >>>> are not the intended recipient of this transmission, please > DESTROY > > > ALL > > > >>>> COPIES RECEIVED and confirm destruction to the sender via return > > > >>>> transmittal. > > > >>>> > > > >>> > > > >> > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > Warning: The sender of this message could not be validated and may not > be > > > the actual sender. > > > > > > Warning: The sender of this message could not be validated and may not be > the actual sender. >
