Wrong choice of words, didn't mean "plugin" but was referring to a more
generic term.

Thank you!

On Wed, Oct 25, 2023 at 5:51 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> > The alternatives suggested by @Jarek Potiuk <ja...@potiuk.com> are
> something which is doable. We need to realise that these plugins are for
> the community and we can only support it if
> "majority" of the community uses it and is willing to maintain it :)
>
> Just to clarify - it's not about plugins, it's about "core" metastore
> support for MSSQL. The MSSQL provider will remain as is, but Airflow will
> lose the ability of using MsSQL as backend for
> scheduling/tasks/UI/executors/communication with workers via DB.
>
> The overhead / extra maintenance there is to make sure that all the
> queries, performance, locking and anything else airflow uses to make "core"
> works with MsSQL.
>
> In Airflow 2.8 the only remaining backends will be MySQL 8 (as MySQL 5.7
> reaches end of life in a week), and Postgres 12-16 (in 2 weeks Postgres 11
> reaches EOL and we will drop it).
> Those will be the only choices for Airflow 2.8 - providing that we will
> remove MsSQL as planned (condition for that is that we describe a viable
> migration path for MsSQL users).
>
> J.
>
>
>
> On Wed, Oct 25, 2023 at 1:23 PM Amogh Desai <amoghdesai....@gmail.com>
> wrote:
>
>> I agree with the comments and where this discussion has led to.
>>
>> The alternatives suggested by @Jarek Potiuk <ja...@potiuk.com> are
>> something which is doable. We need to realise that these plugins are for
>> the community and we can only support it if
>> "majority" of the community uses it and is willing to maintain it :)
>>
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>> On Wed, Oct 25, 2023 at 4:13 PM Kaxil Naik <kaxiln...@gmail.com> wrote:
>>
>>> Yeah agreed, I don’t think it is worth keeping the support of MSSQL given
>>> the amount of usage vs the maintenance effort required.
>>>
>>> On Tue, 24 Oct 2023 at 23:07, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> > Yes. Agree with Andrey.  I think our experience from the last few
>>> years was
>>> > "very" bad. The number of mssql users is very small. And the time that
>>> > maintainers and community members lose on various problems with it is
>>> huge.
>>> > Quite often every time we added a new feature requiring some new db
>>> > functionality, quite a lot of overhead was spent by the one adding new
>>> > features to solve the problems coming just from MSSQL support. It's not
>>> > "existing" issues - it's that it generally slows us down with making
>>> > changes
>>> >
>>> > I think there are two options for you when you. It's not "open issues",
>>> > it's the maintenance
>>> >
>>> > * switch to another backend (recommended). And it's not as complex as
>>> you
>>> > think. You can also use managed DB with all that is needed
>>> > (backup/maintenance), you do not have to manage it yourself. There are
>>> some
>>> > excellent postgres options available.
>>> >
>>> > * have your own fork airflow and keep the tests running and make your
>>> copy
>>> > works for MSSQL if you insist on keeping it. Since you already seem to
>>> be
>>> > ready to spend your engineering time on it, that seems doable.
>>> >
>>> > The second option I think might even be a business opportunity - for
>>> your
>>> > company or for anyone who would like to do it. Someone could even
>>> offer it
>>> > as a service or as a version to support it for others and make a small
>>> > business out of it if you are really so committed to it I guess,
>>> including
>>> > support for any mssql problems.
>>> >
>>> > That would actually be awesome if someone does it.
>>> >
>>> > J.
>>> >
>>> >
>>> > On Tue, Oct 24, 2023 at 11:44 PM Andrey Anshin <
>>> andrey.ans...@taragol.is>
>>> > wrote:
>>> >
>>> > > I don’t think there is any possibility left to keep MS SQL Server as
>>> DB
>>> > > backend for Airflow.
>>> > >
>>> > > I add Elad's message from the original discussion:
>>> > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4
>>> > > Because it cleary describe what is happen with MS SQL as DB backend
>>> for
>>> > the
>>> > > last 1.5 years
>>> > >
>>> > > > During this time we hoped it would become stable and widely
>>> adopted.
>>> > > > To my taste MsSQL a backend has left a niche and is *not* worth
>>> the >
>>> > > maintenance
>>> > > of it in our CI.
>>> > >
>>> > > I also want to note the following points
>>> > > - MS SQL have unstable tests in Airflow CI, and some cases we even
>>> don't
>>> > > run them for the last couple months (or even longer)
>>> > > - In additional it taking 2x memory than any other backend
>>> > > - Lack of ARM support, this is also quite important because it
>>> prevent
>>> > > maintainers to check some sort of things in their M1/M2 laptops
>>> > > - Additional backend required extra effort for any contributors who
>>> want
>>> > to
>>> > > add new feature that touches DB
>>> > >
>>> > >
>>> > > This has always been an experimental feature which are described in
>>> > AIrflow
>>> > > Release Process:
>>> > >
>>> > >
>>> >
>>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features
>>> > > , I would recommend your team focuses on Airflow on Postgres rather
>>> than
>>> > > hanging on to vague hope that MS SQL keeping in Airflow.
>>> > >
>>> > > Quite a few companies provide Managed Airflow, see:
>>> > > https://airflow.apache.org/ecosystem/#airflow-as-a-service (this is
>>> not
>>> > > complete list) and AFAIK none of them use any other backend rather
>>> than
>>> > > Postgres, maybe one exception with Google Composer v1 which seems use
>>> > > MySQL, even on Azure Data Factory Managed Airflow use Postgres as DB
>>> > > backend, see:
>>> > >
>>> > >
>>> >
>>> https://learn.microsoft.com/en-us/azure/data-factory/concept-managed-airflow#architecture
>>> > >
>>> > > ----
>>> > > Best Wishes
>>> > > *Andrey Anshin*
>>> > >
>>> > >
>>> > >
>>> > > On Mon, 23 Oct 2023 at 23:01, agateaaa <agate...@gmail.com> wrote:
>>> > >
>>> > > > Hi All:
>>> > > >
>>> > > > Mssql support was voted to be dropped.
>>> > > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4
>>> > > >
>>> > > > One of our product requirements is that we can only use the Mssql
>>> > > database.
>>> > > > The product that uses airflow is installed with a suite of 8-10
>>> other
>>> > > > products that all use Mssql database as their database. Preferably
>>> we
>>> > do
>>> > > > not want our customers to install another database like postgres or
>>> > MySQL
>>> > > > since it would involve extra overhead on their DBA team to maintain
>>> > (with
>>> > > > backup/restore functions) yet another database
>>> > > >
>>> > > > This has been already discussed and voted on but is there any way
>>> we
>>> > can
>>> > > > keep experimental support if we pitch in to fix any mssql related
>>> > issues?
>>> > > >
>>> > > > List of current mssql issues are here
>>> > > > *
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://github.com/apache/airflow/issues?q=is%3Aissue+label%3Abackend-mssql-experimintal+is%3Aopen
>>> > > >
>>> > > > Are there any other outstanding issues or can you please let us
>>> know a
>>> > > way
>>> > > > to identify mssql related bugs/problems that need to be addressed?
>>> > > >
>>> > > > e.g.
>>> > > > * https://github.com/apache/airflow/discussions/35114
>>> > > >
>>> > > >
>>> > > > We are just trying to understand how much effort will be required
>>> to
>>> > keep
>>> > > > supporting Mssql and if we can help with that in any way.
>>> > > >
>>> > > > Thanks
>>> > > > Ashwin
>>> > > >
>>> > >
>>> >
>>>
>>

Reply via email to