+1 Many people will be interested in contributing to these providers in the future; I believe maintaining them will not be a challenge.
On Sat, Oct 21, 2023 at 4:25 PM Pierre Jeambrun <pierrejb...@gmail.com> wrote: > +1 from me as well. > > Indeed LLMs for Apache Airflow is a no brainer, this will open a lot of > opportunities for our users. > > Having Astronomer supporting this initiative is great. I like the idea of > system tests dashboard, especially for those that are not open > source/easily testable without a paid account. > > Also, now that we have the ability to suspend providers we can still do > that in case we change our mind in the future. > > Best regards, > Pierre > > Le ven. 20 oct. 2023 à 08:35, Elad Kalif <elad...@apache.org> a écrit : > > > +1 from me > > I think the dashboard idea is great! > > > > On Thu, Oct 19, 2023 at 7:05 PM Andrey Anshin <andrey.ans...@taragol.is> > > wrote: > > > > > Because 4 out 5 new providers have a draft PR I would like to raise a > > > question about which related to all new providers. Just to avoid the > > same > > > question in all PRs. > > > > > > Do we actually want to make new operators kindish of like > > "PythonOperator"? > > > Maybe I miss some important thing and can't see why it would work > better > > > rather than run hooks methods inside of PythonOperator / TaskFlow? > > > > > > For the reference Reference: > > > Add Cohere Provider: > > > https://github.com/apache/airflow/pull/34921#discussion_r1358525838 > > > Enable pgvector support for Postgres provider: > > > https://github.com/apache/airflow/pull/34891#discussion_r1362910782 > > > Add OpenAI Provider: > > > https://github.com/apache/airflow/pull/35023#discussion_r1365235167 > > > Add Weaviate Provider: > > > https://github.com/apache/airflow/pull/35060/files#r1365765741 > > > > > > ---- > > > Best Wishes > > > *Andrey Anshin* > > > > > > > > > > > > On Tue, 17 Oct 2023 at 22:42, Kaxil Naik <kaxiln...@apache.org> wrote: > > > > > > > Hey Everyone, > > > > > > > > As a follow-up to my Keynote talk, Building and deploying LLM > > > applications > > > > with Apache Airflow < > https://www.youtube.com/watch?v=mgA6m3ggKhs&t=4s > > >, > > > I > > > > am formally proposing the addition of these 5 providers to the Apache > > > > Airflow repo: > > > > > > > > - > > > > > > > > PgVector <https://github.com/pgvector/pgvector> > > > > - > > > > > > > > Weaviate <https://weaviate.io/> > > > > - > > > > > > > > Pinecone <https://www.pinecone.io/> > > > > - > > > > > > > > OpenAI <https://openai.com/> > > > > - > > > > > > > > Cohere <https://cohere.com/> > > > > > > > > > > > > Advancements in LLMs are moving at a rapid pace & transforming the > way > > we > > > > work and our industry. Although LLMs are simple to use in > prototyping, > > > > using LLM for enterprise applications and for production still > > presents a > > > > lot of challenges. These > > > > < > > > > > > > > > > https://speakerdeck.com/kaxil/building-and-deploying-llm-applications-with-apache-airflow?slide=8 > > > > > > > > > are some of the same problems that we tackle in Data Engineering, and > > > > Airflow is a natural fit for them. > > > > > > > > We at Astronomer would like to add first-class support for the > popular > > > LLMs > > > > (OpenAI & Cohere) and vector DBs (PgVector, Weaviate & Pinecone) so > > that > > > > Data Scientists and ML engineers can utilize them natively with > > > easy-to-use > > > > Operator & Hook abstractions while providing a native (and > > > > Production-ready) approach for Authentication, retries, logging etc. > > > > > > > > We also think this is vital for the Apache Airflow project as we, the > > > > project, embrace the LLM tide and continue to be a great example of > > > > balancing innovation and maintaining backward-compatibility. > > > > > > > > The first versions of these providers will enable building one of the > > > most > > > > common use cases of LLMs i.e. Question and Answering / Chatbots using > > > > Retrieval-augmented generation (RAG) done with the help of > embeddings. > > > > > > > > Everyone is welcome and encouraged to contribute once the PRs are > > merged. > > > > Astronomer is committed to maintaining these providers in the Airflow > > > repo, > > > > including reviewing PRs, maintaining code quality, testing and > keeping > > > the > > > > APIs up-to-date. > > > > > > > > Note: PgVector <https://github.com/pgvector/pgvector> is an > > open-source > > > > project, so we don’t need a formal vote for it as per our guidelines > > > > < > > > > > > > > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers > > > > >. > > > > So please consider this email as seeking a Lazy Consensus for it. > > > > > > > > I will open up a VOTING thread after discussing this for a few days. > > > > > > > > Thanks. > > > > > > > > Regards, > > > > > > > > Kaxil > > > > > > > > > >
