+1 from me as well, this should enable a lot of new usecases -- Regards, Aritra Basu
On Tue, Oct 24, 2023, 11:47 AM Amogh Desai <amoghdesai....@gmail.com> wrote: > +1 from me as well. Looks like something that the dev community would be > interested in using and hence contributing as well! > > Thanks & Regards, > Amogh Desai > > On Tue, Oct 24, 2023 at 2:37 AM utkarsh sharma <utkarshar...@gmail.com> > wrote: > > > +1 From my side as well. Also, having a dashboard to publish system test > > results for the providers will ensure the expected working in the long > run. > > > > Thanks, > > Utkarsh Sharma > > > > On Sun, Oct 22, 2023 at 4:44 PM Hussein Awala <huss...@awala.fr> wrote: > > > > > +1 Many people will be interested in contributing to these providers in > > the > > > future; I believe maintaining them will not be a challenge. > > > > > > On Sat, Oct 21, 2023 at 4:25 PM Pierre Jeambrun <pierrejb...@gmail.com > > > > > wrote: > > > > > > > +1 from me as well. > > > > > > > > Indeed LLMs for Apache Airflow is a no brainer, this will open a lot > of > > > > opportunities for our users. > > > > > > > > Having Astronomer supporting this initiative is great. I like the > idea > > of > > > > system tests dashboard, especially for those that are not open > > > > source/easily testable without a paid account. > > > > > > > > Also, now that we have the ability to suspend providers we can still > do > > > > that in case we change our mind in the future. > > > > > > > > Best regards, > > > > Pierre > > > > > > > > Le ven. 20 oct. 2023 à 08:35, Elad Kalif <elad...@apache.org> a > écrit > > : > > > > > > > > > +1 from me > > > > > I think the dashboard idea is great! > > > > > > > > > > On Thu, Oct 19, 2023 at 7:05 PM Andrey Anshin < > > > andrey.ans...@taragol.is> > > > > > wrote: > > > > > > > > > > > Because 4 out 5 new providers have a draft PR I would like to > > raise a > > > > > > question about which related to all new providers. Just to avoid > > the > > > > > same > > > > > > question in all PRs. > > > > > > > > > > > > Do we actually want to make new operators kindish of like > > > > > "PythonOperator"? > > > > > > Maybe I miss some important thing and can't see why it would work > > > > better > > > > > > rather than run hooks methods inside of PythonOperator / > TaskFlow? > > > > > > > > > > > > For the reference Reference: > > > > > > Add Cohere Provider: > > > > > > > > https://github.com/apache/airflow/pull/34921#discussion_r1358525838 > > > > > > Enable pgvector support for Postgres provider: > > > > > > > > https://github.com/apache/airflow/pull/34891#discussion_r1362910782 > > > > > > Add OpenAI Provider: > > > > > > > > https://github.com/apache/airflow/pull/35023#discussion_r1365235167 > > > > > > Add Weaviate Provider: > > > > > > https://github.com/apache/airflow/pull/35060/files#r1365765741 > > > > > > > > > > > > ---- > > > > > > Best Wishes > > > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 17 Oct 2023 at 22:42, Kaxil Naik <kaxiln...@apache.org> > > > wrote: > > > > > > > > > > > > > Hey Everyone, > > > > > > > > > > > > > > As a follow-up to my Keynote talk, Building and deploying LLM > > > > > > applications > > > > > > > with Apache Airflow < > > > > https://www.youtube.com/watch?v=mgA6m3ggKhs&t=4s > > > > > >, > > > > > > I > > > > > > > am formally proposing the addition of these 5 providers to the > > > Apache > > > > > > > Airflow repo: > > > > > > > > > > > > > > - > > > > > > > > > > > > > > PgVector <https://github.com/pgvector/pgvector> > > > > > > > - > > > > > > > > > > > > > > Weaviate <https://weaviate.io/> > > > > > > > - > > > > > > > > > > > > > > Pinecone <https://www.pinecone.io/> > > > > > > > - > > > > > > > > > > > > > > OpenAI <https://openai.com/> > > > > > > > - > > > > > > > > > > > > > > Cohere <https://cohere.com/> > > > > > > > > > > > > > > > > > > > > > Advancements in LLMs are moving at a rapid pace & transforming > > the > > > > way > > > > > we > > > > > > > work and our industry. Although LLMs are simple to use in > > > > prototyping, > > > > > > > using LLM for enterprise applications and for production still > > > > > presents a > > > > > > > lot of challenges. These > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://speakerdeck.com/kaxil/building-and-deploying-llm-applications-with-apache-airflow?slide=8 > > > > > > > > > > > > > > > are some of the same problems that we tackle in Data > Engineering, > > > and > > > > > > > Airflow is a natural fit for them. > > > > > > > > > > > > > > We at Astronomer would like to add first-class support for the > > > > popular > > > > > > LLMs > > > > > > > (OpenAI & Cohere) and vector DBs (PgVector, Weaviate & > Pinecone) > > so > > > > > that > > > > > > > Data Scientists and ML engineers can utilize them natively with > > > > > > easy-to-use > > > > > > > Operator & Hook abstractions while providing a native (and > > > > > > > Production-ready) approach for Authentication, retries, logging > > > etc. > > > > > > > > > > > > > > We also think this is vital for the Apache Airflow project as > we, > > > the > > > > > > > project, embrace the LLM tide and continue to be a great > example > > of > > > > > > > balancing innovation and maintaining backward-compatibility. > > > > > > > > > > > > > > The first versions of these providers will enable building one > of > > > the > > > > > > most > > > > > > > common use cases of LLMs i.e. Question and Answering / Chatbots > > > using > > > > > > > Retrieval-augmented generation (RAG) done with the help of > > > > embeddings. > > > > > > > > > > > > > > Everyone is welcome and encouraged to contribute once the PRs > are > > > > > merged. > > > > > > > Astronomer is committed to maintaining these providers in the > > > Airflow > > > > > > repo, > > > > > > > including reviewing PRs, maintaining code quality, testing and > > > > keeping > > > > > > the > > > > > > > APIs up-to-date. > > > > > > > > > > > > > > Note: PgVector <https://github.com/pgvector/pgvector> is an > > > > > open-source > > > > > > > project, so we don’t need a formal vote for it as per our > > > guidelines > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers > > > > > > > >. > > > > > > > So please consider this email as seeking a Lazy Consensus for > it. > > > > > > > > > > > > > > I will open up a VOTING thread after discussing this for a few > > > days. > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Kaxil > > > > > > > > > > > > > > > > > > > > > > > > > > > >
