Thanks everyone for the discussion, I will create a voting thread and lazy consensus for the open-sources ones.
Regards, Kaxil On Tue, 24 Oct 2023 at 18:19, Pankaj Koti <pankajk...@apache.org> wrote: > Hi Andrey, > > We discussed internally (at Astronomer) regarding your > suggestion on the PRs and we will incorporate the suggestion > in our PRs to remove the callable approach and include > examples in the docs to achieve the same using TaskFlow > and using hook methods in those. Thank you for > the suggestion, I believe that's easier to maintain and also > gives the power of the TaskFlow API. > > On 2023/10/19 16:03:47 Andrey Anshin wrote: > > Because 4 out 5 new providers have a draft PR I would like to raise a > > question about which related to all new providers. Just to avoid the > same > > question in all PRs. > > > > Do we actually want to make new operators kindish of like > "PythonOperator"? > > Maybe I miss some important thing and can't see why it would work better > > rather than run hooks methods inside of PythonOperator / TaskFlow? > > > > For the reference Reference: > > Add Cohere Provider: > > https://github.com/apache/airflow/pull/34921#discussion_r1358525838 > > Enable pgvector support for Postgres provider: > > https://github.com/apache/airflow/pull/34891#discussion_r1362910782 > > Add OpenAI Provider: > > https://github.com/apache/airflow/pull/35023#discussion_r1365235167 > > Add Weaviate Provider: > > https://github.com/apache/airflow/pull/35060/files#r1365765741 > > > > ---- > > Best Wishes > > *Andrey Anshin* > > > > > > > > On Tue, 17 Oct 2023 at 22:42, Kaxil Naik <kaxiln...@apache.org> wrote: > > > > > Hey Everyone, > > > > > > As a follow-up to my Keynote talk, Building and deploying LLM > applications > > > with Apache Airflow <https://www.youtube.com/watch?v=mgA6m3ggKhs&t=4s>, > I > > > am formally proposing the addition of these 5 providers to the Apache > > > Airflow repo: > > > > > > - > > > > > > PgVector <https://github.com/pgvector/pgvector> > > > - > > > > > > Weaviate <https://weaviate.io/> > > > - > > > > > > Pinecone <https://www.pinecone.io/> > > > - > > > > > > OpenAI <https://openai.com/> > > > - > > > > > > Cohere <https://cohere.com/> > > > > > > > > > Advancements in LLMs are moving at a rapid pace & transforming the way > we > > > work and our industry. Although LLMs are simple to use in prototyping, > > > using LLM for enterprise applications and for production still > presents a > > > lot of challenges. These > > > < > > > > https://speakerdeck.com/kaxil/building-and-deploying-llm-applications-with-apache-airflow?slide=8 > > > > > > > are some of the same problems that we tackle in Data Engineering, and > > > Airflow is a natural fit for them. > > > > > > We at Astronomer would like to add first-class support for the popular > LLMs > > > (OpenAI & Cohere) and vector DBs (PgVector, Weaviate & Pinecone) so > that > > > Data Scientists and ML engineers can utilize them natively with > easy-to-use > > > Operator & Hook abstractions while providing a native (and > > > Production-ready) approach for Authentication, retries, logging etc. > > > > > > We also think this is vital for the Apache Airflow project as we, the > > > project, embrace the LLM tide and continue to be a great example of > > > balancing innovation and maintaining backward-compatibility. > > > > > > The first versions of these providers will enable building one of the > most > > > common use cases of LLMs i.e. Question and Answering / Chatbots using > > > Retrieval-augmented generation (RAG) done with the help of embeddings. > > > > > > Everyone is welcome and encouraged to contribute once the PRs are > merged. > > > Astronomer is committed to maintaining these providers in the Airflow > repo, > > > including reviewing PRs, maintaining code quality, testing and keeping > the > > > APIs up-to-date. > > > > > > Note: PgVector <https://github.com/pgvector/pgvector> is an > open-source > > > project, so we don’t need a formal vote for it as per our guidelines > > > < > > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers > > > >. > > > So please consider this email as seeking a Lazy Consensus for it. > > > > > > I will open up a VOTING thread after discussing this for a few days. > > > > > > Thanks. > > > > > > Regards, > > > > > > Kaxil > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
