utkarsharma2 opened a new pull request, #34921: URL: https://github.com/apache/airflow/pull/34921
This PR is part of our larger effort to add first-class integrations to support LLMOps that was presented at Airflow Summit. This PR specifically adds the Cohere Provider. Cohere is a renowned platform offering a range of AI Models tailored for various NLP tasks. In this iteration, we are integrating with their Embeddings Model. The primary objective of this Provider is to present users with an alternative embedding model. This allows them to generate vectors for their proprietary data, a pivotal step towards establishing integrations with LLM models like ChatGPT. Example DAG: The `CohereEmbeddingOperator` can accept either a list of strings or a callable returning a list of strings. ``` from datetime import datetime from airflow import DAG from airflow.providers.cohere.operators.embedding import CohereEmbeddingOperator with DAG("example_cohere_embedding", schedule=None, start_date=datetime(2023, 1, 1), catchup=False) as dag: texts = [ "On Kernel-Target Alignment. We describe a family of global optimization procedures", " that automatically decompose optimization problems into smaller loosely coupled", " problems, then combine the solutions of these with message passing algorithms.", ] def get_text(): return texts CohereEmbeddingOperator(input_text=texts, task_id="embedding_via_text") CohereEmbeddingOperator(input_callable=get_text, task_id="embedding_via_callable") ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org