utkarsharma2 opened a new pull request, #35060: URL: https://github.com/apache/airflow/pull/35060
This PR is part of our larger effort to add first-class integrations to support LLMOps that was [presented at Airflow Summit](https://www.youtube.com/watch?v=mgA6m3ggKhs&t=4s). This PR adds explicitly the Weaviate Provider. Weaviate is an open-source vector database. It allows you to store data objects and vector embeddings. Storing vector embedding is one of the vital steps in LLM ops. This provider adds the ability to interact the with vector database using the phyton client. Example DAG: The `WeaviateIngestOperator` can accept either JSON or a callable that returns JSON. input_callable. ``` data = [ { "Answer": "Liver", "Category": "SCIENCE", "Question": "This organ removes excess glucose from the blood & stores it as glycogen" }, { "Answer": "Elephant", "Category": "ANIMALS", "Question": "It's the only living mammal in the order Proboseidea" } ] WeaviateIngestOperator( task_id="batch_data_without_vectors_xcom_data", conn_id="weaviate_default", class_name="QuestionWithOpenAIVectorizerUsingOperator", input_json=data, trigger_rule="all_done", ) ``` Email Discussion related to the effort can be found here - https://lists.apache.org/thread/0d669fmy4hn29h5c0wj0ottdskd77ktp -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
