Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-08 Thread via GitHub
utkarsharma2 merged PR #36085: URL: https://github.com/apache/airflow/pull/36085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-08 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1420282153 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -144,22 +148,63 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client =

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-08 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1420266830 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -30,7 +34,7 @@ from airflow.hooks.base import BaseHook if TYPE_CHECKING: -from typing import Any

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-08 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1420266560 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -17,10 +17,14 @@ from __future__ import annotations +import contextlib +import json import warnings

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-07 Thread via GitHub
ephraimbuddy commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1419153313 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -17,10 +17,14 @@ from __future__ import annotations +import contextlib +import json import warnings

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-07 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1418875440 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +141,52 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client =

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-07 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1418799391 ## airflow/providers/weaviate/operators/weaviate.py: ## @@ -51,21 +57,38 @@ def __init__( self, conn_id: str, class_name: str, -

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-07 Thread via GitHub
uranusjr commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1418707392 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +141,52 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client = self.conn

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-07 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1418700900 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +141,52 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client =

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-07 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1418700900 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +141,52 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client =

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-07 Thread via GitHub
uranusjr commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1418636135 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +141,52 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client = self.conn

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-07 Thread via GitHub
uranusjr commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1418634989 ## airflow/providers/weaviate/operators/weaviate.py: ## @@ -51,21 +57,38 @@ def __init__( self, conn_id: str, class_name: str, -

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417258114 ## airflow/providers/weaviate/operators/weaviate.py: ## @@ -51,21 +57,38 @@ def __init__( self, conn_id: str, class_name: str, -

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417256443 ## airflow/providers/weaviate/operators/weaviate.py: ## @@ -51,21 +57,38 @@ def __init__( self, conn_id: str, class_name: str, -

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417165337 ## airflow/providers/weaviate/operators/weaviate.py: ## @@ -40,9 +44,11 @@ class WeaviateIngestOperator(BaseOperator): :param conn_id: The Weaviate

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417162923 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +139,40 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client =

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417162074 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +139,40 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client =

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
pankajkoti commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417131009 ## airflow/providers/weaviate/operators/weaviate.py: ## @@ -40,9 +44,11 @@ class WeaviateIngestOperator(BaseOperator): :param conn_id: The Weaviate

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417082359 ## docs/apache-airflow-providers-weaviate/operators/weaviate.rst: ## @@ -28,7 +28,7 @@ into the database. Using the Operator ^^ -The

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417047431 ## docs/apache-airflow-providers-weaviate/operators/weaviate.rst: ## @@ -28,7 +28,7 @@ into the database. Using the Operator ^^ -The

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417047431 ## docs/apache-airflow-providers-weaviate/operators/weaviate.rst: ## @@ -28,7 +28,7 @@ into the database. Using the Operator ^^ -The

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417034666 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +139,40 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client =

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
uranusjr commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417031209 ## docs/apache-airflow-providers-weaviate/operators/weaviate.rst: ## @@ -28,7 +28,7 @@ into the database. Using the Operator ^^ -The

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
uranusjr commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417028234 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +139,40 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client = self.conn

Re: [PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
uranusjr commented on code in PR #36085: URL: https://github.com/apache/airflow/pull/36085#discussion_r1417027820 ## airflow/providers/weaviate/hooks/weaviate.py: ## @@ -135,22 +139,40 @@ def create_schema(self, schema_json: dict[str, Any]) -> None: client = self.conn

[PR] Add retry mechanism and dataframe support for WeaviateIngestOperator [airflow]

2023-12-06 Thread via GitHub
utkarsharma2 opened a new pull request, #36085: URL: https://github.com/apache/airflow/pull/36085 The PR add the retry mechanism while ingesting data into Weaviate, this is needed to ensure the data that we intend to ingest is not affected by the possible network or ratelimiting issue. We