utkarsharma2 commented on code in PR #36177:
URL: https://github.com/apache/airflow/pull/36177#discussion_r1431800832
##########
airflow/providers/weaviate/hooks/weaviate.py:
##########
@@ -396,28 +399,46 @@ def batch_data(
.. seealso:: `batch_config_params options
<https://weaviate-python-client.readthedocs.io/en/v3.25.3/weaviate.batch.html#weaviate.batch.Batch.configure>`__
:param vector_col: name of the column containing the vector.
:param retry_attempts_per_object: number of time to try in case of
failure before giving up.
+ :param tenant: The tenant to which the object will be added.
+ :param uuid_col: Name of the column containing the UUID.
"""
client = self.conn
if not batch_config_params:
batch_config_params = {}
client.batch.configure(**batch_config_params)
data = self._convert_dataframe_to_list(data)
+ insertion_errors = []
with client.batch as batch:
# Batch import all data
- for index, data_obj in enumerate(data):
- for attempt in Retrying(
- stop=stop_after_attempt(retry_attempts_per_object),
- retry=(
- retry_if_exception(lambda exc:
check_http_error_is_retryable(exc))
- | retry_if_exception_type(REQUESTS_EXCEPTIONS_TYPES)
- ),
- ):
- with attempt:
- self.log.debug(
- "Attempt %s of importing data: %s",
attempt.retry_state.attempt_number, index + 1
- )
- vector = data_obj.pop(vector_col, None)
- batch.add_data_object(data_obj, class_name,
vector=vector)
+ try:
Review Comment:
@mpgreg @sunank200 I have added a callback to capture the third kind of
error. Please take a look.
Also, handled timeout and connection errors via Weavaite's batch options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
