Taragolis commented on code in PR #27775:
URL: https://github.com/apache/airflow/pull/27775#discussion_r1027743485
##########
airflow/providers/amazon/aws/hooks/batch_client.py:
##########
@@ -363,10 +363,8 @@ def get_job_description(self, job_id: str) -> dict:
return self.parse_job_description(job_id, response)
except botocore.exceptions.ClientError as err:
- error = err.response.get("Error", {})
- if error.get("Code") == "TooManyRequestsException":
- pass # allow it to retry, if possible
- else:
+ # Allow it to retry in case of exceeded quota limit of
requests to AWS API
+ if err.response.get("Error", {}).get("Code") !=
"TooManyRequestsException":
raise AirflowException(f"AWS Batch job ({job_id})
description error: {err}")
Review Comment:
I would also like remove this `try.. except` entirely because retry on
`TooManyRequestsException`
could be controlled by `boto3` configurations
-
https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#avoid-throttling-exceptions
-
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#retries
Unfortunately by default `boto3` work with `legacy` retry mode which due to
documentation does not retry request on `TooManyRequestsException`.
So this might be actually replaced by construction:
- Retry in hook level but notify user
- Re-raise actual error rather than AirflowException
```python
try:
...
except botocore.exceptions.ClientError as err:
if err.response.get("Error", {}).get("Code") ==
"TooManyRequestsException":
self.log.warning(
"TooManyRequestsException ignored. Please consider to setup
retries mode. "
"See:
https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#avoid-throttling-exceptions";
)
else:
raise
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
