Taragolis commented on issue #35164: URL: https://github.com/apache/airflow/issues/35164#issuecomment-1788106289
My 50 cents. It is a classical Chicken And Egg problem. For write into the CloudWatch Logs you need to have Log Groups, so you need to check is this log group exists or not and create it but both requests to limited by 5 TPS https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html as result with default retries strategy it is quite easy to have a Throttling error if spawn multiple task and since mapping task quite popular it become easier to got this error. The suggestion here, and just give optional opportunity do not check if is cloudwatch group exist or not, which is already supported by watchtower. No need any other stuff because if we would try to manually check is this cloudwatch group exists or not then we would spawn same error. Just because there is no better way to check it. Another option is configure retries, side effect of this it might take significant amount of time due to exponential backoff just for logger initialization Yet another option would be request AWS Support to increase quotes for this API Call, according to documentation it could be done because it is not hard quota. And final solution it is switch from cloudwatch remote logger to s3 remote logger -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
