fdemiane opened a new issue, #73196: URL: https://github.com/apache/airflow/issues/73196
### Description Hello all, I would appreciate your input about the below enhancement to the functionality/semantics of "[workers] execution_api_retries": ### Problem In Airflow 3, there is this configuration parameter **[workers] execution_api_retries** (as well as execution_api_retry_wait_min and execution_api_retry_wait_max). This variable controls how many attempts to retry an API request to the API server before failing it, with an exponential backoff (See https://github.com/apache/airflow/blob/3.2.2/task-sdk/src/airflow/sdk/api/client.py#L998). To increase the resiliency of an Airflow Environment against outages in the API server, [workers] execution_api_retries can be increased, and **it should be in sync** with [scheduler]task_instance_heartbeat_timeout (or else, the task can be declared zombie before actually failing, and potentially get retried, and in rarer race conditions, have a split-brain problem where 2 same tasks are executing code in parallel). See https://github.com/apache/airflow/blob/3.2.2/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L2201 for a good starting point. Yet, there is a **problem**: [workers] execution_api_retries **also controls** the API retries for requests sent to the in-process API server of the **triggerer** (https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/jobs/triggerer_job_runner.py#L601) and the **dag processor**. So now, to avoid errors, and also split brain problems (which can occur for triggerer asset watchers in certain race conditions, see https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/models/trigger.py#L277), we must **sync this variable** with [scheduler] task_instance_heartbeat_timeout and [triggerer] triggerer_health_check_threshold This means that changing [workers] execution_api_retries **requires syncing it with many other variables**, or else, although rare, unwanted functionality can occur. ### Proposal What do you think about having **different variables controlling the retries for different components**? Example: having [workers] execution_api_retries, [triggerer] execution_api_retries and [dag_processor] execution_api_retries. The same for execution_api_retry_wait_min and execution_api_retry_wait_max. This will make changing API retries for one component much safer than have one variable controlling retries for all currently supported components. Thanks! ### Use case/motivation _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
