Haapalaj opened a new issue, #67939: URL: https://github.com/apache/airflow/issues/67939
### Under which category would you file this issue? Airflow Core ### Apache Airflow version 3.2.2 ### What happened and how to reproduce it? ** Issue Description ** With the LocalExecutor having a DAG executing an task that runs e.g for 3 hours or more. In about 100 minutes, the task is killed for missing / expired heartbeat. The task itself fails to send the heartbeat because of expired JWT token. Seems that the task does not refresh the JWT token. If the EXECUTION_API__JWT_EXPIRATION_TIME is set to much longer value from the default 600s, the task will continue running. ** Steps to Reproduce: ** With similar setup as described in: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#concepts-task-instance-heartbeat-timeout The default configuration: AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_SEC=0 AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT=300 AIRFLOW__EXECUTION_API__JWT_EXPIRATION_TIME=600 Run the DAG: dag_TEST_TIMEOUT.py: ``` from airflow.sdk import DAG, dag from airflow.providers.standard.operators.bash import BashOperator from datetime import datetime, timedelta import os name='TEST_TIMEOUT' default_args = { 'owner': 'airflow', 'depends_on_past': False, 'email_on_failure': False, 'email_on_retry': False, 'retries': 0, 'retry_delay': timedelta(minutes=5), 'type': 'utf-8' } # https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#concepts-task-instance-heartbeat-timeout @dag(start_date=datetime(2021, 1, 1), schedule="@once", catchup=False, default_args=default_args) def sleep_dag(): t1 = BashOperator( task_id="sleep_180_minutes", bash_command=""" echo AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_SEC: $AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_SEC echo AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT: $AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT echo AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL: $AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT_DETECTION_INTERVAL echo AIRFLOW__EXECUTION_API__JWT_EXPIRATION_TIME: $AIRFLOW__EXECUTION_API__JWT_EXPIRATION_TIME sleep 10800""", ) sleep_dag() ``` ### What you think should happen instead? The task should renew the JWT token. ### Operating System debian ### Deployment Other Docker-based deployment ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers apache-airflow-providers-standard==1.13.1 apache-airflow-providers-snowflake==6.13.0 apache-airflow-providers-amazon==9.29.0 apache-airflow-providers-google==21.3.0 apache-airflow-providers-microsoft-azure==13.3.0 apache-airflow-providers-odbc==4.12.2 apache-airflow-providers-postgres==6.7.0 apache-airflow-providers-databricks==7.15.0 apache-airflow-task-sdk==1.2.2 apache-airflow-providers-fab==3.6.4 apache-airflow-core[gunicorn] ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
