chAnvesh opened a new issue, #54479:
URL: https://github.com/apache/airflow/issues/54479
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Hi,
**CONFIGURATION**
This is our airflow setup configuration and we are running it inside a K8s
pod with Postgres as metadataDB and CeleryExecutor.
airflow info:
Apache Airflow
version | 3.0.2
executor | CeleryExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn | postgresql+psycopg2://user:@localhost:5432/airflow
dags_folder | /opt/airflow/dags
plugins_folder | /opt/airflow/plugins
base_log_folder | /opt/airflow/logs
remote_base_log_folder |
System info
OS | Linux
architecture | x86_64
uname | uname_result(system='Linux', node='*********',
release='6.2.0-32-generic', version='#32~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri
Aug 18 10:40:13 UTC 2', machine='x86_64')
locale | ('en_US', 'UTF-8')
python_version | 3.9.21 (main, Jun 27 2025, 00:00:00) [GCC 11.5.0 20240719
(Red Hat 11.5.0-5)]
python_location | /usr/bin/python3
Tools info
git | NOT AVAILABLE
ssh | OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024
kubectl | NOT AVAILABLE
gcloud | NOT AVAILABLE
cloud_sql_proxy | NOT AVAILABLE
mysql | NOT AVAILABLE
sqlite3 | NOT AVAILABLE
psql | psql (PostgreSQL) 13.20
Paths info
airflow_home | /opt/airflow
system_path |
/home/appuser/.local/bin:/home/appuser/bin:/opt/app/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:usr/bin/psql
We updated the JWT Expiration timers to 1 week to avoid authentication
errors.
export AIRFLOW__EXECUTION_API__JWT_EXPIRATION_TIME=604860
export AIRFLOW__API_AUTH__JWT_EXPIRATION_TIME=604860
The Celery executor "visibility_timeout" we see is the default hard-coded
value(NOT the default None value described in airflow.cfg or [celery providers
documentation](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.11.0/configurations-ref.html#visibility-timeout)
)
broker_transport_options | {'visibility_timeout': 84600}
**PROBLEM**
We have a dag which has a 36hrs sleep period task. This task is getting
killed after around ~23 hrs. And we do not have any CPU/Memory
problems/deficiencies in the pod.
The same task was working perfectly fine in our older Airflow 2.10 release.
**LOGS:**
**SCHEDULER LOGS IN LOCALTIME(UTC+3:00):**
[2025-08-12T13:50:06.745+0300] {scheduler_job_runner.py:450} INFO - 1 tasks
up for execution:
<TaskInstance: ETL_overload_dag.sleep_until_ETL_overload
manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7 [scheduled]>
[2025-08-12T13:50:06.745+0300] {scheduler_job_runner.py:522} INFO - DAG
ETL_overload_dag has 0/10 running and queued tasks
[2025-08-12T13:50:06.746+0300] {scheduler_job_runner.py:661} INFO - Setting
the following tasks to queued state:
<TaskInstance: ETL_overload_dag.sleep_until_ETL_overload
manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7 [scheduled]>
[2025-08-12T13:50:06.747+0300] {scheduler_job_runner.py:767} INFO - Trying
to enqueue tasks: [<TaskInstance: ETL_overload_dag.sleep_until_ETL_overload
manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7 [scheduled]>] for executor:
CeleryExecutor(parallelism=24)
[2025-08-12T13:50:06.781+0300] {scheduler_job_runner.py:810} INFO - Received
executor event with state queued for task instance
TaskInstanceKey(dag_id='ETL_overload_dag', task_id='sleep_until_ETL_overload',
run_id='manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7', try_number=1,
map_index=-1)
[2025-08-12T13:50:06.785+0300] {scheduler_job_runner.py:842} INFO - Setting
external_id for <TaskInstance: ETL_overload_dag.sleep_until_ETL_overload
manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7 [queued]> to
1712b1a1-683d-4aa3-b8e4-b942823d017e
[2025-08-13T13:08:09.917+0300] {scheduler_job_runner.py:2128} INFO -
Adopting or resetting orphaned tasks for active dag runs
[2025-08-13T13:13:09.975+0300] {scheduler_job_runner.py:2128} INFO -
Adopting or resetting orphaned tasks for active dag runs
[2025-08-13T13:18:10.052+0300] {scheduler_job_runner.py:2128} INFO -
Adopting or resetting orphaned tasks for active dag runs
[2025-08-13T13:20:07.481+0300] {scheduler_job_runner.py:810} INFO - Received
executor event with state failed for task instance
TaskInstanceKey(dag_id='ETL_overload_dag', task_id='sleep_until_ETL_overload',
run_id='manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7', try_number=1,
map_index=-1)
[2025-08-13T13:20:07.485+0300] {scheduler_job_runner.py:852} INFO -
TaskInstance Finished: dag_id=ETL_overload_dag,
task_id=sleep_until_ETL_overload,
run_id=manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7, map_index=-1,
run_start_date=2025-08-12 10:50:06.983034+00:00, run_end_date=None,
run_duration=None, state=running, executor=CeleryExecutor(parallelism=24),
executor_state=failed, try_number=1, max_tries=10, pool=default_pool,
queue=default, priority_weight=22, operator=PythonOperator,
queued_dttm=2025-08-12 10:50:06.746562+00:00, scheduled_dttm=2025-08-12
10:50:06.725380+00:00,queued_by_job_id=1, pid=164019
[2025-08-13T13:20:07.486+0300] {scheduler_job_runner.py:925} ERROR - DAG
'ETL_overload_dag' for task instance <TaskInstance:
ETL_overload_dag.sleep_until_ETL_overload
manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7 [running]> not found in
serialized_dag table
[2025-08-13T13:20:07.486+0300] {taskinstance.py:1882} ERROR - Executor
CeleryExecutor(parallelism=24) reported that the task instance <TaskInstance:
ETL_overload_dag.sleep_until_ETL_overload
manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7 [running]> finished with
state failed, but the task instance's state attribute is running. Learn more:
https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally
[2025-08-13T13:20:07.524+0300] {taskinstance.py:2005} INFO - Marking task as
UP_FOR_RETRY. dag_id=ETL_overload_dag, task_id=sleep_until_ETL_overload,
run_id=manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7, logical_date=,
start_date=20250812T105006, end_date=20250813T102007
**CELERY EXECUTOR LOGS (UTC):**
STARTED AT
**2025-08-12 10:50:06.750815** [info ] Task
execute_workload[1712b1a1-683d-4aa3-b8e4-b942823d017e] received
[celery.worker.strategy]
2025-08-12 10:50:06.959275 [info ]
[1712b1a1-683d-4aa3-b8e4-b942823d017e] Executing workload in Celery:
token='eyJ***' ti=TaskInstance(id=UUID('01989de6-d041-7eec-80b8-e1c6c5da5a30'),
task_id='sleep_until_ETL_overload', dag_id='ETL_overload_dag',
run_id='manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7', try_number=1,
map_index=-1, pool_slots=1, queue='default', priority_weight=22,
executor_config=None, parent_context_carrier={}, context_carrier={},
queued_dttm=None)
dag_rel_path=PurePosixPath('overload_control/ETL_overload_dag.py')
bundle_info=BundleInfo(name='dags-folder', version=None)
log_path='ETL_overload_dag/sleep_until_ETL_overload/manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7/1.log'
type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2025-08-12 10:50:06.972550 [info ] Secrets backends loaded for worker
[supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1
KILLED AFTER
**2025-08-13 10:20:07.013034** [info ] Task
execute_workload[1712b1a1-683d-4aa3-b8e4-b942823d017e] received
[celery.worker.strategy]
2025-08-13 10:20:07.019987 [info ]
[1712b1a1-683d-4aa3-b8e4-b942823d017e] Executing workload in Celery:
token='eyJ***' ti=TaskInstance(id=UUID('01989de6-d041-7eec-80b8-e1c6c5da5a30'),
task_id='sleep_until_ETL_overload', dag_id='ETL_overload_dag',
run_id='manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7', try_number=1,
map_index=-1, pool_slots=1, queue='default', priority_weight=22,
executor_config=None, parent_context_carrier={}, context_carrier={},
queued_dttm=None)
dag_rel_path=PurePosixPath('overload_control/ETL_overload_dag.py')
bundle_info=BundleInfo(name='dags-folder', version=None)
log_path='ETL_overload_dag/sleep_until_ETL_overload/manual__2025-08-12T10:50:06.216504+00:00_MNRKWnM7/1.log'
type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2025-08-13 10:20:07.031539 [info ] Secrets backends loaded for worker
[supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1
**2025-08-13 10:20:07.051392 [warning ] Server error
[airflow.sdk.api.client] detail={'detail': {'reason':
[celery_error.log](https://github.com/user-attachments/files/21759536/celery_error.log)
'invalid_state', 'message': 'TI was not in a state where it could be marked
as running', 'previous_state': 'running'}}
2025-08-13 10:20:07.061038 [info ] Process exited
[supervisor] exit_code=-9 pid=1877111 signal_sent=SIGKILL
2025-08-13 10:20:07.108748 [error ] Task
execute_workload[1712b1a1-683d-4aa3-b8e4-b942823d017e] raised unexpected:
ServerResponseError('Server returned error') [celery.app.trace]**
### What you think should happen instead?
The sleep task should have completed successfully after 36hrs and continued
to the downstream tasks.
### How to reproduce
1. Run Airflow with Celery executor with the configuration provided
2. Create a task with more than 24 hrs sleep and run it
### Operating System
NAME="Red Hat Enterprise Linux" VERSION="9.6 (Plow)"
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
We install airflow using PIP
### Anything else?
Everytime
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
