iw-pavan opened a new issue, #43440:
URL: https://github.com/apache/airflow/issues/43440
### Apache Airflow version
2.10.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Scheduler was running and launching tasks normally.
Suddenly there was auth error on database operations.
```
psycopg2.OperationalError: connection to server at "<Host>" (<IP>), port
6432 failed: FATAL: server login has been failing, try again later
(server_login_retry)
connection to server at "<HOST>" (<IP>), port 6432 failed: FATAL: server
login has been failing, try again later (server_login_retry)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/opt/app-root/lib64/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py",
line 984, in _execute
self._run_scheduler_loop()
```
After few retries it exited scheduler loop but process was not terminated.
### What you think should happen instead?
After shutting down all executor and dag_processer process should exit.
### How to reproduce
Using hybrid executors with Celery, Kubernetes
Introduce db errors.
### Operating System
Mac/Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else?
There are below logs repeated which indicates some threads not exited.
```
[2024-10-27T06:17:10.658+0000] {kubernetes_executor_utils.py:101} INFO -
Kubernetes watch timed out waiting for events. Restarting watch.
[2024-10-27T06:17:11.658+0000] {kubernetes_executor_utils.py:140} INFO -
Event: and now my watch begins starting at resource_version: 0
[2024-10-27T06:17:11.702+0000] {kubernetes_executor_utils.py:309} INFO -
Event: 666aac59b268675b6b2590ff-bs-8ace-s4sjuxfo is Running, annotations:
<omitted>
[2024-10-27T06:17:11.712+0000] {kubernetes_executor_utils.py:309} INFO -
Event: 666aac59b268675b6b2590ff-bs-44fe-iwuzjfao is Running, annotations:
<omitted>
[2024-10-27T06:17:41.715+0000] {kubernetes_executor_utils.py:101} INFO -
Kubernetes watch timed out waiting for events. Restarting watch.
```
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]