hussein-awala commented on issue #29557:
URL: https://github.com/apache/airflow/issues/29557#issuecomment-1436106024
Airflow raises `AirflowTaskTimeout` exception when the task timed out, and
your code can catch this exception and handle it if needed.
Since you have `[2023-02-15, 20:58:20 IST] {timeout.py:68} ERROR - Process
timed out, PID: 11392` in the log, the timeout exception is raised, but it
seems like your code is stuck in one of the finally blocks, where python calls
finally before raising the exception. I cannot check whats wrong with your
operator, but I can recommend some steps to debug the problem.
First, here is a simple exemple:
```python
import datetime
import time
from typing import Any
import pendulum
from airflow.models import BaseOperator
from airflow.models.dag import dag
from airflow.utils.context import Context
class MyOperator(BaseOperator):
def execute(self, context: Context) -> Any:
try:
print("try")
time.sleep(120)
except Exception as e:
print(e)
raise e
finally:
print("finally")
time.sleep(120)
@dag(
schedule=None,
start_date=pendulum.yesterday(),
)
def timeout_dag():
MyOperator(task_id="test_task",
execution_timeout=datetime.timedelta(seconds=30))
timeout_dag()
```
And here is the log
```
[2023-02-19, 21:27:29 UTC] {logging_mixin.py:149} INFO - try
[2023-02-19, 21:27:59 UTC] {timeout.py:68} ERROR - Process timed out, PID:
98045
[2023-02-19, 21:27:59 UTC] {logging_mixin.py:149} INFO - Timeout, PID: 98045
[2023-02-19, 21:27:59 UTC] {logging_mixin.py:149} INFO - finally
[2023-02-19, 21:29:59 UTC] {taskinstance.py:1837} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/files/dags/dag18.py", line 19, in execute
raise e
File "/files/dags/dag18.py", line 16, in execute
time.sleep(120)
File "/opt/airflow/airflow/utils/timeout.py", line 69, in handle_timeout
raise AirflowTaskTimeout(self.error_message)
airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 98045
```
To detect the problem, you can create a simple test to execute your operator
in the IDE in debug mode, and use breakpoints at each step to follow the call
stack. If you find this complicated, you can add a `print` after each line and
read the log from the UI to find where the task is stuck.
Also if you have some special actions to do when the task is timed out, you
can add this in a new except block:
```python
except AirflowTaskTimeout as timeout_exception:
do_something()
raise timeout_exception
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
