[
https://issues.apache.org/jira/browse/AIRFLOW-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AIRFLOW-3378.
---------------------------------------
Resolution: Fixed
Assignee: Victor
Fix Version/s: 2.0.0
> KubernetesPodOperator does not delete on timeout failure
> --------------------------------------------------------
>
> Key: AIRFLOW-3378
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3378
> Project: Apache Airflow
> Issue Type: Bug
> Components: kubernetes
> Affects Versions: 1.10.1
> Reporter: Victor
> Assignee: Victor
> Priority: Major
> Fix For: 2.0.0
>
>
> I am runnig airflow 1.10.1rc2 on kubernetes with the LocalExecutior and
> KubernetesPodOperator to run pods.
>
> When a failure happens because the pod can't be created (see logs below), the
> whole operator fails and if is_delete_operator_pod is set to True, the pod is
> NOT deleted as expected.
>
> Logs:
> {noformat}
> *** Reading local file:
> /airflow/logs/demo/datapipe/2018-11-21T08:29:57.206456+00:00/1.log
> [2018-11-21 08:30:02,027] {models.py:1361} INFO - Dependencies all met for
> <TaskInstance: demo.datapipe 2018-11-21T08:29:57.206456+00:00 [queued]>
> [2018-11-21 08:30:02,040] {models.py:1361} INFO - Dependencies all met for
> <TaskInstance: demo.datapipe 2018-11-21T08:29:57.206456+00:00 [queued]>
> [2018-11-21 08:30:02,041] {models.py:1573} INFO -
> --------------------------------------------------------------------------------
> Starting attempt 1 of 1
> --------------------------------------------------------------------------------
> [2018-11-21 08:30:02,066] {models.py:1595} INFO - Executing
> <Task(KubernetesPodOperator): datapipe> on 2018-11-21T08:29:57.206456+00:00
> [2018-11-21 08:30:02,067] {base_task_runner.py:118} INFO - Running: ['bash',
> '-c', 'airflow run demo datapipe 2018-11-21T08:29:57.206456+00:00 --job_id 4
> --raw -sd /usr/local/airflow/dags_volume/..data/demo.py --cfg_path
> /tmp/tmp5qa5qyhs']
> [2018-11-21 08:30:03,065] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe [2018-11-21 08:30:03,065] {settings.py:174} INFO -
> setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
> [2018-11-21 08:30:03,734] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe [2018-11-21 08:30:03,732] {__init__.py:51} INFO - Using executor
> LocalExecutor
> [2018-11-21 08:30:04,038] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe [2018-11-21 08:30:04,038] {models.py:271} INFO - Filling up the
> DagBag from /usr/local/airflow/dags_volume/..data/demo.py
> [2018-11-21 08:30:04,590] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe [2018-11-21 08:30:04,590] {cli.py:484} INFO - Running <TaskInstance:
> demo.datapipe 2018-11-21T08:29:57.206456+00:00 [running]> on host
> infra-airflow-6d78c56489-r9trl
> [2018-11-21 08:30:04,684] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:04,676] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:05,698] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:05,698] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:06,706] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:06,706] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:07,715] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:07,714] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:08,723] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:08,723] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:09,730] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:09,730] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:10,736] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:10,736] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:11,742] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:11,742] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:12,748] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:12,748] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:13,754] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:13,754] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:14,764] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:14,764] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:15,769] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:15,769] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:16,777] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:16,777] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:17,783] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:17,783] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:18,790] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:18,790] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:19,796] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:19,796] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:20,802] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:20,802] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:21,808] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:21,808] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:22,815] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:22,815] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:23,822] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:23,822] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:24,833] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:24,833] {pod_launcher.py:121} INFO - Event: datapipe-a5591f0c had an
> event of type Pending
> [2018-11-21 08:30:24,841] {models.py:1760} ERROR - Pod Launching failed: Pod
> took too long to start
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 123, in execute
> get_logs=self.get_logs)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 86, in run_pod
> raise AirflowException("Pod took too long to start")
> airflow.exceptions.AirflowException: Pod took too long to start
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1659,
> in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 135, in execute
> raise AirflowException('Pod Launching failed: {error}'.format(error=ex))
> airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long
> to start
> [2018-11-21 08:30:24,844] {models.py:1791} INFO - Marking task as FAILED.
> [2018-11-21 08:30:25,001] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:25,001] {email.py:127} INFO - Sent an alert email to
> ['victor.n...@xxx.com']
> [2018-11-21 08:30:26,646] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe Traceback (most recent call last):
> [2018-11-21 08:30:26,646] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 123, in execute
> [2018-11-21 08:30:26,646] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe get_logs=self.get_logs)
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 86, in run_pod
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe raise AirflowException("Pod took too long to start")
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe airflow.exceptions.AirflowException: Pod took too long to start
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe During handling of the above exception, another exception occurred:
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe Traceback (most recent call last):
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe File "/usr/local/bin/airflow", line 32, in <module>
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe args.func(args)
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe File
> "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in
> wrapper
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe return f(*args, **kwargs)
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py",
> line 490, in run
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe _run(args, dag, ti)
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py",
> line 406, in _run
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe pool=args.pool,
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py",
> line 74, in wrapper
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe return func(*args, **kwargs)
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe File "/usr/local/lib/python3.6/site-packages/airflow/models.py",
> line 1659, in _run_raw_task
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe result = task_copy.execute(context=context)
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",
> line 135, in execute
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe raise AirflowException('Pod Launching failed:
> {error}'.format(error=ex))
> [2018-11-21 08:30:26,647] {base_task_runner.py:101} INFO - Job 4: Subtask
> datapipe airflow.exceptions.AirflowException: Pod Launching failed: Pod took
> too long to start
> [2018-11-21 08:30:27,054] {logging_mixin.py:95} INFO - [2018-11-21
> 08:30:27,053] {jobs.py:2627} INFO - Task exited with return code 1
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
