[
https://issues.apache.org/jira/browse/AIRFLOW-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949731#comment-16949731
]
ASF subversion and git services commented on AIRFLOW-5581:
----------------------------------------------------------
Commit 39f20a0b3168e045c5ae1a277981ddff03c3ae93 in airflow's branch
refs/heads/v1-10-stable from Kiran Pathak
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=39f20a0 ]
[AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler
shutdown on SIGTERM (#6237)
(cherry picked from commit 12f916a432e7937f6f72d1b556196e863920f334)
> KubernetesJobWatcher dies prematurely with num_runs or run_duration
> -------------------------------------------------------------------
>
> Key: AIRFLOW-5581
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5581
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, executor-kubernetes, executors, scheduler
> Affects Versions: 1.10.5
> Reporter: Kiran
> Assignee: Kiran
> Priority: Major
> Fix For: 1.10.6
>
>
> If run_duration or num_runs is enabled, KubernetesJobWatcher dies prematurely
> causing below error:
> [2019-10-01 19:40:35,323] \{kubernetes_executor.py:327} ERROR - Unknown error
> in KubernetesJobWatcher. Failing
> Traceback (most recent call last):
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 325, in run
> self.worker_uuid, self.kube_config)
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 359, in _run
> task.metadata.resource_version
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 391, in process_status
> self.watcher_queue.put((pod_id, None, labels, resource_version))
> File "<string>", line 2, in put
> File "/usr/lib64/python3.6/multiprocessing/managers.py", line 756, in
> _callmethod
> conn.send((self._id, methodname, args, kwds))
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 206, in send
> self._send_bytes(_ForkingPickler.dumps(obj))
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 404, in
> _send_bytes
> self._send(header + buf)
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 368, in _send
> n = write(self._handle, buf)
> BrokenPipeError: [Errno 32] Broken pipe
> Process KubernetesJobWatcher-3:
> Traceback (most recent call last):
> File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 325, in run
> self.worker_uuid, self.kube_config)
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 359, in _run
> task.metadata.resource_version
> File
> "/home/vccorp/.local/lib/python3.6/site-packages/airflow/contrib/executors/kubernetes_executor.py",
> line 391, in process_status
> self.watcher_queue.put((pod_id, None, labels, resource_version))
> File "<string>", line 2, in put
> File "/usr/lib64/python3.6/multiprocessing/managers.py", line 756, in
> _callmethod
> conn.send((self._id, methodname, args, kwds))
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 206, in send
> self._send_bytes(_ForkingPickler.dumps(obj))
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 404, in
> _send_bytes
> self._send(header + buf)
> File "/usr/lib64/python3.6/multiprocessing/connection.py", line 368, in _send
> n = write(self._handle, buf)
> BrokenPipeError: [Errno 32] Broken pipe
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
