arjunanan6 commented on issue #33402:
URL: https://github.com/apache/airflow/issues/33402#issuecomment-1678965803
Unfortunately, I do not know how to reproduce it yet. I've only seen this
problem happen since the change to job_heartbeat_sec yesterday - but that was
applied to all environments.
The thing is that everything works perfectly if I were to delete the
scheduler, but then it ends up back in this situation again in about 20-30 min.
I can see a lot of errors relating to KubernetesJobWatcher, but I can also see
instances of this error in my other environments - and there is so far no
problem there. Here's the log for that:
```
[2023-08-15T13:00:05.201+0000] {scheduler_job_runner.py:677} INFO - Received
executor event with state queued for task instance
TaskInstanceKey(dag_id='ApttusADEmail', task_id=...skipping...
[2023-08-15T13:43:46.333+0000] {kubernetes_executor.py:114} ERROR - Unknown
error in KubernetesJobWatcher. Failing
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
710, in _error_catcher
yield
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
1077, in read_chunked
self._update_chunk_length()
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
1005, in _update_chunk_length
line = self._fp.fp.readline() # type: ignore[union-attr]
File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.9/ssl.py", line 1242, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.9/ssl.py", line 1100, in read
return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py",
line 105, in run
self.resource_version = self._run(
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py",
line 161, in _run
for event in self._pod_events(kube_client=kube_client,
query_kwargs=kwargs):
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/watch/watch.py",
line 165, in stream
for line in iter_resp_lines(resp):
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/watch/watch.py",
line 56, in iter_resp_lines
for seg in resp.stream(amt=None, decode_content=False):
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
937, in stream
yield from self.read_chunked(amt, decode_content=decode_content)
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
1106, in read_chunked
self._original_response.close()
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
727, in _error_catcher
raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ("Connection broken:
ConnectionResetError(104, 'Connection reset by peer')",
ConnectionResetError(104, 'Connection reset by peer'))
Process KubernetesJobWatcher-3:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
710, in _error_catcher
yield
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
1077, in read_chunked
self._update_chunk_length()
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
1005, in _update_chunk_length
line = self._fp.fp.readline() # type: ignore[union-attr]
File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.9/ssl.py", line 1242, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.9/ssl.py", line 1100, in read
return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in
_bootstrap
self.run()
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py",
line 105, in run
self.resource_version = self._run(
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py",
line 161, in _run
for event in self._pod_events(kube_client=kube_client,
query_kwargs=kwargs):
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/watch/watch.py",
line 165, in stream
for line in iter_resp_lines(resp):
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/watch/watch.py",
line 56, in iter_resp_lines
for seg in resp.stream(amt=None, decode_content=False):
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
937, in stream
yield from self.read_chunked(amt, decode_content=decode_content)
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
1106, in read_chunked
self._original_response.close()
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line
727, in _error_catcher
raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ("Connection broken:
ConnectionResetError(104, 'Connection reset by peer')",
ConnectionResetError(104, 'Connection reset by peer'))
[2023-08-15T13:43:47.588+0000] {kubernetes_executor.py:335} ERROR - Error
while health checking kube watcher process for namespace airflow-test-ns.
Process died for unknown reasons
[2023-08-15T13:43:47.597+0000] {kubernetes_executor.py:150} INFO - Event:
and now my watch begins starting at resource_version: 0
[2023-08-15T13:43:47.645+0000] {kubernetes_executor.py:243} INFO - Event:
iga-hfm-upload-to-sftp-9dvhxqyd Succeeded
[2023-08-15T13:43:47.664+0000] {kubernetes_executor.py:243} INFO - Event:
iga-hfm-cleanup-temp-path-3t2zea2y Succeeded
[2023-08-15T13:43:47.671+0000] {kubernetes_executor.py:243} INFO - Event:
sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-l77bh8z8 Succeeded
[2023-08-15T13:43:47.678+0000] {kubernetes_executor.py:243} INFO - Event:
iga-hfm-hfm-accounts-p2cptgu0 Succeeded
[2023-08-15T13:43:47.685+0000] {kubernetes_executor.py:243} INFO - Event:
it-erpbut-grac-role-usage-sync-h-it-erpbut-grac-role-usage-sync-h-9vpyp2xu
Succeeded
[2023-08-15T13:43:47.693+0000] {kubernetes_executor.py:243} INFO - Event:
iga-hfm-hfm-entitilement-76rlv9y6 Succeeded
[2023-08-15T13:43:50.024+0000] {kubernetes_executor.py:679} INFO - Changing
state of (TaskInstanceKey(dag_id='IGA_HFM', task_id='UPLOAD_TO_SFTP',
run_id='manual__2023-08-15T13:38:24.337649+00:00', try_number=1, map_index=-1),
None, 'iga-hfm-upload-to-sftp-9dvhxqyd', 'airflow-test-ns', '359365852') to None
[2023-08-15T13:43:50.051+0000] {kubernetes_executor.py:238} INFO - Skipping
event for Succeeded pod iga-hfm-upload-to-sftp-9dvhxqyd - event for this pod
already sent to executor
[2023-08-15T13:43:50.068+0000] {kubernetes_executor.py:238} INFO - Skipping
event for Succeeded pod iga-hfm-upload-to-sftp-9dvhxqyd - event for this pod
already sent to executor
[2023-08-15T13:43:50.070+0000] {kubernetes_executor.py:764} INFO - Deleted
pod: TaskInstanceKey(dag_id='IGA_HFM', task_id='UPLOAD_TO_SFTP',
run_id='manual__2023-08-15T13:38:24.337649+00:00', try_number=1, map_index=-1)
in namespace airflow-test-ns
```
So it looks to me like somehow it recovers and continues to be able to
delete pods - but this activity completely comes to a standstill at a later
point - unfortunately no specific time for that either from my observation.
Does this log for kubewatcher issues give you more insight into the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]