arjunanan6 commented on issue #33402:
URL: https://github.com/apache/airflow/issues/33402#issuecomment-1678965803

   Unfortunately, I do not know how to reproduce it yet. I've only seen this 
problem happen since the change to job_heartbeat_sec yesterday - but that was 
applied to all environments. 
   
   The thing is that everything works perfectly if I were to delete the 
scheduler, but then it ends up back in this situation again in about 20-30 min. 
I can see a lot of errors relating to KubernetesJobWatcher, but I can also see 
instances of this error in my other environments - and there is so far no 
problem there. Here's the log for that:
   
   ```
   [2023-08-15T13:00:05.201+0000] {scheduler_job_runner.py:677} INFO - Received 
executor event with state queued for task instance 
TaskInstanceKey(dag_id='ApttusADEmail', task_id=...skipping...
   [2023-08-15T13:43:46.333+0000] {kubernetes_executor.py:114} ERROR - Unknown 
error in KubernetesJobWatcher. Failing
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
710, in _error_catcher
       yield
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
1077, in read_chunked
       self._update_chunk_length()
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
1005, in _update_chunk_length
       line = self._fp.fp.readline()  # type: ignore[union-attr]
     File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
       return self._sock.recv_into(b)
     File "/usr/local/lib/python3.9/ssl.py", line 1242, in recv_into
       return self.read(nbytes, buffer)
     File "/usr/local/lib/python3.9/ssl.py", line 1100, in read
       return self._sslobj.read(len, buffer)
   ConnectionResetError: [Errno 104] Connection reset by peer
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py",
 line 105, in run
       self.resource_version = self._run(
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py",
 line 161, in _run
       for event in self._pod_events(kube_client=kube_client, 
query_kwargs=kwargs):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/watch/watch.py", 
line 165, in stream
       for line in iter_resp_lines(resp):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/watch/watch.py", 
line 56, in iter_resp_lines
       for seg in resp.stream(amt=None, decode_content=False):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
937, in stream
       yield from self.read_chunked(amt, decode_content=decode_content)
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
1106, in read_chunked
       self._original_response.close()
     File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
       self.gen.throw(typ, value, traceback)
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
727, in _error_catcher
       raise ProtocolError(f"Connection broken: {e!r}", e) from e
   urllib3.exceptions.ProtocolError: ("Connection broken: 
ConnectionResetError(104, 'Connection reset by peer')", 
ConnectionResetError(104, 'Connection reset by peer'))
   Process KubernetesJobWatcher-3:
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
710, in _error_catcher
       yield
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
1077, in read_chunked
       self._update_chunk_length()
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
1005, in _update_chunk_length
       line = self._fp.fp.readline()  # type: ignore[union-attr]
     File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
       return self._sock.recv_into(b)
     File "/usr/local/lib/python3.9/ssl.py", line 1242, in recv_into
       return self.read(nbytes, buffer)
     File "/usr/local/lib/python3.9/ssl.py", line 1100, in read
       return self._sslobj.read(len, buffer)
   ConnectionResetError: [Errno 104] Connection reset by peer
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in 
_bootstrap
       self.run()
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py",
 line 105, in run
       self.resource_version = self._run(
     File 
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py",
 line 161, in _run
       for event in self._pod_events(kube_client=kube_client, 
query_kwargs=kwargs):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/watch/watch.py", 
line 165, in stream
       for line in iter_resp_lines(resp):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/watch/watch.py", 
line 56, in iter_resp_lines
       for seg in resp.stream(amt=None, decode_content=False):
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
937, in stream
       yield from self.read_chunked(amt, decode_content=decode_content)
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
1106, in read_chunked
       self._original_response.close()
     File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
       self.gen.throw(typ, value, traceback)
     File 
"/home/airflow/.local/lib/python3.9/site-packages/urllib3/response.py", line 
727, in _error_catcher
       raise ProtocolError(f"Connection broken: {e!r}", e) from e
   urllib3.exceptions.ProtocolError: ("Connection broken: 
ConnectionResetError(104, 'Connection reset by peer')", 
ConnectionResetError(104, 'Connection reset by peer'))
   [2023-08-15T13:43:47.588+0000] {kubernetes_executor.py:335} ERROR - Error 
while health checking kube watcher process for namespace airflow-test-ns. 
Process died for unknown reasons
   [2023-08-15T13:43:47.597+0000] {kubernetes_executor.py:150} INFO - Event: 
and now my watch begins starting at resource_version: 0
   [2023-08-15T13:43:47.645+0000] {kubernetes_executor.py:243} INFO - Event: 
iga-hfm-upload-to-sftp-9dvhxqyd Succeeded
   [2023-08-15T13:43:47.664+0000] {kubernetes_executor.py:243} INFO - Event: 
iga-hfm-cleanup-temp-path-3t2zea2y Succeeded
   [2023-08-15T13:43:47.671+0000] {kubernetes_executor.py:243} INFO - Event: 
sap-wmd-send-mail-to-releaser-sap-wmd-send-mail-to-releaser-l77bh8z8 Succeeded
   [2023-08-15T13:43:47.678+0000] {kubernetes_executor.py:243} INFO - Event: 
iga-hfm-hfm-accounts-p2cptgu0 Succeeded
   [2023-08-15T13:43:47.685+0000] {kubernetes_executor.py:243} INFO - Event: 
it-erpbut-grac-role-usage-sync-h-it-erpbut-grac-role-usage-sync-h-9vpyp2xu 
Succeeded
   [2023-08-15T13:43:47.693+0000] {kubernetes_executor.py:243} INFO - Event: 
iga-hfm-hfm-entitilement-76rlv9y6 Succeeded
   [2023-08-15T13:43:50.024+0000] {kubernetes_executor.py:679} INFO - Changing 
state of (TaskInstanceKey(dag_id='IGA_HFM', task_id='UPLOAD_TO_SFTP', 
run_id='manual__2023-08-15T13:38:24.337649+00:00', try_number=1, map_index=-1), 
None, 'iga-hfm-upload-to-sftp-9dvhxqyd', 'airflow-test-ns', '359365852') to None
   [2023-08-15T13:43:50.051+0000] {kubernetes_executor.py:238} INFO - Skipping 
event for Succeeded pod iga-hfm-upload-to-sftp-9dvhxqyd - event for this pod 
already sent to executor
   [2023-08-15T13:43:50.068+0000] {kubernetes_executor.py:238} INFO - Skipping 
event for Succeeded pod iga-hfm-upload-to-sftp-9dvhxqyd - event for this pod 
already sent to executor
   [2023-08-15T13:43:50.070+0000] {kubernetes_executor.py:764} INFO - Deleted 
pod: TaskInstanceKey(dag_id='IGA_HFM', task_id='UPLOAD_TO_SFTP', 
run_id='manual__2023-08-15T13:38:24.337649+00:00', try_number=1, map_index=-1) 
in namespace airflow-test-ns
   ```
   
   So it looks to me like somehow it recovers and continues to be able to 
delete pods - but this activity completely comes to a standstill at a later 
point - unfortunately no specific time for that either from my observation. 
Does this log for kubewatcher issues give you more insight into the problem? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to