NikolaySokolov commented on issue #18989:
URL: https://github.com/apache/airflow/issues/18989#issuecomment-1006669281
We are seeing it on `apache/airflow:2.2.3-python3.8`. Any advice on how to
troubleshoot it?
```
[2022-01-06 15:08:44,900] {kubernetes_executor.py:147} INFO - Event:
airflowdbcleanupprintconfiguration.a31e0e08678046d1878d5419d3eb70eb had an
event of type MODIFIED
[2022-01-06 15:08:44,901] {kubernetes_executor.py:201} INFO - Event:
airflowdbcleanupprintconfiguration.a31e0e08678046d1878d5419d3eb70eb Pending
[2022-01-06 15:08:45,898] {kubernetes_executor.py:147} INFO - Event:
airflowdbcleanupprintconfiguration.a31e0e08678046d1878d5419d3eb70eb had an
event of type MODIFIED
[2022-01-06 15:08:45,899] {kubernetes_executor.py:213} INFO - Event:
airflowdbcleanupprintconfiguration.a31e0e08678046d1878d5419d3eb70eb is Running
[2022-01-06 15:08:56,092] {scheduler_job.py:161} INFO - Exiting gracefully
upon receiving signal 15
[2022-01-06 15:08:57,097] {process_utils.py:120} INFO - Sending
Signals.SIGTERM to group 55. PIDs of all processes in the group: [4659, 4664,
4670, 4677, 4680, 4681, 55]
[2022-01-06 15:08:57,098] {process_utils.py:75} INFO - Sending the signal
Signals.SIGTERM to group 55
[2022-01-06 15:09:02,433] {process_utils.py:70} INFO - Process
psutil.Process(pid=4659, status='terminated', started='15:08:54') (4659)
terminated with exit code None
[2022-01-06 15:09:02,434] {process_utils.py:70} INFO - Process
psutil.Process(pid=4670, status='terminated', started='15:08:54') (4670)
terminated with exit code None
[2022-01-06 15:09:02,519] {process_utils.py:70} INFO - Process
psutil.Process(pid=4680, status='terminated', started='15:08:55') (4680)
terminated with exit code None
[2022-01-06 15:09:02,607] {process_utils.py:237} INFO - Waiting up to 5
seconds for processes to exit...
Exception ignored in: <function _releaseLock at 0x7f3d1b320940>
Traceback (most recent call last):
File "/usr/local/lib/python3.8/logging/__init__.py", line 227, in
_releaseLock
def _releaseLock():
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/dag_processing/manager.py",
line 500, in _exit_gracefully
sys.exit(os.EX_OK)
SystemExit: 0
Process ForkProcess-4:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in
_bootstrap
self.run()
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in
run
self._target(*self._args, **self._kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/dag_processing/manager.py",
line 287, in _run_processor_manager
processor_manager.start()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/dag_processing/manager.py",
line 520, in start
return self._run_parsing_loop()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/dag_processing/manager.py",
line 609, in _run_parsing_loop
self.collect_results()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/dag_processing/manager.py",
line 935, in collect_results
self._collect_results_from_processor(processor)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/dag_processing/manager.py",
line 904, in _collect_results_from_processor
if processor.result is not None:
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/dag_processing/processor.py",
line 322, in result
if not self.done:
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/dag_processing/processor.py",
line 287, in done
if self._parent_channel.poll():
File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 255,
in poll
self._check_closed()
File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 136,
in _check_closed
raise OSError("handle is closed")
OSError: handle is closed
[2022-01-06 15:09:02,652] {process_utils.py:70} INFO - Process
psutil.Process(pid=55, status='terminated', exitcode=1, started='15:06:11')
(55) terminated with exit code 1
[2022-01-06 15:09:02,653] {process_utils.py:70} INFO - Process
psutil.Process(pid=4664, status='terminated', started='15:08:54') (4664)
terminated with exit code None
[2022-01-06 15:09:02,653] {process_utils.py:70} INFO - Process
psutil.Process(pid=4681, status='terminated', started='15:08:56') (4681)
terminated with exit code None
[2022-01-06 15:09:02,653] {process_utils.py:70} INFO - Process
psutil.Process(pid=4677, status='terminated', started='15:08:55') (4677)
terminated with exit code None
[2022-01-06 15:09:02,654] {kubernetes_executor.py:790} INFO - Shutting down
Kubernetes executor
[2022-01-06 15:09:02,654] {scheduler_job.py:650} ERROR - Exception when
executing Executor.end
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py",
line 628, in _execute
self._run_scheduler_loop()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py",
line 730, in _run_scheduler_loop
time.sleep(min(self._scheduler_idle_sleep_time, next_event))
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py",
line 164, in _exit_gracefully
sys.exit(os.EX_OK)
SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py",
line 648, in _execute
self.executor.end()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py",
line 792, in end
self._flush_task_queue()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py",
line 745, in _flush_task_queue
self.log.debug('Executor shutting down, task_queue approximate size=%d',
self.task_queue.qsize())
File "<string>", line 2, in qsize
File "/usr/local/lib/python3.8/multiprocessing/managers.py", line 834, in
_callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 206,
in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 411,
in _send_bytes
self._send(header + buf)
File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 368,
in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
[2022-01-06 15:09:02,656] {process_utils.py:120} INFO - Sending
Signals.SIGTERM to group 55. PIDs of all processes in the group: []
[2022-01-06 15:09:02,657] {process_utils.py:75} INFO - Sending the signal
Signals.SIGTERM to group 55
[2022-01-06 15:09:02,657] {process_utils.py:89} INFO - Sending the signal
Signals.SIGTERM to process 55 as process group is missing.
[2022-01-06 15:09:02,657] {scheduler_job.py:655} INFO - Exited execute loop
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
