Jarek Potiuk created AIRFLOW-4440:
-------------------------------------
Summary: Avoid terminating processes using shared resources
Key: AIRFLOW-4440
URL: https://issues.apache.org/jira/browse/AIRFLOW-4440
Project: Apache Airflow
Issue Type: Bug
Reporter: Jarek Potiuk
According to
[https://docs.python.org/3.5/library/multiprocessing.html#programming-guidelines]
we should avoid terminating processes when shared resources (Locks, Queues
etc. are used)
We are using terminate() in a few places in the code and we should replace it
with more graceful way of stopping the processes (sending poison pills) - see
the last example in
[https://docs.python.org/3.5/library/multiprocessing.html|https://docs.python.org/3.5/library/multiprocessing.html#programming-guidelines]
where 'STOP' is sent to shutdown the process.
This also prevents graceful joins in a number of cases - for example in
DagFileProcessorAgent.end() where we had to comment it out until this problem
is solved.
We should review and remove all terminates() unless they are absolutely
necessary and replae them with more graceful mechanism of shutting down.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
