[
https://issues.apache.org/jira/browse/AIRFLOW-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830207#comment-16830207
]
Jarek Potiuk commented on AIRFLOW-4440:
---------------------------------------
CC: [~ash]
> Avoid terminating processes using shared resources
> --------------------------------------------------
>
> Key: AIRFLOW-4440
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4440
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Jarek Potiuk
> Priority: Major
>
> According to
> [https://docs.python.org/3.5/library/multiprocessing.html#programming-guidelines]
> we should avoid terminating processes when shared resources (Locks, Queues
> etc. are used)
> We are using terminate() in a few places in the code and we should replace it
> with more graceful way of stopping the processes (sending poison pills) - see
> the last example in
> [https://docs.python.org/3.5/library/multiprocessing.html|https://docs.python.org/3.5/library/multiprocessing.html#programming-guidelines]
> where 'STOP' is sent to shutdown the process.
> This also prevents graceful joins in a number of cases - for example in
> DagFileProcessorAgent.end() where we had to comment it out until this problem
> is solved.
>
> We should review and remove all terminates() unless they are absolutely
> necessary and replae them with more graceful mechanism of shutting down.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
