[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Terry J. Reedy]

Change by Terry J. Reedy :


--
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Pascal Chambon]

Pascal Chambon  added the comment:

The latest doc has a quick mention about the fact that daemon is not used in 
the Unix sens, so it seems fine now  B-)

https://docs.python.org/3/library/multiprocessing.html?#multiprocessing.Process.daemon

"""Additionally, these are not Unix daemons or services, they are normal 
processes that will be terminated (and not joined) if non-daemonic processes 
have exited."""

My paragraph was just my one attempt at distinguishing concepts, it was never 
part of the official docs

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Is the paragraph "Usually, daemon processes ..." in Pascal's initial post 
supposed to be a quote from 
https://docs.python.org/3/library/multiprocessing.html?  'daemon p' is nowhere 
in the corrent doc and I find nothing like this paragraph.  Is this issue still 
relevant?

--
assignee: jnoller -> 
components: +Documentation
nosy: +terry.reedy
versions: +Python 3.10 -Python 2.6, Python 2.7, Python 3.0, Python 3.1

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Tim Peters]

Tim Peters added the comment:

Again, I don't care about orphans.  In the usual cases people are running code 
with no concern about what happens in case of forced termination.  The only 
thing stopping it now is that the code goes out of its way to prevent it (or, 
alternatively, goes out of its way to force Pool workers to be daemonic).  So, 
e.g., people publish recipes for replacing multiprocessing.Pool to remove this 
restriction, like here:

http://stackoverflow.com/questions/6974695/python-process-pool-non-daemonic/8963618#8963618

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Eryk Sun]

Eryk Sun added the comment:

> e.g., a Pool worker dies with an assert(!) error if it tries to create 
> its own Pool for something

A daemon process could create a child daemon when the OS can ensure that no 
orphans are left behind (e.g. call Linux prctl to set PR_SET_PDEATHSIG as 
SIGKILL). This would allow a pool worker to create another process pool.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Tim Peters]

Tim Peters added the comment:

@Eryk, not me ;-)  I find the "daemonic or not?" distinction useless for 
processes - it just gets in the way at times (e.g., a Pool worker dies with an 
assert(!) error if it tries to create its own Pool for something).

I don't care about orphans either.

It's one reason `concurrent.futures.ProcessPoolExecutor` is sometimes more 
attractive (it has no "daemon" concept, the processes it creates don't vanish 
midstream at sometimes surprising times, and neither do they refuse to create 
additional cf process pools).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Eryk Sun]

Eryk Sun added the comment:

> I've personally found this `multiprocessing` restriction (daemonic 
> processes can't have children) to be nothing but a pain in the ass 

To make that reliable on Windows you could create a silent-breakaway, 
kill-on-close Job object in each process for its daemon children. Closing the 
job handle would cascade terminate all descendant daemon processes in that 
branch of the process tree, i.e. when a process is terminated it closes the Job 
handle, which terminates all of its daemon children, which closes their Job 
handles, which terminates all of their daemon children, and so on. Non-daemons 
would still be orphaned when the parent is terminated.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Tim Peters]

Tim Peters added the comment:

I'll take a crack at these ancient comments ;-)

"""
daemon
The process’s daemon flag, a Boolean value. This must be set before start() 
is called.
The initial value is inherited from the creating process. [1]
When a process exits, it attempts to terminate all of its daemonic child 
processes. [2]

[1] this sentence is weird: since daemonic processes are not allowed to have 
children, isn't this flag always False in a new Process instance ?
"""

Yes.  I agree it's confusing; looks like it was pasted from the `threading` 
docs a long time ago, in which context it had a non-degenerate meaning.

I've personally found this `multiprocessing` restriction (daemonic processes 
can't have children) to be nothing but a pain in the ass, especially since 
`Pool` worker processes are (although this doesn't appear to be documented) 
created as daemonic processes.  (Note that 
`concurrent.futures.ProcessPoolExecutor` doesn't have this restriction.)

"""
[2] typo, it meant "all of its NON-daemonic child processes" instead, didn't it 
?
"""

No, this was correct.  When a process exits, it waits to join its non-daemonic 
children, but brutally terminates its daemonic children midstream.  For 
example, run this:

import multiprocessing as mp
from time import sleep

def e(tag):
for i in range(10):
print(tag, i)
sleep(1)

if __name__ == '__main__':
p = mp.Process(target=e, args=("daemon",), daemon=True)
q = mp.Process(target=e, args=("normal",))
p.start()
q.start()
sleep(5)

You'll see that, after 5 seconds, the main process exits and terminates the 
daemonic process immediately, but waits for the non-daemonic process to finish:

daemon 0
normal 0
daemon 1
normal 1
daemon 2
normal 2
daemon 3
normal 3
daemon 4
normal 4
normal 5
normal 6
normal 7
normal 8
normal 9

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Pascal Chambon]

Pascal Chambon chambon.pas...@gmail.com added the comment:

I've just crossed again the doc of the daemon flag 
(http://docs.python.org/library/multiprocessing.html), and it's still quite 
confusing to newcomers.

daemon
The process’s daemon flag, a Boolean value. This must be set before start() 
is called.
The initial value is inherited from the creating process. [1]
When a process exits, it attempts to terminate all of its daemonic child 
processes.

[1] this sentence is weird: since daemonic processes are not allowed to have 
children, isn't this flag always False in a new Process instance ?
[2] typo, it meant all of its NON-daemonic child processes instead, didn't it 
?

--
resolution: fixed - 
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5906
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

committed, r73693 on trunk

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5906
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Vaibhav Mallya]

Vaibhav Mallya mally...@umich.edu added the comment:

I understand pakal's line of reasoning. The term 'daemon' in the general
Unix sense has a specific meaning that is at odds with the
multiprocessing module's usage of 'daemon'. Clarification would be
useful, I feel, especially if an outright rename of that part of the API
is out of the question.

--
nosy: +mallyvai

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5906
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

I'm more than familiar with the Unix meaning of Daemon - and the API will 
*not* be changed or renamed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5906
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Pascal Chambon]

Pascal Chambon chambon.pas...@gmail.com added the comment:

I guess we all agree on the fact that a renaming of the API would be
highly disproportionate ;-)
A simple line of warning in the doc, next to the daemonic-related
methods, should be sufficient to prevent people like me from wondering
weird stuffs for hours ^^

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5906
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Pascal Chambon]

New submission from Pascal Chambon chambon.pas...@gmail.com:

Usually, daemon processes are processes which got disconnected from
their parent process, and work in the background, often under a
different user identity.
The multiprocessing module has the concept of daemon too, but this
time in reference to the threading module, in which dameons are just
threads that wont prevent the application termination, even if they are
still running. Thus, daemonic processes launched through multiprocessing
API are normal processes that will be terminated (and not joined) if 
non-dameonic processes are all over.

I guess this difference of concepts would deserve a little paragraph of
clarification, in both multiprocessing and threading APIs (does the
paragraph above fit for that ?)

--
components: Library (Lib)
messages: 86957
nosy: pakal
severity: normal
status: open
title: Risk of confusion in multiprocessing module - daemonic processes
versions: Python 2.6, Python 2.7, Python 3.0, Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5906
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Benjamin Peterson]

Changes by Benjamin Peterson benja...@python.org:


--
assignee:  - jnoller
nosy: +jnoller

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5906
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

The multiprocessing lib mimics the threading library, and the 
threading.Thread.daemon has always maintained these are not 
services/daemons/etc.

I don't see that the clarification is needed, but let me think about it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5906
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5906] Risk of confusion in multiprocessing module - daemonic processes

[Pascal Chambon]

Pascal Chambon chambon.pas...@gmail.com added the comment:

I agree that for someone who discovers the multiprocessing api as a
generalization of the threading api, there won't be problems ; 

I'm just worried about those (like me) who will see daemonic as coming
from unix processes, and not the threading library B-)

Cheers, 
pascal

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5906
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com