Re: PID lockfile (was: PEP 3143: Standard daemon process library)
(replying in ‘comp.lang.python’ for wider feedback on this issue) On 26-Mar-2009, Francis Irving wrote: On Thu, Mar 26, 2009 at 12:51:06AM +1100, Ben Finney wrote: The ‘python-daemon’ distribution includes a module, ‘daemon.pidlockfile’. The ‘daemon.pidlockfile.PIDLockFile’ class is intended to be used for this purpose. I am working with the developer of ‘lockfile’ to incorporate the ‘PIDLockFile’ class into that library. Ah! I didn't know about that! Can you update: a) The example in the PEP to use it. No, PEP 3143 is deliberately not tied to that implementation. PID file handling is purely an interface from the point of view of that PEP. When I said “the PIDLockFile class is intended to be used for this purpose”, that's strictly one-way. PEP 3143 is not meant to refer to the PIDLockFile class, only to a generic interface (context manager) for a ‘pidfile’ object. If the ‘pidlockfile’ module improves enough, and the ‘lockfile’ maintainer gets my changes into a release in time, I may update the PEP to recommend that; but so far, it's outside the scope. b) In the documentation, give a list of different classes that might be appropriate to use here. Recommending that one, but pointing to the ones in lockfile also. I don't have one to recommend yet: c) Its own documentation (maybe some doctests in the source? or some help?) which show how it works. I'm still not having much luck with it though :( This is part of why it's not recommended in the PEP; it's not really ready. Thank you for testing it :-) It doesn't seem to have a constructor which sets its path, so do I do: ourlockfile = daemon.pidlockfile.PIDLockFile Presuming you mean ‘daemon.pidlockfile.PIDLockFile()’ here (that is, get the return value of the class constructor, not the class itself). ourlockfile.path = '/tmp/mydaemon.pid' context = daemon.DaemonContext( pidfile=ourlockfile, stdout=logout, stderr=logout ) If so, it doesn't work, it just exits without an error. Can you please provide the full traceback? Another thing: I've noticed there are some ^L characters in daemon.py, e.g. just before class DaemonError, def change_working_directory etc. Yes. That is a page break (ASCII FF), useful for printing the file or navigating the text file by “page”. Python, like most languages, treats them like any other whitespace in the source. So, I think that PIDLockFile will leave the lockfile there if the daemon is killed abruptly (say with kill -9 or due to some bug). Certainly, all the lock classes in lockfile.py have the problem that it stays locked int hat circumstance. I have my programs delete the lockfile on start-up if it's stale (i.e. check for the existence of the file on start-up and delete it if the referenced PID is not running). Perhaps there are better ways. This page (which seems pretty good anyway, and I'm sure you've seen!) section 6) suggests using lockf, although I believe from elsewhere that fcntl will do also. http://www.enderunix.org/docs/eng/daemon.php No, I've not seen that, but I have seen others; they tend to differ in the details. I have looked for a more canonical reference for the intricacies of PID file handling, but it seems to be much more ad hoc than the definition of the daemonisation procedure. My impression is that the lockf is linked to the process, so if the process is killed the kernel will automatically free it. So my suggestion would be to store the pid in a pidfile, and lockf it. Not sure that is the exact convention used by most daemons on Debian, but it might be. On Linux at least, ‘lockf’ is not defined to alter the file at all; it doesn't cause it to be created nor removed. It is purely for a lock on an existing file. It wouldn't be cross platform though. I imagine Windows code for this should be very different from Unix, however - making a service. Explicitly outside the scope of PEP 3143; though it is hoped that the described functionality will make a good basis on which to *build* such a service, on Unix. Would be lovely to have something that provided one interface to both eventually, but probably too wild for now! There is a skeletal PEP on the ‘python-ideas’ list for this purpose URL:http://mail.python.org/pipermail/python-ideas/2009-January/002606.html. Anyone should feel free to develop it further. Thank you for your testing and feedback! And thank you! Your project is well worth doing, Unix daemons are so arcane, and to make them more Pythonic is lovely. -- \ “When we talk to God, we're praying. When God talks to us, | `\ we're schizophrenic.” —Jane Wagner, via Lily Tomlin, 1985 | _o__) | Ben Finney b...@benfinney.id.au -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
Ben Finney ben+pyt...@benfinney.id.au writes: I've submitted PEP 3143 URL:http://www.python.org/dev/peps/pep-3143/ to meet this need, and have re-worked an existing library into a new ‘python-daemon’ URL:http://pypi.python.org/pypi/python-daemon/ library, the reference implementation. Now I need wider testing and scrutiny of the implementation and specification. Thank you to those who have submitted bug reports so far. I have addressed some bugs and uploaded version 1.4.4 of ‘python-daemon’ to PyPI. Changes include: * Catch and report some OSError exceptions thrown by various steps. * Wait until later in the daemonisation process to cloase all open file descriptors. This gives a chance to see errors reported earlier in the process! * Redirect standard streams to the null device if no stream specified. Testing and feedback is still welcome, I want to knock this PEP and implementation into better shape. -- \ “I have never imputed to Nature a purpose or a goal, or | `\anything that could be understood as anthropomorphic.” —Albert | _o__)Einstein, unsent letter, 1955 | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
On Tue, 24 Mar 2009 15:42:46 +1100, Ben Finney bignose+hates-s...@benfinney.id.au wrote: Jean-Paul Calderone exar...@divmod.com writes: [snip] An additional feature which would be useful for the library to provide, however, would be the setting of euid and egid instead of uid and gid. This is necessary, for example, to write an SSH daemon which gives out user shells. That sounds rather more specific than is needed for the generic library being proposed here. I'm wary of adding features to an API that is already quite complex. Isn't setting the EUID and EGID something that is just as easily done *after* the program achieves a daemon process? That depends. If you mean that one can ignore the uid and gid setting features of the proposed library so that they are not changed during daemonization and then make the appropriate calls from the application afterwards, then yes. Otherwise, no. Since this means all of your daemon startup code is forced to run as a privileged process when it might otherwise have run without those privileges, I think it's worth the tiny additional complexity it will bring to the API (and it really is pretty tiny, something on the order of a new `set_effective=True´ flag). Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
On Mar 21, 11:06 pm, Ben Finney ben+pyt...@benfinney.id.au wrote: Floris Bruynooghe floris.bruynoo...@gmail.com writes: Had a quick look at the PEP and it looks very nice IMHO. Thank you. I hope you can try the implementation and report feedback on that too. One of the things that might be interesting is keeping file descriptors from the logging module open by default. Hmm. I see that this would be a good idea. but it raises the question of how to manage the set of file handles that should not be closed on becoming a daemon. So far, the logic of closing the file descriptors is a little complex: * Close all open file descriptors. This excludes those listed in the `files_preserve` attribute, and those that correspond to the `stdin`, `stdout`, or `stderr` attributes. Extending that by saying “… and also any file descriptors for ``logging.FileHandler`` objects” starts to make the description too complex. I have a strong instinct that it the description is complex, the design might be bad. Can you suggest an alternative API that will ensure that all file descriptors get closed *except* those that should not be closed? Not an answer yet, but I'll try to find time in the next few days to play with this and tell you what I think. logging.FileHandler would be too narrow in any case I think. Regards Floris -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
Jean-Paul Calderone exar...@divmod.com writes: On Tue, 24 Mar 2009 15:42:46 +1100, Ben Finney bignose+hates-s...@benfinney.id.au wrote: That sounds rather more specific than is needed for the generic library being proposed here. I'm wary of adding features to an API that is already quite complex. Isn't setting the EUID and EGID something that is just as easily done *after* the program achieves a daemon process? That depends. If you mean that one can ignore the uid and gid setting features of the proposed library so that they are not changed during daemonization and then make the appropriate calls from the application afterwards, then yes. Yes, that's what I meant. Otherwise, no. Since this means all of your daemon startup code is forced to run as a privileged process when it might otherwise have run without those privileges Er? You can still set the real UID and GID via the DaemonContext API, and then also set the EUID and EGID. I think it's worth the tiny additional complexity it will bring to the API (and it really is pretty tiny, something on the order of a new `set_effective=True´ flag). It leads immediately to the request to set *both* real UID/GID *and* effective UID/GID to separate values. Can you describe the use case more, so I can understand better how common it might be? In what circumstances must one not change the real UID/GID but instead change the effective UID/GID, *and* must change them during daemonisation? -- \ “One thing vampire children have to be taught early on is, | `\ don't run with a wooden stake.” —Jack Handey | _o__) | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
Jean-Paul Calderone exar...@divmod.com writes: Here is a demonstration of the problem: # python -c ' from __future__ import with_statement import sys, daemon, os with daemon.DaemonContext(stdout=sys.stdout, stdin=sys.stdin, stderr=sys.stderr, uid=1, gid=1) as c: pass ' Traceback (most recent call last): File string, line 5, in module File daemon/daemon.py, line 342, in __enter__ File daemon/daemon.py, line 325, in open OSError: [Errno 1] Operation not permitted This hadn't occurred during my testing, but I can see the logic of it. I've now added a test and fixed this; it will be in the next release. Thank you! An additional feature which would be useful for the library to provide, however, would be the setting of euid and egid instead of uid and gid. This is necessary, for example, to write an SSH daemon which gives out user shells. That sounds rather more specific than is needed for the generic library being proposed here. I'm wary of adding features to an API that is already quite complex. Isn't setting the EUID and EGID something that is just as easily done *after* the program achieves a daemon process? -- \“If it ain't bust don't fix it is a very sound principle and | `\ remains so despite the fact that I have slavishly ignored it | _o__) all my life.” —Douglas Adams | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
Floris Bruynooghe floris.bruynoo...@gmail.com writes: Had a quick look at the PEP and it looks very nice IMHO. Thank you. I hope you can try the implementation and report feedback on that too. One of the things that might be interesting is keeping file descriptors from the logging module open by default. Hmm. I see that this would be a good idea. but it raises the question of how to manage the set of file handles that should not be closed on becoming a daemon. So far, the logic of closing the file descriptors is a little complex: * Close all open file descriptors. This excludes those listed in the `files_preserve` attribute, and those that correspond to the `stdin`, `stdout`, or `stderr` attributes. Extending that by saying “… and also any file descriptors for ``logging.FileHandler`` objects” starts to make the description too complex. I have a strong instinct that it the description is complex, the design might be bad. Can you suggest an alternative API that will ensure that all file descriptors get closed *except* those that should not be closed? -- \“The Bermuda Triangle got tired of warm weather. It moved to | `\ Alaska. Now Santa Claus is missing.” —Steven Wright | _o__) | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
Jean-Paul Calderone exar...@divmod.com writes: The biggest shortcoming seems to be a complete lack of unit tests. A full unit test suite is in the source distribution's ‘tests/’ directory. You can run it with ‘python ./setup.py test’. A quick skim of the code suggests that part of it don't even work at all and have never been tested, even interactively, since they must surely fail. For example, uid/gid setting is broken. This doesn't help identify the problem. Can you explain what you see as broken, preferably after running the code to observe its behaviour? -- \ “Science shows that belief in God is not only obsolete. It is | `\also incoherent.” —Victor J. Stenger, 2001 | _o__) | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
On Sun, 22 Mar 2009 10:19:58 +1100, Ben Finney ben+pyt...@benfinney.id.au wrote: Jean-Paul Calderone exar...@divmod.com writes: The biggest shortcoming seems to be a complete lack of unit tests. A full unit test suite is in the source distribution's ‘tests/’ directory. You can run it with ‘python ./setup.py test’. Of course this is correct. My apologizes for my incorrect statement. I was probably looking for the tests in the daemon directory, not used to seeing them outside the package they are testing. In my hurry I didn't see them. A quick skim of the code suggests that part of it don't even work at all and have never been tested, even interactively, since they must surely fail. For example, uid/gid setting is broken. This doesn't help identify the problem. Can you explain what you see as broken, preferably after running the code to observe its behaviour? Here is a demonstration of the problem: # python -c ' from __future__ import with_statement import sys, daemon, os with daemon.DaemonContext(stdout=sys.stdout, stdin=sys.stdin, stderr=sys.stderr, uid=1, gid=1) as c: pass ' Traceback (most recent call last): File string, line 5, in module File daemon/daemon.py, line 342, in __enter__ File daemon/daemon.py, line 325, in open OSError: [Errno 1] Operation not permitted This happens because setuid is called before setgid. This means that by the time setgid is called, it is no longer allowed. Reversing the order is a simple fix. An additional feature which would be useful for the library to provide, however, would be the setting of euid and egid instead of uid and gid. This is necessary, for example, to write an SSH daemon which gives out user shells. Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
PEP 3143: Standard daemon process library (was: Writing a well-behaved daemon)
Ben Finney b...@benfinney.id.au writes: Writing a Python program to become a Unix daemon is relatively well-documented: there's a recipe for detaching the process and running in its own process group. However, there's much more to a Unix daemon than simply detaching. […] My searches for such functionality haven't borne much fruit though. Apart from scattered recipes, none of which cover all the essentials (let alone the optional features) of 'daemon', I can't find anything that could be relied upon. This is surprising, since I'd expect this in Python's standard library. I've submitted PEP 3143 URL:http://www.python.org/dev/peps/pep-3143/ to meet this need, and have re-worked an existing library into a new ‘python-daemon’ URL:http://pypi.python.org/pypi/python-daemon/ library, the reference implementation. Now I need wider testing and scrutiny of the implementation and specification. One point to note: This is only intended to address the task of a program transforming *itself* into a daemon process. If you want to spawn off *extra* processes and manage them through a “service” channel, you want something this spec was never meant to cover. You may be interested in discussing that further on a separate thread at URL:http://mail.python.org/pipermail/python-ideas/2009-January/002606.html. If you want to turn your program into a well-behaved daemon process, I'd like to know how well PEP 3143 works for you. Please try it out for your daemon programs and discuss! -- \“The whole area of [treating source code as intellectual | `\property] is almost assuring a customer that you are not going | _o__) to do any innovation in the future.” —Gary Barnett | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
Ben Finney ben+pyt...@benfinney.id.au writes:
I've submitted PEP 3143 URL:http://www.python.org/dev/peps/pep-3143/
to meet this need, and have re-worked an existing library into a new
‘python-daemon’ URL:http://pypi.python.org/pypi/python-daemon/
library, the reference implementation.
Now I need wider testing and scrutiny of the implementation and
specification.
PEP: 3143
Title: Standard daemon process library
Version: $Revision: 1.1 $
Last-Modified: $Date: 2009-03-19 12:51 $
Author:Ben Finney ben+pyt...@benfinney.id.au
Status:Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2009-01-26
Python-Version:3.2
Post-History:
Abstract
Writing a program to become a well-behaved Unix daemon is somewhat
complex and tricky to get right, yet the steps are largely similar for
any daemon regardless of what else the program may need to do.
This PEP introduces a package to the Python standard library that
provides a simple interface to the task of becoming a daemon process.
.. contents::
..
Table of Contents:
Abstract
Specification
Example usage
Interface
``DaemonContext`` objects
Motivation
Rationale
Correct daemon behaviour
A daemon is not a service
Reference Implementation
Other daemon implementations
References
Copyright
=
Specification
=
Example usage
=
Simple example of direct `DaemonContext` usage::
import daemon
from spam import do_main_program
with daemon.DaemonContext() as daemon_context:
do_main_program()
More complex example usage::
import os
import grp
import signal
import daemon
import lockfile
from spam import (
initial_program_setup,
do_main_program,
program_cleanup,
reload_program_config,
)
context = daemon.DaemonContext(
working_directory='/var/lib/foo',
umask=0o002,
pidfile=lockfile.FileLock('/var/run/spam.pid'),
)
context.signal_map = {
signal.SIGTERM: program_cleanup,
signal.SIGHUP: 'terminate',
signal.SIGUSR1: reload_program_config,
}
mail_gid = grp.getgrnam('mail').gr_gid
context.gid = mail_gid
important_file = open('spam.data', 'w')
interesting_file = open('eggs.data', 'w')
context.files_preserve = [important_file, interesting_file]
initial_program_setup()
with context:
do_main_program()
Interface
=
A new package, `daemon`, is added to the standard library.
A class, `DaemonContext`, is defined to represent the settings and
process context for the program running as a daemon process.
``DaemonContext`` objects
=
A `DaemonContext` instance represents the behaviour settings and
process context for the program when it becomes a daemon. The
behaviour and environment is customised by setting options on the
instance, before calling the `open` method.
Each option can be passed as a keyword argument to the `DaemonContext`
constructor, or subsequently altered by assigning to an attribute on
the instance at any time prior to calling `open`. That is, for
options named `wibble` and `wubble`, the following invocation::
foo = daemon.DaemonContext(wibble=bar, wubble=baz)
foo.open()
is equivalent to::
foo = daemon.DaemonContext()
foo.wibble = bar
foo.wubble = baz
foo.open()
The following options are defined.
`files_preserve`
:Default: ``None``
List of files that should *not* be closed when starting the
daemon. If ``None``, all open file descriptors will be closed.
Elements of the list are file descriptors (as returned by a file
object's `fileno()` method) or Python `file` objects. Each
specifies a file that is not to be closed during daemon start.
`chroot_directory`
:Default: ``None``
Full path to a directory to set as the effective root directory of
the process. If ``None``, specifies that the root directory is not
to be changed.
`working_directory`
:Default: ``'/'``
Full path of the working directory to which the process should
change on daemon start.
Since a filesystem cannot be unmounted if a process has its
current working directory on that filesystem, this should either
be left at default or set to a directory that is a sensible “home
directory” for the daemon while it is running.
`umask`
:Default: ``0``
File access creation mask (“umask”) to set for the process on
daemon start.
Since a process inherits its umask from its parent process,
starting the daemon will reset the umask to this value so that
files are created by the daemon with access modes as it expects.
`pidfile`
:Default: ``None``
Context manager for a PID lock file. When the daemon context opens
and closes, it enters
Re: PEP 3143: Standard daemon process library (was: Writing a well-behaved daemon)
On Fri, 20 Mar 2009 20:58:58 +1100, Ben Finney ben+pyt...@benfinney.id.au wrote: Ben Finney b...@benfinney.id.au writes: Writing a Python program to become a Unix daemon is relatively well-documented: there's a recipe for detaching the process and running in its own process group. However, there's much more to a Unix daemon than simply detaching. […] My searches for such functionality haven't borne much fruit though. Apart from scattered recipes, none of which cover all the essentials (let alone the optional features) of 'daemon', I can't find anything that could be relied upon. This is surprising, since I'd expect this in Python's standard library. I've submitted PEP 3143 URL:http://www.python.org/dev/peps/pep-3143/ to meet this need, and have re-worked an existing library into a new ‘python-daemon’ URL:http://pypi.python.org/pypi/python-daemon/ library, the reference implementation. Now I need wider testing and scrutiny of the implementation and specification. The biggest shortcoming seems to be a complete lack of unit tests. A quick skim of the code suggests that part of it don't even work at all and have never been tested, even interactively, since they must surely fail. For example, uid/gid setting is broken. I'd recommend adding an automated test suite, fixing all the issues that come up during that process, and then asking for scrutiny again. Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library
On Fri, 20 Mar 2009 21:47:00 +1100, Ben Finney bignose+hates-s...@benfinney.id.au wrote: [snip] Somewhat by accident I noticed this other part of the PEP: Other Python daemon implementations that differ from this PEP: [snip] * Twisted [twisted]_ includes, perhaps unsurprisingly, an implementation of a process daemonisation API that is integrated with the rest of the Twisted framework; it differs significantly from the API in this PEP. What do you mean be integrated? Twisted's daemonization code is in a free function which depends only on the os module. It's basically as un-integrated as could be (it's also only 14 lines long). Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 3143: Standard daemon process library (was: Writing a well-behaved daemon)
On Mar 20, 9:58 am, Ben Finney ben+pyt...@benfinney.id.au wrote: Ben Finney b...@benfinney.id.au writes: Writing a Python program to become a Unix daemon is relatively well-documented: there's a recipe for detaching the process and running in its own process group. However, there's much more to a Unix daemon than simply detaching. […] My searches for such functionality haven't borne much fruit though. Apart from scattered recipes, none of which cover all the essentials (let alone the optional features) of 'daemon', I can't find anything that could be relied upon. This is surprising, since I'd expect this in Python's standard library. I've submitted PEP 3143 URL:http://www.python.org/dev/peps/pep-3143/ to meet this need, and have re-worked an existing library into a new ‘python-daemon’ URL:http://pypi.python.org/pypi/python-daemon/ library, the reference implementation. Now I need wider testing and scrutiny of the implementation and specification. Had a quick look at the PEP and it looks very nice IMHO. One of the things that might be interesting is keeping file descriptors from the logging module open by default. So that you can setup your loggers before you daemonise --I do this so that I can complain on stdout if that gives trouble-- and are still able to use them once you've daemonised. I haven't looked at how feasable this is yet so it might be difficult, but useful anyway. Regards Floris -- http://mail.python.org/mailman/listinfo/python-list
