I am working on a PEP and would appreciate comment. The proposal is available at
http://python.timehorse.com/PEP_-_Application_Priority.reST and is repeated below: --------- :PEP: <Unassigned> :Title: Adding Priority Scheduling feature to the subprocess module :Version: $Rev: 93 $ :Last Modified: $Date: 2008-02-20 08:37:00 -0500 (Wed, 20 Feb 2008) $ :Author: Jeffrey C. Jacobs :Discussions-To: python at timehorse dot com :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 19-Feb-2008 :Python-Version: 2.5.1 :Post-History: <None> ---------- Introduction ~~~~~~~~~~~~ The ``subprocess`` module is intended to create a platform-independent method of *spawning* or *popening* or *execing* an OS-level sub- process. This module provides the ability to spawn processes on a variety of platforms by creating an instance of the ``subprocess.Popen`` class. Currently, only the Windows version of the ``Popen.__init__`` method can set the OS priority of a spawned process directly using the ``createFlags`` parameter. Clients operating under UNIX must use some version of the ``preexec_fn`` to either wrap a C-call to the ``setpriority`` POSIX function or do so by prepending ``nice <u>`` to the ``args`` or in the ``executable``, which would only work for sub- processes spawned from a shell. Under a Windows environment, however, the ``preexec_fn`` parameter is ignored and there is no ``nice`` shell program. This thus precludes the easy setting of a spawned process's priority in the UNIX environment, as well as creates a disconnect in terms of the way priorities are set between Windows and UNIX platforms, which somewhat defeats the purpose of ``subprocess.py`` in terms of being relatively platform-independent. In addition to these issues, because Windows processes are referenced by handles, not by process IDs, although this would divide the UNIX- specific and Windows specific code, the Handle for the windows would none the less be useful in the ``win32process`` module and therefore it is proposed that this handle be officially exposed from the ``Popen`` instance via the name ``handle``. Currently, this property is unofficially accessible via the name ``_handle``, so this proposes to rename that value and make the name official for python running under Windows. The value of ``handle`` for other platforms would then be ``None``. In addition to officially exposing the Handle to Process, it is proposed that the Main Thread Handle, also internally exposed within the ``Popen.__init__`` process, be externally exposed via the name ``hthread`` for use by ``win32process`` and the priority methods. Because UNIX uses priorities between +20 and -20 and Windows, via Process and Thread priorities, allows settings between 0 and 31, a uniform setting for each system should be derived. This would be accomplished by giving process priority in terms of a floating-point value between 0.0 and 1.0 for lowest and highest possible priority, respectively. This priority would be added to the ``Popen`` constructor as an optional parameter, with a default value of ``None``. If and only if something other than ``None`` is set, that value would override any setting in ``createFlags``, and the Process Priority flags of ``createFlags`` would be masked off under the Windows environment. In all cases a non-``None`` setting would cause the spawned process to run at the indicated priority converted to the platform-specific units. The Macintosh OS X / BSD subsystem should be compatible with the UNIX- specific sections of this proposal. Specifics ~~~~~~~~~ The following steps would accomplish this proposal: 1) Add a C-Python wrapper for the POSIX ``getpriority`` / ``setpriority`` methods to the ``os`` module. These methods would only be available in a UNIX environment and would mesh well with the existing file/process functions specified therein. The ``win32process`` module already exposes the corresponding Windows methods. 2) Officially expose the Process Handle as ``handle`` and the Process's Main Thread Handle as ``hThread`` within the ``Popen`` instance interface. 3) Add ``setPriority`` / ``getPriority`` methods to the ``Popen`` class which will use the platform-specific priority function either from ``os`` for UNIX or ``win32process`` for Windows. The input priority to ``setPriority`` will be checked and raise a ``ValueError`` exception for values not in the range 0.0 to 1.0. The 0.0 to 1.0 scale will be converted to the 32-point Windows scale by linear mapping, e.g. for a floating-point priority *p*, the Windows priority is given by: ``int(p*31 + .5)`` Windows is limited to a non-linear, 7-point scale for process priorities, with thread priorities making up the difference relative to the 32-point internal priority scale. Thus, the Windows versions of these functions will use a combination of ``SetPriorityClass`` and ``SetThreadPriority`` in order to set the requested process priority, using the ``handle`` and ``hThread`` properties of the ``Popen`` instance. The 0.0 to 1.0 scale will be converted to the 41-point UNIX scale (values between +20 and -20, inclusive) via an inverted linear mapping, e.g. for a floating-point priority *p*, the UNIX priority is given by: ``20 - int(p*40 + .5)`` Under the UNIX platform, only the super-user (root) is allowed to set priorities below 0 (*p* > 0.5). This condition will not be checked in the ``setPriority`` code; it is the responsibility of the caller to ensure the script making the call to ``setpriority`` has sufficient user privileges, i.e. via the ``os.setuid`` function. The conversion from the platform-specific Priority scale to the floating-point scale are used by ``getPriority`` and are as follows for a given UNIX priority *u* and Windows priority *w*: ``(20 - u) / 40.0`` ``w / 31.0`` As with the ``setPriority`` method for windows, the ``getPriority`` method shall use a combination of ``GetPriorityClass`` / ``GetThreadPriority`` to determine the true Windows priority based on the 32-point scale before converting to floating point. The specific numeric conversion for the Windows ``getPriority`` method is given in `Table 1`_. See the `Appendix`_ for details about the 32-point Windows scale. Note that some priorities have equivalent settings. In this case, raw Process Priority Class (without regard for Foreground and Background Process settings) will have preferential setting and Thread Priority Levels shall be used to make up the difference. E.g. priorities from 1 to 4 will have ``IDLE_PRIORITY_CLASS``, 5 to 7 will have ``BELOW_NORMAL_PRIORITY_CLASS``, 8 and 9 will have ``NORMAL_PRIORITY_CLASS``, 10 to 12 will have ``ABOVE_NORMAL_PRIORITY_CLASS``, 13 to 15 will have ``HIGH_PRIORITY_CLASS`` and 16 to 31 will be accessed through ``REALTIME_PRIORITY_CLASS``, potentially with a non-standard Thread Priority level. Also note that Priority 0 is reserved for the Zero- Page Thread, which is responsible for zeroizing memory when no other process is running. It is impossible for a user process to have this priority. Therefore, on Windows, any priority set request below about 0.016 will be treated as if it was requesting Windows Priority Level 1. 4) Add a priority parameter to the ``Popen.__init__`` method that shall be the final parameter of the function and default to ``None`` in order to preserve backwards compatibility. When priority is not ``None``, it shall be used in order to set the priority of the spawned process either at the time of the creation, or soon after. The setting of priority may be accomplished via a ``nice <u>`` prefix to shell commands (``shell = True``) or via a call to ``setpriority`` after the process is spawned. When a priority is set via the Windows version, a mask is applied to the ``createFlags`` parameter to remove any priority related bits. The calculated priority is then used to determine a new priority in the 7-point Windows Process scale and set the corresponding flag in the ``createFlags`` parameter. Then ``SetThreadPriority`` shall be called on the returned Main Thread Handle with the remaining priority points to further adjust the priority toward the 32-point scale. Opened Questions ~~~~~~~~~~~~~~~~ 1) Should ``Popen.__init__`` for UNIX check for shell-based spawns and automatically apply ``nice`` or should it always first spawn the process and then set its priority via ``setpriority`` once created. Note, the later technique would technically not be available using ``preexec_fn``. 2) The ``startinfo`` parameter may also need adjusting to set whether the process is spawned as a background or foreground process (See `Table 1`_). 3) Should windows use the Priority Boost APIs: ``SetProcessPriorityBoost``, ``GetProcessPriorityBoost``, ``SetThreadPriorityBoost`` and ``GetThreatPriorityBoost``? 4) How does one determine if a given Windows process with handle ``handle`` is running as a *Background* or a *Foreground* process at any given time, for use in the Windows version of ``getPriority``? 5) With the current version of ``subprocess`` I have been unable to spawn a process with a higher priority than ``NORMAL_PRIORITY_CLASS`` under Windows; python is always running as ``NORMAL_PRIORITY_CLASS``. Appendix ~~~~~~~~ .. _Table 1: +----+---------------------------------- +-------------------------------+ | Pt | Process Priority Class | Thread Priority Level | +====+================================== +===============================+ | 1 | IDLE_PRIORITY_CLASS | THREAD_PRIORITY_IDLE | +----+---------------------------------- +-------------------------------+ | 1 | BELOW_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_IDLE | +----+---------------------------------- +-------------------------------+ | 1 | NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_IDLE | +----+---------------------------------- +-------------------------------+ | 1 | ABOVE_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_IDLE | +----+---------------------------------- +-------------------------------+ | 1 | HIGH_PRIORITY_CLASS | THREAD_PRIORITY_IDLE | +----+---------------------------------- +-------------------------------+ | 2 | IDLE_PRIORITY_CLASS | THREAD_PRIORITY_LOWEST | +----+---------------------------------- +-------------------------------+ | 3 | IDLE_PRIORITY_CLASS | THREAD_PRIORITY_BELOW_NORMAL | +----+---------------------------------- +-------------------------------+ | 4 | IDLE_PRIORITY_CLASS | THREAD_PRIORITY_NORMAL | +----+---------------------------------- +-------------------------------+ | 4 | BELOW_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_LOWEST | +----+---------------------------------- +-------------------------------+ | 5 | IDLE_PRIORITY_CLASS | THREAD_PRIORITY_ABOVE_NORMAL | +----+---------------------------------- +-------------------------------+ | 5 | BELOW_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_BELOW_NORMAL | +----+---------------------------------- +-------------------------------+ | 5 | Background NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_LOWEST | +----+---------------------------------- +-------------------------------+ | 6 | IDLE_PRIORITY_CLASS | THREAD_PRIORITY_HIGHEST | +----+---------------------------------- +-------------------------------+ | 6 | BELOW_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_NORMAL | +----+---------------------------------- +-------------------------------+ | 6 | Background NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_BELOW_NORMAL | +----+---------------------------------- +-------------------------------+ | 7 | BELOW_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_ABOVE_NORMAL | +----+---------------------------------- +-------------------------------+ | 7 | Background NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_NORMAL | +----+---------------------------------- +-------------------------------+ | 7 | Foreground NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_LOWEST | +----+---------------------------------- +-------------------------------+ | 8 | BELOW_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_HIGHEST | +----+---------------------------------- +-------------------------------+ | 8 | NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_ABOVE_NORMAL | +----+---------------------------------- +-------------------------------+ | 8 | Foreground NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_BELOW_NORMAL | +----+---------------------------------- +-------------------------------+ | 8 | ABOVE_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_LOWEST | +----+---------------------------------- +-------------------------------+ | 9 | NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_HIGHEST | +----+---------------------------------- +-------------------------------+ | 9 | Foreground NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_NORMAL | +----+---------------------------------- +-------------------------------+ | 9 | ABOVE_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_BELOW_NORMAL | +----+---------------------------------- +-------------------------------+ | 10 | Foreground NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_ABOVE_NORMAL | +----+---------------------------------- +-------------------------------+ | 10 | ABOVE_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_NORMAL | +----+---------------------------------- +-------------------------------+ | 11 | Foreground NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_HIGHEST | +----+---------------------------------- +-------------------------------+ | 11 | ABOVE_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_ABOVE_NORMAL | +----+---------------------------------- +-------------------------------+ | 11 | HIGH_PRIORITY_CLASS | THREAD_PRIORITY_LOWEST | +----+---------------------------------- +-------------------------------+ | 12 | ABOVE_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_HIGHEST | +----+---------------------------------- +-------------------------------+ | 12 | HIGH_PRIORITY_CLASS | THREAD_PRIORITY_BELOW_NORMAL | +----+---------------------------------- +-------------------------------+ | 13 | HIGH_PRIORITY_CLASS | THREAD_PRIORITY_NORMAL | +----+---------------------------------- +-------------------------------+ | 14 | HIGH_PRIORITY_CLASS | THREAD_PRIORITY_ABOVE_NORMAL | +----+---------------------------------- +-------------------------------+ | 15 | HIGH_PRIORITY_CLASS | THREAD_PRIORITY_HIGHEST | +----+---------------------------------- +-------------------------------+ | 15 | HIGH_PRIORITY_CLASS | THREAD_PRIORITY_TIME_CRITICAL | +----+---------------------------------- +-------------------------------+ | 15 | IDLE_PRIORITY_CLASS | THREAD_PRIORITY_TIME_CRITICAL | +----+---------------------------------- +-------------------------------+ | 15 | BELOW_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_TIME_CRITICAL | +----+---------------------------------- +-------------------------------+ | 15 | NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_TIME_CRITICAL | +----+---------------------------------- +-------------------------------+ | 15 | ABOVE_NORMAL_PRIORITY_CLASS | THREAD_PRIORITY_TIME_CRITICAL | +----+---------------------------------- +-------------------------------+ | 16 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_IDLE | +----+---------------------------------- +-------------------------------+ | 17 | REALTIME_PRIORITY_CLASS | -7 | +----+---------------------------------- +-------------------------------+ | 18 | REALTIME_PRIORITY_CLASS | -6 | +----+---------------------------------- +-------------------------------+ | 19 | REALTIME_PRIORITY_CLASS | -5 | +----+---------------------------------- +-------------------------------+ | 20 | REALTIME_PRIORITY_CLASS | -4 | +----+---------------------------------- +-------------------------------+ | 21 | REALTIME_PRIORITY_CLASS | -3 | +----+---------------------------------- +-------------------------------+ | 22 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_LOWEST | +----+---------------------------------- +-------------------------------+ | 23 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_BELOW_NORMAL | +----+---------------------------------- +-------------------------------+ | 24 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_NORMAL | +----+---------------------------------- +-------------------------------+ | 25 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_ABOVE_NORMAL | +----+---------------------------------- +-------------------------------+ | 26 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_HIGHEST | +----+---------------------------------- +-------------------------------+ | 27 | REALTIME_PRIORITY_CLASS | 3 | +----+---------------------------------- +-------------------------------+ | 28 | REALTIME_PRIORITY_CLASS | 4 | +----+---------------------------------- +-------------------------------+ | 29 | REALTIME_PRIORITY_CLASS | 5 | +----+---------------------------------- +-------------------------------+ | 30 | REALTIME_PRIORITY_CLASS | 6 | +----+---------------------------------- +-------------------------------+ | 31 | REALTIME_PRIORITY_CLASS | THREAD_PRIORITY_TIME_CRITICAL | +----+---------------------------------- +-------------------------------+ Table 1: 32-Point Windows Priority Scale [#MSPriorityTable]_. ------- .. [#MSPriorityTable] From `Microsoft Windows Scheduling Priorities <http://msdn2.microsoft.com/en-us/library/ms685100.aspx>`_. -- http://mail.python.org/mailman/listinfo/python-list