Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
P.J. Eby wrote:
At 05:16 PM 7/15/2009 +0200, Joachim König wrote:
f you have m different versions of n packages then
you could have n**m different combinations for an application so you
need a
possiblilty to select one combination from n**m possible ones at
application
startup time. Is this really worth it?
Obviously yes, as neither buildout nor setuptools would exist
otherwise. ;-)
Nor would Fedora be packaging certain library versions as eggs
specifically to get certain multi-version scenarios to work.
The specific solutions for handling n*m problems aren't fantastic, but
they are clearly needed.
I still do not see the need.
IMO the whole obfuscation comes from fact that all versions of all
packages are installed into
one location where python automaticallly looks for packages and then
with a lot of magic the
packages are hidden from the interpreter and only specific requested
versions are made "visible"
to the interpreter at runtime.
Why do the package have to be installed there at the first place?
For an application it would be enough to have an additional directory on
its PYTHONPATH where
the packages required for this application would be installed. So a
package could be installed either
to the common directory ("site-packages") or an application specific
directory (e.g. something like
"app-packages//"). This approach has been used by Zope2 with
its "private" lib/python
directory for years.
So one would have to set up the application specific packages before
running the application, but the
whole clutter of uncounted versions of the same package in one directory
could go away. The
"drawback" of this approach would be, that the same version of a package
would have to be installed
multiple times if needed by different applications.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
On Thu, May 18, 2006 at 20:02, "Martin v. Löwis" wrote: > Nick Coghlan wrote: > > And if I understand it correctly, it falls under the category that > > waiting for another thread while holding the import lock is a *really* > > bad idea from a thread safety point of view. > > > > The thing with the import-after-fork deadlock is that you can trigger it > > without even doing anything that's known not to be thread-safe. > > Right. With some googling, I found that one solution is pthread_atexit: > a pthread_atexit handler is a triple (before, in_parent, in_child). > You set it to (acquire, release, release). When somebody forks, > the pthread library will first acquire the import lock in the thread > that wants to fork. Then the fork occurs, and the import lock gets > then released both in the parent and in the child. > > I would like to see this approach implemented, but I agree with you > that a test case should be created first. > Picking up a rather old discussion... We encountered this bug at Google and I'm now "incentivized" to fix it. For a short recap: Python has an import lock that prevents more than one thread from doing an import at any given time. However, unlike most of the locks we have lying around, we don't clear that lock in the child after an os.fork(). That means that doing an os.fork() during an import means the child process can't do any other imports. It also means that doing an os.fork() *while another thread is doing an import* means the child process can't do any other imports. Since this three-year-old discussion we've added a couple of post-fork-cleanups to CPython (the TLS, the threading module's idea of active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we already do simply discard the memory for other locks held during fork (the GIL, see Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock, except when the platform is AIX. I don't see any particular reason why we aren't doing the same thing to the import lock that we do to the other locks, on all platforms. It's a quick fix for a real problem (see http://bugs.python.org/issue1590864 and http://bugs.python.org/issue1404925 for two bugreports that seem to be this very issue.) It also seems to me, since we have two or three (depending on how you count) uses for it, that we should either add a way to free the old locks (to the various threading implementations) or add a way to use pthread_atexit portably -- or at least portably across systems with fork(). I don't think we should wait with fixing the big issue (making threads and fork() being unnecessarily flakey) until we have a good fix for the small issue (a tiny bit of a memory leak after a fork() -- in the child process only.) Especially since the good fix for the small issue might require co-ordination between all the threading implementations we have and nobody knows about. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
Thomas Wouters wrote: > Picking up a rather old discussion... We encountered this bug at Google > and I'm now "incentivized" to fix it. > > For a short recap: Python has an import lock that prevents more than one > thread from doing an import at any given time. However, unlike most of > the locks we have lying around, we don't clear that lock in the child > after an os.fork(). That means that doing an os.fork() during an import > means the child process can't do any other imports. It also means that > doing an os.fork() *while another thread is doing an import* means the > child process can't do any other imports. > > Since this three-year-old discussion we've added a couple of > post-fork-cleanups to CPython (the TLS, the threading module's idea of > active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we > already do simply discard the memory for other locks held during fork > (the GIL, see Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in > Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock, > except when the platform is AIX. I don't see any particular reason why > we aren't doing the same thing to the import lock that we do to the > other locks, on all platforms. It's a quick fix for a real problem (see > http://bugs.python.org/issue1590864 and > http://bugs.python.org/issue1404925 for two bugreports that seem to be > this very issue.) +1. Leaving deadlock issues around that people can run into without doing anything wrong in their application is unfriendly. > It also seems to me, since we have two or three (depending on how you > count) uses for it, that we should either add a way to free the old > locks (to the various threading implementations) or add a way to use > pthread_atexit portably -- or at least portably across systems with > fork(). I don't think we should wait with fixing the big issue (making > threads and fork() being unnecessarily flakey) until we have a good fix > for the small issue (a tiny bit of a memory leak after a fork() -- in > the child process only.) Especially since the good fix for the small > issue might require co-ordination between all the threading > implementations we have and nobody knows about. One-off memory leaks for relatively rare operations like fork() don't bother me all that much, so I agree that having fork() leak another lock in the child process is preferable to it having a chance to mysteriously deadlock. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
On Thu, May 18, 2006 at 3:02 PM, "Martin v. Löwis" wrote: > > Right. With some googling, I found that one solution is pthread_atexit: > a pthread_atexit handler is a triple (before, in_parent, in_child). > You set it to (acquire, release, release). Did you mean pthread_atfork() ? > When somebody forks, > the pthread library will first acquire the import lock in the thread > that wants to fork. Then the fork occurs, and the import lock gets > then released both in the parent and in the child. > > I would like to see this approach implemented, but I agree with you > that a test case should be created first. > Whould some new C-API functions do the trick? PyOS_AtForkPrepare() # acquire import lock pid = fork() if (pid == 0) PyOS_AtForkChild() # current PyOS_AfterFork() + release import lock else PyOS_AtForkParent() # release import lock -- Lisandro Dalcín --- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
Joachim König wrote: > So one would have to set up the application specific packages before > running the application, but the > whole clutter of uncounted versions of the same package in one directory > could go away. The > "drawback" of this approach would be, that the same version of a package > would have to be installed > multiple times if needed by different applications. While this is a very common practice in the Windows world, it is far less common in the *nix world of vendor managed packaging systems. As for why it can be a problem, it (bundling libraries with applications) makes security vulnerability management a *lot* more difficult for system administrators. If a bug is found in a key library (e.g. openssl) a dependency based system just needs to update the single shared copy of that library. In a bundling system, you first have to work out which of your applications contain an instance of that library and then see if the application vendors have provided a security patch. If any one of them hasn't released a patch and you can't patch it yourself, then you either have to stop using that application or else accept remaining exposed to the vulnerability. The bundling approach also leads to applications being much bigger than they need to be. That isn't much of a problem for desktop or server systems these days, but can still be an issue in the embedded world. As far as the idea of making bundling easier goes, we already implemented that in 2.6 and 3.0. It's the whole reason that zip files and directories are directly executable now: the named zip file or directory itself is automatically added to sys.path, so the top level "__main__.py" in that location can freely import any other co-located modules and packages. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
