Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes:
Thanks for the additional digging in here. I think your analysis is
right - it actually occurred to me yesterday that this could be the
problem, and I filed a bug to track it here:
3. The fully-reliable fix would be to somehow give the copied binary a
hint where to find the right standard library, and this would involve
adding something to the algorithm in getpath.c. The hint could take the
form of a key in the config file, but I'd really like to avoid fully
parsing the config-file in C and then again in Python later on. The hint
could also be some kind of specially-formatted comment line at the top
of the config file, which would require less C code to find and parse?
Any thoughts on this (or alternative solutions I haven't thought of) are
most welcome.
Hi Carl,
I've been thinking about the C-level configuration some more (cpythonv issue
#6). Suppose we have a virtual env at absolute path testenv, and in
testenv/bin we have a copied, non-installed Python, obtained from absolute
path pythonsrc. Say we provide the location of the source in the
configuration, and that this is read in getpath.c. Logically, it should just
replace the argv0_path value, and if the subsequent logic is unchanged, at the
point that site.py's main() runs, you would have (on Linux i686):
sys.prefix, sys.exec_prefix would have values set from the configure script
('/usr/local' by default).
sys.path would be ['/usr/local/lib/python33.zip', 'pythonsrc/Lib',
'pythonsrc/Lib/plat-linux2', 'pythonsrc/build/lib.linux-i686-3.3']
which seems OK.
If a virtual environment is in absolute path testenv, then after the call to
virtualize() in site.py, we would have sys.prefix and sys.exec_prefix both set
to testenv. Since sys.executable still points to testenv/bin, the code in
sysconfig.py appears not to be doing the right thing, e.g.
sysconfig._PROJECT_BASE would be testenv/bin, which seems wrong.
So it seems as if site.py and/or sysconfig.py might need further changes, as
well as changes to getpath.c. What do you think?
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Hi Vinay,
On 03/29/2011 09:45 AM, Vinay Sajip wrote:
I've been thinking about the C-level configuration some more (cpythonv issue
#6). Suppose we have a virtual env at absolute path testenv, and in
testenv/bin we have a copied, non-installed Python, obtained from absolute
path pythonsrc. Say we provide the location of the source in the
configuration, and that this is read in getpath.c. Logically, it should just
replace the argv0_path value, and if the subsequent logic is unchanged, at the
point that site.py's main() runs, you would have (on Linux i686):
sys.prefix, sys.exec_prefix would have values set from the configure script
('/usr/local' by default).
sys.path would be ['/usr/local/lib/python33.zip', 'pythonsrc/Lib',
'pythonsrc/Lib/plat-linux2', 'pythonsrc/build/lib.linux-i686-3.3']
which seems OK.
If a virtual environment is in absolute path testenv, then after the call to
virtualize() in site.py, we would have sys.prefix and sys.exec_prefix both set
to testenv. Since sys.executable still points to testenv/bin, the code in
sysconfig.py appears not to be doing the right thing, e.g.
sysconfig._PROJECT_BASE would be testenv/bin, which seems wrong.
So it seems as if site.py and/or sysconfig.py might need further changes, as
well as changes to getpath.c. What do you think?
In general, I think the copied binary case should be as similar as
possible to the symlinked binary case. Python doesn't dereference
symlinks in setting sys.executable, so in either case sys.executable
will point to the virtual environment's binary, which I think is clearly
what we want (think code using sys.executable to decide how to re-exec
itself).
The changes I made to sysconfig thus far were the minimum necessary to
be able to successfully install stuff in a pythonv virtualenv. On a
brief glance, it looks to me like you are right; we'll need to modify
the setting of _PROJECT_BASE in sysconfig.py to be virtual-aware (this
is part -- maybe all -- of what's needed to fix issue #1).
Also, see issue #10 - I'm wavering on whether changing sys.prefix and
sys.exec_prefix to point to the virtualenv is actually the right
approach. We're already forced to modify sysconfig.py to be
virtual-aware in order to avoid having to hackily copy/symlink bits of
the stdlib and include directories and the Makefile and config.h and
whatnot into the virtualenv. Once we've taken the step of making
sysconfig virtual-aware, it seems to me that perhaps sys.prefix is
better left pointing to the actual Python installation, and we add a
sys.virtual_prefix or some such to tell site.py and sysconfig.py where
virtualized stuff belongs.
There are a variety of possible meanings of sys.prefix, and the only one
we really want to modify is where should I install third-party
packages? Looking at sys.prefix references in the stdlib, I see them in
trace, tkinter, and idlelib. It appears to me that in all of those
cases, having sys.prefix pointing to the virtualenv would be wrong.
If we switch to sys.virtual_prefix and leave sys.prefix alone, the
danger would be that some installer that's using sys.prefix directly,
and ignoring sysconfig.py, would try to install stuff to the global
Python installation. But I'm not aware of any such installer, and I
think we could safely call any installer of Python code that ignores the
layout schemes in sysconfig thoroughly broken anyway.
Your thoughts? (Or anyone else?)
Carl
P.S. The elephant in the room here is the fact that if we want pythonv
to be backwards-compatible with distutils/distribute, which we do, we
can't get away with just modifying sysconfig.py, we also have to modify
the old distutils/sysconfig.py. In general distutils is frozen; I'm
hoping that Tarek will let me get away with a few changes for pythonv if
they aren't changing APIs at all, just modifying some install-scheme
paths to be virtual-aware.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] buildout, setuptools, sourceforge problems
There seem to be some problems with packages on sourceforge not working with buildout. Something about sourceforge is sending two content-length headers. Anything the buildout gurus here can add? Description of problem. Suggests just eliminating sourceforge from allowed hosts: http://plone.293351.n2.nabble.com/buildout-dies-due-quot-invalid-literal-for-int-with-base-10-quot-td6200410.html Ticket at sourceforge: http://sourceforge.net/apps/trac/sourceforge/ticket/18486 Thanks. -- Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout, setuptools, sourceforge problems
At 04:25 PM 3/29/2011 -0400, Jeff Kunce wrote: There seem to be some problems with packages on sourceforge not working with buildout. Something about sourceforge is sending two content-length headers. Update your setuptools to 0.6c12dev-r88975 for the fix. (If you're using a forked setuptoools, you'll need to uninstall it first so it doesn't block your update.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes: In general, I think the copied binary case should be as similar as possible to the symlinked binary case. Python doesn't dereference symlinks in setting sys.executable, so in either case sys.executable will point to the virtual environment's binary, which I think is clearly what we want (think code using sys.executable to decide how to re-exec itself). I agree. The changes I made to sysconfig thus far were the minimum necessary to be able to successfully install stuff in a pythonv virtualenv. On a brief glance, it looks to me like you are right; we'll need to modify the setting of _PROJECT_BASE in sysconfig.py to be virtual-aware (this is part -- maybe all -- of what's needed to fix issue #1). Also, see issue #10 - I'm wavering on whether changing sys.prefix and sys.exec_prefix to point to the virtualenv is actually the right approach. We're already forced to modify sysconfig.py to be virtual-aware in order to avoid having to hackily copy/symlink bits of the stdlib and include directories and the Makefile and config.h and whatnot into the virtualenv. Once we've taken the step of making sysconfig virtual-aware, it seems to me that perhaps sys.prefix is better left pointing to the actual Python installation, and we add a sys.virtual_prefix or some such to tell site.py and sysconfig.py where virtualized stuff belongs. This sounds better, too. If we switch to sys.virtual_prefix and leave sys.prefix alone, the danger would be that some installer that's using sys.prefix directly, and ignoring sysconfig.py, would try to install stuff to the global Python installation. But I'm not aware of any such installer, and I think we could safely call any installer of Python code that ignores the layout schemes in sysconfig thoroughly broken anyway. Your thoughts? (Or anyone else?) What about the possibility of third-party code which uses sys.prefix/sys.exec_prefix, though? (I mean apart from the installer code.) P.S. The elephant in the room here is the fact that if we want pythonv to be backwards-compatible with distutils/distribute, which we do, we can't get away with just modifying sysconfig.py, we also have to modify the old distutils/sysconfig.py. In general distutils is frozen; I'm hoping that Tarek will let me get away with a few changes for pythonv if they aren't changing APIs at all, just modifying some install-scheme paths to be virtual-aware. Here's hoping :-) On the question of a C-readable configuration file, I've started work on this on the assumption that it's a UTF8-encoded file which contains lines of the form key = value. Having thought about it, since this file is only for bootstrapping the virtualenv anyway, there's no strong case for it to be ConfigParser-compatible. I've modified calculate_path in getpath.c to look for the file adjacent to the executable or one level up. If found, we look for a line home = /path/to/source/of/copied/python and, if found, replace argv0_path with the value, then let the existing code take its course. I've modified your virtualize() in site.py to cater for the new config file format: we default the prefix to the parent of the directory containing the executable (hence, the virtualenv root), then set sys.virtual_prefix to this prefix, and optionally disallow the system site-packages from the system path. For now (since [distutils.]sysconfig is not sys.virtual_prefix aware) I continue to set sys.prefix and sys.exec_prefix to sys.virtual_prefix. So as not to cause confusion with existing config files you might have, I changed the config file name to 'env.cfg'. This can be changed to the appropriate bikeshed colour in two places - site.py and getpath.c. Progress seems good. To test: 1. Clone the repo at https://bitbucket.org/vinay.sajip/pythonv 2. In that repo, run ./configure and make 3. Download the pmv.py script at https://gist.github.com/893507 4. Run pmv.py with the python you just built - see the header comment in that script. Note that the pmv.py (Poor Man's virtualize.py) script deliberately runs with a copied rather than a symlinked Python. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout, setuptools, sourceforge problems
On Tue, Mar 29, 2011 at 4:54 PM, P.J. Eby p...@telecommunity.com wrote: Update your setuptools to 0.6c12dev-r88975 for the fix. Egads, Philip! Do regular version numbers cost more in your neighborhood? -Fred -- Fred L. Drake, Jr. fdrake at acm.org A storm broke loose in my mind. --Albert Einstein ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
