Re: [Distutils] buildout, setuptools, sourceforge problems
On 2011-3-29 22:54, P.J. Eby wrote: At 04:25 PM 3/29/2011 -0400, Jeff Kunce wrote: There seem to be some problems with packages on sourceforge not working with buildout. Something about sourceforge is sending two content-length headers. Update your setuptools to 0.6c12dev-r88975 for the fix. Would you be willing to make a 0.6c12 release as well? Wichert. -- Wichert Akkerman wich...@wiggy.net It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout, setuptools, sourceforge problems
On Tue, Mar 29, 2011 at 04:25:51PM -0400, Jeff Kunce wrote: There seem to be some problems with packages on sourceforge not working with buildout. Something about sourceforge is sending two content-length headers. Anything the buildout gurus here can add? Description of problem. Suggests just eliminating sourceforge from allowed hosts: http://plone.293351.n2.nabble.com/buildout-dies-due-quot-invalid-literal-for-int-with-base-10-quot-td6200410.html Ticket at sourceforge: http://sourceforge.net/apps/trac/sourceforge/ticket/18486 One workaround people have suggested is to add [buildout] allow-hosts = *.python.org # plus other machines that are *not* sourceforge.net, if you use eggs # that aren't uploaded to PyPI to your buildout.cfg. Marius Gedminas -- I am a computer. I am dumber than any human and smarter than any administrator. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [buildout] private releases
On Fri, Mar 25, 2011 at 3:24 PM, Chris Withers ch...@simplistix.co.uk wrote: Hi Christian, On 25/03/2011 16:49, Christian Theune wrote: the German speaking Zope Users Group (DZUG e.V.) organizes a series of 4 sprints this year to support feature development within the proximity of the ZTK and solve problems encountered by Zope, Plone and Python developers. snip Topics == * Discussing how to deal with private releases FWIW, I've had no problems with this, here's a sample buildout.cfg: ... We do something similar with sftp (zc.buildoutsftp). To publish eggs, we just use scp. The advantage of this is that it leverages ssh infrastructure, so *no* additional password management is needed. This is wildly better, IMO, than keeping passwords in clear text in your buildout configuration or in a dot file. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Hi Vinay, On 03/29/2011 07:00 PM, Vinay Sajip wrote: If we switch to sys.virtual_prefix and leave sys.prefix alone, the danger would be that some installer that's using sys.prefix directly, and ignoring sysconfig.py, would try to install stuff to the global Python installation. But I'm not aware of any such installer, and I think we could safely call any installer of Python code that ignores the layout schemes in sysconfig thoroughly broken anyway. Your thoughts? (Or anyone else?) What about the possibility of third-party code which uses sys.prefix/sys.exec_prefix, though? (I mean apart from the installer code.) Right. The question is, for third-party code using sys.prefix/sys.exec_prefix, which value would be the right one, the installation prefix or the virtual prefix? It depends on what they're using it for. I did a quick Google Code Search for sys.prefix, and unfortunately right on the front page I can find examples where either approach we take would be wrong. The majority of uses I found are people copying code from trace.py which is intended to exclude the standard library and system-wide packages from coverage tracing. In this case, leaving sys.prefix pointing to the installation prefix would give the desired result. Also, for instance, here where matplotlib's setup looks for DLLs on Windows, the system prefix would be correct: http://bit.ly/eQSVYc On the other hand, here scons uses sys.prefix to try to find site-packages: http://bit.ly/ik0PwJ If we leave sys.prefix pointing to the installation directory, this would leave scons unable to find its config file in the virtualenv's site-packages directory. Unless they fix their code to use the get_python_lib() function in sysconfig, which is what they ought to be doing instead of duplicating all that logic. So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. On the question of a C-readable configuration file, I've started work on this on the assumption that it's a UTF8-encoded file which contains lines of the form key = value. Having thought about it, since this file is only for bootstrapping the virtualenv anyway, there's no strong case for it to be ConfigParser-compatible. Agreed. I only used ConfigParser initially because that was easiest from Python code :-) I've modified calculate_path in getpath.c to look for the file adjacent to the executable or one level up. If found, we look for a line home = /path/to/source/of/copied/python and, if found, replace argv0_path with the value, then let the existing code take its course. I've modified your virtualize() in site.py to cater for the new config file format: we default the prefix to the parent of the directory containing the executable (hence, the virtualenv root), then set sys.virtual_prefix to this prefix, and optionally disallow the system site-packages from the system path. For now (since [distutils.]sysconfig is not sys.virtual_prefix aware) I continue to set sys.prefix and sys.exec_prefix to sys.virtual_prefix. This sounds good. So as not to cause confusion with existing config files you might have, I changed the config file name to 'env.cfg'. This can be changed to the appropriate bikeshed colour in two places - site.py and getpath.c. Progress seems good. To test: 1. Clone the repo at https://bitbucket.org/vinay.sajip/pythonv 2. In that repo, run ./configure and make 3. Download the pmv.py script at https://gist.github.com/893507 4. Run pmv.py with the python you just built - see the header comment in that script. Note that the pmv.py (Poor Man's virtualize.py) script deliberately runs with a copied rather than a symlinked Python. Hmm, it doesn't seem to be working for me. sys.prefix and sys.exec_prefix are still /usr/local using the python binary in my pmv, and sys.virtual_prefix is not set at all, despite env.cfg being present and looking as it should. Oddly when I run the compiled python binary directly from the checkout, it does set sys.virtual_prefix to point one level above the checkout, despite there being no env.cfg in the vicinity at all. You can see both of these things (in reverse order) here: http://paste.pocoo.org/show/362798/ I'll dig in a bit more and see if I can figure out what's happening. Is there a reason you didn't base your changesets on mine (in the cpythonv repo), and instead apparently copied over a bunch of the changes manually as a diff? I can transplant or copy your work back, or try to do a full merge, but in the long run if we'll both be working on this it seems best if we are easily able to merge our work back and forth. Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] reservations about pythonv
kiorky wrote: You have rpath to record search paths for shared libraries at link time. This enables you not to set LD_LIBRARY_PATH at runtime. Something like cc -Wl,-rpath,/path/to/libdir -L/path/to/libdir -lfoo is what you want. You left out the hard part: How do I make distutils do that? Le 18/03/2011 18:46, Mark Sienkiewicz a écrit : The above just means my memory is too faulty to of much use ;). I'll just echo Carl's request for specific cases where $LD_LIBRARY_PATH needs to be set. Here is a case that might resemble what you are talking about: Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right. This is not really different from what happens in a compiled language, except in one way: In C, I can compile it -static or I can give the full path to the .so file. Either results in a thing that works without LD_LIBRARY_PATH. With distutils, you can't. It goes to great lengths to ensure that you can only compile a C extension with cc ... -L/some/directory -lname -- I can't find any way to make it do cc ... /some/directory/libname.so So, the real problem here is that distutils uses cc -L, but it demonstrates a case where LD_LIBRARY_PATH can be important to a python program even when Python itself can run without it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
Carl Meyer carl at oddbird.net writes: So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. That's problematic, of course (third-party breakage, I mean). It may be that despite using virtual_prefix being a cleaner approach, it may be necessary to alter sys.prefix to ensure compatibility, a la virtualenv. Hmm, it doesn't seem to be working for me. sys.prefix and sys.exec_prefix are still /usr/local using the python binary in my pmv, and sys.virtual_prefix is not set at all, despite env.cfg being present and looking as it should. Oddly when I run the compiled python binary directly from the checkout, it does set sys.virtual_prefix to point one level above the checkout, despite there being no env.cfg in the vicinity at all. You can see both of these things (in reverse order) here: http://paste.pocoo.org/show/362798/ I'll dig in a bit more and see if I can figure out what's happening. It may be that the thing isn't working properly, but it's masked on my system because I have an installed 3.3 in /usr/local (from when I was doing my pip/virtualenv tests). This means that there is a Makefile in a location which would not be there if I'd never installed 3.3 using sudo make install. However, the getpath.c stuff should work - to test this, insert an import pdb; pdb.set_trace() in site.main, and check sys.path. This should contain entries from the virtualenv. The problems are more likely to be in the Python code: the C code always sets sys.prefix and sys.exec_prefix to /usr/local (or whatever configure was invoked with) and the sys.prefix / sys.exec_prefix stuff is done in site.virtualize, as before. I'll try to test this on a machine which never had 3.3 installed - it seems you can't uninstall Python using sudo make uninstall :-( Is there a reason you didn't base your changesets on mine (in the cpythonv repo), and instead apparently copied over a bunch of the changes manually as a diff? I can transplant or copy your work back, or try to do a full merge, but in the long run if we'll both be working on this it seems best if we are easily able to merge our work back and forth. Sorry - I got in a muddle because I've got too many clones around and ran into free disk space issues, and goofed somewhere. Don't bother to do a full merge for now, as I'm not sure if my code's working yet. Since I don't expect to change more than 4 files (getpath.c, site.py, distutils/sysconfig.py and sysconfig.py) I don't think we'll have too much trouble merging in due course, and if/when we can agree that my experiment in pythonv is worth taking further, I'll do the work of merging into cpythonv. For now, though, my changes might be treading on your toes, making merging needlessly annoying. I'll try to test on a Python-clean machine in a day or two, will report back then. Regards, Vinay ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] reservations about pythonv
You have rpath to record search paths for shared libraries at link time. This enables you not to set LD_LIBRARY_PATH at runtime. Something like cc -Wl,-rpath,/path/to/libdir -L/path/to/libdir -lfoo is what you want. Le 18/03/2011 18:46, Mark Sienkiewicz a écrit : The above just means my memory is too faulty to of much use ;). I'll just echo Carl's request for specific cases where $LD_LIBRARY_PATH needs to be set. Here is a case that might resemble what you are talking about: Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right. This is not really different from what happens in a compiled language, except in one way: In C, I can compile it -static or I can give the full path to the .so file. Either results in a thing that works without LD_LIBRARY_PATH. With distutils, you can't. It goes to great lengths to ensure that you can only compile a C extension with cc ... -L/some/directory -lname -- I can't find any way to make it do cc ... /some/directory/libname.so So, the real problem here is that distutils uses cc -L, but it demonstrates a case where LD_LIBRARY_PATH can be important to a python program even when Python itself can run without it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF Pensez à l’environnement. N’imprimez ce courriel que si vous en avez vraiment besoin. signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
At 01:35 PM 3/30/2011 -0400, Carl Meyer wrote: So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. Looking at it from a software distribution POV, I would say that the virtual prefix is what it should point to, since that means things won't get installed to the wrong place. (Of course, a configuration option could be used to override it, if really necessary.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On 03/30/2011 07:18 PM, Vinay Sajip wrote: So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. That's problematic, of course (third-party breakage, I mean). It may be that despite using virtual_prefix being a cleaner approach, it may be necessary to alter sys.prefix to ensure compatibility, a la virtualenv. Well, as I said, I think there will be compatibility problems either way. We are splitting site-packages from the base Python installation, and when third-party code assumes that site-packages and the rest of the base Python installation are both found relative to sys.prefix, neither choice we make can possibly be right. Virtualenv tries to avoid this problem by copying or symlinking in enough stuff from the base Python install in order to convince third parties using sys.prefix that they really are looking at a full Python installation. But that approach is hacky and far from problem-free, and requires keeping up with various things that third-party code might try to look for under sys.prefix (there is at least one current open bug on virtualenv, Tkinter failing in a Windows virtualenv, that is the result of this problem). The logical conclusion of that approach, if you want full compatibility with third-party code, is to just copy the entire Python installation to the new location. So three options as I see it: 1) Add sys.virtual_prefix (or perhaps sys.site_prefix would be a better name), and require third-party code to use that (or sysconfig) to find site-packages, if it wants to be virtual-compatible. 2) Repoint sys.prefix, and require third-party code to understand some new sys.system_prefix or sys.installed_prefix or equivalent to find the rest of the base Python installation. 3) Attempt the virtualenv approach of trying to fool third-party code into thinking the virtualenv IS a full Python installation, even though it's not. I think #1 is far preferable to #2, and in the long run preferable to #3 even if it involves some short-term pain. But maybe I'm being too idealistic, and #3 is the pragmatic choice even though it's wrong. It may be that the thing isn't working properly, but it's masked on my system because I have an installed 3.3 in /usr/local (from when I was doing my pip/virtualenv tests). This means that there is a Makefile in a location which would not be there if I'd never installed 3.3 using sudo make install. However, the getpath.c stuff should work - to test this, insert an import pdb; pdb.set_trace() in site.main, and check sys.path. This should contain entries from the virtualenv. The problems are more likely to be in the Python code: the C code always sets sys.prefix and sys.exec_prefix to /usr/local (or whatever configure was invoked with) and the sys.prefix / sys.exec_prefix stuff is done in site.virtualize, as before. No, if I break early in site.py (or run python -S), sys.path doesn't contain any virtualenv paths. Sorry - I got in a muddle because I've got too many clones around and ran into free disk space issues, and goofed somewhere. Don't bother to do a full merge for now, as I'm not sure if my code's working yet. Since I don't expect to change more than 4 files (getpath.c, site.py, distutils/sysconfig.py and sysconfig.py) I don't think we'll have too much trouble merging in due course, and if/when we can agree that my experiment in pythonv is worth taking further, I'll do the work of merging into cpythonv. For now, though, my changes might be treading on your toes, making merging needlessly annoying. I'll try to test on a Python-clean machine in a day or two, will report back then. Ok, thanks! Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] reservations about pythonv
export LDFLAGS=what i want to be in linker flags.
You have also play the LD_RUN_PATH to modify the RPATH.
See [1]
You can have a look to minitage.recipe.{common, eggs, scripts} for a good
buildout integration.
[1] - http://goo.gl/oPPj0
Le 30/03/2011 23:41, Mark Sienkiewicz a écrit :
kiorky wrote:
You have rpath to record search paths for shared libraries at link time.
This enables you not to set LD_LIBRARY_PATH at runtime.
Something like cc -Wl,-rpath,/path/to/libdir -L/path/to/libdir -lfoo is
what
you want.
You left out the hard part: How do I make distutils do that?
Le 18/03/2011 18:46, Mark Sienkiewicz a écrit :
The above just means my memory is too faulty to of much use ;). I'll just
echo Carl's request for specific cases where $LD_LIBRARY_PATH needs to be
set.
Here is a case that might resemble what you are talking about:
Compile a C extension that requires a shared library that is not in the
standard
system path. To import it, LD_LIBRARY_PATH needs to be right.
This is not really different from what happens in a compiled language,
except in
one way: In C, I can compile it -static or I can give the full path to the
.so
file. Either results in a thing that works without LD_LIBRARY_PATH.
With distutils, you can't. It goes to great lengths to ensure that you can
only
compile a C extension with cc ... -L/some/directory -lname -- I can't
find any
way to make it do cc ... /some/directory/libname.so
So, the real problem here is that distutils uses cc -L, but it
demonstrates a
case where LD_LIBRARY_PATH can be important to a python program even when
Python
itself can run without it.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
--
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF
Pensez à l’environnement.
N’imprimez ce courriel que si vous en avez vraiment besoin.
signature.asc
Description: OpenPGP digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pythonv, take two
On 03/30/2011 11:23 PM, P.J. Eby wrote: At 01:35 PM 3/30/2011 -0400, Carl Meyer wrote: So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position. Looking at it from a software distribution POV, I would say that the virtual prefix is what it should point to, since that means things won't get installed to the wrong place. Indeed. The issue is that from every point of view other than software distribution (code trying to find anything other than site-directories), sys.prefix pointing to the virtualenv is wrong. Unless, like virtualenv, we try to make it just right enough by copying/symlinking things into the virtualenv that otherwise wouldn't need to be there. But this may be moot; I didn't realize until I checked just now that setuptools (well, easy_install) builds its own list of site-dirs based on sys.prefix. It doesn't look like that's used for installation, but it is used for pre-flight checking pth-capability and finding pth files. So if easy_install directly relies on site-directories always being sys.prefix-based, that probably forces the issue. (Of course, a configuration option could be used to override it, if really necessary.) This seems like the worst option of all - then third-party code really would have no idea at all what sys.prefix is supposed to mean, or what can be reliably found there. Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
