Re: [Distutils] numpy's distutils fork circa 2001
On 30/11/2012 13:30, Chris Withers wrote: On 29/11/2012 23:00, Dag Sverre Seljebotn wrote: But while I know the problem you mentioned has come up on the SciPy list, it really is a trifle in this context. Having to remember to install NumPy before SciPy is not what people have issues with. The pain points SciPy deals with are just from a different world. I think the most common problems are of the sort a) linking with the right LAPACK in the right way, b) using Fortran compilers with distutils and in particular combining various Fortran compilers with various C runtime libraries on Windows. Going beyond SciPy and just to provide another example, a non-trivial number of scientific projects use Fortran and Cython and NumPy together; that's basically unachievable with either distutils or numpy.distutils or Cython.Distutils (so one uses an actual build tool such as CMake or scons of waf instead; distutils doesn't really qualify (or was meant to be) a build tool after all). I wonder if this is where the wheel format fits in? If I understand, it lets you use build tools to do the build, and then wheel is just the distribution format. Not much. Like you said, it's just the distribution format. It doesn't let you use build tools to do the so much as not be involved in that process at all. What's more on target is a common command that will build any conforming Python package from source, but letting it defer to other build tools to do the actual build. Wheels are probably the best standard for the end result of that process, but eggs or even just bdist_dumb are good enough for most cases. I don't really recall if distutils2's pysetup command quite gets us this far, but it's on the right track. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] numpy's distutils fork circa 2001
On 29/11/2012 21:51, Alex Clark wrote: Is anyone familiar with this situation enough to comment on it, in a few sentences? numpy-discussion would be a better mailing list for this question: http://www.scipy.org/Mailing_Lists Based on: - https://github.com/numpy/numpy/commit/f1a2d6376c430f65550efa235209b86c1a0967e3 It looks like: - Numpy forked distutils (2001) prior to setuptools existence/popularity (2004) It's not a fork. It's just a set of extensions to distutils' functionality. setuptools doesn't really implement anything numpy.distutils does, so that's not really relevant. - As a result, SciPy users must now pay a terrible price (i.e. SciPy does not `install_requires` numpy as you might expect; i suspect because it relies on numpy's distutils… which it can't rely on until it's installed… which it can't `install_requires` until it's installed…) Yup. No real way around it until we have such metadata in a declarative, non-executable data file rather than executed code. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] the 'wheel' binary package format
On 7/1/12 6:52 AM, Daniel Holth wrote: I got tired of waiting for lxml to compile over and over again, so I invented a binary packaging format called 'wheel' (of cheese) that uses Metadata 1.2 and .dist-info directories instead of .egg-info, patched pkg_resources.py to be able to load .dist-info directories, implemented python setup.py bdist_wheel, and patched pip to be able to install .whl files. The gist of the spec is that it is a zip file with the .whl extension containing the contents of 'purelib' for a distribution, plus a Name-1.0.dist-info/ directory with the metadata files, plus Name-1.0.data/subdir directories for scripts, platlib, packaging's categories, ... My specification so far is at https://docs.google.com/document/d/1mWPyvoeiqCrAy4UPNnvaz7Cgrqm4s_cfaTauAeJWABI/edit and an lxml compiled for linux-x86-64 is at https://docs.google.com/open?id=0BxHz5bC4iN5TN0VWTFNrZGtCbWs http://bitbucket.org/dholth/distribute http://github.com/dholth/pip http://pypi.python.org/pypi/wheel Perhaps it will be useful. The implementation is still pretty rough, for example it does not check the architecture while installing, but it could be a handy way to speed up repeated virtualenv builds. Very nice. I like how much information the file names provide. Do you think it would be possible to add (optional) build numbers to the specification? At Enthought, we make a Python distribution with access to a repository of binary eggs. When regularly building third party eggs for downstream consumption, it's essential to have build numbers in the file name since we need to fix bugs in our builds while keeping the upstream version of the package itself the same. Linux distributions have the same issue and put in build numbers into their package file names. I'm not sure I'd go so far as to force every .whl to have a build number in it, but it would be great if we could figure out a way to have a compatible convention that could include build numbers when needed. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] At least one package management tool for 2.7
On 2010-03-24 17:19 PM, Tarek Ziadé wrote: BTW, That makes me wonder how hard it would be to make it use a plain python setup.py install call instead, to remove the easy_install dependency ? Since it could simply loop into each dependency Some packages use setuptools and some don't. Since setuptools changes the behavior of the install command to install .eggs, those packages need extra flags to use pip's preferred flat installs. Using easy_install (with the right flags) yields consistent behavior for all packages. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] At least one package management tool for 2.7
On 2010-03-24 17:53 PM, Tarek Ziadé wrote: On Wed, Mar 24, 2010 at 11:27 PM, Robert Kernrobert.k...@gmail.com wrote: On 2010-03-24 17:19 PM, Tarek Ziadé wrote: BTW, That makes me wonder how hard it would be to make it use a plain python setup.py install call instead, to remove the easy_install dependency ? Since it could simply loop into each dependency Some packages use setuptools and some don't. Since setuptools changes the behavior of the install command to install .eggs, those packages need extra flags to use pip's preferred flat installs. Using easy_install (with the right flags) yields consistent behavior for all packages. That's just an environment problem. Notice that I've never tried it, but this could be done by properly configuring the default behavior of easy_install globally (with the distutils.cfg file for instance since its a distutils command) so it behaves like pip wants when python setup.py install is called. The problem with doing that currently is that distutils complains when it receives an option that it doesn't recognize. Packages that do not use setuptools will fail if you try to provide setuptools-specific configuration: $ python setup.py install running install error: error in /Users/rkern/.pydistutils.cfg: command 'install' has no such option 'single_version_externally_managed' Making those warnings instead of errors would probably work. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] At least one package management tool for 2.7
On 2010-03-24 18:30 PM, Tarek Ziadé wrote: On Thu, Mar 25, 2010 at 12:20 AM, Robert Kernrobert.k...@gmail.com wrote: [??] The problem with doing that currently is that distutils complains when it receives an option that it doesn't recognize. No because these are options you pass to easy_install only in your distutils.cfg file: [easy_install] foo=bar [bdist_rpm] baz=1 ... Packages that do not use setuptools will fail if you try to provide setuptools-specific configuration: $ python setup.py install running install error: error in /Users/rkern/.pydistutils.cfg: command 'install' has no such option 'single_version_externally_managed' You need to use [easy_install] in your .pydistutils.cfg, not [install]. The easy_install command is called internally by the install command in distribute/setuptools. The --single-version-externally-managed and --record options are only on the install command, not the easy_install command. $ cat ~/.pydistutils.cfg [easy_install] single-version-externally-managed=True record=/dev/null $ python setup.py install running install error: error in /Users/rkern/.pydistutils.cfg: command 'easy_install' has no such option 'single_version_externally_managed' Please show me a configuration that you have tested. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] At least one package management tool for 2.7
On 2010-03-24 19:56 , P.J. Eby wrote: At 05:27 PM 3/24/2010 -0500, Robert Kern wrote: Since setuptools changes the behavior of the install command to install .eggs, those packages need extra flags to use pip's preferred flat installs. Using easy_install (with the right flags) yields consistent behavior for all packages. Technically, those flags cause setuptools to *not* use easy_install, but rather to use more-or-less plain vanilla distutils installation. I haven't looked at pip's code, but I doubt it's relying on easy_install at all. It does the 'python -c import setuptools;execfile(...) install --flags' trick. It uses setuptools to make sure that it can pass the same flags for both setuptools-using and non-setuptools-using packages. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] modules in different folders but under same namespace
On 2010-02-04 10:53 AM, Riccardo-Maria BIANCHI wrote: Hi, I have a package structured like this: package/__init__.py src/ __init__.py mod1.py share/__init__.py mod2.py Now I can import them as: package.src.mod1 package.share.mod2 How can I use Distutils to be able to import both of them under the same package namespace as: import package.mod1 import package.mod2 ? Thanks a lot in advance for your kind help! Remove the package/src/__init__.py and package/share/__init__.py . In your package/__init__.py, append the src/ and share/ directories to the __path__ list. E.g. import os import pkgutil for subdir in ['src', 'share']: __path__.append(os.path.join(os.path.dirname(__file__), subdir)) -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] modules in different folders but under same namespace
On 2010-02-04 11:06 AM, Robert Kern wrote: On 2010-02-04 10:53 AM, Riccardo-Maria BIANCHI wrote: Hi, I have a package structured like this: package/__init__.py src/ __init__.py mod1.py share/__init__.py mod2.py Now I can import them as: package.src.mod1 package.share.mod2 How can I use Distutils to be able to import both of them under the same package namespace as: import package.mod1 import package.mod2 ? Thanks a lot in advance for your kind help! Remove the package/src/__init__.py and package/share/__init__.py . In your package/__init__.py, append the src/ and share/ directories to the __path__ list. See this essay for more documentation on this feature: http://www.python.org/doc/essays/packages.html -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] modules in different folders but under same namespace
On 2010-02-04 11:05 AM, Tarek Ziadé wrote: On Thu, Feb 4, 2010 at 5:53 PM, Riccardo-Maria BIANCHI riccardo.maria.bian...@cern.ch wrote: Hi, Hi, I have a package structured like this: package/__init__.py src/ __init__.py mod1.py share/__init__.py mod2.py Now I can import them as: package.src.mod1 package.share.mod2 How can I use Distutils to be able to import both of them under the same package namespace as: import package.mod1 import package.mod2 unrelated to Distutils but, What you can do is add these line in package/__init__.py : from package.src import mod1 from package.share import mod2 Beware though, that this means the modules will be loaded even if import package is called, which could have some unwanted side effects depending on the code. This does not work. You cannot import package.mod1 under this scenario. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] modules in different folders but under same namespace
On 2010-02-04 11:24 AM, Tarek Ziadé wrote: Not sure why you want to remove the __init__.py files in sub and share in your solution though. It prevents accidentally importing both package.src.mod1 and package.mod1 and helps readers of the code understand that only package.mod1 is correct. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] modules in different folders but under same namespace
On 2010-02-04 11:37 AM, Floris Bruynooghe wrote: On Thu, Feb 04, 2010 at 05:53:16PM +0100, Riccardo-Maria BIANCHI wrote: Now I can import them as: package.src.mod1 package.share.mod2 How can I use Distutils to be able to import both of them under the same package namespace as: import package.mod1 import package.mod2 In package/__init__.py: import src.mod1 as mod1 import share.mod2 as mod2 And then you can use just one import package statement. If you want src and share to be separately installable then it gets more complicated (you'd need namespace packages etc), but why complicated when simple works? Because simple doesn't work. You cannot do import package.mod1 or from package.mod1 import foo with that scheme. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distribute doesn't use pip?
On 2010-01-29 16:03 PM, Tom Davis wrote: File /tmp/easy_install-m7gqed/numpy-1.4.0/numpy/distutils/misc_util.py, line 1668, in add_scripts AttributeError: 'NoneType' object has no attribute 'extend' I think I may have fixed it in numpy r8080. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distribute doesn't use pip?
On 2010-01-29 16:46 PM, Tom Davis wrote: I downloaded a snapshot, added it to dependency_links, and ran /setup.py install/ again. It threw the same exception. Traceback, please? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distribute doesn't use pip?
On 2010-01-29 16:47 PM, Tom Davis wrote: Oops, my apologies! The new error location is: File /tmp/easy_install-JJUS9R/numpy/numpy/distutils/misc_util.py, line 1278, in add_include_dirs AttributeError: 'NoneType' object has no attribute 'extend' Thank you for the traceback! Try r8081. I fixed a couple of other places where lists may not be initialized properly. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distribute doesn't use pip?
On 2010-01-29 17:08 PM, Tom Davis wrote:
Tested; failed with the following:
File
/tmp/easy_install-cmuJYo/numpy/numpy/distutils/command/build_src.py,
line 385, in generate_sources
File /deploy/repo/setup.py, line 390, in generate_config_h
File /deploy/repo/setup.py, line 41, in check_types
File /deploy/repo/setup.py, line 255, in check_types
SystemError: Cannot compiler 'Python.h'. Perhaps you need to install
python-dev|python-devel.
I'm not sure if the former is an issue with numpy or something else
(issue with Python.h). Although i am positive my dev headers are
installed (and numpy still installs fine with pip), I don't know if
distutils is having trouble finding them for some unrelated-to-numpy
reason. I can't think of another package offhand that requires the
headers or I'd test that theory.
Try this patch to see some more output about the failed compile. It will be the
second piece of code dumped. You should see something like the following:
Generating build/src.macosx-10.3-i386-2.5/numpy/core/include/numpy/config.h
compiling '_configtest.c':
#include Python.h
/* we need a dummy line to make distutils happy */
C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk
-fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd
-fno-common -dynamic -DNDEBUG -g -O3
compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core
-Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath
-Inumpy/core/include
-I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c'
gcc: _configtest.c
success!
Please verify that Python.h exists in one of the -I locations. It is possible
that we are passing something weird.
Index: numpy/distutils/misc_util.py
===
--- numpy/distutils/misc_util.py(revision 8081)
+++ numpy/distutils/misc_util.py(working copy)
@@ -1733,8 +1733,8 @@
cmd = get_cmd('config')
cmd.ensure_finalized()
-cmd.dump_source = 0
-cmd.noisy = 0
+cmd.dump_source = 1
+cmd.noisy = 1
old_path = os.environ.get('PATH')
if old_path:
path = os.pathsep.join(['.',old_path])
You may want to join us over numpy-discussion until we identify real issues with
Distribute or pip.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On 2009-12-22 12:48 PM, Martin v. Löwis wrote: By installer, I mean things produced by bdist_*. A significant portion of windows users don't like eggs, and prefer .exe-based (or .msi-based) installers. Currently, it is not possible to (reliably) convert from one to the other (e.g. eggs-wininst), but there is no reason why this is so. There is most certainly a reason. The binary distribution may be lacking pieces of source code that would be needed to build another (different) binary distribution. For example, if you have a Linux RPM or .deb, it is, in general, not possible to create a Windows installer out of this, as you may need to recompile extension modules. Including the full source code in the binary distribution just to support this rare use case is unreasonable: people who want a different package format can build from the source package. Yes, that use case is rare (and bizarre), but the desire to convert Windows-platform eggs to .msi or .exe installers is not so rare and does not present the same level of difficulty. The latter is the use case David was referring to, not the one you outlined. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On 2009-12-22 13:20 PM, Martin v. Löwis wrote: Yes, that use case is rare (and bizarre), but the desire to convert Windows-platform eggs to .msi or .exe installers is not so rare and does not present the same level of difficulty. The latter is the use case David was referring to, not the one you outlined. Why do people want to do that? Can't they build the type of package they want from the source package? Not necessarily on Windows. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 2009-11-11 15:02 PM, Greg Ewing wrote: Tarek Ziadé wrote: If the answer to that solution is just: Distutils sucks anyways.., it is not really helpfull imho.. I don't see the point to write Distutils from scratch, instead of making it evolve. If you can see a way to get from the current distutils code to something with a well-designed and well-documented API and a clean implementation, that would be fine by me. When I say it's fundamentally broken, I'm really talking about the API. My idea of what an API for a build system should be like is more like make or scons, which slice the functionality up in a completely orthogonal direction to the way distutils does. Maybe it would be possible to plug such a system in under the existing build_ext class. I don't know. In fact, David has. http://pypi.python.org/pypi/numscons/ I think I would like the same philosophy applied to other areas of distutils, not just compiling extensions. Otherwise it would feel like two incompatible systems bolted together. It does feel something like that. The build system is just one of the problems with distutils' internals, in my experience. You can think of the rest of distutils as a little application framework for command line utilities. I think this framework simply fails to provide in very fundamental ways, like the extend commands by subclassing design pattern. That choice makes it fundamentally difficult to combine extensions together. I really don't see a way to evolve away from that (and believe me, over the last decade, I've tried). You just need to redesign the internals if you want to get away from that. You can't get from any point A to any point B by evolving in small steps that are functional (not to mention backwards compatible!) all the way through. With all respect to Greg Ward and the rest of the original distutils authors, it was a fantastic improvement over the state of affairs ten years ago, but we've learned a lot about application frameworks and about building software since then. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On 2009-11-11 15:59 PM, Tarek Ziadé wrote: On Wed, Nov 11, 2009 at 10:36 PM, Robert Kernrobert.k...@gmail.com wrote: [..] It does feel something like that. The build system is just one of the problems with distutils' internals, in my experience. You can think of the rest of distutils as a little application framework for command line utilities. I think this framework simply fails to provide in very fundamental ways, like the extend commands by subclassing design pattern. That choice makes it fundamentally difficult to combine extensions together. I really don't see a way to evolve away from that (and believe me, over the last decade, I've tried). You just need to redesign the internals if you want to get away from that. You can't get from any point A to any point B by evolving in small steps that are functional (not to mention backwards compatible!) all the way through. I am very surprised about this statement. What did you tried for the paste decade and failed to do ? I hear some complaints since a week, but beside's David examples I didn't read any other precise use cases. We're looking through the build_ext use case, and we are making some improvement on the other thread. So why not doing this in other issues ? Let's discuss your use case. And if it means adding new options to run arbitrary commands like post/pre hooks to a given command, to avoid subclassing an existing command, let's do it. http://svn.scipy.org/svn/numpy/trunk/numpy/distutils/command/ All of it. Now consider that here we are also trying to play nicely with the setuptools extensions, and Pyrex, and David is working on integrating Cython support. To get to one real specific problem, let's consider build_src. build_src is a new subcommand in numpy.distutils that builds C extension sources from other files. We use this to hook in f2py's wrapper generator and other more ad hoc forms of generating wrappers. When build_ext uses --inplace, we need build_src to use --inplace, too, because it will often output some final products in addition to intermediate wrapper sources. In order to integrate this with setuptools' develop command (which invokes build_ext --inplace but not build_src --inplace because setuptools knows nothing about numpy.distutils), we need to create a subclass of setuptool's develop command that will reinitialize build_src with the appropriate option. Then we need to conditionally place the develop command into the set of command classes so as not to introduce a setuptools dependency on those people who don't want to use it. This is nuts. numpy.distutils shouldn't have to know anything about setuptools to accomplish this if the framework were properly designed. And this doesn't even get into the fact that many of the numpy.distutils command classes that are shared with setuptools are conditional subclasses and probably still buggily cobbled together. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils vs redesigning it (was people want CPAN)
On 2009-11-11 17:18 PM, David Cournapeau wrote: On Thu, Nov 12, 2009 at 6:59 AM, Tarek Ziadéziade.ta...@gmail.com wrote: And let's drop the backward compat issues in these discussions, so we don't burn out in details. That's the part I don't understand. If backward compatibility is not a concern, why keeping distutils ? If you change the command and Distribution class design, what remains of the original code ? You are changing the API and the implementation (which are quite tangled with each other in distutils case), almost none of the original code would remain. It really feels to me like you are getting the pain of backward compatibility without the gains. What am I missing ? I think Tarek wants to avoid the Second System Effect and related problems. http://en.wikipedia.org/wiki/Second-system_effect http://www.joelonsoftware.com/articles/fog69.html While that is usually a good habit to cultivate and a good default position, it's not an unyielding law or anything. You have think deeply about whether the code is the way it is because it contains useful knowledge or if it is just constrained by ossified decisions from the past. I tend to think that the useful knowledge can be extracted from distutils and applied well in a rewrite. The most important useful knowledge is the extension building flags, and I think you have done a good job of transplanting that information into the entirely different build system of SCons. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils vs redesigning it (was people want CPAN)
On 2009-11-11 18:04 PM, Tarek Ziadé wrote: On Thu, Nov 12, 2009 at 12:18 AM, David Cournapeaucourn...@gmail.com wrote: On Thu, Nov 12, 2009 at 6:59 AM, Tarek Ziadéziade.ta...@gmail.com wrote: And let's drop the backward compat issues in these discussions, so we don't burn out in details. That's the part I don't understand. If backward compatibility is not a concern, why keeping distutils ? If you change the command and Distribution class design, what remains of the original code ? You are changing the API and the implementation (which are quite tangled with each other in distutils case), almost none of the original code would remain. It really feels to me like you are getting the pain of backward compatibility without the gains. What am I missing ? What you are missing is that : - you are convinced that distutils should be written from scratch. I am not for many reasons. Some others are not either. it won't happen. the only thing that could make it happen is the replacement of distutils by another tool that is used by the majority of the community for several years. That's basically what David is suggesting when he says rewrite. He doesn't mean to replace distutils with another package named distutils. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils vs redesigning it (was people want CPAN)
On 2009-11-11 18:48 PM, Tarek Ziadé wrote: I want to improve Distutils, but not for the price of a complete drop. I don't think the edge cases we are discussing worth it, and I still fail to see why we can't work them out in the context of the existing tool. Mostly because I'm entirely uninterested in helping you make incremental improvements that are going to break all the hard work we've already done just to get things working as it is. I find that prospect incredibly frustrating. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils vs redesigning it (was people want CPAN)
Tarek Ziadé wrote: On Thu, Nov 12, 2009 at 1:59 AM, Robert Kern robert.k...@gmail.com wrote: On 2009-11-11 18:48 PM, Tarek Ziadé wrote: I want to improve Distutils, but not for the price of a complete drop. I don't think the edge cases we are discussing worth it, and I still fail to see why we can't work them out in the context of the existing tool. Mostly because I'm entirely uninterested in helping you make incremental improvements that are going to break all the hard work we've already done just to get things working as it is. I find that prospect incredibly frustrating. I can understand that. I am very frustrated too because in the last threads, whenever we are speaking about design, it seems that on your side and David side, dropping Distutils seems like a post-condition to everything. Even if I have made some proposals on some concrete design changes. In our considered opinion, piecemeal changes probably aren't going to solve the significant problems that we face. At best, they simply aren't going to help; we wouldn't be able to use the new features until we can drop support for Python 2.6. numpy and scipy still need to support Python 2.4. At worst, they would introduce incompatibilities that we will have to work around somehow. Who's we by the way ? The Scons project ? or the numpy.distutils project ? numpy and scipy. While I hesitate to speak for an entire community, I must say that David and my opinions on distutils are shared by a good portion of our community that has to deal with building and packaging. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils vs redesigning it (was people want CPAN)
Lennart Regebro wrote: 2009/11/12 David Cournapeau da...@ar.media.kyoto-u.ac.jp: I am trying to understand what is 'nebulous' about our claims. We have given plenty of hard and concrete examples of things which are problematic in distutils. The major progress in our build issues have been achieved by dropping distutils. Up to now, the only people who have claimed that distutils can solve our problems are the people who are not involved at all with our projects, and the people who claim distutils cannot solve our problems are the people involved with it. That's odd to say the least. Now, I am ready to accept that we are missing the big picture and the rest of the community knows more about it. But I certainly have not seen strong arguments to believe it so far. Look, there is only one way this argument can be solved, and that is by building something better than distutils. Honestly. Now, I'm sure as heck not going to spend time on that, and Tarek don't think it's a good idea, so it's up to you guys. I think you're missing the point of our statements. We're not asking you to work or not work on anything. Guido asked, in response to a comment from our community: Is the work on distutils-sig going to be enough? And our answer is that no, it's probably not. It's not addressing our most significant problems. David's work, e.g. on numscons, is helping, but it is still constrained by the requirement of working within distutils' framework. Bitching that distutils needs to be scratched and rewritten is not going to help. You need to DO it. Pardon us for sincerely answering the questions asked of us. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils vs redesigning it (was people want CPAN)
Lennart Regebro wrote: 2009/11/12 David Cournapeau courn...@gmail.com: On Thu, Nov 12, 2009 at 2:28 PM, Lennart Regebro rege...@gmail.com wrote: Bitching that distutils needs to be scratched and rewritten is not going to help. You need to DO it. Nobody asked you (or anyone else) to do anything. Guido asked a question, we answered with our rationale. There is no need to be rude. It was not my intention to be rude, and I don't believe I was. This discussion has been ongoing for a long time, and I have just recently asked questions (which of course has been ignored) about this. This is a discussion that has been going on for a long time, more than a year, at least, yet noone who says distutils needs to be scratched and something else written does something about it. I've pointed this out repeatedly, and been ignored. Perhaps I used to strong language this time, but on the other hand, I wasn't ignored this time. The discussion is pointless. If you want something better than distutils, do it. David's working on it: http://pypi.python.org/pypi/numscons -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils vs redesigning it (was people want CPAN)
Lennart Regebro wrote: 2009/11/12 Robert Kern robert.k...@gmail.com: The discussion is pointless. If you want something better than distutils, do it. David's working on it: http://pypi.python.org/pypi/numscons Enable to use scons within distutils to build extensions I'm confused now. In fact, David wouldn't have bothered with the distutils bits at all if I hadn't been so insistent that the standard python setup.py install, etc. would continue to work. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils vs redesigning it (was people want CPAN)
Lennart Regebro wrote: 2009/11/12 Robert Kern robert.k...@gmail.com: The discussion is pointless. If you want something better than distutils, do it. David's working on it: http://pypi.python.org/pypi/numscons Enable to use scons within distutils to build extensions I'm confused now. It's a start. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils vs redesigning it (was people want CPAN)
Glyph Lefkowitz wrote: Still, it would have been more helpful to point out how exactly this problem could be solved, and to present (for example) a description of similar objects politely interacting and delegating responsibility to one another to accomplish the same task. Sorry, I edited out the bit at the last minute where I explained that it would be great to have a centralized option-managing object such that any command can ask what options were set on any other regardless of the dependencies between commands. I would definitely characterize these assertion from Robert as nebulous, given that the prior messages in the thread (as far as I can tell) do not describe the kind of massive-overhaul changes which would fix things, only the problems that currently exist: In our considered opinion, piecemeal changes probably aren't going to solve the significant problems that we face. Why not? The whole of computer history is the story of piecemeal improvements of one kind or another; despite perennial claims that, for example, hierarchical filesystems or bit-mapped displays fundamentally cannot support one type of data or another, here we are. Perhaps in my head the analogy with biological evolution is unjustifiably strong. Species can't always get from point A to point B while making viable intermediates with incremental changes. Evolutionary deadends happen frequently. In software, design decisions early on affect how much change the software can tolerate (which is why we are told to design for change). I think that distutils exemplifies a design that is particularly adverse to evolution. Or this one, also from Robert: Mostly because I'm entirely uninterested in helping you make incremental improvements that are going to break all the hard work we've already done just to get things working as it is. Why do incremental improvements have to break all the hard work that has already been done? Surely this is what a compatibility policy is about. Since Tarek keeps asking us to make proposals without thinking about compatibility, I wonder what policy is being kept in mind. My comment stems from my worry about that attitude. In any case, I think that keeping compatibility while making improvements to the code in situ is going to be quite difficult. The distutils API is not clean. Using distutils beyond setup() and Extension() involves too much intimate knowledge of the detailed implementation of distutils as-is. That's why we got a 2.6.4 release. This is why I think the piecemeal evolution of distutils is not going to work. The act of fixing distutils to make a cleaner API for modification and extension *triggers* the problem that the fix is supposed to address. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 376 -- support for release versions?
On 2009-11-09 16:28 PM, David Lyon wrote: On Mon, 09 Nov 2009 13:45:29 -0800, Sridhar Ratnakumar sridh...@activestate.com wrote: Same with PyPM. Which is why release numbers need to be stored elsewhere. Given the current design/proposals - what makes sense to me now is to store it in .egg-info/PyPM.txt. But release numbers are already stored in the PKG_INFO metadata file. Why store it somewhere else? No, they're not. http://www.python.org/dev/peps/pep-0241/ -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 376 -- support for release versions?
On 2009-11-09 16:53 PM, David Lyon wrote: On Mon, 09 Nov 2009 16:39:40 -0600, Robert Kernrobert.k...@gmail.com wrote: But release numbers are already stored in the PKG_INFO metadata file. Why store it somewhere else? No, they're not. http://www.python.org/dev/peps/pep-0241/ Ok, well they're part of http://www.python.org/dev/peps/pep-0345/ I'll revise my statement back to saying that doing it according to PEP-345 makes the most sense imho. Where? I don't see it. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
David Cournapeau wrote: I don't understand what's there to buy. Several people reported distutils errors without any backtrace, though a fair shared of those were caused by our own extensions. distutils specifically swallows exceptions and formats them for users by default. After all, it is trying to behave like a regular command line program that interacts with users, not developers. This is easily overridable for developers who are trying debug problems by setting the environment variable DISTUTILS_DEBUG=1. This will make distutils just give the traceback. Is this what you are referring to? -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 376 -- support for release versions?
On 2009-11-05 15:03 PM, Tarek Ziadé wrote:
On Thu, Nov 5, 2009 at 9:31 PM, Sridhar Ratnakumar
sridh...@activestate.com wrote:
Hi,
While reading the PEP I noticed that there seems to be no straightforward to
specify the package release versions.
egginfo_dirname('python-ldap', '2.3.10')
'python_ldap-2.3.10.egg-info'
Consider the following PyPM package file name:
python-ldap-2.3.10_linux-x86_2.6_2.pypm
Here 2.3.10 is python-ldap's version, 2.6 is the Python version .. and 2
(the last component) is the binary release number (for ActiveState
repository). This is to upgrade previous releases of the same python-ldap
(say, python-ldap-2.3.10_linux-x86_2.6_1.pypm).
If I understand correctly, ActiveState has its own releases of
python-ldap repackaged in pypm files,
and has several releases per python-ldap versions.
So, why it doesn't have its own versions ? like: 2.3.10.2 in that
case. (following a PEP 386-compatible scheme)
So this would be:
egginfo_dirname('python-ldap', '2.3.10.2')
'python_ldap-2.3.10.2.egg-info'
Generally speaking, it is desirable to distinguish between the upstream
package's version and the build number rather than munging them together. You
never know when the upstream package will decide that they need to make a really
quick bugfix release and just append .1. The main package version is and
should remain entirely under the control of the upstream developers, not the
binary packagers. However, the binary packagers need a place to mark revisions
of the binary packages distinct from the upstream sources.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distutils and Distribute roadmap (and some words on Virtualenv, Pip)
On 2009-10-27 18:41 PM, David Lyon wrote: On Tue, 27 Oct 2009 20:51:10 +0100, Tarek Ziadéziade.ta...@gmail.com wrote: $ easy_install your_bdist_wininst_dist.exe will install it and process the dependencies from the install_requires option. And pip should be compatible soon too. That makes this format a perfect binary format for win32. I'm not sure about that Tarek.. An .exe installer as a perfect binary format for python packages? Are you serious? That is the biggest security threat I can think of, asking python users to run unverified, unsigned, un-trusted executable files on their systems. easy_install does not execute the executable. bdist_wininst installers are zip files concatenated with an executable header. easy_install just unzips the file as if it were a zipped egg and ignores the executable part. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Cache PYTHONPATH? (Re: make unzipped eggs be the default)
On 2009-07-29 16:47, David Lyon wrote: Anyway, you're kindof biting the hairs on the tail here.. because 3rd party packages don't impact the size of the whole python installation that much. My site-packages directory would like a word with you: [~]$ cd /Library/Frameworks/Python.framework/Versions/Current [Current]$ du -hsc . 1.5G. 1.5Gtotal [Current]$ du -hsc lib/python2.5/site-packages 1.4Glib/python2.5/site-packages 1.4Gtotal -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distutils on embedded devices - was Cache PYTHONPATH? (Re: make unzipped eggs be the default)
On 2009-07-29 18:20, David Lyon wrote: On Wed, 29 Jul 2009 16:58:13 -0500, Robert Kernrobert.k...@gmail.com wrote: My site-packages directory would like a word with you: [~]$ cd /Library/Frameworks/Python.framework/Versions/Current [Current]$ du -hsc . 1.5G. 1.5Gtotal [Current]$ du -hsc lib/python2.5/site-packages 1.4Glib/python2.5/site-packages 1.4Gtotal ok - so what are you complaining about? you have a 1.4G drive or more... I am not complaining about anything. I am disputing your assertion that 3rd party packages don't impact the size of the whole python installation that much. That's not a universally true statement. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distutils on embedded devices - was Cache PYTHONPATH? (Re: make unzipped eggs be the default)
On 2009-07-29 18:44, David Lyon wrote: On Wed, 29 Jul 2009 18:36:49 -0500, Robert Kernrobert.k...@gmail.com wrote: I am not complaining about anything. I am disputing your assertion that 3rd party packages don't impact the size of the whole python installation that much. That's not a universally true statement. ok but is it a real problem or a perceived problem ? If it's a real problem then spend some time to trim some fat.. or spend an extra $20 and get a bigger memory card.. At least your embedded device isn't a 4k one... then we could moan and groan together about the bad old good old days I really don't know enough about your hardware to say any more... I'm not on an embedded device. I don't have a particular problem that I am asking you to solve. I am challenging your assumption that third party packages don't impact the size of the whole Python installation. It's just not true. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Data files with setuptools
On 2009-07-27 11:29, Adeel Ahmad Khan wrote: The README is not package data because it's not inside a package. You can't install package data in a project that only includes modules. In any case, there's no point in shipping documentation inside an egg, because only your project's *code* will be able to read it, not humans. Human-readable documentation only belongs in a source distribution. I'm writing a command-line program and I was using the README to print help. Should I just copy the README into my module then or is there a better way? Most likely, the README and the help text should be different. READMEs need information like how to install the program. Help text does not. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Announcing the 'Distribute' project
On 2009-07-16 15:29, Barry Warsaw wrote: On Jul 16, 2009, at 4:21 PM, P.J. Eby wrote: At 10:29 AM 7/16/2009 -0400, Barry Warsaw wrote: One thing I'd like to see for this new project is more built-in support for the popular revision control systems. Arguably, the existing built-in support is a bug that should be fixed. If SVN support was a plug-in, it could be more easily fixed when stuff changes. Using a plugin for this is still a pain, but revision control parity would still be a good thing, whether that argues for inclusion or exclusion in the core. I suspect one could have implementations in the core that could be overridden by plugins. Then you can have the best of both worlds at the cost of some additional complexity. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Help making setuptools install more like plain distutils one
On 2009-04-20 15:49, Christian Hudon wrote:
The only thing I'd need would be way to reliably determine if I'm
dealing with setuptools-based setup.py or not. I can always call python
setup.py installl --help and look if
--single-version-externally-managed is present in the output. Is there a
cleaner version of doing this?
You could ensure that you always have a setuptools-based setup.py:
python -c import setuptools; execfile('setup.py') install
--single-version-.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deprecate MANIFEST.in
On 2009-04-07 00:20, Lennart Regebro wrote: On Tue, Apr 7, 2009 at 06:17, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: Ben Finney wrote: An sdist is *not* just a tarball of the source files. It is if you consider any non-tracked file to be an anti-usecase, which is what I was answering to :) Except for files generated by the sdist/build-process, yes. Why would you want to include files in a distribution that are neither controlled source files or a result of the sdist/build process? Unfortunately, the setup.py file is not always capable of performing all of the build processes (at least, not easily). One might use a Makefile or other build tool to build the docs or other assets. So you need a manual way of telling the setup.py file what additional files to include. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] console scripts in distutils
On 2009-02-14 06:26, Tarek Ziadé wrote: Hello I really like the way setuptools wraps scripts depending on the platform. http://peak.telecommunity.com/DevCenter/setuptools#automatic-script-creation I am not talking about the entry_point feature, but the way it creates executables. What would people think about making that feature go into distutils itself to enhance the current script behavior? http://bugs.python.org/issue870479 I have a couple of issues with this feature as implemented in setuptools. I think they should be addressed if the feature makes its way into the stdlib: 1. The pkg_resources import can make startup time quite large. For small command-line utilities like my grin tool and DivMod's PyFlakes, this can make them unusable. If you bring the feature into the stdlib, please pay attention to the overhead of locating the real script however you end up doing it. 2. In the generated script, please use if __name__ == '__main__': to block out the executable bits. Currently, multiprocessing does not work on Windows when the application is started by a console_script. Because Windows does not have a true fork, multiprocessing will start up clean Python subprocesses that import the __main__ of the parent process. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Solaris and distutils: Need to pass LIBDIR explicitly with -L when building extensions?
On 2009-01-30 17:08, Floris Bruynooghe wrote:
Hi Dave
On Mon, Jan 26, 2009 at 06:48:06PM -0600, Dave Peterson wrote:
I am trying to build a number of projects that use Python extensions on
Solaris 10 and I've discovered that nothing with extensions will link
unless I explicitly pass in a '-L/path/to/python/lib/dir' because
libpython2.5.so is not otherwise found when the '-lpython2.5' argument
is specified during linking.
[...]
If that all seems correct, then it appears the issue is the
finalize_options() source in lib/distutils/commands/build_ext.py. There
are a number of if blocks that explicitly append the value of
distutils.sysconfig.get_config_vars('LIBDIR') to the list of
library_dirs used to link built extensions with. However, there doesn't
seem to be one of these for Solaris / sunos.
Could you point to one of the projects you're having trouble with? I
haven't had any such problems with Python 2.5 on Solaris 10, distutils
always finds the right things when I'm building extension modules.
It's not specific to any one project. Mostly, it seems to be tied to using a
custom --prefix where $prefix/lib is not on the default library search path. Are
you using such a custom --prefix?
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Help with easy_install
Martin Manns wrote: Phillip J. Eby pje at telecommunity.com wrote: From reading your setup script, I don't understand why your package needs a .pth file of its own, let alone why it's writing it directly in the setup script. I would expect it to work correctly if you simply delete that portion of the setup script. In fact, it works for me that way. Without the .pth file, the libraries in the folder pyspread are not found. If I leave it out, the program fails to start (after installation finished successfully). Rename pyspread.py to something else, then use package imports. E.g. from pyspread import _widgets from pyspread._choicebars import MainMenu, MainToolbar from pyspread._dialogs import MacroDialog, CsvImportDialog, DimensionsEntryDialog, \ AboutDialog from pyspread._interfaces import CsvImport, string_match from pyspread.config import DEBUG, LIBPREFIX, ICONPREFIX, icon_size -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Simultaneous multi-version support
Ben Finney wrote: Ian Bicking [EMAIL PROTECTED] writes: I'll also note that the require in setuptools-generated scripts causes pretty frequent problems for people, all to support this multi-version feature that no one really uses. I agree heartily that it seems to cause more trouble than it's worth — for my assessment of its worth, anyway. Is it true that “no-one” (to some epsilon value) actually uses this feature? There is one person on enthought-dev who does this for everything. He says it keeps him honest about his dependencies. And consequently keeps us at Enthought honest about ours. I typically have multiple versions of things installed, and switch between them, but I never use pkg_resources.require() to do so. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Availability of setuptools installer for python2.6?
Paul Moore wrote: 2008/10/6 Phillip J. Eby [EMAIL PROTECTED]: At 12:05 PM 10/6/2008 -0700, Chris Mahan wrote: Phillip, At http://pypi.python.org/pypi/setuptoolshttp://pypi.python.org/pypi/setuptools there is no MS windows installer for python 2.6. Do you have an idea of when that might be available? No, it'll likely be a while before I'm doing anything with 2.6. Ian Bicking now has privs for the setuptools PyPI entry, so perhaps he could upload one. (There actually isn't anything Windows-specific about it; all the .exe installers that I upload are actually built on a Linux machine.) This is a really frustrating aspect of setuptools, that pure-Python packages produce version-specific installers. I'm sure I've seen an explanation of why this is necessary somewhere before, but I can't recall precisely where (and I don't really have time to wade through all the setuptools documentation to see if it's in there - it wasn't obvious from reading the contents). Can anyone give me a quick pointer to the explanation? .pyc files are minor-version-specific. Eggs contain .pyc files. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] virtualenv, OS X, and GUIs
Ronald Oussoren wrote: On 9 Jan, 2008, at 1:47, Ian Bicking wrote: Robert Kern wrote: Is there an appropriate mailing list for talking about virtualenv? I've run into a problem where virtualenvs on OS X cannot access the screen to run GUIs. I've found a solution that appears to work and am going to write a patch to let virtualenv do it. I'd like to get some feedback from other OS X virtualenv users, though. What's the solution? It is currently not possible to use GUI api's from the commandline using a virtualenv interpreter, but that's just basic OSX lameness. It works in the system install because that uses some trickery to let the system think that the interpreter is inside an application bundle (at the cost of an extra fork(2)). Basically, I replicated this procedure for the ~/virtualenv/. I copied the Python.app/ into the ~/virtualenv/, used the install_name_tool to point it ~/virtualenv/.Python, and then compiled pythonw.c with a -D flag to point it to the executable in the copied Python.app. It's not particularly pretty, but it worked for me with python.org's Python 2.5 on Leopard. That's good enough for me, but I'm wondering how robust that is going to be for others. Which framework builds do not have a Python.app? I notice that Leopard's 2.3 doesn't, so I'm guessing that Tiger's didn't either. Were there releases of 2.4 that didn't? Should I bother trying to support Python 2.3? It doesn't look like virtualenv tries to. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Sandbox violation help
Dave Peterson wrote: Robert Kern wrote: Dave Peterson wrote: Most of us here are not seeing this error so I'm thinking it isn't something obviously wrong with the setup.py. You can look at the setup.py at : https://svn.enthought.com/enthought/browser/tags/enthought.interpolate_2.0.0b3/setup.py Note that this project does use numpy.distutils' setup function. Perhaps the error has something to do with that? enthought.interpolate uses weave to built its extension module. weave uses ~/.python24_compiled as its location for the temporaries. We need to fix weave to not do this, or rewrite the extension not to use weave. The latter is probably best. I'm probably just being slow here, but how come the build of enthought.interpolate works for some (I'll assert most here) people but not others? I don't believe it has ever worked for anyone from easy_install. The sandbox is only active during the easy_install command, not when just running bdist_egg. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Sandbox violation help
Dave Peterson wrote: Most of us here are not seeing this error so I'm thinking it isn't something obviously wrong with the setup.py. You can look at the setup.py at : https://svn.enthought.com/enthought/browser/tags/enthought.interpolate_2.0.0b3/setup.py Note that this project does use numpy.distutils' setup function. Perhaps the error has something to do with that? enthought.interpolate uses weave to built its extension module. weave uses ~/.python24_compiled as its location for the temporaries. We need to fix weave to not do this, or rewrite the extension not to use weave. The latter is probably best. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to easy_install from source package with mingw32 on Windows?
zooko wrote: A deeper question is: wouldn't it be nice if ./setup.py install automatically used the build -c mingw32 feature when necessary? Would anyone be interested in extending distutils to do that? You can configure it to do so already. Making distutils automatically figure out that this is desired and safe is probably more magic than it's worth. http://docs.python.org/inst/config-syntax.html Use a section in your pydistutils.cfg file like this: [build_ext] compiler=mingw32 -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How can I get setuptools to include files beyond what find_packages() finds?
Stanley A. Klein wrote: I tried to do something to fix the numpy distutils bdist_rpm.py (by trying to follow what was done in install.py) but it didn't work and I got an error message I didn't understand. I'd like to help, but if you don't copy the exact error message here, I can't. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How can I get setuptools to include files beyond what find_packages() finds?
Stanley A. Klein wrote: Robert Kern wrote: Stanley A. Klein wrote: I tried to do something to fix the numpy distutils bdist_rpm.py (by trying to follow what was done in install.py) but it didn't work and I got an error message I didn't understand. I'd like to help, but if you don't copy the exact error message here, I can't. OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- obviously not well -- to follow what was done in numpy/distutils/commands/install.py: import os import sys if 'setuptools' in sys.modules: import setuptools.command.bdist_rpm as old_bdist_rpm class bdist_rpm(old_bdist_rpm): pass else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm Note the difference between these two lines. You imported a module, not the class inside it. Try this instead: if 'setuptools' in sys.modules: from setuptools.command.bdist_rpm import bdist_rpm as old_bdist_rpm else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How can I get setuptools to include files beyond what find_packages() finds?
Phillip J. Eby wrote: At 03:15 PM 7/30/2007 -0400, Stanley A. Klein wrote: I don't need to build the .so files. They are already built. That had to be done using the build-in-place and the numpy distutils for reasons I don't fully understand but are related to the use of numpy. Have you tried building them with setuptools, using the numpy distutils 'build_ext' command, using: setup( cmdclass = dict(build_ext = numpy_build_ext_class_here), ext_modules = list_of_Extension_objects, ... ) Unless there is a radical difference between numpy distutils and the regular distutils, you should be able to do this. Just find numpy's build_ext class, and define the appropriate Extension() objects (for the things to build) in your setup script. Setuptools will then delegate the building to numpy, but handle the installing itself. Again, this is assuming that numpy's distutils extensions don't do anything unfriendly like completely redefine how extension objects work or assume that their commands will be only mixed with other numpy commands. (Setuptools doesn't make such assumptions, and tries to leave the normal distutils stuff alone as much as possible.) I think we're getting into confusing territory by trying to get workarounds for workarounds. Let me try to take us a step back and focus on the initial problem which is that bdist_rpm is not working with enthought.kiva. The existing setup script already does build extensions just fine; they're just not being picked up by bdist_rpm. A suggestion from a coworker of mine prompted Stanley to look at using a script that we have for building enthought.kiva inplace (there are a few more options that are needed beyond python setup.py develop); however, it wasn't really a suggestion to use that as basis for building an RPM. numpy.distutils extends distutils in three ways which are important for enthought.kiva: * automatically adds the location of the numpy headers to the include_dirs of Extensions. (easily replaced) * adds a build_src command that allows users to give Python functions in the sources list of an Extension. These functions will be called to actually generate the real source files. (hard to replace) * allows subpackages to have their own build information which is assembled by the top-level setup.py script. This is mostly legacy from when the enthought package was monolithic and doesn't strictly need to continue. I won't go into details since I don't think it's part of the problem. (straightforward, but time-consuming to replace) numpy.distutils tries hard to not step on setuptools' toes. We actually check if setuptools is in sys.modules and use its command classes instead of distutils' as the base classes for our commands. However, it's possible that neglect of our bdist_rpm implementation has caused the implementations to diverge and some toe-stepping has taking place. The main problem is that bdist_rpm is not working on enthought.kiva. Most likely, this is the fault of numpy.distutils. However, this is a bug that needs to be caught and fixed. Working around it by doing an --inplace build and then trying to include the extension modules as package_data is not likely to work and is not a solution. I'm not usually a Redhat guy, so I don't have much experience with bdist_rpm; however, numpy.distutils has had problems with bdist_rpm in the past. I'm trying to get an environment working on a Redhat machine, and will try to build an RPM for enthought.kiva and try to see the problem first-hand. I've looked over Stanley's emails on the subject, and don't see enough information for me to really pin down the problem. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] (no subject)
Stanley A. Klein wrote: Robert - Thanks for illuminating the issue. The problem I had was as follows. Fedora (also RedHat) uses SE-Linux, which needs to know all the files expected to be in sensitive directories such as the Python site-packages. This includes the pyc and pyo files ordinarily generated by Python as the .py files are executed. It turns out that to do a bdist_rpm for Fedora, it is necessary to create a setup.cfg file containing the lines: [install] optimize = 1 or to add these lines to the existing setup.cfg file. If that is not done, the result in Fedora is an unpackaged files error. This is due to the fact that if distutils/setuptools doesn't cause the pyc and pyo files to be created, Fedora will create them but they won't be properly handled in the spec file created by distutils/setuptools. In trying to do bdist_rpm with kiva, I got the unpackaged files error. This implies that numpy distutils did not properly handle the optimize=1 in the setup.cfg (when I did python setup.py bdist_rpm). That's when I went to the workaround that resulted in this thread. I hope this clarifies the problem. Not quite. I don't know what the unpackaged files error looks like. Can you try Phillip's suggestion using --root and --record and show us the results? Did you run python setup.py build before python setup.py bdist_rpm? We've often seen problems with the dependency-handling between distutils commands. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Questions regarding the compilation of a non-extension shared library
James Dominy wrote: Hi, I'm have a shared library that I would like to distribute along with my python package, written in pure C. It requires the correct installation of other software at linktime, and at runtime. The package a ctypes wrapper for SUNDIALS (a mathematical C library). You might want to take a look at OOF2. If includes a distutils command for building a shared library. I believe it tries to install the library to $prefix/lib which may not be what you want, but it might help you to find the right incantation to at least build the thing. http://www.ctcms.nist.gov/oof/oof2/index.html Will your code be open source? We have many requests for SUNDIALS wrappers on the scipy list, and it would be great to satisfy them. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Docs re debian distributions for pure Python programs?
Edward Ream wrote: Does anyone know of documentation for creating debian packags for pure Python projects? In particular, I would like to bypass the make-centric build process and the debian/rules file. The official Debian Python Policy document is the best starting point. http://www.debian.org/doc/packaging-manuals/python-policy/ The Wiki seems to have some more information, too. http://wiki.debian.org/DebianPython/NewPolicy You cannot really get around debian/rules, though. The debian-python mailing list would be the place to ask further questions along this line. http://lists.debian.org/debian-python/ -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Parsing setup()
Rob wrote: Ian Bicking wrote: If you have a built egg (which you can do with python setup.py egg_info, without actually installing anything) you can read the requires.txt file with pkg_resources. I had tried that but I was having problems with un-installed packages. I have this so far, which works with *nux, but I'll have to figure out how to find the correct Python executable in Windows (I think using different Python versions could yield different requirements?). pkg_name = pycopia-net os.chdir(setup_dir) os.system(/usr/bin/env python %s egg_info % setup_file) You probably want to use sys.executable (and the subprocess module if you have Python = 2.4): import subprocess import sys subprocess.call([sys.executable, 'setup.py', 'egg_info']) -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] automating __version__' ing issues
Phillip J. Eby wrote: At 04:47 PM 5/15/2007 -0500, Robert Kern wrote: Phillip J. Eby wrote: Reading that page, I'm not sure whether you want a scikits namespace package in the first place. A namespace package is used to group several projects under a single owner, ala the various zope.* and peak.* and plone.* packages. However, nobody but Zope Corp. should be releasing a zope.* package, and nobody but me releasing a peak.* one. The whole point is to claim a namespace for packages being released by a single owner - either an organization or umbrella project. This corresponds to things like the 'org.apache' and 'com.sun' package namespaces in Java - it's just a namespace to allow authors free rein to put whatever modules and packages they like under that namespace. This is exactly what we are aiming for. While different people will be responsbile for each subpackage, they are all operating under the aegis of the scikits infrastructure. But who adds things to the namespace? Is there a single organization that decides what packages or modules will exist in scikits? Yes, the scipy developer team. mlabwrap is just the first scikits project. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] conditionally compiling extensions
Christopher Armstrong wrote: Twisted has a few extension modules that it only wants to build if certain things are available on the platform - like epoll support if the epoll.h header file is available, for example. We can call a function to run some code to decide which extensions to build, but in certain cases having access to the compiler object in distutils would be preferable -- to test certain things about the platform. Currently we have some fairly heinous hacks to get our code to run *after* that compiler object is available (i.e., after it's constructed in build_ext.run), but these hacks are becoming problematic especially for easy_install support. Before showing what these hacks are (if you really care, you can check the various setup.py files in the Twisted repository), I'd like to ask if there is any other best-practice knowledge in the community for conditionally building extensions based on knowledge only available by using a compiler. We've been doing something along these lines in the numpy project. We've extended the distutils command config to allow building and executing small C programs to test for platform support of certain C functions. Looking at the code right now ... it's possibly at the same level of heinous hackishness that yours are, but it works for us. We use it to build a config.h file using the build_src command which we added. build_src allows us to pass a callable as an item in an Extension's sources list, and the callable will generate a file. Inside the function that generates config.h, we grab the config command, call .ensure_finalized() on it and then call the methods that we added to try compiling, linking, and/or running the test files. By the time the functions are called in build_src, the config command exists and is ready to be used. You might be able to compile the extensions unconditionally, but through the use of the #defines in config.h and #ifdefs in the extension module only compile an empty module if the feature were unsupported. That would turn the try: import _c_accelerator func = _c_accelerator.func except ImportError: pass into import _c_accelerator if hasattr(_c_accelerator, 'func'): func = _c_accelerator.func I leave it to you to decide which is better, but I find try: except ImportError: to be too fragile. I spent a good amount of time at a customer's site trying to figure out why Cheetah was unbearably slow behind Apache, but worked like blazes when we ran the script manually on the same box. It turns out that the permissions on the accelerator extension module were incorrect for mod_python to import them. The setup scripts for Marc-André Lemburg's mx extensions seem to define an mx_autoconf distutils command that might be useful to look at, but I don't know the details about how he uses it. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] numpy build fails in distutils code
Jay Parlar wrote: I've tried asking this on the numpy list (a few times), but unfortunately no response. Since the error *appears* to be inside distutils, I thought maybe someone here would recognize it. No, it's not a problem in distutils. Please go back to the numpy list where I will respond to your message. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Some negative press for easy_install
Andrew Straw wrote:
Note that matplotlib tried essentially this for a while, but apparently
some folks really didn't like it. I'm not sure what exactly broke on
their systems (they didn't complain to the mailing list), but when
setup.py reverted to a plain distutils script, they cheered.
That's right, they complained to me, instead. :-)
The issue was that they had setuptools on their system because they were
building eggs for another project, but specifically *didn't* want to install
matplotlib as an egg.
Matplotlib
now has setupegg.py:
from setuptools import setup
execfile('setup.py',
{'additional_params' :
{'namespace_packages' : ['matplotlib.toolkits']}})
I guess the point is that just because someone has setuptools installed
on their system doesn't mean they want to use it.
Another option, for a single setup.py, is something like the following:
if 'setuptools' in sys.modules:
use_setuptools_options()
This way, setup.py can be setuptools-aware without doing 'import
setuptools', but the user would have to do:
python -c import setuptools; execfile('setup.py')
And this is my preferred method for those packages that want to provide the
appropriate information for egg builds but not (yet) *require* setuptools or egg
builds. It means that easy_install somepackage==dev works (a failing of the
setupegg.py approach) and allows python setup.py install to work as standard
distutils does.
--
Robert Kern
[EMAIL PROTECTED]
In the fields of hell where the grass grows high
Are the graves of dreams allowed to die.
-- Richard Harter
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Some negative press for easy_install
Phillip J. Eby wrote: At 01:31 PM 2/9/2006 -0600, Robert Kern wrote: Andrew Straw wrote: Note that matplotlib tried essentially this for a while, but apparently some folks really didn't like it. I'm not sure what exactly broke on their systems (they didn't complain to the mailing list), but when setup.py reverted to a plain distutils script, they cheered. That's right, they complained to me, instead. :-) The issue was that they had setuptools on their system because they were building eggs for another project, but specifically *didn't* want to install matplotlib as an egg. setuptools-based packages can be forced to install the old-fashioned way using: setup.py install --single-version-externally-managed as long as you also specify a --root directory or a --record file. This is of course not upgradeable or uninstallable without help from a packaging tool that can utilize the results of --root or --record. And this particular user did not want that, either. -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Some negative press for easy_install
Phillip J. Eby wrote: At 02:10 PM 2/9/2006 -0600, Robert Kern wrote: Phillip J. Eby wrote: setuptools-based packages can be forced to install the old-fashioned way using: setup.py install --single-version-externally-managed as long as you also specify a --root directory or a --record file. This is of course not upgradeable or uninstallable without help from a packaging tool that can utilize the results of --root or --record. And this particular user did not want that, either. I'm confused. They didn't want to be able to uninstall? Didn't want to point --record to /dev/null, or --root to /? Or something else? Didn't want setuptools involved at all. He's wasted more hours on it than he ever really wanted to in the days before non-root installs were reasonably documented. He just doesn't trust it. He only has setuptools installed at all because I've convinced him to distribute eggs of his project (and that I will help him troubleshoot setuptools issues). -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] static linking in distutils/setuptools?
Stefan Seefeld wrote: Robert Kern wrote: Not really, no. In many cases (e.g., GNU ld), there's simply no way to tell the linker that you prefer static libraries to shared libraries when you are building a shared library like a Python extension. You simply have to make sure that the static libraries are found first. I believe this is wrong. The (GNU ld) man pages mention '-Bdynamic' and '-Bstatic' as a means to instruct the linker to link to a particular library dynamically or statically. (To only link statically to libbar.a you could write e.g. ... -lfoo -Bstatic -lbar -Bdynamic -lbaz etc.) Ah, yes, you are correct. Unfortunately, OS X's ld does not have these options (and is not GNU; I am an idiot). -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] namespace_packages
Charlie Moad wrote: Well here are th cvs links to them. http://cvs.sourceforge.net/viewcvs.py/matplotlib/matplotlib/setup.py?rev=1.143view=auto http://cvs.sourceforge.net/viewcvs.py/matplotlib/toolkits/basemap/setup.py?rev=1.33view=auto The matplotlib.toolkits module does not exist in the mpl cvs and it is not in the setup.py file. I have tried adding both. In basemap however, the matplotlib.toolkits module does exist and is listed in the setup.py file. Both of these setup.py files are buggy. The packages list needs to have entries in dotted form (matplotlib.toolkits), not filesystem form (matplotlib/toolkits). Correcting them, and adding an empty lib/matplotlib/toolkits/__init__.py to the matplotlib checkout allows me to build eggs with namespace_packages set appropriately. -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking on Linux, Mac OS, et al
Jeff Pitman wrote: Are these just extensions that are imported? Because if they are, then you don't need to mess with LD_LIBRARY_PATH and you can put them anywhere on Linux as long as PYTHONPATH is correct. No, they are things like libpng.so that the extension modules use. -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking on Linux, Mac OS, et al
Phillip J. Eby wrote: Hi folks. I just implemented preliminary shared library build support for setuptools, but I ran into a bit of a snag. I was operating under the assumption that you could simply put shared libraries in the directory alongside the extensions that use them, but in practice it turns out that this only works on Windows. On Linux I found that you could only put shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar issue will be found under OS X and other Unixes. On OS X, the headers of the dylib can be rewritten to point to a specific path. IIRC, this can be a relative path. Bob Ippolito's py2app does this kind of rewriting for making relocatable .app bundles. I'm sure he'll chime in soon, so I'm not entirely sure why I bothered replying myself. -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Choice of dependencies?
Phillip J. Eby wrote: At 08:53 PM 12/3/2005 +, John J Lee wrote: Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. Interesting; I'd be curious to know what P, B and C translate to in real packages. :) P = matplotlib B = Numeric C = numarray D = scipy_core Until everyone ports their stuff to scipy_core, there are still people who need to use numarray or Numeric to be compatible with other packages. -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] formencode as .egg in Debian ??
Phillip J. Eby wrote: Note, by the way, that those two things are the only essentials here, as best I can tell, and I've already stated my willingness to change *how* those two things get accomplished. For clarity, I will repeat yet again, in yet another way: 1. Egg-based projects need to install their published metadata, in a well-known location relative to the installation location of their code, so that it can be found by searching sys.path, so that it and other projects can locate the metadata for currently-importable projects, *without* needing to first import the project's code. 2. Egg-based projects need to be able to identify whether another Python project package is installed and what version it is, without requiring modification to that other project's code or needing to import it. (And this is independent of whether the depended-on project was packaged as an egg by its author.) As far as I'm aware, those are the irreducible technical minimum requirements for making an egg-based project work. *How* these requirements are met is quite flexible, as there are already three working layouts that achieve this. As I said before, I'm quite willing to implement a fourth. But nobody has been proposing anything that meets these requirements, because they're too busy trying to prove the requirements don't exist or are somehow not real. [Note: I am a happy Debian user, though not a DD. I am also one of the developers of a Debian-packaged Python package, and we're considering using pkg_resources to implement certain new features. I swear, this is like watching two parents fight. Anyways...] I think one of the sticking points with the Debian developers has been that the .egg-info metadata is being put into /usr/lib/... when according to Debian policy and general UNIX lore, such should be placed somewhere in /usr/share/ Would it be possible to treat /usr/share/pythonX.Y-egginfo/ as a proxy for /usr/lib/pythonX.Y/site-packages/ when searching for .egg-info directories? -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Non-root install for testing
Ian Bicking wrote:
I've created a script to set up a non-root installation of Python,
suitable for testing installation procedures. It follows the
instructions located here:
http://peak.telecommunity.com/DevCenter/EasyInstall#non-root-installation
I really dislike this approach. I'm afraid that I don't understand what
problem it's trying to solve. On OS X, the instructions tell you to
install to ~/Library/Python/2.4/site-packages . Why do the general UNIX
instructions tell you to make a ton of symlinks? IMHO, better advice
would be (thanks in part to Prabhu Ramachandran):
Make a directory to hold your EasyInstall'ed packages (e.g.
~/lib/python2.4/site-packages). easy_install uses .pth tricks.
Therefore, the directory where eggs are installed require that .pth
files work there. On non-system wide installs, this can be achieved
using a sitecustomize.py somewhere inside PYTHONPATH with the
following lines:
import site, os
site.addsitedir(os.path.expanduser('~/lib/python2.4/site-packages'))
easy_install needs to be told to use this directory and that it is
okay to use. Add the following options to your ~/.pydistutils.cfg
file:
[install]
install_lib=~/lib/python$py_version_short/site-packages
install_scripts=~/bin
[easy_install]
site_dirs=~/lib/python$py_version_short/site-packages
At this point in time, I think this approach is cleaner. Eventually it
might run into conflicts with a distribution-provided easy_install, but
AFAICT there aren't any of those yet.
In any case, neither of these approaches is going to be tenable in the
long term. Having the non-root user recreate the whole python
distribution via symlinks isn't a reasonable approach. Controlling the
installation/activation of eggs via .pth files was a cool approach, but
it appears that it's not terribly flexible because of the way that
Python inserts them into sys.path.
A more general way forward might be to have sitecustomize.py (or site.py
if your distro is kind enough to take care of such things for you) read
data files that list the eggs that are activated. sitecustomize.py could
then insert the eggs wherever it sees fit into sys.path. On Debian, for
example, there would be a file in /usr/lib/pythonX.Y/site-packages/ for
eggs that are provided by real dpkg-installed .deb's. There would be
another file in /usr/local/lib/pythonX.Y/site-packages/ for eggs that
are installed by root but aren't real Debian packages. There might be a
final one somewhere canonical in a user's home directory
(~/.easy_install perhaps) for eggs that are local to the user (and
probably installed to ~/lib/pythonX.Y/site-packages/ but not
necessarily). These data files would be searched in reverse order: ~/,
/usr/local/, /usr/. It would be nice if the data files could specify
that certain eggs in the later directories are to be ignored.
Does this sound like a workable idea?
On a similar note: easy_install does a lot of sanity checks to determine
if it ought to install to the given install_lib directory.
Unfortunately, those sanity checks can be too restrictive. For example,
when I use Debian, I manage my /usr/local/ tree with GNU Stow. I want to
put everything managed by easy_install under the prefix
/usr/local/stow/easy_install/ (with subdirectories bin/,
lib/python2.4/site-packages/, and so on). When I run Stow, everything
will be symlinked into /usr/local/. Because Debian was thoughtful,
/usr/local/lib/python2.4/site-packages/ is already on the PYTHONPATH and
is .pth-enabled so everything would work just fine. Unfortunately,
/usr/local/stow/easy_install/lib/python2.4/site-packages/ isn't on the
PYTHONPATH and easy_install will refuse to install there even when it
has been specified as a valid --site-dir.
Paranoia is healthy, but we need a way to override the paranoia. I am
requesting a --force option to override this sanity check. Preferably
there would be one --force option per sanity check and one overall
--force that entails all of the others, but I'll take what I can get.
I'm willing to work on a patch if given a little guidance as to where
all of the sanity checks are.
Thank you for your attention.
--
Robert Kern
[EMAIL PROTECTED]
In the fields of hell where the grass grows high
Are the graves of dreams allowed to die.
-- Richard Harter
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools: site_dirs
Ian Bicking wrote: With easy_install I can specify --site-dirs to make it setup package with .pth files. But with setuptools alone I can't, particularly python setup.py develop. I'm not sure if there's a setting in distutils.cfg I could use...? There doesn't seem to be any appropriate command-line option. I'm also not sure how the site directories are identified...? In this case, on Debian, /usr/local/lib/pythonX.Y/site-packages isn't recognized as such a directory, even after I added it with site.addsitedir in sitecustomize. On Debian that directory is added to the list of site directories in site.addsitepackages(), which is automatically executed through site.main(). If it's not working on your machine, then I think it's a problem with your particular configuration. -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils' script and GUI app handling
Kevin Dangoor wrote: On 9/15/05, Phillip J. Eby [EMAIL PROTECTED] wrote: As for Mac OS, I have almost no experience with it, so I'm not sure what GUI applications there need. Does everything need py2app? If you have a wx-based app, would you just make a #! script? Bob Ippolito previously mentioned that you don't install applications there, that people just drag applications wherever they want them rather than using shortcuts, so at least that part isn't a problem. :) GUI apps on the Mac need py2app. It depends. wx-based programs don't have a technical need to be in a .app bundle. They will run fine from the command line. Now, the *user* might want it packaged in a .app bundle, and we as developers might need to respond to that desire. I think that PyObjC apps might actually need to be in a .app bundle to work reliably, though. I have a feeling that there isn't a pressing need to be able to install .app bundles from easy_install. If the package maintainer is going to go to the trouble of writing a setup.py that can be used with py2app, he'll almost certainly actually build the .app binary and distribute it. Since they're standalone, they don't really interact with the other installed packages. -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving distutils' script and GUI app handling
Phillip J. Eby wrote: At 11:10 AM 9/15/2005 -0700, Robert Kern wrote: I have a feeling that there isn't a pressing need to be able to install .app bundles from easy_install. If the package maintainer is going to go to the trouble of writing a setup.py that can be used with py2app, he'll almost certainly actually build the .app binary and distribute it. Since they're standalone, they don't really interact with the other installed packages. If you can run a wx or tkinter (or other cross-platform GUI) app on Mac OS without needing anything special, but users would prefer to have the .app, it would probably be a good idea to have it, even if the original developer hasn't targeted that platform specifically. For example, packages like idle or the pydoc tkinter GUI would seem to merit something like this. Oh, certainly it would be nice. But AFAICT, you really do have to write a setup.py specifically for py2app. The information you need isn't something you can simply divine from a generic setup.py . -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] question about setuptools
Phillip J. Eby wrote:
At 11:27 PM 9/13/2005 -0700, Robert Kern wrote:
Index: setuptools/command/easy_install.py
=== RCS
file:
/cvsroot/python/python/nondist/sandbox/setuptools/setuptools/command/easy_install.py,v
retrieving revision 1.28 diff -u -r1.28 easy_install.py ---
setuptools/command/easy_install.py 3 Sep 2005 04:50:05 - 1.28
+++ setuptools/command/easy_install.py 14 Sep 2005 06:17:28 - @@
-141,6 +141,10 @@ self.set_undefined_options('install',
('record', 'record')) normpath = map(normalize_path,
sys.path) self.all_site_dirs = get_site_dirs()
+install_cmd = self.distribution.get_command_obj('install')
+if install_cmd.prefix is not None:
+
self.all_site_dirs.append(normalize_path(install_cmd.install_lib))
+ if self.site_dirs is not None: site_dirs =
[ os.path.expanduser(s.strip()) for s in
self.site_dirs.split(','___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Your patch came through horribly mangled, as you can see. You probably
need to make it an attachment rather than pasting it into your email.
Odd. It *was* sent as an attachment. Thunderbird on OS X sees it fine
through GMane. Maybe GMane mangled it somewhere along the line. Or
perhaps line-endings are getting confused. I'm cc'ing this to you with
an attached patch that uses CRLF line endings.
--
Robert Kern
[EMAIL PROTECTED]
In the fields of hell where the grass grows high
Are the graves of dreams allowed to die.
-- Richard Harter
Index: setuptools/command/easy_install.py
===
RCS file:
/cvsroot/python/python/nondist/sandbox/setuptools/setuptools/command/easy_install.py,v
retrieving revision 1.28
diff -u -r1.28 easy_install.py
--- setuptools/command/easy_install.py 3 Sep 2005 04:50:05 - 1.28
+++ setuptools/command/easy_install.py 14 Sep 2005 06:17:28 -
@@ -141,6 +141,10 @@
self.set_undefined_options('install', ('record', 'record'))
normpath = map(normalize_path, sys.path)
self.all_site_dirs = get_site_dirs()
+install_cmd = self.distribution.get_command_obj('install')
+if install_cmd.prefix is not None:
+self.all_site_dirs.append(normalize_path(install_cmd.install_lib))
+
if self.site_dirs is not None:
site_dirs = [
os.path.expanduser(s.strip()) for s in
self.site_dirs.split(',')
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] question about setuptools
Phillip J. Eby wrote: This code will break on other people's installations, in that it will not be able to tell you're installing to a non-site directory unless you explicitly provide an --install-dir on the command line. It will thus create a non-working easy-install.pth, and will fail to warn the user that 'require()' is needed. I know. I said it was not satisfactory and that the real tricky part wasn't calculating the path but determining whether --prefix was explicitly provided or not. Ooh! But if install_cmd.prefix != sys.prefix, then --prefix must have been explicitly set by the user! Eureka! Now I just have to figure out a way to find out if --install-dir wasn't set -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding egg support (Resource API) to existing packages
Phillip J. Eby wrote: So give them a patch that's egg-safe *without* using pkg_resources! It'll be slightly more complicated to implement, but won't introduce *any* dependencies. But first see if they'll drink the Kool Aid. Only if they really hate the idea of depending on setuptools should you bother with this strategy. -- Robert Kern [EMAIL PROTECTED] In the fields of hell where the grass grows high Are the graves of dreams allowed to die. -- Richard Harter ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
