Re: [Distutils] [Fwd: Re: Annoucing distribute project]
Phillip J. Eby wrote: the domain knowledge embedded in the distutils are of fairly limited scope and kind: * Extension building, compile/link options and defines * Wildly-differing installation path schemes across platforms * Platform distribution formats like bdist_rpm, bdist_wininst, and bdist_msi Seems to me that if there were a well-defined API for plugging in platform-dependent modules, it shouldn't be too hard to find people willing to contribute modules for the platforms they're familiar with. It's not really that the knowledge is particularly arcane, just that there's no single person who knows it all. This makes it very unlikely that one perso could come up with an entire viable distutils replacement just by scratching their own itches. We need a communal itch-scratching session. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
Guido van Rossum wrote: You're not building a brand new bridge where there was no bridge before. You're trying to upgrade an existing bridge that people use for their daily commute. But that hasn't been decided. Whether to upgrade the existing bridge or build a new one is precisely what we're debating. And it seems to me that the easiest way to avoid disrupting existing traffic *is* to build a new bridge alongside and then scrap the old one. If the existing bridge isn't too badly run down and is basically still the kind of bridge you want, upgrading it may make sense. But that doesn't seem to be the case with the distutils bridge. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
Guido van Rossum wrote: Yeah, but *is* dropping backward compatibility an option here? I'm not sure the concept of backward compatibility makes sense here. The only kind of distutils replacement I would be interested in would have a completely different API, completely different structure and completely different implementation. Anything less would fail to fix the problems we want to fix. Given that, what would it even *mean* for the new tool to be backward compatible with distutils? The same might be happen to distutils 2 in a few years. You are going to have to plan making it more maintainable. A modular structure with an easy-to-follow implementation would certainly be part of the plan, I hope. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [pyconuk] "Python Package Management Sucks"
Floris Bruynooghe <[EMAIL PROTECTED]> writes: > Hmm, I've never read the packaging sections of the LSB (and maybe I > should before making a comment like this) http://refspecs.linux-foundation.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/packagefmt.html> > but I've always heard complaints that it is very RPM-centric and > ignores DEBs etc. The LSB core specification does recommend (and specify the format of) RPM packages for distribution; but it doesn't require the system to use RPMs natively, only that such packages should be installable. Note: Supplying an RPM format package is encouraged because it makes systems easier to manage. This specification does not require the implementation to use RPM as the package manager; it only specifies the format of the package file. http://refspecs.linux-foundation.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/swinstall.html> So, a Debian system is LSB-compliant if it enables the installation of externally-produced LSB-compliant RPM packages. This is easily done by making an appropriate tool available; I think 'alien' is the one used on a Debian system. The LSB folks have (IMHO as an outside observer) worked quite well with the Debian project and vice versa. A Debian system now has various 'lsb-*' packages available that make it simple to bring a Debian system into compliance with the various specifications of the LSB (note: I don't know what gaps in compliance may remain even with those packages installed). -- \ “Everything is futile.” —Marvin of Borg | `\ | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [pyconuk] "Python Package Management Sucks"
On Fri, Sep 26, 2008 at 10:33:22PM +0200, Nicolas Chauvat wrote: > On Tue, Sep 23, 2008 at 11:37:53AM +0100, Chris Withers wrote: > > - all the package management systems behave differently and expect > > packages to be set up differently for them > > But there is the Linux Standard Base to bind them. Hmm, I've never read the packaging sections of the LSB (and maybe I should before making a comment like this), but I've always heard complaints that it is very RPM-centric and ignores DEBs etc. So maybe even the LSB is not quite up to the task of binding them all yet... Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [pyconuk] "just use debian"
Hi, [sorry for slowly drifting away from pure distutils-related topics] On Tue, Sep 23, 2008 at 11:47:03AM +0100, Chris Withers wrote: >> My main tool is Python, but I have many other tools on my system. I >> do not want to have as many package management utilities as >> "subsystems". > > Then I suggest you volunteer to maintain the debian packages for every > single python package. Do you really think every single Python package in PyPI deserves to be packaged for every distribution? I don't. How do I make a difference? When I need something I download it. When I find it really useful and plan on using it I package it. Many others are behaving in the same way and the result is "apt-cache search python". > If you have projects this large, then you likely want to roll your own > OS packages anyway. I am not sure what you mean by "OS packages". Do you mean "roll your own distribution" as in "roll your own thing based on Debian and adapt packages to your needs as Ubuntu is doing"? >> [Please note that for an experienced Debian developer, making the >> initial package of a Python module can be a matter of half an hour to >> a couple hours and releasing a new version a matter of minutes.] > > ...and for someone not using Debian or not an experienced Debian > developer? Despite being a fan of Debian, I'm well aware of just how > "friendly" a community it can be to the new user... Do you expect someone who is not proficient at programming to quickly design and develop a piece of software? What makes you think it would be different for integrating pieces of software into a consistent system? As I said on the pyconuk list, packaging software requires some work and currently there is no way around it. Tools get better over time, but automation is out of reach. As usual "user != developer". For someone not using Debian: just be happy with whatever tool you choose to use. For someone not an experienced Debian developer: just wait for someone to do the work you want to benefit from, or learn to do it yourself and get it done. In my company's case, we worked for years on setting up an efficient environment for software development and system administration and we are now able to move python code from a mercurial repository to a production system running Debian in a matter of minutes. The tools we built to support this process have been free software from the very beginning and can be found on our website http://www.logilab.org/project/logilab-devtools -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [pyconuk] "just use debian"
Hi Stephen, On Tue, Sep 23, 2008 at 11:59:18AM +0100, Stephen Pascoe wrote: > I'm interested in hearing what you find so annoying about the egg format > because for me it's the one part of the setuptools system that I would keep. I suppose the best answer is to point you to this thread: http://teams.debian.net/lurker/message/20070904.152810.4f84c924.fr.html There is also useful information at: http://wiki.debian.org/DebianPythonFAQ and http://www.debian.org/doc/packaging-manuals/python-policy/ but I would not bet on the latter being up to date. Baseline is "no problem with providing egg-info metadata, but pretty please Python developers, do not code *for* distutils/setuptools/etc. Just find a way to provide useful dependency/meta information then let your users choose how they install your code on *their* system". -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [pyconuk] "Python Package Management Sucks"
Hi Tarek, On Tue, Sep 23, 2008 at 01:38:00PM +0200, Tarek Ziadé wrote: > I just wanted to react on that: I have created a setuptools-enabled package > for Pylint (logilab.pylintinstaller, see http://pypi.python.org/pypi/ > logilab.pylintinstaller) with the bless of Logilab, so people under Windows > could use the tool with a one-liner installation process... Thanks again for doing it. We provide Debian packages for the free software we make and we will support anyone wanting to package our free software for other targets than Debian. -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [pyconuk] "Python Package Management Sucks"
On Tue, Sep 23, 2008 at 11:37:53AM +0100, Chris Withers wrote: >> Install debian and get back to productive tasks. > > This is an almost troll-like answer. True. But it worked, didn't it? Hopefully the thread will end up with some positive output. :) > See page 35 of the presentation. > > If you're too lazy, here's a re-hash: > > - there are lots of different operating systems Agreed. > - even with Linux, there are many different package management systems I would not link package management systems that strongly to operating systems (or kernels): http://www.debian.org/ports/index#nonlinux > - all the package management systems behave differently and expect > packages to be set up differently for them But there is the Linux Standard Base to bind them. > - expecting package developers to shoulder this burden is unfair and > results in various bitrotting repackaged versions of python packages > rather than just one up-to-date version maintained by the entity > originating the python package PyPI says it holds 4818 packages. I doubt all of these are actually worth being packaged. Please note that my point of view is not the one of the developer or single-user who wants to try out something easily and quickly, but the one of the developer that has to deploy his software many times in many places and maintain them in production for a long time. I understand easy_install and similar tools make it easy to try something out in one's home directory and I have nothing to say against that. I complain that some advocate this as the One True Way to distribute their code and end up with code that depends on it, thus complicating matters for the ones who have been happily using tools that manage entire systems for years. > - Adobe Photoshop Plugins, Firefox Add-ons, etc do not delegate their > management to an OS package manager. Packages are Python's "plugins" and > so should get the same type of consistent, cross-platform package > management targetted at the application in question, which is Python in > this case. I strongly disagree with this. I guess this is why you may like a Python-specific package management system whereas I never will. To me, there is no such thing as a clear boundary between a "Python subsystem" and the rest of a computing system. A have only one (complex) system, in my case Debian, and want only one tool to manage it. -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
Phillip J. Eby wrote: In the case of the distutils, the people who are capable of extending it are tired of doing so, and the people who have the energy and time are very unlikely to be able to work on it much without breaking something significant. Distutils is also far too flexible in some areas to be able to improve much while maintaining 100% backward compatibility -- it doesn't enforce One Obvious Way To Do It. Basically volunteer projects like this just won't move forward when doing so is painful -- I think distutils is like this, and though you suffered through that with the development of setuptools I think part of this recent discussion is that it's still not really self-sustaining. So I think we need a bit higher bar of code quality (and brevity) than in a commercial project. What's more, there are very few people who've even said they like the distutils API or think it's a good fit for the application area. And, frankly, the domain knowledge embedded in the distutils are of fairly limited scope and kind: * Extension building, compile/link options and defines * Wildly-differing installation path schemes across platforms * Platform distribution formats like bdist_rpm, bdist_wininst, and bdist_msi Most other things the distutils do are either well-specified, obsolete (e.g. its internal logging and option parsing libs), or probably not worth keeping (e.g. bdist_dumb). Yes -- what distutils really does just isn't that interesting or complicated. There's a lot of arcane knowledge built into the build process, but it's not well expressed or particularly useful, and for pure-Python packages there really isn't any arcane knowledge to be had. There's other features, but a lot of them aren't well enough implemented to be reliable, or they are adapted in eclectic ways by individual packages in a way that makes them unpredictable. The arcane knowledge is distutils is broken and unusable -- I don't think distutils reform would actually save much that was useful. > Maybe a "distutils 2" project could start outside Python, using distutils > and setuptools code as > legacy infrastructure, and taking back pieces of it, > > Then it could be re-integrated in Python as a replacement for distutils when > it is mature ? Only if much effort and study went into the planning of this re-integration. That's why at least some of the discussion has been around requirements gathering and PEP-writing as a first step. There is a much better sense of what the scope of distutils should be now than when it was written, and the PEP process could make that clarity explicit. I suspect that when the scope is made clear that the implementation just won't be that complex. My own inclination is that a scalable future for distutils means an improved sdist format, the end of setup.py as an command-line interface, and community-maintained platform-specific installation tools that process source or binary distributions. Most complaints about distutils (and setuptools, for that matter) are focused on installation policy&preference issues. Making it possible and practical for a variety of tools to flourish around a standardized format (ala WSGI) seems like the way to go. I'd like to see an improvement of sdist, toward a more declarative and introspectable system. Just including EGG-INFO in sdist would be a big help, and allow the easy reading of the egg metadata without having to deal with the binary aspects of .egg (which remain very challenging to use reliably). In pyinstall I'm unpacking and running egg_info to get this data, but that's very awkward and that metadata should really just be there without having to invoke any code. I'm not sure I agree with removing setup.py, though I dislike how it functions currently -- that you kind of have to poke it from the side to get basic information out of it, instead of something more declarative. In some ways it's the distutils/setuptools interface that most people actually use -- pyinstall for instance only calls setup.py in subprocesses to install the package, and never touches it directly. So anything that acts fairly similar to the current setup.py's will be compatible. This is part of why I don't think the backward compatibility issue is as difficult as Guido suggests. Though... really if pyinstall could easily detect a new source format and only setup.py with the old source format, it wouldn't be hard to support both. There does need to be room for custom code, but mostly for the build/compilation process. Notice that the existence of eggs has already allowed buildout, virtualenv, and pyinstall to appear. But eggs don't handle installing tests or documentation, and they have to be prebuilt by platform. An improved sdist format, on the other hand, with standardized layout and metadata would address all of those issues. The tools for building this format and APIs for inspecting it would be
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
Guido van Rossum wrote: > > Can you think of a specific way to evolve distutils to provide the API > you would like to see, without (for the time being) losing backwards > compatibility? > Here is what I did with numscons: numscons is called through distutils scons command, which is called during the build dance. By default, the scons command does nothing, but you add scons scripts through a distutils command, and in that case only scons does something. Those hooks do not break anything in numpy.distutils. The best proof is that the default build system is still numpy.distutils, and numscons is not in numpy sources (only the hooks are). Some people build with numscons, other - the majority - use the current system. Same source tree, same setup.py files (except for C extensions of course; they have to be synchronised, but the common part between the two build could easily be factorized in a common_setup I think). That's the idea I was suggesting and may be applicable to python as a whole: - design a small self contained build system, which does nothing but is extensible (there is a lot of prior art: vellum, rake/rant, etc...), and "hook" it into distutils through a new distutils command. People smarter than me will enjoy this part, and will hopefully come up with something nice. - The new command would not nothing if you don't use the new features: you don't break any existing setup.py - Another thing would be to gradually move all the configuration from code into a separate module in a sane way: putting the flags in config files, not in the code, make an API to discover and reuse those settings, etc... Basically, all the arcane stuff. This would be independant of distutils, but would be callable from distutils (by the new command and the current distutils). A good way to design the API here would be to make sure it work in distutils and in e.g. scons (so a python extension builder in scons could be easily done importing this module only: all the compiler specific flags, etc...). It would be a distutils.sysconfig on steroids, which works for every platform - including windows with VS. So that every other python build tool, being vellum, scons, whatever, could access those. I believe this can be made general enough to be usable by almost any tool. At least I started something toward this direction for scons, and if it works for scons, it should work for a lot of tools. That would enable refactoring something like 1/3 of current distutils code, and provide a better build tool for people who use the new 'stuff'. I realize this is really vague at that point, but up to today, I never thought about what could be applied to python in general from my work numscons. If you think the above makes any sense, I will think more about this, cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
Phillip J. Eby wrote: > The economic factors are a bit different, here. Joel himself has > previously pointed out that where Netscape failed, Mozilla won - i.e., > the economics of open source can mean that it's sometimes easier to > get volunteers for a new project than for fixing an old one, or at > least for a project where dropping backward compatibility is allowed > (e.g. Py3K). Yes, I forgot to mention that earlier: I don't think Joel's point is applicable to distutils. Distutils is by almost all acount a small project (sloccount tells me Python-2.5.2/Lib/distutils is 9241 loc); one of Joel's point is that it is more costly to write from scratch than to refactor. But I think distutils is not "refactorable" (sorry for hurting native English speakers): a lot of things are fundamentally broken *by design* (option handling, running one command after the other, etc...), and cannot be changed in a backward compatible way. It is also small enough so that a single person can have a pretty good idea of the whole design. So it maybe less costly to do something from scratch, simply because starting from scratch would brings much more resources. For example, some people from OS distributors complain, rightfully, about distutils: my understanding is that they would put ressources for code which would (optionally) make python packages "OS compliant" (FHS compliant on linux, etc...). But not in the current state of affairs. Joel's point is valid when you have a fixed amount of resources, e.g. a company; in open source, the dynamics can be different [1] > What's more, there are very few people who've even said they like the > distutils API or think it's a good fit for the application area. And, > frankly, the domain knowledge embedded in the distutils are of fairly > limited scope and kind: > > * Extension building, compile/link options and defines This can be done without too much work: the hard work really is the options and the corner cases. Corner cases can be solved gradually (and there are not that many corner-cases; mostly VS stuff and some mac os X stuff in my experience). I have done it for numscons (scons did made this a bit easier, but I needed to read and understand almost all the distutils and numpy.distutils code to build C code on all supported platforms, including Visual Studio). I would be glad to help there, that's a part I now feel relatively confident with. > * Wildly-differing installation path schemes across platforms I did not touch this part too much. Would it be possible to refactor internally this part in distutils, such as we can provide an alternative implementation, which is not backward compatible, but would be specified from a set of requirements (FHS compliant option, etc...). By default, setup.py users would get the current default, but if they want, they would get a better model here. cheers, David [1] Again, talking about numpy/scipy: numpy.distutils itself was ~ 1 loc, and it was much easier for me to start from scratch (but using numpy.distutils knowledge) than to solve numpy.distutils problems. At least in the case of numpy, people smarter than me tried to improve numpy.distutils, but never did it because everytime they did something, they broke something else. Now, it may just be that I have way more free-time than them. But the point remain that in open source, the dynamics can be different. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
On Fri, Sep 26, 2008 at 8:47 AM, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 03:24 PM 9/25/2008 -0700, Guido van Rossum wrote: >> >> > Right now there's a momentum in the community, including framework >> > gurus, >> > that are >> > willing to work on a new distutils package. They are not core developers >> > but >> > they are really >> > good in distribution matters. Even Phillip Eby said that starting a new >> > distutils could be a good >> > pick in this thread earlier. >> >> I wasn't there. I'd like to refer to a post by Joel Spolsky about the >> problem with total rewrites: >> http://www.joelonsoftware.com/articles/fog69.html > > The economic factors are a bit different, here. Joel himself has previously > pointed out that where Netscape failed, Mozilla won - i.e., the economics of > open source can mean that it's sometimes easier to get volunteers for a new > project than for fixing an old one, or at least for a project where dropping > backward compatibility is allowed (e.g. Py3K). Yeah, but *is* dropping backward compatibility an option here? > In the case of the distutils, the people who are capable of extending it are > tired of doing so, and the people who have the energy and time are very > unlikely to be able to work on it much without breaking something > significant. The same might be happen to distutils 2 in a few years. You are going to have to plan making it more maintainable. It might also help if there was a team of people working on the replacement instead of a single one -- reduces the risk of that one person getting tired or engaged or whatever. > Distutils is also far too flexible in some areas to be able to > improve much while maintaining 100% backward compatibility -- it doesn't > enforce One Obvious Way To Do It. That's a good criticism, and can be dealt with through careful deprecation. > What's more, there are very few people who've even said they like the > distutils API or think it's a good fit for the application area. And, > frankly, the domain knowledge embedded in the distutils are of fairly > limited scope and kind: > > * Extension building, compile/link options and defines > * Wildly-differing installation path schemes across platforms > * Platform distribution formats like bdist_rpm, bdist_wininst, and bdist_msi So maybe this part is worth salvaging? > Most other things the distutils do are either well-specified, obsolete (e.g. > its internal logging and option parsing libs), or probably not worth keeping > (e.g. bdist_dumb). It's fine to gradually replace these with better stuff. Deprecated backward compatible APIs can be offered for a few Python releases. >> > Maybe a "distutils 2" project could start outside Python, using >> > distutils >> > and setuptools code as >> > legacy infrastructure, and taking back pieces of it, >> > >> > Then it could be re-integrated in Python as a replacement for distutils >> > when >> > it is mature ? >> >> Only if much effort and study went into the planning of this >> re-integration. > > That's why at least some of the discussion has been around requirements > gathering and PEP-writing as a first step. That's good to hear. > My own inclination is that a scalable future for distutils means an improved > sdist format, the end of setup.py as an command-line interface, and > community-maintained platform-specific installation tools that process > source or binary distributions. Most complaints about distutils (and > setuptools, for that matter) are focused on installation policy&preference > issues. Making it possible and practical for a variety of tools to flourish > around a standardized format (ala WSGI) seems like the way to go. Given the success of WSGI (which I use every day) this sounds like a very good plan! > Notice that the existence of eggs has already allowed buildout, virtualenv, > and pyinstall to appear. But eggs don't handle installing tests or > documentation, and they have to be prebuilt by platform. An improved sdist > format, on the other hand, with standardized layout and metadata would > address all of those issues. Beware however of too large a scope -- the project may crumble under its ambition, or be mired in border conflicts with language-independent distribution management tools like RPM or the Debian/Ubuntu tools. > The tools for building this format and APIs for inspecting it would be > candidates for the stdlib, in much the same way that wsgiref was - the spec > is/should be stable, and those parts that are compiler/install-layout > specific will need to be maintained in the stdlib anyway, for Python's own > build infrastructure. Good thinking. This is also similar to the conclusion we came to with respect to NumPy -- NumPy itself is a separate package and will always remain so, but core Python contains some things that make this possible. > In that sense, "distutils 2" would not be so much a rewrite of the > distutils, as the separation of them into tools for distributing,
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
On Fri, Sep 26, 2008 at 5:47 PM, Phillip J. Eby <[EMAIL PROTECTED]>wrote: > > My own inclination is that a scalable future for distutils means an > improved sdist format, the end of setup.py as an command-line interface, and > community-maintained platform-specific installation tools that process > source or binary distributions. Most complaints about distutils (and > setuptools, for that matter) are focused on installation policy&preference > issues. Making it possible and practical for a variety of tools to flourish > around a standardized format (ala WSGI) seems like the way to go. > [cut] > In that sense, "distutils 2" would not be so much a rewrite of the > distutils, as the separation of them into tools for distributing, and tools > for installing, where some of the tools for installing may be > community-maintained. > There's something we could probably take out of distutils really easily : the register and upload commands. Those commands work together with a PyPI-like server, and deal with a .pypirc file. This code is really simple to take off distutils. What about putting it in a separate package or project ? (maybe "releaser" or something) We could have a separate project that deals with this idea of registering uploading a package to PyPI or elsewhere, with a set of scripts distributed with Python maybe. I can think of many features à la CPAN, to query PyPI repositories through XML-RPC, like ppmtools, so users can upload but also browse what is available. This is probably a project where pyinstaller and/or easy_installer could live as well: a set of tools to browse, install, register and upload packages. Now maybe this is too ambitious, but at least, separate register and upload from distutils is definitely doable and not too risky. I could defenitely write a PEP on that particular part if it sounds good. Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
At 11:47 AM 9/26/2008 -0400, Phillip J. Eby wrote: and those parts that are compiler/install-layout specific will need to be maintained in the stdlib anyway, for Python's own build infrastructure. I just realized the above is not too clear: what I mean is that anything that's specific to *Python's* installation layout on a given platform should be in the stdlib. (Not the things that depend on how vendors want libraries to be installed or how users want to install them; that should be left to community-supplied installation tools, except for a single, simple "standard" install tool, analagous to wsgiref's SimpleHTTPServer -- able to do the basic stuff, but if you want fancy, you go roll your own or download someone else's.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
At 03:24 PM 9/25/2008 -0700, Guido van Rossum wrote: > Right now there's a momentum in the community, including framework gurus, > that are > willing to work on a new distutils package. They are not core developers but > they are really > good in distribution matters. Even Phillip Eby said that starting a new > distutils could be a good > pick in this thread earlier. I wasn't there. I'd like to refer to a post by Joel Spolsky about the problem with total rewrites: http://www.joelonsoftware.com/articles/fog69.html The economic factors are a bit different, here. Joel himself has previously pointed out that where Netscape failed, Mozilla won - i.e., the economics of open source can mean that it's sometimes easier to get volunteers for a new project than for fixing an old one, or at least for a project where dropping backward compatibility is allowed (e.g. Py3K). In the case of the distutils, the people who are capable of extending it are tired of doing so, and the people who have the energy and time are very unlikely to be able to work on it much without breaking something significant. Distutils is also far too flexible in some areas to be able to improve much while maintaining 100% backward compatibility -- it doesn't enforce One Obvious Way To Do It. What's more, there are very few people who've even said they like the distutils API or think it's a good fit for the application area. And, frankly, the domain knowledge embedded in the distutils are of fairly limited scope and kind: * Extension building, compile/link options and defines * Wildly-differing installation path schemes across platforms * Platform distribution formats like bdist_rpm, bdist_wininst, and bdist_msi Most other things the distutils do are either well-specified, obsolete (e.g. its internal logging and option parsing libs), or probably not worth keeping (e.g. bdist_dumb). > Maybe a "distutils 2" project could start outside Python, using distutils > and setuptools code as > legacy infrastructure, and taking back pieces of it, > > Then it could be re-integrated in Python as a replacement for distutils when > it is mature ? Only if much effort and study went into the planning of this re-integration. That's why at least some of the discussion has been around requirements gathering and PEP-writing as a first step. My own inclination is that a scalable future for distutils means an improved sdist format, the end of setup.py as an command-line interface, and community-maintained platform-specific installation tools that process source or binary distributions. Most complaints about distutils (and setuptools, for that matter) are focused on installation policy&preference issues. Making it possible and practical for a variety of tools to flourish around a standardized format (ala WSGI) seems like the way to go. Notice that the existence of eggs has already allowed buildout, virtualenv, and pyinstall to appear. But eggs don't handle installing tests or documentation, and they have to be prebuilt by platform. An improved sdist format, on the other hand, with standardized layout and metadata would address all of those issues. The tools for building this format and APIs for inspecting it would be candidates for the stdlib, in much the same way that wsgiref was - the spec is/should be stable, and those parts that are compiler/install-layout specific will need to be maintained in the stdlib anyway, for Python's own build infrastructure. In that sense, "distutils 2" would not be so much a rewrite of the distutils, as the separation of them into tools for distributing, and tools for installing, where some of the tools for installing may be community-maintained. That's the general idea, anyway. (I'm going to be away from email for a few days, so I'll probably be out of this thread 'till Tuesday.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
On Thu, Sep 25, 2008 at 7:53 PM, David Cournapeau <[EMAIL PROTECTED]> wrote: > Guido van Rossum wrote: >> I don't think anyone loves distutils, or even likes it. Unfortunately >> replacing it will be very tough, because distutils has arcane but >> important knowledge built into it about many platforms, and a new >> from-scratch system is unlikely to reach that level of coverage for >> years. > > Well, yes and no. Yes, there is some arcane knowledge in distutils, but > the information is already there. Incidentally, a problem with distutils > is that it is very difficult to reuse this knowlege in a programmatic > way. For example, my first 'real' contact with distutils was to try > extending it to build ctypes extensions (that is pure C dynamically > loaded libraries loadable from ctypes). It should have been easy, since > from a compile/build POV, that's the same than a C python extension. > Unfortunately, the options are buried into the code, and worse, their > location depend on the used platform/compiler (for example, > distutils.sysconfig is not usable with Visual Studio). At the end, I > gave up, doing it in a cross-platform way without copying all the > knowledge from distutils was too difficult (for me at least). > > I was fed up with distutils complexity in numpy/scipy, so I started > using scons within distutils to build all compiled code in numpy/scipy. > I spent more time dealing with distutils idiosyncraties than writing the > core new system. I am certainly not advocating using scons, but the fact > is that in a couple of months, I, a not that knowledgable person about > python and cross-platforms issues at that point, manage to build numpy > and scipy without using distutils *at all* for C/C++/Fortran extensions. > And numpy is certainly far from trivial to build (numpy distutils > extensions is ~ 10 kloc), and deal with quite a variety of compilers and > platforms. > > Maybe there is something to be learnt here: we could have a new small > subproject to deal with building extensions, and plug it into distutils > (e.g. have an extremely simplified scons; scons is overkill for the vast > majority of python packages, and certainly not integrable in python > core; several small projects in that space already exist). Something > like rake; the arcane knowledge (for compiled extensions at least) could > be moved gradually from distutils to this submodule. Would that fit into > your vision or not ? Can you think of a specific way to evolve distutils to provide the API you would like to see, without (for the time being) losing backwards compatibility? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
Uwe Schmitt wrote: > Hi, > > is anybody aware of numscons ? > http://conference.scipy.org/static/wiki/numscons.pdf At least I am, since I am the main - and sole author at this point - of numscons :) Numscons was created to solve a specific problem: distutils is a nightmare for extensions which need to control the compilation process in a fine-grained manner. But numpy/scipy is quite special, and numscons would be overkill for most python packages. And it does not solve at all the problems people are talking about here (that is mostly deployment issues). The reasons why numscons was created are the same as why people want to get rid of distutils, though, that is distutils is very inflexible if you have special needs (and numpy/scipy needs are special; how many python packages need fortran ?), which is why I think it is somewhat relevant to this discussion. Numscons does show some points which are worth being pointed out IMHO: - it is possible to add a new command which handle a pretty big portion of distutils (the arcane part Guido was talking about I believe) in a sane manner, and it could be done gradually (scons is called from distutils, and scons is a distutils command: you do python setup.py scons to build your extensions, which means it is transparent for users at least). - It took me a few weeks to get simple python extensions built on most platforms (of which a big portion was to understand distutils and numpy.distutils). Of course, it would take more time to deal with all platforms python supports (more exactly all the corner cases), but I don't think it is that difficult. - A "miniscons" would be needed, but there are already many projects which do that, indicating that it should be relatively easy. The first rake prototype was 100 lines I believe; vellum is 300 lines. Something like vellum would be enough for most needs I believe. Of course, doing it for python itself means some issues which were non-existent in numpy's case become big problems (bootstrapping issues, for once). cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] [Fwd: Re: Annoucing distribute project]
Hi, is anybody aware of numscons ? http://conference.scipy.org/static/wiki/numscons.pdf I thinke they have some good ideas, maybe those ideas help. Greetings, Uwe -- Dr. rer. nat. Uwe Schmitt F&E Mathematik mineway GmbH Science Park 2 D-66123 Saarbrücken Telefon: +49 (0)681 8390 5334 Telefax: +49 (0)681 830 4376 [EMAIL PROTECTED] www.mineway.de Geschäftsführung: Dr.-Ing. Mathias Bauer Amtsgericht Saarbrücken HRB 12339 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
