Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
> The Python sandbox has a setuptools directory. Is this the canonical > location for the code? Yes, it is. > If so, then anybody who has Python commit > privileges can commit to it and help further develop setuptools. They can, but they shouldn't. Nothing should be committed there without pje's approval (in whatever form he choses to give such approval). > If not, why not and what is the sandbox setuptools used for? I think it shouldn't be in sandbox, but toplevel, but that's a minor detail. Maybe I misunderstand the English word "sandbox". Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
At 05:15 PM 3/19/2008 -0500, Jeff Rush wrote: >Phillip J. Eby wrote: > > At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: > >> Are you open to giving certain others patch view/commit privileges to > >> setuptools? > > > > Jim Fulton has such already. I'm open to extending that to others who > > have a good grasp of the subtleties involved. > > > > Truthfully, if we can just get 0.6 put to bed, I could probably open up > > the trunk a lot wider. > >What is needed to put 0.6 to bed? How can we help accelerate this? Get a tracker set up. I'm already in the main Python one, might as well use that. >It certainly is possible for someone to create a parallel packaging moduleset >that uses the existing eggs format and PyPI but without the currently >codebase, and then, once proven to work, lobby for it as distutils 3000. Yep. And I believe that something will look rather more like zc.buildout than setuptools, actually. Specifically in being data-driven rather than script-driven, and in the flexibility of what sort of parts get build and by what methods. Setuptools is still too rooted in distutils' world, the world where you can't depend on any other components being around to build things with. >Frankly I'd like to see setuptools exploded, with those parts of general use >folded back into the standard library, the creation of a set of >non-implementation-specific documents of the distribution formats and >behavior, leaving a small core of one implementation of how to do it and the >door open for others to compete with their own implementation. Apart from the exploding part, there are already documents. The only thing that makes them implementation-specific is that they haven't passed through any magic blessing process to make them standards. >You should document those ideas someplace and start getting community input. >There are a lot of diverse opinions on the right way to do this and the way >ahead is quite unclear. We might be talking about different things, as I'm more concerned with replacing setuptools and distutils on the build-and-distribute side. What's needed there is more the weeding out of too many ways to do simple things, and fixing the complete absence of ways to do complex things. :) For simple things the distutils are too hard, and for slightly-more-complex things, the entry barrier encourages people to abandon and replace them. On the package management side, I'm somewhat more inclined to agree with the need for a community approach, though. > > btw, offtopic question: are you by any chance the same Jeff Rush who > > invented EchoMail? > >Yep, that's me. Not many remember the Fidonet days. I designed >EchoMail on a >napkin during a DFW Sysop pizza party during a conversation on what >to do with >the unused capability of inter-BBS private file transfers. It too >escaped its >ecosystem and spread like wildfire, almost getting banned from Fidonet. ;-) Ah, so you *do* know what it's like to develop setuptools, then. I might even have met you at the one DFW sysop pizza party I ever attended. Back then, I ran the FreeZone, and before that, "Ferris Bueller's Fine Arts Forum", back in the late 80's and early 90's. My wife met me through the D/FW BBS list in the back of Computer Shopper, with a modem she bought at Software, Etc., up in Allen or wherever that place was. Not the chain store, the little consignment shop. Those were the days. But now we're *really* getting off-topic. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby wrote: > At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: >> Are you open to giving certain others patch view/commit privileges to >> setuptools? > > Jim Fulton has such already. I'm open to extending that to others who > have a good grasp of the subtleties involved. > > Truthfully, if we can just get 0.6 put to bed, I could probably open up > the trunk a lot wider. What is needed to put 0.6 to bed? How can we help accelerate this? > Probably the most frustrating thing (or "chief amongst the most > frustrating things") about setuptools development is that it's a black > hole. By which I mean that backward compatibility and cruft accretion > make it difficult to get out of. > > In the beginning, there was the distutils. Distutils begat setuptools, > and setuptools begat virtualenv and zc.buildout and source control > plugins. Etc., etc. I've found in the past a revisiting of basic principles and objectives, communicated in enhanced documentation, can help to clear out such black holes. ;-) I'm pulling something together, from the recent emails and some archived threads -- it definitely is tangled though, I'll agree. > What I think is really needed in the long run is to keep eggs, but get > rid of setuptools and the distutils in their current form. There's a > lot of brokenness there, and also a lot of accumulated cruft. We really > need a distutils 3000, and it needs to be built on a better approach. That will require a lot of concensus building as well as collection of use cases so that the architecture team can encompass aspects they are not personally aware of. As you've said, it's hard to address itches that are not your own. It certainly is possible for someone to create a parallel packaging moduleset that uses the existing eggs format and PyPI but without the currently codebase, and then, once proven to work, lobby for it as distutils 3000. Frankly I'd like to see setuptools exploded, with those parts of general use folded back into the standard library, the creation of a set of non-implementation-specific documents of the distribution formats and behavior, leaving a small core of one implementation of how to do it and the door open for others to compete with their own implementation. > In truth, my *real* motivation for PEP 365's bootstrap tool isn't so > much to support the package management tools we have today, as it is to > support a new one tomorrow. I have a few ideas for ways to shift the > paradigm of how individual projects get built, to incorporate many > scenarios that don't work well now. But to implement those things in > such a next-generation tool, I will not want to be restricted to just > what's in the stdlib or what can be bundled in the tool. You should document those ideas someplace and start getting community input. There are a lot of diverse opinions on the right way to do this and the way ahead is quite unclear. > And I think it's probably getting close to time I stepped down from > day-to-day management of the codebase (which is more like month-to-month > or quarter-to-quarter for me lately). It will probably be a lot easier > for me to step back and critique stuff that goes in, after the fact, > than to go over the stuff beforehand. :) > > I'm not sure exactly how to go about such a handoff though. My guess is > that we need a bug/patch tracker, and a few people to review, test, and > apply. Maybe a transitional period during which I just say yea or nay > and let others do the test and apply, before opening it up entirely. > That way, we can perhaps solidify a few principles that I'd like to have > stay in place. (Like no arbitrary post-install code hooks.) I'll see about a tracker and identify some people to help out. > btw, offtopic question: are you by any chance the same Jeff Rush who > invented EchoMail? Yep, that's me. Not many remember the Fidonet days. I designed EchoMail on a napkin during a DFW Sysop pizza party during a conversation on what to do with the unused capability of inter-BBS private file transfers. It too escaped its ecosystem and spread like wildfire, almost getting banned from Fidonet. ;-) -Jeff ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby writes: > >7. Many wanted to ability to install files anywhere in the install tree and > > not just under the Python package. Under distutils this was possible > > but > > it was removed in setuptools for security reasons. > > It wasn't security, it was manageability. Egg-based installation > means containment, (analagous to GNU stow) and therefore portability > and disposability of plugins. (Which again is what eggs were really > developed for in the first place.) defining containment this way doesn't help when preparing eggs for inclusion in a linux distribution. E.g. users on these distributions are used to find log files in /var/log (maybe in a subdir), documentation in /usr/share/doc/. You probably will get different views about manageability depending on your background (used to linux distribution standards or used to standards set by setuptools/cheeseshop). Packagers currently move these files manually to the standard locations and often have to keep symlinks in the egg dirs to these locations. Installation on linux distributions is handled by existing package tools which is unlikely to change. So it would be nice to find a common layer which can be used for both distribution methods, optionally enabling this with some kind of option like --install-files-in-places-not-handled-by-setuptools ;) Matthias ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Jeff Rush writes: > I was in a Packaging BoF yesterday and, although not very relevant to the > packager bootstrap thread, Guido has asked me to post some of the concerns. We did address many topics on both days, I added the following topics which were addressed on the Friday BoF only, see http://wiki.python.org/moin/PackagingBOF - Linux distributions try to ship only one version of a package/egg/module in one release, only shipping more than one version if necessary. eggs (as least as shipped with Debian, Fedora, Ubuntu) are all built using --single-version-externally-managed. - import foo should work wether installed as an egg or installed with distutils, and without using pkg_resources.require - pkg_resources should handle the situation of one egg version installed as --single-version-externally-managed (default version) and one or more eggs installed not using --single-version-externally-managed. Currently these additional versions cannot be imported. - It would be useful if setuptools could handle separate build and install steps like most configure/make/make install systems do. Access to external resources should optionally be disabled during a build. - The idea was brought up to use a to-be-defined api-version to describe dependencies between eggs. Version numbers are generally used for more than api changes; the idea follows existing practice for shared object names, only changing when the API is changed. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: >Are you open to giving certain others patch view/commit privileges >to setuptools? Jim Fulton has such already. I'm open to extending that to others who have a good grasp of the subtleties involved. Truthfully, if we can just get 0.6 put to bed, I could probably open up the trunk a lot wider. One of the things that slows me down is that patches usually don't come with tests, so I usually have to manually smoke-test them for scenarios I think they'll effect. There isn't really any automated procedure. Probably the most frustrating thing (or "chief amongst the most frustrating things") about setuptools development is that it's a black hole. By which I mean that backward compatibility and cruft accretion make it difficult to get out of. In the beginning, there was the distutils. Distutils begat setuptools, and setuptools begat virtualenv and zc.buildout and source control plugins. Etc., etc. What I think is really needed in the long run is to keep eggs, but get rid of setuptools and the distutils in their current form. There's a lot of brokenness there, and also a lot of accumulated cruft. We really need a distutils 3000, and it needs to be built on a better approach. In truth, my *real* motivation for PEP 365's bootstrap tool isn't so much to support the package management tools we have today, as it is to support a new one tomorrow. I have a few ideas for ways to shift the paradigm of how individual projects get built, to incorporate many scenarios that don't work well now. But to implement those things in such a next-generation tool, I will not want to be restricted to just what's in the stdlib or what can be bundled in the tool. (Btw, by "real" motivation, I don't mean I've been deceptive about my intentions, I mean that my strong intuition that such a bootstrap facility is needed, is probably being fueled by the long term desire to replace the entire distutils-based infrastructure with something better.) > I'd be willing to help out, and keep a carefully balanced hand in > what is accepted. And I think it's probably getting close to time I stepped down from day-to-day management of the codebase (which is more like month-to-month or quarter-to-quarter for me lately). It will probably be a lot easier for me to step back and critique stuff that goes in, after the fact, than to go over the stuff beforehand. :) I'm not sure exactly how to go about such a handoff though. My guess is that we need a bug/patch tracker, and a few people to review, test, and apply. Maybe a transitional period during which I just say yea or nay and let others do the test and apply, before opening it up entirely. That way, we can perhaps solidify a few principles that I'd like to have stay in place. (Like no arbitrary post-install code hooks.) btw, offtopic question: are you by any chance the same Jeff Rush who invented EchoMail? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 19, 2008, at 3:57 AM, Jeff Rush wrote: > > I and others appreciate your call for more patches on various > topics. However > a long delay in applying them will discourage contribution. Are you > open to > giving certain others patch view/commit privileges to setuptools? > I'd be > willing to help out, and keep a carefully balanced hand in what is > accepted. The Python sandbox has a setuptools directory. Is this the canonical location for the code? If so, then anybody who has Python commit privileges can commit to it and help further develop setuptools. If not, why not and what is the sandbox setuptools used for? - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR+Eca3EjvBPtnXfVAQLabwP9F8NtQX6YsDXJMHiByCGILPAQ2NgtaIzg en6yYbhl5IAweTr0DtWzxRXjSGMifK/D4PmtRSWWUTy3VY+8cRUkYuBjIxPOHJRF 4TA4dYoW4f2+qM1IO/l59FIAJgUyrXKhv3aznpXBFl+PaRCW9qP9G1lur3lolipB h4i8ya+I7zU= =2/iq -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby wrote: > > I'm actually happy to hear that there's this much energy available -- > hopefully some of it can be harnessed towards positive solutions. > > When I began developing setuptools, I often asked for the input of > packagers, developers, etc., through the distutils-sig... and was met > with overwhelming silence. So the fact that there is now a group of > people who are ready to work for some solutions seems like a positive > change, to me. I can appreciate how frustrating silence is when you call for input. Let's see if we can keep the volunteer energy going this time around. > It's hard to make design decisions regarding itches you don't personally > have, and which other people won't help scratch. Unfortunately, a lot > of the proposals from packaging system people have been of the form of, > "fix this for us by breaking things for other people". Not all of them, > though. Many have been very helpful, contributing troubleshooting help > and good patches. > > That some of those good patches took nearly a year to get into > setuptools (some from Fedora just got into 0.6c8 that were sent to me > almost a year ago) is because I'm the only person reviewing setuptools > patches, and I've spent only a few days in the last year doing focused > development work on setuptools (as opposed to answering questions about > it on the SIG). > > It's never a good thing when people's patches sit around, regardless of > where they come from. But that's not the same thing as *rejecting* the > patches. I and others appreciate your call for more patches on various topics. However a long delay in applying them will discourage contribution. Are you open to giving certain others patch view/commit privileges to setuptools? I'd be willing to help out, and keep a carefully balanced hand in what is accepted. -Jeff ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
At 05:10 PM 3/17/2008 -0500, Jeff Rush wrote: >I was in a Packaging BoF yesterday and, although not very relevant to the >packager bootstrap thread, Guido has asked me to post some of the concerns. > >The BoF drew about 15 people, many of whom were packagers for Red Hat, Ubuntu >and such. Everyone had strong expressions of frustration with the status quo >and most had tried to resolve their issues but had their patches rejected. I >am not taking either side and whether those rejections were >justified I cannot >say, but the general feeling of their concerns intentionally not being >addressed isn't healthy. Several had abandoned setuptools, deeming it a >failed solution and others called for a fork. > >To start, I am not a leader of the group nor do I claim I accurately captured >and expressed all their concerns. I apologize to those in the BoF for any >misrepresentations. I'm actually happy to hear that there's this much energy available -- hopefully some of it can be harnessed towards positive solutions. When I began developing setuptools, I often asked for the input of packagers, developers, etc., through the distutils-sig... and was met with overwhelming silence. So the fact that there is now a group of people who are ready to work for some solutions seems like a positive change, to me. It's hard to make design decisions regarding itches you don't personally have, and which other people won't help scratch. Unfortunately, a lot of the proposals from packaging system people have been of the form of, "fix this for us by breaking things for other people". Not all of them, though. Many have been very helpful, contributing troubleshooting help and good patches. That some of those good patches took nearly a year to get into setuptools (some from Fedora just got into 0.6c8 that were sent to me almost a year ago) is because I'm the only person reviewing setuptools patches, and I've spent only a few days in the last year doing focused development work on setuptools (as opposed to answering questions about it on the SIG). It's never a good thing when people's patches sit around, regardless of where they come from. But that's not the same thing as *rejecting* the patches. >1. Many felt the existing dependency resolver was not correct. They wanted a > full tree traversal resulting in an intersection of all restrictions, > instead of a first-acceptable-solution approach taking now, which can > result in top-level dependencies not being enforced upon lower > levels. The > latter is faster however. One solution would be to make the resolver > pluggable. Patches welcome, on both counts. Personally, Bob and I originally wanted a full-tree intersection, too, but it turned out to be hairier to implement than it seems at first. My guess is that none of the people who want it, have actually tried to implement it without a factorial or exponential O(). But that doesn't mean I'll be unhappy if somebody succeeds. :) Intuitively, it seems easy, just gather the requirements and intersect. In practice, different versions of a package may have different dependencies, so the intersection is not nearly as simple as that. We ended up just going for a depth-first version of the current algorithm (switched to breadth-first later, after field tests showed some problems with that), being greedy by testing latest-version-first, on the assumption that more recent versions would be likely to have the most-restrictive version requirements. In other words, we attempt to achieve heuristically what's being proposed to do algorithmically. And my guess is that whatever cases the heuristic is failing at, would probably not be helped by an algorithmic approach either. But I would welcome some actual data, either way. Again, though, patches are welcome. :) (Specifically, for the trunk; I don't see a resolver overhaul as being suitable for the 0.6 stable branch.) >2. People want a solution for the handling of documentation. The distutils > module has had commented out sections related to this for several years. As with so many other things, this gets tossed around the distutils-sig every now and then. A couple of times I've thrown out some options for how this might be done, but then the conversation peters out around the time anybody would have to actually do some work on it. (Me included, since I don't have an itch that needs scratching in this area.) In particular, if somebody wants to come up with a metadata standard for including documentation in eggs, we've got a boatload of hooks by which it could be done. Nothing's stopping anybody from proposing a standard and building a tool, here. (e.g. using the setuptools command hook, .egg-info writer hook, etc.) >3. A more flexible internal handing of the different types of files is needed. > Currently the code, data, lib, etc. files are aggregated at > build time and > people woul
