Re: [Distutils] [Python-Dev] platform management
Great idea! Sounds like a PEP (informational, probably) would be good idea.
On Tue, Mar 18, 2008 at 4:59 PM, Bill Janssen <[EMAIL PROTECTED]> wrote:
> I don't think this is bike-shedding.
>
> The debate about "AMD64" vs. "amd64" vs. "x86_64" reminded me that
> I've been bit more and more frequently by bits of platform-specific
> knowledge scattered around the standard library. The latest is the
> code in distutils.unixccompiler that tries to figure out what flags to
> send to the linker in order to add a dynamic library path lookup to a
> shared library.
>
> There are lots of different ways of figuring out which platform is
> being used, and they're all over the place. The code in
> distutils.unixccompiler uses "sys.platform[:6]", and looks for
> "darwin"; the code in urllib.py uses "os.name", and expects it to be
> "mac", but later looks for "sys.platform == 'darwin'; posixfile
> believes that platforms come with version numbers ("linux2", "aix4");
> pydoc and tarfile have tests for "sys.platform == 'mac'". tempfile
> looks at os.sys.platform *and* os.name.
>
> Could well be that all of these are correct (I believe that "mac", for
> instance, refers to the generations of the Mac OS before OS X). But
> it means that when someone tries to update "Python" to a new major
> version release for, say, OS X, it's really easy to miss things. And
> the fact that the platform version is sometimes included and sometimes
> not is also problematic; darwin9 is different from darwin8 in some
> important aspects.
>
> It would be nice to
>
> (a) come up with a list of standard platform symbols,
> (b) put those symbols somewhere in the documentation,
> (c) have one way of discovering them, via sys or os or platform or
> whichever module,
> (d) add a standard way of discovering the platform version, and
> (e) stamp out all the other ways that have crept in over the years.
>
> Bill
> ___
> Python-Dev mailing list
> [EMAIL PROTECTED]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
I was using the human interface at python.org/pypi. There are two prominent links at the top of the page: "Browse the tree of packages" and "Submit package information" followed by the 30 most recently changed packages. What I was looking for was the page for a specific package. The "Browse the tree of packages" link was no help. Finally I realized that in the side bar, in a small unobtrusive font, is a link to "List packages" which links to a list of *all* packages, in alphabetical order. I found my package there. I think repeating that link right below "browse the tree" would have been sufficient. But it would have been cool if there had been a search box (also in the start page) where I could type (part of) the name of the package and it would have given me the nearest matches. On Wed, Mar 19, 2008 at 6:05 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > I don't understand PyPI all that well; it seems poor design that the > > browsing via keywords is emphasized but there is no easy way to > > *search* for a keyword (the list of all packages is not emphasized > > enough on the main page -- it occurs in the side bar but not in the > > main text). > > I don't understand. What is "browsing via keywords" and how is that > emphasized? (one I know that, I can look into ways for searching > for keywords) > > > > I assume there's a programmatic API (XML-RPC?) but I > > haven't found it yet. > > The recommended "programmatic" API is > > > http://pypi.python.org/simple/ > > Not sure what you were trying to achieve programmatically; > "typically" people know what they want to install (e.g. > "threadedcomments"), and then the tool goes directly to > > http://pypi.python.org/simple/threadedcomments/ > > Regards, > Martin > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 365 (Adding the pkg_resources module)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido van Rossum wrote: > On Wed, Mar 19, 2008 at 12:54 PM, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > [a long message] > > I'm back at Google and *really* busy for another week or so, so I'll > have to postpone the rest of this discussion for a while. If other > people want to chime in please do so; if this is just a dialog between > Phillip and me I might incorrectly assume that nobody besides Phillip > really cares. I care, a lot, enough to have volunteered to help with maintenance / development of setuptols back in September 2007. I think that, warts an all, setuptools is a *huge* improvement over bare distutils for nearly every use case I know about. A lot of setuptools warts are driven by related design problems in the distutils, such as the choice to use imperative / procedural code for everything: a declarative approach, with hooks for cases which actually need them (likely 5% of existing packages) would have made writing tools on top of the framework much simpler. It is ironic that Python is *too powerful* a tool for the tasks normally done by distutils / setuptools: a more restricted, and thererfore introspectable, configuration-driven approoach seems much cleaner. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH4e7m+gerLs4ltQ4RAt+hAKDBqIrashlgf8U6XRtfMHjTOaiy4gCeO1Zn UfdjDYIb2P6vDCcUGSjITTo= =JTok -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for Python 3000?
Martin v. Löwis wrote: >> specific code in PyPI. Are developers for Python 3.x encouraged in >> 3.x guidelines to release 'fat' distributions that combine 2.x and 3.x >> usable versions? > > Passive voice is misleading here: encouraged by whom? "... encouraged in __3.x guidelines__ to ...": I presume although I've not found them yet that there is some kind of document for developers titled something like, "how to migrate your Python code from 2.x to 3.x". That document would be a logical place for advice and consideration of the tradeoffs of jumping to 3.x, maintaining two synced versions using 2to3 or 3to2, and the risks of keeping two independent releases. Identifying best practices would help them make good choices for the community. > *I* encourage people to consider that option, rather than assuming it > couldn't possibly work. Whether it actually works, I don't know. > I hope it would work, and I hope it would not be fat at all. So we don't have an actual success story of a dual-version distribution, even as a prototype, using 2to3 within a distutils package? I would not encourage a practice without at least one such example. >>> Can you kindly refer to some archived discussion for that? >> >> http://mail.python.org/pipermail/python-dev/2006-April/063943.html >> > Thanks. I still have the same position as I had then - if > distutils is broken, it should be fixed, not ignored. Since the precise API was not documented well and many people began to make use of ambiguous internal interfaces, such fixes would indeed break them. So your vote would be to do the right thing, even if it results in some breakage. I can respect that philosophy. >> A controversial point, I'm afraid. Perhaps it is time for a parallel >> rewrite, so that those setup.py who import distutils get the old >> behavior, and those who import distutils2 get the new, and we let >> attrition and the community decide which is standard. > > Is there a list of the problems with distutils somewhere? Unfortunately no. Much of it is anecdotal, much of it occurs on lists outside the Python community by those attempting to package things. And some of it are comments by developers who peeked into the distutils source and blanched. And some of the problems are not bugs, per se, but disagreement on scope of functionality and a lack of well-known alternatives. So just "fix it if broken" doesn't work when there is no agreement on how to expand that scope. I am working on pulling together such a list however, and getting it into the tracker, so that debate with a record of decisions can occur. I agree that it is hard to fix problems if no one is _clearly_ reporting them to us. Too much smoke, not enough light. > It always worked fine for me, so I see no reason to fix it in the > first place. Pardon my lack of knowledge of your background; when you say it always worked fine for you, are you referring to personal experiences using it to _install_ software or to experiences as a packager in actually distributing complex collections of modules on different platforms? -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mar 19, 2008, at 3:23 PM, Guido van Rossum wrote: > If other people want to chime in please do so; if this is just a > dialog between Phillip and me I might incorrectly assume > that nobody besides Phillip really cares. I really care. I've used setuptools, easy_install, eggs, and pkg_resources extensively for the past year or so (and contributed a few small patches). There have been plenty of problems, but I find them to be overall useful tools. It is a great boon to a programming community to lower the costs of re-using other people's code. The Python community will benefit greatly once a way to do that becomes widely enough accepted to reach a tipping point and become ubiquitous. Setuptools is already the de facto standard, but it hasn't become ubiquitous, possibly in part because of "egg hatred", about which more below. I've interviewed several successful Python hackers who "hate eggs" in order to understand what they hate about them, and I've taken notes from some of these interviews. (The list includes MvL, whose name was invoked earlier in this thread.) After filtering out yer basic complaining about bugs (which complaints are of course legitimate, but which don't indict setuptools as worse than other software of comparable scope and maturity), their objections seem to fall into two categories: 1. "The very notion of package dependency resolution and programmable or command-line installation of packages at the language level is a bad notion." This can't really be the case. If the existence of such functionality at the programming language level were an inherently bad notion, then we would be hearing some complaints from the Ruby folks, where the Gems system is standard and ubiquitous. We hear no complaints -- only murmurs of satisfaction. One person recently reported to me that while there are more packages in Python, he finds himself re-using other people's code more often when he works in Ruby, because almost all Ruby software is Gemified, but only a fraction of Python software is Eggified. Often this complaint comes with the idea that eggs conflict with their system-level package management tools. (These are usually Debian/Ubuntu users.) Note that Ruby software is not too hard to include in operating system packaging schemes -- my Ubuntu Hardy apt-cache shows plenty of Ruby software. A sufficiently mature and widely supported setuptools could actually make it easier to integrate Python software into Debian -- see stdeb [1]. 2. "Setuptools/eggs give me grief." What can really be the case is that setuptools causes a host of small, unnecessary problems for people who prefer to do things differently than PJE does. Personally, I prefer to use GNU stow, and setuptools causes unnecessary, but avoidable, problems for me. Many people object (rightly enough) to a "./setup.py install" automatically fetching new software over the Internet by default. The fact that easy_install creates a site.py that changes the semantics of PYTHONPATH is probably the most widely and deservedly hated example of this kind of thing [2]. I could go on with a few other common technical complaints of this kind. These type-2 problems can be fixed by changing setuptools or they can be grudgingly accepted by users, while retaining compatibility with the large and growing ecosystem of eggy software. Certainly fixing setuptools to play better with others is a more likely path to success than setting out to invent a non-egg-compatible alternative. Such a project might never be implemented well enough to serve, and if it were it would probably never overtake eggs's lead in the Python ecosystem, and if it did it would probably not turn out to be a better tool. So, since you asked for my chime, I advise you to publically bless eggs, setuptools, and easy_install as plausible future standards and solicit patches which address the complaints. For that matter, soliciting specific complaints would be a good start. I've done so in private many times with only partial success as to the "specific" part. One promising approach is to request objections in the form of automated tests that setuptools fails, e.g. [3]. Regards, Zooko O'Whielacronx [1] http://stdeb.python-hosting.com/ [2] http://www.rittau.org/blog/20070726-02 And no, PJE's suggested "trivial fix" does not satisfy the objectors, as it can't support the use case of "cd somepkg ; python ./ setup.py install ; cd .. ; python -c 'import somepkg'". [3] http://twistedmatrix.com/trac/ticket/2308#comment:5 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Request for Input re Packaging
Jeff Rush <[EMAIL PROTECTED]> writes: > In researching the state of packaging, I've been reading the > archives and all the bug reports filed against distutils. > > I'd like though to get some examples of particularly troublesome > uses of setup.py, to pull together and propose some changes to make > their use case a bit easier. So far such cases I've been made aware > of are Twisted, numpy and SciPy. If you know of a tough case where > the developer had to jump through hoops to make it work, please > point me to it. > > I'd also like to get suggestions of improvements to PyPI, which I've > not seen much discussion about. I know that those in Debian who are involved with packaging Python modules and applications often complain about the state of distutils and setuptools. I don't know what the specific complaints are, though. I'll forward this request to the debian-python list in an attempt to rouse them for feedback. -- \"Why was I with her? She reminds me of you. In fact, she | `\ reminds me more of you than you do!" -- Groucho Marx | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for Python 3000?
> Yes, I am assuming that existing projects would at some point introduce a 3.x > version and maybe continue a 2.x version as separate distros, similar to > Python itself. Then the large number of existing unqualified dependencies > on, > say SQLObject, would pull in the higher 3.x version and crash. It's the > older > projects that don't get updated often that are at risk of being destabilized > by the arrival of 3.x specific code in PyPI. Are developers for Python 3.x > encouraged in 3.x guidelines to release 'fat' distributions that combine 2.x > and 3.x usable versions? Passive voice is misleading here: encouraged by whom? *I* encourage people to consider that option, rather than assuming it couldn't possibly work. Whether it actually works, I don't know. I hope it would work, and I hope it would not be fat at all. > Perhaps a convention of a keyword or more likely a new trove classifier that > spells outs 3.x stuff, with indicators on package info pages and query > filters > on PyPI against that? I'm fine with adding more trove classifiers if that solves the problem (although I still assume the problem doesn't actually exist). As always, a classifier should not be added until there actually are two packages that want to use it. >> Can you kindly refer to some archived discussion for that? > > http://mail.python.org/pipermail/python-dev/2006-April/063943.html > >"I started looking at this. The number of complaints I > got when I started on this that it would break the > existing distutils based installers totally discouraged > me. In addition, the existing distutils codebase is ... > not good. > > It is flatly not possible to "fix" distutils and > preserve backwards compatibility." -Anthony Baxter Thanks. I still have the same position as I had then - if distutils is broken, it should be fixed, not ignored. > A controversial point, I'm afraid. Perhaps it is time for a parallel > rewrite, > so that those setup.py who import distutils get the old behavior, and those > who import distutils2 get the new, and we let attrition and the community > decide which is standard. Is there a list of the problems with distutils somewhere? It always worked fine for me, so I see no reason to fix it in the first place. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for Python 3000?
> While not quite to the same scale as the 2 to 3 transition, this problem > seems like one that would generally already exist. If one writes code > that uses newer 2.4/2.5 features (say decorators for example,) it won't > build/run on 2.3 or earlier installs. How have people been handling > that sort of situation? Is it only by not using the newer features or > is there some situation I'm just not seeing in a brief review of a few > projects pages on PyPI where there is only one source tarball? I think packages have taken all sorts of responses to this issue. Some will list the minimum required Python version in their README, some might put a test in setup.py that aborts installation if the Python version is too old, some may just install and let the user find out at runtime. Typically, packages try to support all the Python versions that their users still use. If a user of an older Python version comes along, they'll just need to fetch the older release (which hopefully is still online, or can be extracted from the source repository). Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
> I don't understand PyPI all that well; it seems poor design that the > browsing via keywords is emphasized but there is no easy way to > *search* for a keyword (the list of all packages is not emphasized > enough on the main page -- it occurs in the side bar but not in the > main text). I don't understand. What is "browsing via keywords" and how is that emphasized? (one I know that, I can look into ways for searching for keywords) > I assume there's a programmatic API (XML-RPC?) but I > haven't found it yet. The recommended "programmatic" API is http://pypi.python.org/simple/ Not sure what you were trying to achieve programmatically; "typically" people know what they want to install (e.g. "threadedcomments"), and then the tool goes directly to http://pypi.python.org/simple/threadedcomments/ Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
> The Python sandbox has a setuptools directory. Is this the canonical > location for the code? Yes, it is. > If so, then anybody who has Python commit > privileges can commit to it and help further develop setuptools. They can, but they shouldn't. Nothing should be committed there without pje's approval (in whatever form he choses to give such approval). > If not, why not and what is the sandbox setuptools used for? I think it shouldn't be in sandbox, but toplevel, but that's a minor detail. Maybe I misunderstand the English word "sandbox". Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for Python 3000?
Jeff Rush wrote: > Perhaps it is time for a parallel rewrite, > so that those setup.py who import distutils get the old behavior, and those > who import distutils2 get the new, That sounds good to me. If anyone wants to have a go at this, I have some ideas on how to structure it that I'd be happy to discuss. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Request for Input re Packaging
In researching the state of packaging, I've been reading the archives and all the bug reports filed against distutils. I'd like though to get some examples of particularly troublesome uses of setup.py, to pull together and propose some changes to make their use case a bit easier. So far such cases I've been made aware of are Twisted, numpy and SciPy. If you know of a tough case where the developer had to jump through hoops to make it work, please point me to it. I'd also like to get suggestions of improvements to PyPI, which I've not seen much discussion about. A few I've collected are: - move to https/ssl - improvements re package signing - internal parsing/aggregation of metadata for better queries, and to stop using the filename for version/platform/etc. information. - moving of requirements logic from client into PyPI, where it has db access to the dependency, resolves what packages are needed and delivers a list back to the client for prompting the user for permission, similar to how "yum" interacts today. - a db lint-picking walker that looks for problems on PyPI, such as binary distros w/o a source distro, lack of binaries for those platforms often without compilers, failure to provide a link to a version repo for use with "project==dev". - some auto-generated reports of access statistics and the mix of distutils vs setuptools, those who "registered" w/o "uploading", and perhaps if we get a new classifier assigned, some idea of Python 2.x vs 3.x packages. Last, some of the issues with distutils/setuptools can be solved with zc.buildout. If you have found zc.buildout lacking, please tell me where it fell short so we can see if anything can be done. Thanks for your involvement, -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
At 05:15 PM 3/19/2008 -0500, Jeff Rush wrote: >Phillip J. Eby wrote: > > At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: > >> Are you open to giving certain others patch view/commit privileges to > >> setuptools? > > > > Jim Fulton has such already. I'm open to extending that to others who > > have a good grasp of the subtleties involved. > > > > Truthfully, if we can just get 0.6 put to bed, I could probably open up > > the trunk a lot wider. > >What is needed to put 0.6 to bed? How can we help accelerate this? Get a tracker set up. I'm already in the main Python one, might as well use that. >It certainly is possible for someone to create a parallel packaging moduleset >that uses the existing eggs format and PyPI but without the currently >codebase, and then, once proven to work, lobby for it as distutils 3000. Yep. And I believe that something will look rather more like zc.buildout than setuptools, actually. Specifically in being data-driven rather than script-driven, and in the flexibility of what sort of parts get build and by what methods. Setuptools is still too rooted in distutils' world, the world where you can't depend on any other components being around to build things with. >Frankly I'd like to see setuptools exploded, with those parts of general use >folded back into the standard library, the creation of a set of >non-implementation-specific documents of the distribution formats and >behavior, leaving a small core of one implementation of how to do it and the >door open for others to compete with their own implementation. Apart from the exploding part, there are already documents. The only thing that makes them implementation-specific is that they haven't passed through any magic blessing process to make them standards. >You should document those ideas someplace and start getting community input. >There are a lot of diverse opinions on the right way to do this and the way >ahead is quite unclear. We might be talking about different things, as I'm more concerned with replacing setuptools and distutils on the build-and-distribute side. What's needed there is more the weeding out of too many ways to do simple things, and fixing the complete absence of ways to do complex things. :) For simple things the distutils are too hard, and for slightly-more-complex things, the entry barrier encourages people to abandon and replace them. On the package management side, I'm somewhat more inclined to agree with the need for a community approach, though. > > btw, offtopic question: are you by any chance the same Jeff Rush who > > invented EchoMail? > >Yep, that's me. Not many remember the Fidonet days. I designed >EchoMail on a >napkin during a DFW Sysop pizza party during a conversation on what >to do with >the unused capability of inter-BBS private file transfers. It too >escaped its >ecosystem and spread like wildfire, almost getting banned from Fidonet. ;-) Ah, so you *do* know what it's like to develop setuptools, then. I might even have met you at the one DFW sysop pizza party I ever attended. Back then, I ran the FreeZone, and before that, "Ferris Bueller's Fine Arts Forum", back in the late 80's and early 90's. My wife met me through the D/FW BBS list in the back of Computer Shopper, with a modem she bought at Software, Etc., up in Allen or wherever that place was. Not the chain store, the little consignment shop. Those were the days. But now we're *really* getting off-topic. :) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for Python 3000?
Martin v. Löwis wrote: > > I don't see the need to for PyPI. For packages (or "distributions", > to avoid confusion with Python packages), I see two options: > > a) provide a single release that supports both 2.x and 3.x. > b) switch to Python 3 at some point (i.e. burn your bridges). > > You seem to be implying that some projects may release separate > source distributions. I cannot imagine why somebody would want > to do that. Yes, I am assuming that existing projects would at some point introduce a 3.x version and maybe continue a 2.x version as separate distros, similar to Python itself. Then the large number of existing unqualified dependencies on, say SQLObject, would pull in the higher 3.x version and crash. It's the older projects that don't get updated often that are at risk of being destabilized by the arrival of 3.x specific code in PyPI. Are developers for Python 3.x encouraged in 3.x guidelines to release 'fat' distributions that combine 2.x and 3.x usable versions? There is also some hassle with 2.x programmers browsing PyPI for useful modules to incorporate in their programs, downloading them (w/easy_install so they don't see the project website) and getting streams of errors because they unknowningly hit a 3.x-specific version. Perhaps a convention of a keyword or more likely a new trove classifier that spells outs 3.x stuff, with indicators on package info pages and query filters on PyPI against that? >> 2. There have been attempts over the years to fix distutils, >> with the last one being in 2006 by Anthony Baxter. He >> stated that a major hurdle was the strong demand to >> respect backward compatibility and he finally gave up. > > Can you kindly refer to some archived discussion for that? http://mail.python.org/pipermail/python-dev/2006-April/063943.html "I started looking at this. The number of complaints I got when I started on this that it would break the existing distutils based installers totally discouraged me. In addition, the existing distutils codebase is ... not good. It is flatly not possible to "fix" distutils and preserve backwards compatibility." -Anthony Baxter >> One of the purposes of Python 3.0 was the freedom to >> break backward compatibility for the sake of "doing >> the right thing". So is it now permissible to give >> distutils a good reworking and stop letting >> compatibility issues hold us back? > > I don't know what the proposed changes are, but for some > changes; in general, I feel that the need for backwards > compatibility is exaggerated. A controversial point, I'm afraid. Perhaps it is time for a parallel rewrite, so that those setup.py who import distutils get the old behavior, and those who import distutils2 get the new, and we let attrition and the community decide which is standard. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby wrote: > At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: >> Are you open to giving certain others patch view/commit privileges to >> setuptools? > > Jim Fulton has such already. I'm open to extending that to others who > have a good grasp of the subtleties involved. > > Truthfully, if we can just get 0.6 put to bed, I could probably open up > the trunk a lot wider. What is needed to put 0.6 to bed? How can we help accelerate this? > Probably the most frustrating thing (or "chief amongst the most > frustrating things") about setuptools development is that it's a black > hole. By which I mean that backward compatibility and cruft accretion > make it difficult to get out of. > > In the beginning, there was the distutils. Distutils begat setuptools, > and setuptools begat virtualenv and zc.buildout and source control > plugins. Etc., etc. I've found in the past a revisiting of basic principles and objectives, communicated in enhanced documentation, can help to clear out such black holes. ;-) I'm pulling something together, from the recent emails and some archived threads -- it definitely is tangled though, I'll agree. > What I think is really needed in the long run is to keep eggs, but get > rid of setuptools and the distutils in their current form. There's a > lot of brokenness there, and also a lot of accumulated cruft. We really > need a distutils 3000, and it needs to be built on a better approach. That will require a lot of concensus building as well as collection of use cases so that the architecture team can encompass aspects they are not personally aware of. As you've said, it's hard to address itches that are not your own. It certainly is possible for someone to create a parallel packaging moduleset that uses the existing eggs format and PyPI but without the currently codebase, and then, once proven to work, lobby for it as distutils 3000. Frankly I'd like to see setuptools exploded, with those parts of general use folded back into the standard library, the creation of a set of non-implementation-specific documents of the distribution formats and behavior, leaving a small core of one implementation of how to do it and the door open for others to compete with their own implementation. > In truth, my *real* motivation for PEP 365's bootstrap tool isn't so > much to support the package management tools we have today, as it is to > support a new one tomorrow. I have a few ideas for ways to shift the > paradigm of how individual projects get built, to incorporate many > scenarios that don't work well now. But to implement those things in > such a next-generation tool, I will not want to be restricted to just > what's in the stdlib or what can be bundled in the tool. You should document those ideas someplace and start getting community input. There are a lot of diverse opinions on the right way to do this and the way ahead is quite unclear. > And I think it's probably getting close to time I stepped down from > day-to-day management of the codebase (which is more like month-to-month > or quarter-to-quarter for me lately). It will probably be a lot easier > for me to step back and critique stuff that goes in, after the fact, > than to go over the stuff beforehand. :) > > I'm not sure exactly how to go about such a handoff though. My guess is > that we need a bug/patch tracker, and a few people to review, test, and > apply. Maybe a transitional period during which I just say yea or nay > and let others do the test and apply, before opening it up entirely. > That way, we can perhaps solidify a few principles that I'd like to have > stay in place. (Like no arbitrary post-install code hooks.) I'll see about a tracker and identify some people to help out. > btw, offtopic question: are you by any chance the same Jeff Rush who > invented EchoMail? Yep, that's me. Not many remember the Fidonet days. I designed EchoMail on a napkin during a DFW Sysop pizza party during a conversation on what to do with the unused capability of inter-BBS private file transfers. It too escaped its ecosystem and spread like wildfire, almost getting banned from Fidonet. ;-) -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Wed, Mar 19, 2008 at 03:58:58PM -0400, Jim Fulton wrote: > On Mar 19, 2008, at 3:52 PM, Marius Gedminas wrote: > >On Wed, Mar 19, 2008 at 03:37:59PM -0400, Jim Fulton wrote: > >>On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote: > >>... > >>>os.path.join(os.path.dirname(__file__), 'foo') just has too many > >>>problems. > >> > >>Could you be specific? > > > >I suppose I was thinking about the stale .pyc file problem, mostly. > > > >zc.buildout/setuptools/distutils (I'm not sure which part of the stack > >is to blame) likes to compile modules in a temporary directory and > >then > >move the .py and .pyc files to their final location. As a result, > >__file__ points to the now-nonexistent directory. > > You are mistaken. The file information in the code object is wrong, > which is wildly annoying when looking at traceback or using pdb, but > __file__ is correct. Oops, you're right. My brain stopped working (which often happens lately, just about when I decide to write an email or comment in a bug tracker. Late evenings and lack of sleep may have something to do with it.). I retract my statement. The only problem with os.path.join(os.path.dirname(__file__), 'foo') is that it's long and cumbersome to type (and doesn't work in zip files, but I don't much care about those). Marius Gedminas -- If it weren't for the last minute, nothing would get done. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Wed, Mar 19, 2008 at 12:54 PM, Phillip J. Eby <[EMAIL PROTECTED]> wrote: [a long message] I'm back at Google and *really* busy for another week or so, so I'll have to postpone the rest of this discussion for a while. If other people want to chime in please do so; if this is just a dialog between Phillip and me I might incorrectly assume that nobody besides Phillip really cares. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] The Breaking of distutils and PyPI for Python 3000?
Martin v. Löwis wrote: 1. What is the plan for PyPI when Python 3.0 comes out and dependencies start getting satisfied from distribution across the great divide, e.g. a 3.0-specific package pulls from PyPI a 2.x-specific package to meet some need? Are there plans to fork PyPI, apply special tags to uploads or what? I don't see the need to for PyPI. For packages (or "distributions", to avoid confusion with Python packages), I see two options: a) provide a single release that supports both 2.x and 3.x. The precise strategy to do so might vary. If one is going for a single source version, have setup.py run 2to3 (or perhaps 3to2). For dual-source packages, have setup.py just install the one for the right version; setup.py itself needs to be written so it runs on both versions (which is easy to do). b) switch to Python 3 at some point (i.e. burn your bridges). You seem to be implying that some projects may release separate source distributions. I cannot imagine why somebody would want to do that. While not quite to the same scale as the 2 to 3 transition, this problem seems like one that would generally already exist. If one writes code that uses newer 2.4/2.5 features (say decorators for example,) it won't build/run on 2.3 or earlier installs. How have people been handling that sort of situation? Is it only by not using the newer features or is there some situation I'm just not seeing in a brief review of a few projects pages on PyPI where there is only one source tarball? -- Dave ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote: ... > os.path.join(os.path.dirname(__file__), 'foo') just has too many > problems. Could you be specific? It has worked pretty well for us. It makes it hard to provide software in zip files, but that seems overrated. In particular, I recently noticed that it took significantly longer to start an app from zipped eggs than from unzipped eggs. And, of course, zipped eggs are a lot less convenient for looking at code. Jim -- Jim Fulton Zope Corporation ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 06:41 PM 3/19/2008 +, Paul Moore wrote: >I may just go for a simple registry of implementation classes. That's all simplegeneric *is*, in this context. A way of having "simple registries", since more than one was needed there. > Sorry, but if you >couldn't get sufficient agreement to simplegeneric to get it in a >"proper" location, It's a private implementation detail of pkgutil, not a stdlib feature, which is why no such agreement was requested or required. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mar 19, 2008, at 3:52 PM, Marius Gedminas wrote: > On Wed, Mar 19, 2008 at 03:37:59PM -0400, Jim Fulton wrote: >> On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote: >> ... >>> os.path.join(os.path.dirname(__file__), 'foo') just has too many >>> problems. >> >> Could you be specific? > > I suppose I was thinking about the stale .pyc file problem, mostly. > > > zc.buildout/setuptools/distutils (I'm not sure which part of the stack > is to blame) likes to compile modules in a temporary directory and > then > move the .py and .pyc files to their final location. As a result, > __file__ points to the now-nonexistent directory. You are mistaken. The file information in the code object is wrong, which is wildly annoying when looking at traceback or using pdb, but __file__ is correct. BTW, I'm pretty sure the bad code file information is my fault. I definitely plan to fix this. > Aside: I never understood the reason for storing the value of __file__ > in the .pyc/.pyo file itself. Why not set it to the correct value on > import time? Agreed. But this has no impact on resource loading. Jim -- Jim Fulton Zope Corporation ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 10:48 AM 3/19/2008 -0700, Guido van Rossum wrote: >I don't understand PyPI all that well; it seems poor design that the >browsing via keywords is emphasized but there is no easy way to >*search* for a keyword (the list of all packages is not emphasized >enough on the main page -- it occurs in the side bar but not in the >main text). I assume there's a programmatic API (XML-RPC?) but I >haven't found it yet. http://wiki.python.org/moin/CheeseShopXmlRpc There's also a REST API that setuptools uses: http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api The API was originally designed for screen-scraping an older version of PyPI, but that has been replaced with a "lite" version served from: http://pypi.python.org/simple/ The "lite" version is intended for tools such as easy_install to process, as it consists strictly of links and can be statically cached. Zope Corp., for example, maintains a static mirror of this API, to guard themselves against PyPI outages and slowdowns, since their buildouts can involve huge numbers of eggs, both their own and external dependencies. >I'd love it if you could write or point me to code that takes a >package name and optional version and returns the URL for the source >archive, and the type (in case it can't be guessed from the filename >or the Content-type header). You can probably do that with the XML-RPC API. There's a function to get the versions of a package, given a (case-sensitive) name, and there's a function to get information for uploaded archives, given a name and a version. I originally intended to use it for the PEP 365 approach, but you can get the necessary information in just one static roundtrip using the REST (/simple) HTML API, if you're willing to parse the URLs for version information. (The catch of course being that distutils source distributions don't have unambiguously parseable filenames.) >Hm. Why not just use the existing convention for running setup.py >after unpacking? This works great in my experience, and has the >advantage of having an easy fallback if you end up having to do this >manually for whatever reason. Because I want bootstrap-ees to be able to use the bootstrap mechanism. For example, I expect at some point that setuptools will use other, non-self-contained packages, and other package managers such as zc.buildout et al also want to depend on setuptools without bundling it. > > * calling the bootstrap module 'bootstrap', as in 'python -m > > bootstrap projectname optionalversion'. The module would expose an > > API to allow it to be used programmatically as well as the command > > line, so that bootstrapped packages can use the bootstrap process to > > locate dependencies if they so desire. (Today's package management > > tools, at least, are all based on setuptools, so if it's not present > > they'll need to download that before beginning their own > > bootstrapping process.) > >This sounds like going beyond bootstrapping. My vision is that you use >the bootstrap module (with the command line you suggest above) once to >install setuptools or the alternate package manager of your choice, >and then you can use easy_install (or whatever alternative) to install >the rest. Well, I noticed that the other package managers were writing bootstrap scripts that then download setuptools' bootstrap script and run it as part of *their* bootstrap process... and then I got to thinking that it sure would be nice for setuptools to not have to be a giant monolithic download if I wanted to start using other packages in it... and that it sure would be nice to get rid of all these bootstrap scripts downloading other bootstrap scripts... and then I wrote PEP 365. :) One other thing that PEP 365 does for these use cases that your approach doesn't, is that pkg_resources could detect whether a desired package of a usable version was *already* installed, and skip it if so. So, we've already scaled back the intended use cases quite a bit, as people will have to write their own "is it already there?" and "is it the right version?" checks. > > Without one or the other, the bootstrap tool would have to grow a > > version parsing scheme of some type, and play guessing games with > > file extensions. (Which is one reason I limited PEP 365's scope to > > downloading eggs actually *uploaded* to PyPI, rather than arbitrary > > packages *linked* from PyPI.) > >There are two version parsers in distutils, referenced by PEP 345, the >PyPI 1.2 metadata standard. Yes, and StrictVersion doesn't parse release candidates. And neither LooseVersion nor StrictVersion supports handling multiple pre/post-release tags correctly. (E.g. "1.1a1dev-r2753") > > So, if I had to propose something right now, I would be inclined > to propose: > > > > * using setuptools' version parsing semantics for interpretation of > > alpha/beta/dev/etc. releases > >Can you point me to the code fo
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Wed, Mar 19, 2008 at 03:37:59PM -0400, Jim Fulton wrote: > On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote: > ... > >os.path.join(os.path.dirname(__file__), 'foo') just has too many > >problems. > > Could you be specific? I suppose I was thinking about the stale .pyc file problem, mostly. zc.buildout/setuptools/distutils (I'm not sure which part of the stack is to blame) likes to compile modules in a temporary directory and then move the .py and .pyc files to their final location. As a result, __file__ points to the now-nonexistent directory. Aside: I never understood the reason for storing the value of __file__ in the .pyc/.pyo file itself. Why not set it to the correct value on import time? Marius Gedminas -- If you have to run heating in winter, you don't own enough computers. signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Wed, Mar 19, 2008 at 11:34 AM, Paul Moore <[EMAIL PROTECTED]> wrote: > On 19/03/2008, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > On Mon, Mar 17, 2008 at 2:19 PM, zooko <[EMAIL PROTECTED]> wrote: > > > 4. The standard Python library includes a tool to find and read > > > resources (other than Python modules) that came bundled in a Python > > > package. > > I think we're pretty close to this already. PEP 302 defines a > > getdata() method. Hopefully most PEP 302 implementations support it. > > The only thing missing IMO is a little function that does what > > getdata() does when there is no __loader__ object (i.e. when the > > default "import-from-filesystem" import method is used). > > I'm currently working on an addition to pkgutil to provide this type > of function. I'm considering going a little further (adding functions > to get a file-like object, test for existence, and list available > resources, modelled on the pkg_resources functions - but these extra > ones are not supported by the loader protocol, so I'm undecided as to > whether it's worth it, I'll see how complex the code gets). I'd only do what __loader__ offers. People can always wrap a StringIO around it. > Once I have a patch, I'll post it to the tracker. What's the best > approach? Code a patch for 3.0 and backport, or code for 2.6 and let > the merging process do its stuff? Code for 2.6, let the merge do its thing. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby wrote: At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: I'd be willing to help out, and keep a carefully balanced hand in what is accepted. I'm not sure exactly how to go about such a handoff though. My guess is that we need a bug/patch tracker, and a few people to review, test, and apply. Maybe a transitional period during which I just say yea or nay and let others do the test and apply, before opening it up entirely. That way, we can perhaps solidify a few principles that I'd like to have stay in place. (Like no arbitrary post-install code hooks.) +1 to blessing more people to commit. +1 to the transition period idea. These two ought to enable things to move a bit quicker than taking a year to accept a patch. :-) In addition to a bug tracker and patch manager, seems like perhaps a wiki to help document some of these solidified principles and other notes would be a good thing. (Like a patch should almost always include at least one test, possibly more.) Given that the source for setuptools is in the python.org svn, couldn't we just use the python.org roundup and wiki for these facilities? Though looking at the list of components, it seems that things in the sandbox generally aren't tracked in this infrastructure. In which case, I'm sure we could use sf, launchpad, or some such external provider. Enthought could even host this stuff. Like Jeff Rush, I'm also willing to help out as both a writer and reviewer of patches. As you can see from my earlier posts there are a number of things (besides running an arbitrary post-install script) that we'd like to be able to get into the codebase. -- Dave ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 19/03/2008, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > So that future PEP 302 importers can define support for these > features, please use > pkgutil.simplegeneric to implement these, following the pattern set > out by other pkgutil functions. E.g.: > > def resource_listdir(package, resource_name): > return loader_listdir(get_loader(package), package, resource_name) > > Where 'loader_listdir' is a simplegeneric with default handlers > registered for the built-in/stdlib loader types (zipimporter, None, > and ImpLoader). > > If you do this, then new importer/loader types can use > loader_listdir.register() to support the functionality, without > monkeypatching or other hacks. If I did this, I'd certainly make sure it was possible for other loaders to add their own implementations. But I'm not sure I want to get involved in the generic-function politics involved in using pkgutil.simplegeneric - particularly as it's undocumented. I may just go for a simple registry of implementation classes. Sorry, but if you couldn't get sufficient agreement to simplegeneric to get it in a "proper" location, I doubt I'd fare better, (Of course, if Guido pipes up with support for using simplegeneric, I'll go with that - and document it, and propose that it gets moved to a sensible location. But I don't want to be the one to try to persuade Guido...) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 05:04 PM 3/19/2008 +, Paul Moore wrote: >On 19/03/2008, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > > It's so that you can get loaders for modules that aren't imported yet > > -- and your code would need to handle this case too. > >Well, technically, it need not, as I don't *need* to implement the >exact functionality that pkg_resources does. My (personal) goal is to >standardise the functionality, not to cater for every possible use >case. > > > (You could handle it by actually going ahead and importing the module, as > > pkg_resources does, but there are other tools using pkgutil to e.g. > > inspect as-yet-unimported modules.) > >I'm not sure what you mean by that. There are no tools using >pkgutil.resource_stream, as it doesn't exist yet. So whether it >imports the module or not cannot be relevant (although it should be >documented). You asked why pkgutil.get_loader() was more complex than you'd expect. That's the question I answered, in both of the snippets you quoted above. That is, when I said, "your code would need to handle this case", I meant "your code for get_loader()", not "your code for resource_foo() methods". IOW, I was trying to encourage you to use get_loader() as-is rather than modifying or replacing it with something "simpler" that would break other uses of get_loader() in the stdlib. > > You don't really need to, because even though they're technically > > "undocumented", the intent was for them to be published APIs (apart > > from simplegeneric, which is an implementation detail). The docs > > just never got copied to the official docs. > >If it's just a matter of copying documentation from the PkgResources >page, that's no issue. However, only get_importer is documented on the >PkgResources page. I meant, the documentation was never copied from the *docstrings* of these functions in pkgutil; as you noted, not all of them are direct ports from pkg_resources. I refactored the framework there to include support for pyrun and pydoc's use cases as well. It's true that some additional documentation is needed, too, for anyone who wants to implement a PEP 302 loader or importer that wants to support enhanced features like listing modules or getting resources. >On reviewing the pkg_resources APIs, I only see the following that >look worth porting to pkgutil: > > * resource_exists(package, resource_name) > * resource_stream(package, resource_name) > * resource_string(package, resource_name) > * resource_isdir(package, resource_name) > * resource_listdir(package, resource_name) > >... > >Hmm, these go way beyond the simple (and optional) loader.get_data >interface from PEP 302. And yet the pkg_resources machinery is far >more complex than I'd want to see in the stdlib. Maybe I'll just >reinvent the wheel and see how far I get :-) So that future PEP 302 importers can define support for these features, please use pkgutil.simplegeneric to implement these, following the pattern set out by other pkgutil functions. E.g.: def resource_listdir(package, resource_name): return loader_listdir(get_loader(package), package, resource_name) Where 'loader_listdir' is a simplegeneric with default handlers registered for the built-in/stdlib loader types (zipimporter, None, and ImpLoader). If you do this, then new importer/loader types can use loader_listdir.register() to support the functionality, without monkeypatching or other hacks. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 19/03/2008, Guido van Rossum <[EMAIL PROTECTED]> wrote: > On Mon, Mar 17, 2008 at 2:19 PM, zooko <[EMAIL PROTECTED]> wrote: > > 4. The standard Python library includes a tool to find and read > > resources (other than Python modules) that came bundled in a Python > > package. > > I think we're pretty close to this already. PEP 302 defines a > getdata() method. Hopefully most PEP 302 implementations support it. > The only thing missing IMO is a little function that does what > getdata() does when there is no __loader__ object (i.e. when the > default "import-from-filesystem" import method is used). I'm currently working on an addition to pkgutil to provide this type of function. I'm considering going a little further (adding functions to get a file-like object, test for existence, and list available resources, modelled on the pkg_resources functions - but these extra ones are not supported by the loader protocol, so I'm undecided as to whether it's worth it, I'll see how complex the code gets). Once I have a patch, I'll post it to the tracker. What's the best approach? Code a patch for 3.0 and backport, or code for 2.6 and let the merging process do its stuff? Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Tue, Mar 18, 2008 at 3:36 PM, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 03:43 PM 3/18/2008 -0500, Guido van Rossum wrote: > >Only very few people would care about writing a setup > >script that works with this bootstrap module; basically only package > >manager implementers. > > That's true today, sure, but as soon as it is widely available, > others are sure to want to use it too. I just want a bright-line > distinction between what is and isn't bootstrappable, rather than a > murky region of "maybe, if you're not doing anything too complicated". How about "anything that uses only distutils in its setup.py and doesn't have external dependencies"? See a (horribly incomplete) prototype I added as sandbox/bootstrap/bootstrap.py. I wrote this on the plane last night and have only tested it with file:/// URLs; it needs to add the ability to consult PyPI to find the download URL, and probably more. (PS: just now I also managed to successfully install setuptools from source by giving it the URL to the gar.gz file.) > >There seems to be a misunderstanding about what I am proposing we do > >instead. The boostrap installer should only be powerful enough to > >allow it to be used to install a real package manager like setuptools. > > Which is why PEP 365 proposed only downloading an archive to a cache > directory, and optionally running something from it. It explicitly > disavows "installation" of anything, since the downloaded archive > wouldn't have been added to sys.path except for the duration of the > bootstrap process, and no scripts were to be installed. (Indeed, > apart from the methods it would have used to locate the archive on > PyPI, and to determine what to run from inside it, there was nothing > particularly egg-specific about the proposed bootstrapping process.) My bootstrap.py does exactly that: it downloads and unzips/untars a file and runs its setup.py with "install" as the only command line argument. (It currently looks for setup.py at the toplevel and one level deep in the unpacked archive.) Of course you will likely have to be root or administrator to run it effectively. > So, to fully egg-neutralize the bootstrapping approach, we need only > know how to locate an appropriate archive, and how to determine what > to run from it. Right. > For the latter, we could use the already-in-2.6 convention of running > __main__ from a zipfile or directory. (Too bad distutils source > distributions have an extra directory name embedded in them, so one > can't just execute them directly. Otherwise, we could've just let > people drop in a __main__.py next to setup.py. OTOH, maybe it would > be enough to use setuptools' algorithm for finding setup.py to locate > __main__.py, and I'm fairly sure *that* can be briefly expressed in the PEP.) What's wrong with just running "setup.py install"? I'd rather continue existing standards / conventions. Of course, it won't work when setup.py requires setuptools; but "old style" setup.py files that use only distutils work great (I managed to install Django from a file:/// URL). > The other open question is a naming convention and version detection, > so that the bootstrap tool can identify which of the files listed on > PyPI is suitable for its use. (Both with regard to the version > selection, and file type.) However, if PyPI were to grow support for > designating the appropriate files and/or versions in some other way, > we wouldn't need a naming convention as such. I don't understand PyPI all that well; it seems poor design that the browsing via keywords is emphasized but there is no easy way to *search* for a keyword (the list of all packages is not emphasized enough on the main page -- it occurs in the side bar but not in the main text). I assume there's a programmatic API (XML-RPC?) but I haven't found it yet. > Without one or the other, the bootstrap tool would have to grow a > version parsing scheme of some type, and play guessing games with > file extensions. (Which is one reason I limited PEP 365's scope to > downloading eggs actually *uploaded* to PyPI, rather than arbitrary > packages *linked* from PyPI.) There are two version parsers in distutils, referenced by PEP 345, the PyPI 1.2 metadata standard. > So, if I had to propose something right now, I would be inclined to propose: > > * using setuptools' version parsing semantics for interpretation of > alpha/beta/dev/etc. releases Can you point me to the code for this? What is its advantage over distutils.version? > * having a bdist_bootstrap format that's essentially a bdist_dumb > .zip file with the internal path prefixes stripped off, making it an > importable .zip with a different file extension. (Or maybe just > .pyboot.zip?) The filename convention would use setuptools' > canonicalization and escaping of names and version numbers, to allow > unambiguous machine parsing of the filename. A __main__ module would > have to be pr
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby writes: > >7. Many wanted to ability to install files anywhere in the install tree and > > not just under the Python package. Under distutils this was possible > > but > > it was removed in setuptools for security reasons. > > It wasn't security, it was manageability. Egg-based installation > means containment, (analagous to GNU stow) and therefore portability > and disposability of plugins. (Which again is what eggs were really > developed for in the first place.) defining containment this way doesn't help when preparing eggs for inclusion in a linux distribution. E.g. users on these distributions are used to find log files in /var/log (maybe in a subdir), documentation in /usr/share/doc/. You probably will get different views about manageability depending on your background (used to linux distribution standards or used to standards set by setuptools/cheeseshop). Packagers currently move these files manually to the standard locations and often have to keep symlinks in the egg dirs to these locations. Installation on linux distributions is handled by existing package tools which is unlikely to change. So it would be nice to find a common layer which can be used for both distribution methods, optionally enabling this with some kind of option like --install-files-in-places-not-handled-by-setuptools ;) Matthias ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
On Wed, Mar 19, 2008 at 1:05 PM, Matthias Klose <[EMAIL PROTECTED]> wrote: > - It would be useful if setuptools could handle separate build and > install steps like most configure/make/make install systems do. Access > to external resources should optionally be disabled during a build. What's wrong with "python setup.py bdist_egg"? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mon, Mar 17, 2008 at 2:19 PM, zooko <[EMAIL PROTECTED]> wrote: > 4. The standard Python library includes a tool to find and read > resources (other than Python modules) that came bundled in a Python > package. > > Consider, for example, this snippets of code in Nevow: > > http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786#L10 > http://divmod.org/trac/browser/trunk/Nevow/setup.py?rev=13786 > http://divmod.org/trac/browser/trunk/Nevow/setup_egg.py?rev=2406 > > When Nevow uses pkg_resources to import its files such as > "default.css", then it is able to find at runtime, even if is being > imported from a py2exe or py2app zip, or on other platforms where its > homegrown setup script and homegrown "find my file" function fail. > So using pkg_resources (and setuptools to install it) makes > "test_nevow" pass on all of the allmydata.org buildslaves: > > http://allmydata.org/buildbot/waterfall?show_events=false I think we're pretty close to this already. PEP 302 defines a getdata() method. Hopefully most PEP 302 implementations support it. The only thing missing IMO is a little function that does what getdata() does when there is no __loader__ object (i.e. when the default "import-from-filesystem" import method is used). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Jeff Rush writes: > I was in a Packaging BoF yesterday and, although not very relevant to the > packager bootstrap thread, Guido has asked me to post some of the concerns. We did address many topics on both days, I added the following topics which were addressed on the Friday BoF only, see http://wiki.python.org/moin/PackagingBOF - Linux distributions try to ship only one version of a package/egg/module in one release, only shipping more than one version if necessary. eggs (as least as shipped with Debian, Fedora, Ubuntu) are all built using --single-version-externally-managed. - import foo should work wether installed as an egg or installed with distutils, and without using pkg_resources.require - pkg_resources should handle the situation of one egg version installed as --single-version-externally-managed (default version) and one or more eggs installed not using --single-version-externally-managed. Currently these additional versions cannot be imported. - It would be useful if setuptools could handle separate build and install steps like most configure/make/make install systems do. Access to external resources should optionally be disabled during a build. - The idea was brought up to use a to-be-defined api-version to describe dependencies between eggs. Version numbers are generally used for more than api changes; the idea follows existing practice for shared object names, only changing when the API is changed. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 19/03/2008, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > It's so that you can get loaders for modules that aren't imported yet > -- and your code would need to handle this case too. Well, technically, it need not, as I don't *need* to implement the exact functionality that pkg_resources does. My (personal) goal is to standardise the functionality, not to cater for every possible use case. > (You could handle it by actually going ahead and importing the module, as > pkg_resources does, but there are other tools using pkgutil to e.g. > inspect as-yet-unimported modules.) I'm not sure what you mean by that. There are no tools using pkgutil.resource_stream, as it doesn't exist yet. So whether it imports the module or not cannot be relevant (although it should be documented). > You don't really need to, because even though they're technically > "undocumented", the intent was for them to be published APIs (apart > from simplegeneric, which is an implementation detail). The docs > just never got copied to the official docs. If it's just a matter of copying documentation from the PkgResources page, that's no issue. However, only get_importer is documented on the PkgResources page. On reviewing the pkg_resources APIs, I only see the following that look worth porting to pkgutil: * resource_exists(package, resource_name) * resource_stream(package, resource_name) * resource_string(package, resource_name) * resource_isdir(package, resource_name) * resource_listdir(package, resource_name) For all of these, the first argument would be restricted to a package/module name. The option of specifying a "requirement" is not suitable. Also, automatic extraction and resource_filename is far beyond what I see as sensible in the stdlib. Hmm, these go way beyond the simple (and optional) loader.get_data interface from PEP 302. And yet the pkg_resources machinery is far more complex than I'd want to see in the stdlib. Maybe I'll just reinvent the wheel and see how far I get :-) Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 09:43 AM 3/19/2008 +, Paul Moore wrote: >On 19/03/2008, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > > >I don't particularly want to use undocumented functions - even within > > >the module that defines them. > > > > Er, you could always document them, you know. I just didn't get > > around to it before the whole "setuptools flap of 2005" wiped out my > > motivation to do any further work on Python 2.5. For the most part, > > they do have docstrings. I just never did the LaTeX work on > > them. And now, you can use reST instead of LaTeX. :) > >I could, but the problem is I don't really follow the code. My >motivation is to add useful functions, not document stuff that's >already there. You mentioned using get_loader to implement >resource_string. OK, but I'd have done something a lot simpler than >the code of get_loader, and I don't understand why the code in there >is necessary. It's so that you can get loaders for modules that aren't imported yet -- and your code would need to handle this case too. (You could handle it by actually going ahead and importing the module, as pkg_resources does, but there are other tools using pkgutil to e.g. inspect as-yet-unimported modules.) >Never mind, I'll implement what I'm planning on using my own code, and >ignore trying to understand the (corner cases of the) undocumented >functions. You don't really need to, because even though they're technically "undocumented", the intent was for them to be published APIs (apart from simplegeneric, which is an implementation detail). The docs just never got copied to the official docs. These APIs are also actively used by at least pydoc and pyrun in Python 2.5, so they're unlikely to go away or break. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Question on using distutils.core.run_setup
At 07:00 AM 3/19/2008 +0100, Terry Jones wrote:
> > "Phillip" == Phillip J Eby <[EMAIL PROTECTED]> writes:
>Phillip> It'd be on the 'install_lib' command instance, not the
>Phillip> distribution. Try
>Phillip> dist.get_finalized_command('install_lib').install_dir instead.
> >>
> >> For the record / archives, I think this should be
> >> dist.get_command_obj('install_lib').install_dir
>
>Phillip> No, it shouldn't. Check again. :)
>
>I get this (using the simplistic direct invocation of run_setup):
>
> $ python
> Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
> [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from distutils.core import run_setup
> >>> d = run_setup('setup.py', ['-q', 'install'])
> >>> d.get_finalized_command('install_lib').install_dir
> Traceback (most recent call last):
> File "", line 1, in
> AttributeError: Distribution instance has no attribute
> 'get_finalized_command'
> >>> d.get_command_obj('install_lib').install_dir
>
>'/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/'
>
>I guess I'm missing something here?
Whoops, my bad. 'get_finalized_command' is a distutils *command
object* method, not a distribution method. I usually work with code
in command objects much more than with distribution objects.
The difference, though, between get_finalized_command() and
get_command_obj() is that the latter is not guaranteed to return a
*finalized* command, i.e., one that has processed its config file and
command-line options. I would personally call .ensure_finalized() on
the command object before checking its install_dir. (This is done
automatically by cmd.get_finalized_command(), which is why I usually
use it, hence the confusion.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: >Are you open to giving certain others patch view/commit privileges >to setuptools? Jim Fulton has such already. I'm open to extending that to others who have a good grasp of the subtleties involved. Truthfully, if we can just get 0.6 put to bed, I could probably open up the trunk a lot wider. One of the things that slows me down is that patches usually don't come with tests, so I usually have to manually smoke-test them for scenarios I think they'll effect. There isn't really any automated procedure. Probably the most frustrating thing (or "chief amongst the most frustrating things") about setuptools development is that it's a black hole. By which I mean that backward compatibility and cruft accretion make it difficult to get out of. In the beginning, there was the distutils. Distutils begat setuptools, and setuptools begat virtualenv and zc.buildout and source control plugins. Etc., etc. What I think is really needed in the long run is to keep eggs, but get rid of setuptools and the distutils in their current form. There's a lot of brokenness there, and also a lot of accumulated cruft. We really need a distutils 3000, and it needs to be built on a better approach. In truth, my *real* motivation for PEP 365's bootstrap tool isn't so much to support the package management tools we have today, as it is to support a new one tomorrow. I have a few ideas for ways to shift the paradigm of how individual projects get built, to incorporate many scenarios that don't work well now. But to implement those things in such a next-generation tool, I will not want to be restricted to just what's in the stdlib or what can be bundled in the tool. (Btw, by "real" motivation, I don't mean I've been deceptive about my intentions, I mean that my strong intuition that such a bootstrap facility is needed, is probably being fueled by the long term desire to replace the entire distutils-based infrastructure with something better.) > I'd be willing to help out, and keep a carefully balanced hand in > what is accepted. And I think it's probably getting close to time I stepped down from day-to-day management of the codebase (which is more like month-to-month or quarter-to-quarter for me lately). It will probably be a lot easier for me to step back and critique stuff that goes in, after the fact, than to go over the stuff beforehand. :) I'm not sure exactly how to go about such a handoff though. My guess is that we need a bug/patch tracker, and a few people to review, test, and apply. Maybe a transitional period during which I just say yea or nay and let others do the test and apply, before opening it up entirely. That way, we can perhaps solidify a few principles that I'd like to have stay in place. (Like no arbitrary post-install code hooks.) btw, offtopic question: are you by any chance the same Jeff Rush who invented EchoMail? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 19, 2008, at 3:57 AM, Jeff Rush wrote: > > I and others appreciate your call for more patches on various > topics. However > a long delay in applying them will discourage contribution. Are you > open to > giving certain others patch view/commit privileges to setuptools? > I'd be > willing to help out, and keep a carefully balanced hand in what is > accepted. The Python sandbox has a setuptools directory. Is this the canonical location for the code? If so, then anybody who has Python commit privileges can commit to it and help further develop setuptools. If not, why not and what is the sandbox setuptools used for? - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR+Eca3EjvBPtnXfVAQLabwP9F8NtQX6YsDXJMHiByCGILPAQ2NgtaIzg en6yYbhl5IAweTr0DtWzxRXjSGMifK/D4PmtRSWWUTy3VY+8cRUkYuBjIxPOHJRF 4TA4dYoW4f2+qM1IO/l59FIAJgUyrXKhv3aznpXBFl+PaRCW9qP9G1lur3lolipB h4i8ya+I7zU= =2/iq -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] The Breaking of distutils and PyPI for Python 3000?
On 19/03/2008, Jeff Rush <[EMAIL PROTECTED]> wrote: > 1. What is the plan for PyPI when Python 3.0 comes out and > dependencies start getting satisfied from distribution > across the great divide, e.g. a 3.0-specific package > pulls from PyPI a 2.x-specific package to meet some > need? As distutils (and core Python) doesn't do any automatic dependency management, this is a setuptools issue. As such, it's up to setuptools to deal with it. There may be infrastructure changes that would be generally useful, but there's nothing *needed* for the core. > 2. There have been attempts over the years to fix distutils, > with the last one being in 2006 by Anthony Baxter. He > stated that a major hurdle was the strong demand to > respect backward compatibility and he finally gave up. > One of the purposes of Python 3.0 was the freedom to > break backward compatibility for the sake of "doing > the right thing". So is it now permissible to give > distutils a good reworking and stop letting > compatibility issues hold us back? Sounds reasonable. I'm sure patches would be considered, but past discussions around "including setuptools" have been controversial and generally not reached consensus (for reasons other than pure backward compatibility). Also, while compatibility isn't as important for 3.0, smooth migration *is* - so any incompatible proposal must include some consideration of how to assist people with huge, complex setup.py files which use distutils internals in complex ways. So be prepared to do some work :-) (But I'd be happy to see distutils improved. I just don't have any need for such improvement, personally). Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 19/03/2008, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > >I don't particularly want to use undocumented functions - even within > >the module that defines them. > > Er, you could always document them, you know. I just didn't get > around to it before the whole "setuptools flap of 2005" wiped out my > motivation to do any further work on Python 2.5. For the most part, > they do have docstrings. I just never did the LaTeX work on > them. And now, you can use reST instead of LaTeX. :) I could, but the problem is I don't really follow the code. My motivation is to add useful functions, not document stuff that's already there. You mentioned using get_loader to implement resource_string. OK, but I'd have done something a lot simpler than the code of get_loader, and I don't understand why the code in there is necessary. Never mind, I'll implement what I'm planning on using my own code, and ignore trying to understand the (corner cases of the) undocumented functions. Looking at the pkg_resources code, I'd have had the same issues there trying to disentangle the "simple bits" from the infrastructure, as you suggested, so that's probably a better approach anyway. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] The Breaking of distutils and PyPI for Python 3000?
As I'm digging into packaging issues here at PyCon, a couple of Python 3000 related matters occur to me. As I'm new to the Python 3000 development, if these have already been addressed in prior discussions, I apologize for your time. 1. What is the plan for PyPI when Python 3.0 comes out and dependencies start getting satisfied from distribution across the great divide, e.g. a 3.0-specific package pulls from PyPI a 2.x-specific package to meet some need? Are there plans to fork PyPI, apply special tags to uploads or what? While binary distributions are tagged with the Python version, source distributions are not. And of course a dependency expression as it stands today for "SomePackage > 2.4" may pull 3.0 to satisfy it. 2. There have been attempts over the years to fix distutils, with the last one being in 2006 by Anthony Baxter. He stated that a major hurdle was the strong demand to respect backward compatibility and he finally gave up. One of the purposes of Python 3.0 was the freedom to break backward compatibility for the sake of "doing the right thing". So is it now permissible to give distutils a good reworking and stop letting compatibility issues hold us back? -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby wrote: > > I'm actually happy to hear that there's this much energy available -- > hopefully some of it can be harnessed towards positive solutions. > > When I began developing setuptools, I often asked for the input of > packagers, developers, etc., through the distutils-sig... and was met > with overwhelming silence. So the fact that there is now a group of > people who are ready to work for some solutions seems like a positive > change, to me. I can appreciate how frustrating silence is when you call for input. Let's see if we can keep the volunteer energy going this time around. > It's hard to make design decisions regarding itches you don't personally > have, and which other people won't help scratch. Unfortunately, a lot > of the proposals from packaging system people have been of the form of, > "fix this for us by breaking things for other people". Not all of them, > though. Many have been very helpful, contributing troubleshooting help > and good patches. > > That some of those good patches took nearly a year to get into > setuptools (some from Fedora just got into 0.6c8 that were sent to me > almost a year ago) is because I'm the only person reviewing setuptools > patches, and I've spent only a few days in the last year doing focused > development work on setuptools (as opposed to answering questions about > it on the SIG). > > It's never a good thing when people's patches sit around, regardless of > where they come from. But that's not the same thing as *rejecting* the > patches. I and others appreciate your call for more patches on various topics. However a long delay in applying them will discourage contribution. Are you open to giving certain others patch view/commit privileges to setuptools? I'd be willing to help out, and keep a carefully balanced hand in what is accepted. -Jeff ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
