Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
At 05:32 AM 3/6/2010, Brian Quinlan wrote: Using twisted (or any other asynchronous I/O framework) forces you to rewrite your I/O code. Futures do not. Twisted's "Deferred" API has nothing to do with I/O. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
At 01:19 AM 3/6/2010, Jeffrey Yasskin wrote: On Fri, Mar 5, 2010 at 10:11 PM, Phillip J. Eby wrote: > I'm somewhat concerned that, as described, the proposed API ... [creates] yet another alternative (and > mutually incompatible) event loop system in the stdlib ... Futures are a blocking construct; they don't involve an event loop. And where they block is in a loop, waiting for events (completed promises) coming back from other threads or processes. The Motivation section of the PEP also stresses avoiding reinvention of such loops, and points to the complication of using more than one at a time as a justification for the mechanism. It seems relevant to at least address why wrapping multiprocessing and multithreading is appropriate, but *not* dealing with any other form of sync/async boundary, *or* composition of futures. On which subject, I might add, the PEP is silent on whether executors are reentrant to the called code. That is, can I call a piece of code that uses futures, using the futures API? How will the called code know what executor to use? Must I pass it one explicitly? Will that work across threads and processes, without explicit support from the API? IOW, as far as I can tell from the PEP, it doesn't look like you can compose futures without *global* knowledge of the application... and in and of itself, this seems to negate the PEP's own motivation to prevent duplication of parallel execution handling! That is, if I use code from module A and module B that both want to invoke tasks asynchronously, and I want to invoke A and B asynchronously, what happens? Based on the design of the API, it appears there is nothing you can do except refactor A and B to take an executor in a parameter, instead of creating their own. It seems therefore to me that either the proposal does not define its scope/motivation very well, or it is not well-equipped to address the problem it's setting out to solve. If it's meant to be something less ambitious -- more like a recipe or example -- it should properly motivate that scope. If it's intended to be a robust tool for composing different pieces of code, OTOH, it should absolutely address the issue of writing composable code... since, that seems to be what it says the purpose of the API is. (I.e., composing code to use a common waiting loop.) And, existing Python async APIs (such as Twisted's Deferreds) actually *address* this issue of composition; the PEP does not. Hence my comments about not looking at existing implementations for API and implementation guidance. (With respect to what the API needs, and how it needs to do it, not necessarily directly copying actual APIs or implementations. Certainly some of the Deferred API naming has a rather, um, "twisted" vocabulary.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 3148] futures - execute computations asynchronously
At 01:03 AM 3/5/2010, Brian Quinlan wrote: Hi all, I recently submitted a daft PEP for a package designed to make it easier to execute Python functions asynchronously using threads and processes. It lets the user focus on their computational problem without having to build explicit thread/process pools and work queues. The package has been discussed on stdlib-sig but now I'd like this group's feedback. My immediate reaction is that this would be a lot more useful if it built on an API for coroutine yielding/interaction, similar to what's in say, Eventlet. That would seem to make it easier to write synchronous-looking code that operates on futures, and allow futures to be composed more cleanly. ISTM that if futures were standardized before a coroutine API, it would lead to effective orphaning ala what happened with asyncore, especially since the new library is, well, new. I'm somewhat concerned that, as described, the proposed API adds little over what's relatively easy to do with a mature coroutine framework like Eventlet, while at the same time creating yet another alternative (and mutually incompatible) event loop system in the stdlib, beyond the ones that are already in asyncore, tkinter, and the various SocketServer subclasses. As far as naming goes, Twisted uses the term "Deferred" for this concept (and also has a very mature API for handling them). And speaking of Twisted, it seems to me that the PEP would be much improved in general by learning from some of the lessons of other systems. I don't think that Java's example is really the best one to follow in this instance, compared to the many existing async frameworks that have Python-specific experience and APIs to learn from. Or, to put it another way, something that worries me about this PEP is that nearly all of its Python-related citations are for *discussions* of futures, with the only previous Python implementation cited being a crude sketch of a cookbook recipe. The PEP also doesn't address questions of interoperability with existing solutions, compare features with them, or even so much as say, "There are other production implementations of this concept in Python, but we are going to pretend they don't exist." ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wart which should have been repaired in 3.0?
At 08:57 PM 12/30/2008 -0600, s...@pobox.com wrote: Phillip> At 02:32 PM 12/30/2008 -0800, Scott David Daniels wrote: >> More trouble with the "just take the dirname": >> >> paths = ['/a/b/c', '/a/b/d', '/a/b'] >> os.path.dirname(os.path.commonprefix([ >> os.path.normpath(p) for p in paths])) >> >> give '/a', not '/a/b'. Phillip> ...because that's the correct answer. I don't understand. If you search for os.path.commonprefix at codesearch.google.com you'll find uses like this: if os.path.commonprefix([basedir, somepath]) != basedir: ... which leads me to believe that other people using the current function in the real world would be confused by your interpretation. It never would've occurred to me to use it for that, versus checking for somepath.startswith(basedir+sep). The only thing I've ever used commonprefix for is to find the most-specific directory that contains all the specified paths. Never occurred to me that there was any other use for it, actually. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wart which should have been repaired in 3.0?
At 09:30 PM 12/30/2008 -0500, rdmur...@bitdance.com wrote: On Tue, 30 Dec 2008 at 17:51, Phillip J. Eby wrote: At 02:32 PM 12/30/2008 -0800, Scott David Daniels wrote: More trouble with the "just take the dirname": paths = ['/a/b/c', '/a/b/d', '/a/b'] os.path.dirname(os.path.commonprefix([ os.path.normpath(p) for p in paths])) give '/a', not '/a/b'. ...because that's the correct answer. But not the answer that is wanted. So the challenge now is to write a single expression that will yield '/a/b' when passed the above paths list, and also produce '/a/b' when passed the following paths list: paths = ['/a/b/c', '/a/b/cd'] Change that to [os.path.normpath(p)+'/' for p in paths] and you've got yourself a winner. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wart which should have been repaired in 3.0?
At 02:32 PM 12/30/2008 -0800, Scott David Daniels wrote: More trouble with the "just take the dirname": paths = ['/a/b/c', '/a/b/d', '/a/b'] os.path.dirname(os.path.commonprefix([ os.path.normpath(p) for p in paths])) give '/a', not '/a/b'. ...because that's the correct answer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wart which should have been repaired in 3.0?
At 06:14 AM 12/30/2008 -0600, s...@pobox.com wrote: Paul demonstrates the shortcoming of commonprefix: >>> os.path.commonprefix(["foo\\bar\\baz", "foo/bar/boink"]) 'foo' With the patch in issue4755: >>> import ntpath >>> ntpath.commonpathprefix(["foo\\bar\\baz", "foo/bar/boink"]) 'foo\\bar' But it doesn't handle the fact that Windows paths are case-insensitive, or that Posix paths can have symlinks... or that one path might be relative and another absolute... As soon as you move away from being a string operation, you get an endless series of gotchas... none of which are currently documented. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wart which should have been repaired in 3.0?
You know, all this path separator and list complication isn't really necessary, when you can just take the os.path.dirname() of the return from commonprefix(). Perhaps we could just add that recommendation to the docs? At 04:46 PM 12/29/2008 -0600, s...@pobox.com wrote: Jeff> For those that prefer not to add functions all willy-nilly, would Jeff> it not be better to add a "delimiter" keyword that defaults to Jeff> False? Then "delimiter=False" will function with the current Jeff> functionality unchanged while Jeff> os.path.commonprefix(["bob/export/home", "bob/etc/passwd"], delimiter = "/") Jeff> would properly return Jeff> 'bob/' On Windows what would you do with this crazy, but valid, path? c:/etc\\passwd I don't do Windows, so don't have any idea if there is even an /etc/passwd file on Windows. I'd guess not, but that's not the point. The point is that you can use both / (aka ntpath.sep) and \ (aka ntpath.altsep) in Windows pathnames. See my patch (issue 4755) for a version of os.path. which works as at least I expect and should work cross-platform. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/pje%40telecommunity.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should there be a way or API for retrieving from a code object a loader method and package file where the code comes from?
At 04:00 PM 12/23/2008 +, Paul Moore wrote: PPS Seriously, setuptools and the adoptions of eggs has pushed a lot of code to be much more careful about unwarranted assumptions that code lives in the filesystem. That's an incredibly good thing, and very hard to do right (witness the setuptools "zip_safe" parameter which acts as a get-out clause). Much kudos to setuptools for getting as far as it has. And ironically, if I ever get the time to actually work on a new version of easy_install (as opposed to perpetually tweaking the old one), the default zipping and default sys.path munging will be among the first things to go. ;-) Ironically, my choice of isolated directories and zipfiles for quick-and-dirty uninstall support has ended up costing far too much, compared to if I'd just taken the time to design a decent uninstall feature. Of course, hindsight is 20-20; in order to fully understand the requirements of a problem, you sometimes have to get a rather long way towards solving it the simple, obvious... and wrong way. (And, it didn't help that I had significant time constraints pushing me in the direction of the Seemingly-Simplest-At-The-Moment Thing That Could Possibly Work.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Should there be a way or API for retrieving from a code object a loader method and package file where the code comes from?
At 06:55 AM 12/23/2008 -0500, Rocky Bernstein wrote: Now that there is a package mechanism (are package mechanisms?) like zipimporter that bundle source code into a single file, should the notion of a "file" location should be adjusted to include the package and/or importer? Is there a standard API or routine which can extract this information given a code object? The inspect module (in 2.5 and up) supports retrieving the source lines for any object that has module globals. So you could do it for a class, a function, a method, module-level code, or even a frame, but not for a standalone code object. I believe there are also certain inspect module APIs that will return a pseudo-filename, i.e. the zipfile name followed by the path within the zipfile. Also I'm not sure there *is* a standard print string way to show member inside a package. zipimporter may insert co_filename strings like: /usr/lib/python2.5/site-packages/tracer-0.1.0-py2.5.egg/tracer.py AFAIK, it'll only do this if the zipfile doesn't contain a usable .pyc or .pyo. Ordinarily, co_filename will be the name of the original source file before the zipfile was created. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [ANN] VPython 0.1
At 07:50 AM 10/25/2008 -0400, A.M. Kuchling wrote: On Sat, Oct 25, 2008 at 04:33:23PM +1300, Greg Ewing wrote: > Maybe not, but at least you can follow what it's doing > just by knowing C. Introducing vmgen would introduce another > layer for the reader to learn about. A stray thought: does using a generator for the VM make life easier for the Stackless Python developers in any way? Does it make it possible for stock CPython to become stackless? Dunno about that, but I do know that having stack effect info for the bytecode could help with things like bytecode verification (without having to define a bunch of magic constants that duplicate information from the innards of ceval.c). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [ANN] VPython 0.1
At 10:47 AM 10/24/2008 +0200, J. Sievers wrote: - Right now, CPython's bytecode is translated to direct threaded code lazily (when a code object is first evaluated). This would have to be merged into compile.c in some way plus some assorted minor changes. Don't you mean codeobject.c? I don't see how the compiler relates, as Python programs can generate or transform bytecode. (For example, Zope's Python sandboxing works that way.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Things to Know About Super
At 06:07 AM 8/29/2008 +0200, Michele Simionato wrote: On Thu, Aug 28, 2008 at 8:54 PM, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > I created a "universal metaclass" in > DecoratorTools whose sole function is to delegate metaclass __new__, > __init__, and __call__ to class-level methods (e.g. __class_new__, > __class_call__, etc.), thereby eliminating the need to have custom > metaclasses for most use cases in the first place. Now, wherever possible, > I use that single metaclass in my frameworks, so that there's no need to mix > them. easy_installed DecoratorTools and found it: classy_class. >From the point of view of the code, this is a beautiful and elegant snippet. However, suppose that from tomorrow everybody starts using it. Since metaclasses would become so easy to use, possibly a lot of people would take advantage of them. That was sort of the idea. ;-) Then we would have potentially complex (multiple) inheritance hierarchies with chains of methods (_class__new__/_class__init__) calling themselves cooperatively in the MRO. Would the resulting code be readable? Readability's orthogonal. Some of them might be readable, some not. Depends on who's writing them. :) How easy would be for an average framework user to understand what is happening to his class? You're right, let's abolish inheritance, too, because then you might have to read more than one class to see what's happening. I think class decorators would be a much better solution than classy_class for most use cases Obviously, I disagree. :) You'll notice that DecoratorTools supports class decorators for Python 2.3 and up (actually, I think that particular bit worked in 2.2 as well). So, it's not the absence of class decorators that motivated the 'classy' mixin. Generally speaking I like more solutions bases on functional composition (as in WSGI that you know very well) than on method cooperation. Rather than improve the support for inheritance, I would like (in an ideal world) to reduce it, to make easier the choice for people between inheritance and alternatives (object composition, delegation, functional composition). In the real world, I am content in documenting the pitfalls of super, warn people about the dangers of complex design involving multiple inheritance and cooperation, and suggest alternatives. Naturally, if you can design a system to use delegates instead of class hierarchy to represent a chain of responsibility, it might well be an improvement. But there are tradeoffs, and no matter what you are going to end up coding chains of responsibility. Co-operative inheritance is a nice solution for chains of responsibility that can be expressed in a class hierarchy, and are no more "dangerous" than any other sort of chain of responsibility. In fact, they are in some ways less so since the patterns are likely to be better documented than anything you come up with on your own. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Things to Know About Super
At 05:50 PM 8/28/2008 +0200, Michele Simionato wrote: On Aug 28, 5:30 pm, "Phillip J. Eby" <[EMAIL PROTECTED]> wrote: > How is that making things easier for application programmers? We have different definitions of "application programmer". For me a typical application programmer is somebody who never fiddles with metaclasses, which are the realm of framework builders. Application programmers use frameworks, and sometimes more than one. If they're subclassing from two different frameworks, each using a different metaclass, they will need to also multiple-inherit the metaclass. This is in fact so annoying that I created a "universal metaclass" in DecoratorTools whose sole function is to delegate metaclass __new__, __init__, and __call__ to class-level methods (e.g. __class_new__, __class_call__, etc.), thereby eliminating the need to have custom metaclasses for most use cases in the first place. Now, wherever possible, I use that single metaclass in my frameworks, so that there's no need to mix them. That, IMO, would be a more useful change than getting rid of super(); it would get rid of the explicit metaclass mixing. (It would still not remove the need for co-operative methods, as the class-delegated methods still need to be co-operative for MI to work.) There are, of course, other ways to create co-operative function calls besides super(), and I've certainly created more a few of them in my time. (E.g. generic function method combination, instancemethod() chains, and next-method-iterators, to name the ones that occur to me right off.) But these are more for cases where super() is wholly inadequate to the purpose, and none are anywhere near as convenient as super(). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Things to Know About Super
At 06:35 AM 8/28/2008 +0200, Michele Simionato wrote: Multiple inheritance of metaclasses is perhaps the strongest use case for multiple inheritance, but is it strong enough? I mean, in real code how many times did I need that? I would not mind make life harder for gurus and simpler for application programmers. Then you need to leave MI and co-operation the hell alone. Right now, an application programmer can mix metaclasses like this: class FooBar(Foo, Bar): class __metaclass__(Foo.__class__, Bar.__class__): pass ... Or, in 3.x: class FooBarClass(Foo.__class__, Bar.__class__): pass class FooBar(Foo, Bar, metaclass=FooBarClass): ... Either way, this is useful in cases where Foo and Bar come from different frameworks. That's the *only* way to get such things to co-operate, in fact. I do not think removing cooperation would be so bad in practice. In many practical cases, one could just write the metaclass by hand, How is that making things easier for application programmers? Maybe you would need to duplicate a couple of lines and/or to introduce an helper function, ...which then has to have an agreed-upon protocol that all metaclass authors have to follow... which we already have... but which you're proposing to get rid of... so we can re-invent it lots of times... in mutually incompatible ways. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Things to Know About Super
At 03:16 AM 8/27/2008 +0200, Michele Simionato wrote: It is just a matter of how rare the use cases really are. Cooperative methods has been introduced 6+ years ago. In all this time surely they must have been used. How many compelling uses of cooperation we can find in real life code? For instance in the standard library or in some well known framework? This is a serious question I have been wanting to ask for years. I am sure people here can find some example, so just give me a pointer and we will see. ISTR pointing out on more than one occasion that a major use case for co-operative super() is in the implementation of metaclasses. The __init__ and __new__ signatures are fixed, multiple inheritance is possible, and co-operativeness is a must (as the base class methods *must* be called). I'm hard-pressed to think of a metaclass constructor or initializer that I've written in the last half-decade or more where I didn't use super() to make it co-operative. That, IMO, is a compelling use case even if there were not a single other example of the need for super. However, I'm pretty sure I've had other cases where it was necessary to co-operate in cases where multiple inheritance occurred later; ie. where it was possible for a subclass to add a new class between parents. Remember that subclasses of a new-style class do not always have the same MRO tail as the original class; i.e., a subclass of "class A(B, C):" is only constrained to have [A...B...C] in its MRO; semi-arbitrary classes may be inserted between e.g. A and B. So, a new-style class cannot, as a general rule, statically determine what base class implementation of a method should be invoked. I personally consider the rare case where I have to force such static knowledge to be an unfortunate wart in the design (of that code, not Python). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] lnotab and the AST optimizer
At 12:56 AM 7/25/2008 +1000, Thomas Lee wrote: I'm making some good progress with the AST optimizer, and now the main thing standing in my way is lnotab. Currently lnotab expects bytecode sequencing to be roughly in-sync with the order of the source file and a few things that the optimizer does (e.g. swapping the bodies of an if/else after removing negation such that "if not X: A; else: B" becomes "if X: B; else A") breaks this assumption. This will result in either an assertion failure or incorrect line numbers being reported. It seems that lnotab is used in relatively few places in the source code at the moment, but if I'm going to make a change to how lnotab works I want to do so in a way that's going to allow me to move forward while keeping everybody happy. I'm away for a few days so I probably won't be able to get back to anybody until either Sunday or Monday, but I'd appreciate it if anybody in the know can weigh in on this. I'd personally love it if the lnotab were capable of handling line numbers from different files as well as out-of-order lines. (For function inlining, among other more esoteric things.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing restricted Python in Zope2
At 11:27 AM 7/17/2008 -0700, Brett Cannon wrote: On Thu, Jul 17, 2008 at 10:54 AM, ranjith kannikara <[EMAIL PROTECTED]> wrote: > I have taken the gsoc 08 project of porting zope2 to python2.5. > Through my way to the successful completion of the project I have to > implement Restricted python in Zope2. I could only get the information > that the python AST has not changed on moving from python2.4 to 2.5 > but Restricted Python is not well documented enough for a stident to > test the Zope2 's Restricted Python implentation. > > As a student I am not familiar with Restricted Python and python AST > implementation.And in need of help to start the Restricted Python > implementation. > What do you mean, "Restricted Python"? If you mean rexec and Bastion, they are no longer supported, and that began before 2.5. No, he means the restricted Python compiler and capability-proxy system used by Zope. You know, the one I always bring up whenever anybody says they want to implement capabilities in Python? ;-) Zope's restricted Python is basically a combination of a special compiler, __builtin__ replacements, and a proxy type. Instead of using LOAD_ATTR opcodes, the compiler generates code that calls a special getattr() function instead, and most objects other than relatively-safe builtin types are wrapped in proxies that control what attributes can be accessed and what operations can be performed. The restricted Python framework itself doesn't impose any particular security policy; proxies delegate checks to "checker" objects that are essentially capabilities. Mostly, it focuses on creating a safe sandbox that can be expanded. There are two parts to the implication; one is called RestrictedPython and lives at: http://svn.zope.org/RestrictedPython/trunk The other part is "zope.security.untrustedpython", and it's part of the zope.security distribution; see: http://svn.zope.org/zope.security/trunk/src/zope/security/untrustedpython/ for its specific code and docs. Both packages appear to have automated tests. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python FAQ: Why doesn't Python have a "with" statement?
At 04:54 AM 6/19/2008 -0700, C. Titus Brown wrote:
More generally, I've never understood why some people insist that
certain features make Ruby better for DSLs -- are code blocks really
that important to DSLs? Or is it just the lack of parens??
Comparison to JavaScript suggests that it's the blocks that make the
difference. Even the fact that you have to spell your blocks with
"function(){...}" doesn't stop you from making a usable DSL, it's
just not always as pretty as you'd like.
The lack of parens and simpler syntax for blocks in Ruby just makes
it easier to make *nice-looking* DSLs.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: add odict to collections
At 02:34 PM 6/15/2008 +, Antoine Pitrou wrote: Phillip J. Eby telecommunity.com> writes: > > As for the other uses for ordered dictionaries, I find it simplest to > just use a list of key,value pairs, and only transform it to a > dictionary or dictionary-like structure as needed, using tools like > the cgi module, the email package, or wsgiref.headers. What you are saying is that there are already generally useful container types in the stdlib, but isn't it a good argument in favor of ripping them out of domain-specific packages and provide them as generic classes in the collections module? Someone never using cgi or wsgiref wouldn't know that some of the code there can be useful for other purposes. I didn't say I used them for other purposes, or that they were generally useful. Rather, they're specifically useful for the things they're useful for. More often than not, the use case calls for not merely ordering, but ordering of *values*, with multiple values for each key. But the precise API desired for manipulating such structures tends to be highly app-specific (like email, CGI form values, HTTP headers, etc.), so it's actually IMO an argument *against* a general odict type. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: add odict to collections
At 02:19 PM 6/15/2008 +, Antoine Pitrou wrote: > Ordered dicts, dicts that remember the chronological order of their > insertion, don't sound generally useful. They are generally useful in any case where you want to handle key-value pairs while not confusing a human operator by messing up the original order. Think e.g. configuration files. A common complaint against ConfigParser is that writing a configuration file does not preserve the order of the original file, which is harmless for the computer but very annoying for the human being who maintains that file. You don't need an ordered dictionary for that; you need a save routine that stream-edits the old file contents. That way, you don't lose comments and spacing either. As for the other uses for ordered dictionaries, I find it simplest to just use a list of key,value pairs, and only transform it to a dictionary or dictionary-like structure as needed, using tools like the cgi module, the email package, or wsgiref.headers. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python FAQ: Why doesn't Python have a "with" statement?
At 08:19 AM 6/14/2008 +0200, Cesare Di Mauro wrote: Assignament must work on the object's namespace, of course: def foo(a): on a: x += 1 print x will be equivalent to: def foo(a): a.x += 1 print a.x Er, you need a syntactic disambiguation here to distinguish attributes from locals or globals: def foo(a): on a: .x += 1 print .x Otherwise, this leads to all sorts of craziness. You'd also have to restrict what could be referenced in a nested "on" block, in order to avoid further ambiguities. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug or a feature?
At 12:46 PM 6/12/2008 -0700, Guido van Rossum wrote: The intention was for these dicts to be used as namespaces. By "these" do you mean type object __dict__ attributes, or *all* __dict__ attributes? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug or a feature?
At 01:34 PM 6/12/2008 +0200, Carl Friedrich Bolz wrote: Phillip J. Eby wrote: > As it happens, most objects' __dict__ slots are settable by default, and > *require* that you set it to a dict or subclass thereof. This is wrong for types: Which is why I said "most" - to exclude types, and objects that don't have a __dict__ slot to begin with. I think there are good arguments for not allowing strings keys in type dicts, or at least leaving it up to the implementation. That may well be, but there is nothing in Python's spec that I'm aware of that *forbids* it. For example the type() constructor doc doesn't say anything about using string-only keys in the class dictionary. Using non-string keys in type dicts is relatively awkward and allowing them makes many interesting optimizations (like method caches) a _lot_ harder to get right. Really? Why? Having non-string dict keys is NOT the same thing as having non-string attribute names, so attribute name lookups should be unaffected. (Attribute names *are* required to be strings, and -- as far as I know -- always have been.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug or a feature?
At 02:59 AM 6/12/2008 +0200, Maciej Fijalkowski wrote: It's about abusing locals, which are not even given that they'll modify this dict. Note that class bodies are a special case: as of PEP 3115, it's possible for a class body's locals to be a non-dictionary object, so it makes no sense to make a class body's locals() or f_locals return some *other* object. Meanwhile, as a practicality-beats-purity matter, you're going to run into compatibility problems with a fair number of libraries in the field (including Zope and Twisted, and anything using them) if you *don't* support locals() modification in class bodies. (Those other libraries don't use non-string keys like AddOns does, but they *do* depend on modifying a class body's frame locals.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug or a feature?
At 08:32 PM 6/11/2008 -0400, Terry Reedy wrote: The Data Model chapter of the Reference Manual lists .__dict__ as a special attribute of callables, modules, classes, and instances. It describes __dict__ as a "namespace dictionary" or "implementation of the namespace" thereof. Since namespaces map names (or possibly non-name strings) to objects, this to me implies that an implementation is and should not be required to allow non-strings in __dict__. The same chapter has more than one sentence saying something like "o.x is equivalent to o.__dict__['x']". While one could read this as prohibiting o.__dict__[non_string], one could also read it as being silent, neither allowing nor prohibiting. As it happens, most objects' __dict__ slots are settable by default, and *require* that you set it to a dict or subclass thereof. This seems (to me) to imply that a standard dictionary (i.e. one supporting keys of any type) is required. (In the sense that a dict subclass which rejects non-strings would be violating the Liskov principle.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bug or a feature?
At 03:37 AM 6/11/2008 +0200, Maciej Fijalkowski wrote: On Wed, Jun 11, 2008 at 3:36 AM, Scott Dial <[EMAIL PROTECTED]> wrote: > Maciej Fijalkowski wrote: >> >> What do you think about this code: >> >> class A: >> locals()[42] = 98 >> >> Seems people rely on it working. > > I apologize for my ignorance, but who? Could you please cite something > reputable that relies on this detail? > It's in tests of sqlalchemy. My question is among the lines "should I bug sqlalchemy guys to remove this, or should I change pypy to accept this". That test is there to ensure that it interoperates with code using the AddOns library from the Cheeseshop; SQLAlchemy is not the source of the usage. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Addition of "pyprocessing" module to standard lib.
At 12:19 PM 5/15/2008 +1200, Greg Ewing wrote: Andrew McNabb wrote: If it made people feel better, maybe it should be called threading2 instead of multiprocessing. I think that errs in the other direction, making it sound like just another way of doing single-process threading, which it's not. Maybe "multicore" would help give the right impression? Sounds like a marketing win to me, since it directly addresses the "python doesn't do multicore" meme. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problems with the new super()
At 04:38 PM 5/1/2008 -0300, Facundo Batista wrote: Has super() proved more useful than harmful? For me, yes. I use it all the time. The only time I use explicit-target upcalls is in __init__ methods, and there usually only to skip a subclass' init or to explicitly manage a tricky bit of multiple inheritance. (Note, by the way, that you cannot safely write an upcall in a mixin class without super, so it can't safely be done away with, anyway.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pydoc works with eggs? (python-2.5.1)
At 06:48 AM 4/23/2008 -0400, Neal Becker wrote: Neal Becker wrote: > pydoc blew up when I tried to view doc for pytools module, which is an > egg: > > pydoc -p 8082 > pydoc server ready at http://localhost:8082/ > ... > I see that installing the egg unzipped fixes this. It looks to me that pydoc doesn't work with zipped eggs. What's odd about this is that it *did* at one time. Or at least help() did. The changes I made in 2.5 were mainly so that help(package) would work on zipped eggs. From the traceback, it looks like the issue is that it's trying to parse comments out of the source for an object with no docstring. I didn't know it could do that, so I never tried testing it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to easily consume just the parts of eggs that are good for you
At 12:12 AM 4/10/2008 -0700, Stephen Hansen wrote: >I think PJE's idea here is very good. Just include certain files and >such in the RPM/DEB that will satisfy the >"python-package-management" system. For RPM/DEB users and their OS's >database of packages, its irrelevant largely-- they'll still keep >using their own system. But if a product needs something without a >.deb or .rpm, or if someone's on an operating system without a >native system-- they can still gather everything they need. I've narrowed it a bit from that, actually. It's safest if easy_install simply refuses to touch any files that it can't tell were installed by it (or a compatible system, eg. distutils.) While that won't solve Paul Moore's desire for the One True Package Manager, it will at least make it possible to move forward. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you
At 12:51 AM 4/10/2008 +0200, Gael Varoquaux wrote: >On Wed, Apr 09, 2008 at 11:46:19PM +0100, Paul Moore wrote: > > I find this whole discussion hugely confusing, because a lot of people > > are stating opinions about environments which it seems they don't use, > > or know much about. I don't know how to avoid this, but it does make > > it highly unlikely that any practical progress will get made. > >I find that something that doesn't help at all the discussion move >forward is that everybody has different usecases in mind, on different >platforms, and is not interested in other people's usecases. > >Hopefuly I am wrong, You're not wrong at all. I have to deal with *all* the platforms and use cases, which makes it quite frustrating when people who haven't even read the requirements are making proposals to "solve" things in ways that break for everyone except their own niche platform+usecase combination. Guido, can I borrow the time machine and go back and NOT try to improve on the distutils? Or is there already too much collateral damage to the timeline? ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you
At 11:48 PM 4/9/2008 +0100, Paul Moore wrote: >On 09/04/2008, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > > It would be, if .eggs were a packaging format, rather than a binary > > distribution/runtime format. > > > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > > doesn't contain documentation either, unless it's needed at > > runtime. Same for configuration files. > >And yet, Java doesn't have an equivalent of easy_install for jar >files, to my knowledge. Actually, OSGi and Eclipse plugins and "feature sites" come quite close, and setuptools rips off many of its features from them. OSGi is basically a standard for additional .jar metadata to encompass dependencies and other info. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you
At 03:20 PM 4/9/2008 -0700, zooko wrote: >I've opened a ticket on my setuptools trac about this proposal: > >http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should >come with .py files by default, rather than .pyc files Filling your tracker with already-rejected proposals isn't likely to encourage me to look at it, especially when I've personally rejected them to you in IRC. That goes for your ticket #4 as well. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you
At 04:43 PM 4/9/2008 -0400, Stanley A. Klein wrote: >I don't understand what you mean by "shared environments and development > environments". I mean that in a shared or development environment, a system packager isn't useful, since it expects things to live in *one* place, and usually to have only one *version*, as well. >I agree that we are dealing with a combination of technical and social >issues here. However, I think it takes a lot more understanding for a >publisher to get everything straight. If they provide you with the source distribution, you can make any sort of package you want. > > Eggs don't include documentation or configuration files, and they > > install scripts in script directories, so I don't get what you're > > talking about here. For any other data that a package accesses at > > runtime, my earlier comments apply. > > > >But rpms and debs do include these files, plus manual pages, localization >files and a lot of other ancillary stuff. ...just one of the many reasons that eggs are not a replacement for rpms and debs. :) >Most of the Python tarballs I have downloaded have all kinds of files in >their installation trees. This is a major pain in the you-know-what for >someone trying to use bdist_rpm and get proper, FHS-compliant rpms. If >eggs are supposed to be strictly runtime files, I think very few >developers actually understand that. Better yet, how do you define what >should be included in an installation? It sounds like the egg concept >doesn't include several kinds of files that rpm and deb would include in >an installation. I think that may be an important issue here. It would be, if .eggs were a packaging format, rather than a binary distribution/runtime format. Remember "eggs are to Python as jars are to Java" -- a Java .jar doesn't contain documentation either, unless it's needed at runtime. Same for configuration files. They're not system packages, in other words. The assumption that they are is another marketing failure, due to conflation of "package == distribution of python code" and "package == thing you manage with a system packager". People see, "I put my package in an .egg" and think it's the latter definition, when it's barely even the former. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to easily consume just the parts of eggs that are good for you
At 12:30 PM 4/9/2008 -0700, zooko wrote: >On Apr 8, 2008, at 4:36 PM, Greg Ewing wrote: > > > > I discovered another annoyance with eggs the other day -- it > > seems that tracebacks referring to egg-resident files contain the > > pathname of some temporary directory that existed when the egg > > was being packaged, rather than the one it actually exists in > > at run time. > >Brian Warner and I discovered that issue yesterday, too. We >determined that if you install the egg (with easy_install or with a >setuptools-powered ./setup.py install) in unzipped form then the >source file names get rewritten so that your stack traces come with >source lines. Are you using Python 2.5? As of 2.5, the linecache module should correctly read the source line from the present location of the source file the module was loaded from, regardless of the file name specified in the traceback. >If you have a package which requires stack traces to come with source >lines, then you could pass "zip_safe=False" to the call to setup(). > >I would prefer that zip_safe=False were the default and that either >the producer or the consumer of a package had to specifically choose >zip_safe=True in order to install eggs in zipped form. A better fix would be for Python to use relative paths in co_filename, as this would fix similar problems that occur whenever you move .pyc/.pyo files around. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you
At 10:30 AM 4/9/2008 -0700, zooko wrote: >PEP 262 sounds like a non-starter to me because > >1. It appears to be backwards-incompatible with setuptools/ >easy_install/eggs, thus losing all the recently gained cooperation >that I mentioned in the previous paragraph, and No. It provides a forward-compatibility path for the distutils, so that easy_install doesn't have to install things in .egg format in the future. There's no compatibility breakage at all. >2. It defines a new database file, No, it defines *files*. One file per installed distribution, containing (among other things) an installation manifest. >where I would prefer either: >a. Doing away with database files entirely and relying on the >filesystem alone to hold that information, or ...which is what PEP 262 *does*. Unfortunately, PEP 262's title is bad for marketing, as you've effectively pointed out. It would be better titled something "package installation manifests" or "package contents files", or something of that sort. >b. Continuing to use the current ".pth" database file format, >possibly improved by having native support for .pth files in the >Python import machinery. These mechanisms are orthogonal to this issue. >3. Because of #2, it triggers programmers to exclaim "They are >planning to reinvent apt!", thus making it unlikely that the new >proposal will recapture the cooperation that setuptools has already >(slowly) gained. Yeah, we need a new name. Everybody is going off of "database of installed packages" and thinking "apt", because they aren't paying any closer attention. However, given that we are discussing this on Python-Dev and distutils-sig, I do think it's reasonable to expect (if perhaps not reasonable to receive) that people discussing the PEP have *read* the freaking PEP first, prior to trashing it or offering alternatives. And it's not like I'm personally offended or anything -- I didn't even write the PEP in question. But what's the point of having PEPs if people read nothing but their titles? We could just delete everything but PEP 0. :) >Perhaps PEP 262 and my proposal are not actually alternatives, but >are complementary. As I've already pointed out, your proposal does not address multiple installed versions of a package, and I see no sane way to modify it to do so. >What I want is for the already implemented, tested, and deployed >code- re-use features of setuptools/easy_install to be more widely >accepted. This is best and most easily achieved by fixing the two >most frequent objections to setuptools/easy_install: 1. That you >can't conveniently install into an arbitrary directory, and 2. that >it subverts the meaning of your PYTHONPATH. As I've already stated, the only way for these problems to be fixed is for easy_install to not install files in .egg form -- which also solves the general objection to using .eggs in the first place. And the only way to do that, is to have a way to keep track of what files are installed. Rather than have easy_install come up with its own way of doing that, I would prefer to share a standard with the distutils. Hence, the PEP discussion. For earlier versions of Python, it will still be possible to install and uninstall with setuptools using this approach. You just won't be able to uninstall pure distutils-based packages, unless you installed them using easy_install. Meanwhile, it has occurred to me that the easiest way of handling compatibility is not to require that other packaging tools mark their files for non-removability, but simply not allow easy_install to remove or overwrite anything that *isn't* claimed by a manifest. In that way, easy_install would be immediately usable in the new mode, without any updates to Python or to system packaging tools. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you
At 11:52 AM 4/9/2008 -0400, Stanley A. Klein wrote: >However, are you implying that the installation information for Python egg >packages accesses and coordinates with the rpm database? Yes, when the information isn't stripped out. Try a more recent Fedora. >IMHO, the main system without a package manager is Windows. You're ignoring shared environments and development environments. (Not to mention Mac OS.) > A reasonable >way to deal with Windows would be to create a package manager for it that >could be used by Python and anyone else who wanted to use it. Let us know when you've finished it, along with the one for Mac OS. :) Of course this still won't do anything for shared environments and development environments. >You are talking here about bdist_rpm and not about a tool that would take >a Python package distributed as an egg file and convert the egg to an rpm >or a deb. Unfortunately, some Python packagers are beginning to limit >their focus only to egg distribution. That creates a problem for users >who have native operating system package management. That is indeed a problem -- but it's a social one, not a technical one. It's trivial for the publisher of an egg to change their command line from "setup.py bdist_egg upload" to "setup.py sdist bdist_egg upload", as soon as their users (politely) request that they do so. > > Applying LSB and FHS to the innards of Python packages makes as much > > sense as applying them to the contents of Java .jar files -- i.e., > > none. If it's unchanging data that's part of a program or library, > > then it's a program or library, just like static data declared in a C > > program or library. Whether the file extension is .py, .so, or even > > .png is irrelevant. > >The FHS defines places to put specific kinds of files, such as command >scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation >(/usr/share/doc/package-name), and configuration files (/etc). There are >several kinds of files identified and places defined to put them. >Distribution by eggs has a tendency to scoop up all of those files and put >them in /usr/lib/python/site-packages, regardless of where they belong. Eggs don't include documentation or configuration files, and they install scripts in script directories, so I don't get what you're talking about here. For any other data that a package accesses at runtime, my earlier comments apply. >Having eggs support conformance to FHS would mean recognizing and tagging >the relevant files. A tool for converting eggs to rpms or debs would >essentially reformat the egg to rpm or deb and put files where they >belong. No, because such files as you describe don't exist. If you think they do, then either you have misunderstood the nature of the files in question, or the developer has incorrectly placed non-runtime files in their installation tree. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you
At 10:00 AM 4/9/2008 +0200, Gael Varoquaux wrote: >On Wed, Apr 09, 2008 at 12:41:32AM -0400, Phillip J. Eby wrote: > > >The way to achieve a database for Python would be to provide tools for > > >conversion of eggs to rpms and debs, > > > Such tools already exist, although the conversion takes place from > > source distributions rather than egg distributions. > >What is the status of the deb backend? The only one I know is unofficial >maintained by Andrew Straw, but my information my be lagging behind. I was under the impression that there were 2 .deb tools, neither one "official" in any sense, any more than 'bdist_rpm' is really "official" for RPM-based systems. >By the way, if these tools work well, they are priceless! I haven't had need to use any of them, so I don't really know. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you
At 10:49 PM 4/8/2008 -0400, Stanley A. Klein wrote: >On Tue, April 8, 2008 9:37 pm, Ben Finney ><[EMAIL PROTECTED]> wrote: > > Date: Wed, 09 Apr 2008 11:37:07 +1000 > > From: Ben Finney <[EMAIL PROTECTED]> > > Subject: Re: [Distutils] how to easily consume just the parts of eggs > > thatare good for you > > To: [EMAIL PROTECTED] > > > > > > zooko <[EMAIL PROTECTED]> writes: > >> eyes and said "So they are planning to reinvent apt!". > > > > That's pretty much my reaction, too. > >I have the same reaction. I'm curious. Have any of you actually read PEP 262 in any detail? I have seen precious little discussion so far that doesn't appear to be based on significant misunderstandings of either the purpose of reviving the PEP, or the mechanics of its proposed implementation. >I have tried in the past to use easy_install, but have run into problems >because there is no communication between easy_install and the rpm >database, resulting in failure of easy_install to recognize that >dependencies have already been installed using rpms. This problem doesn't exist with Python 2.5, unless you're using a platform that willfully strips out the installation information that Python 2.5 provides for these packages. >A database focused only on Python packages is highly inappropriate for >Linux systems, violates the Linux standards, and creates problems because >eggs are not coordinated with the operating system package manager. The revamp of PEP 262 is aimed at removing .egg files and directories from the process, by allowing system packagers to tell Python what files belong to them and should not be messed with. And conversely, allowing systems and installation targets *without* package managers to safely manage their Python installations. > The >way to achieve a database for Python would be to provide tools for >conversion of eggs to rpms and debs, Such tools already exist, although the conversion takes place from source distributions rather than egg distributions. >to have eggs support conformance to >the LSB and FHS, Applying LSB and FHS to the innards of Python packages makes as much sense as applying them to the contents of Java .jar files -- i.e., none. If it's unchanging data that's part of a program or library, then it's a program or library, just like static data declared in a C program or library. Whether the file extension is .py, .so, or even .png is irrelevant. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to easily consume just the parts of eggs that are good for you
At 10:01 AM 4/8/2008 -0700, zooko wrote:
>On Mar 26, 2008, at 7:34 PM, Chris McDonough wrote:
> > zooko wrote:
>
>http://mail.python.org/pipermail/python-dev/2008-March/078243.html
>
> >> Here is a simple proposal: make the standard Python "import"
> >> mechanism notice eggs on the PYTHONPATH and insert them (into the
> >> *same* location) on the sys.path.
> >> This eliminates the #1 problem with eggs -- that they don't
> >> easily work when installing them into places other than your site-
> >> packages and that if you allow any of them to be installed on
> >> your system then they take precedence over your non-egg packages
> >> even you explicitly put those other packages earlier in your
> >> PYTHONPATH. (That latter behavior is very disagreeable to more
> >> than a few prorgammers.)
> >
> > Sorry if I'm out of the loop and there's some subtlety here that
> > I'm disregarding, but it doesn't appear that either of the issues
> > you mention is a actually problem with eggs. These are instead
> > problems with how eggs get installed by easy_install (which uses
> > a .pth file to extend sys.path). It's reasonable to put eggs on
> > the PYTHONPATH manually (e.g. sys.path.append('/path/to/some.egg'))
> > instead of using easy_install to install them.
>
>Yes, you are missing something. While many programmers, such as
>yourself and Lennart Regebro (who posted to this thread) find the
>current eggs system to be perfectly convenient and to Just Work, many
>others, such as Glyph Lefkowitz (who posted to a related thread) find
>them to be so annoying that they actively ensure that no eggs are
>ever allowed to touch their system.
>
>The reasons for this latter problem are two:
>
>1. You can't conveniently install eggs into a non-system directory,
>such as ~/my-python-stuff.
Wha?
>2. If you allow even a single egg to be installed into your
>PYTHONPATH, it will change the semantics of your PYTHONPATH.
Only in the same way that manually putting an egg on the front of
PYTHONPATH can be considered to "change the semantics" of your PYTHONPATH.
>Both of these problems are directly caused by the need for eggs to
>hack your site.py. If Python automatically added eggs found in the
>PYTHONPATH to the sys.path, both of these problems would go away.
And add new ones.
>I am skeptical that the current proposals to define a new database
>for installed packages will fare any better than the current eggs
>scheme does in this respect.
The purpose for the installation database is to allow easy_install to
eschew the use of .egg files or directories for anything other than
multi-version installs. Thus, no need to add those .egg files or
directories to the head of the PYTHONPATH. Conflicts would be
handled at install time rather than runtime, in other words.
>I am skeptical that prorgammers are going to be willing to use a new
>database format. They already have a database -- their filesystem --
>and they already have the tools to control it -- mv, rm, and
>PYTHONPATH. Many of them already hate the existence the
>"easy_instlal.pth" database file, and I don't see why a new database
>file would be any different.
PEP 262 does not propose a database file -- it proposes the inclusion
of a metadata file for each installed distribution.
>My proposal makes the current benefits of eggs -- clean, easy code re-
>use among programmers -- more compatible with their current tools --
>mv, rm, and PYTHONPATH. It is also forward-compatible with more
>sophisticated proposals to add features like uninstall and operating
>system integration.
Actually, your current proposal doesn't work, unless you at least
have some way to indicate which *version* of an egg should be
automatically added to sys.path -- and some way to change
that. Otherwise, you might as well use the -m option to
easy_install, and require() the eggs at runtime. (Which needs
neither .pth files nor site.py hacking.)
Meanwhile, my understanding is that the people who dislike eggs,
dislike them because when they install a setuptools-based package,
it's installed as an egg by default. The installation database
proposal (and by the way, people really should read and understand
PEP 262, including the open issues, before trying to compete with
it), will allow setuptools-based packages to install the
"old-fashioned" way by default. That is, not as eggs. Similarly,
easy_install would be able to skip installing .eggs unless you wanted
multi-version support.
So, people who don't like eggs would never see them, since the only
way you'd ever get them would be via easy_install -m, and they would
never use it.
>By the way, since I posted my proposal two weeks ago I have pointed a
>couple of Python hackers who currently refuse to use eggs at the URL:
>
>http://mail.python.org/pipermail/python-dev/2008-March/078243.html
>
>They both agreed that it made perfect sense. I told one of them
>about the alternate proposal to define a new database file to
Re: [Python-Dev] Two questions about jump opcodes
At 10:43 PM 3/22/2008 +, Antoine Pitrou wrote: >- Why are there both relative and absolute jump instructions? The traditional >rationale for relative jumps (apart from position-independent code) >is to allow >for shorter operand sizes; but Python opcodes all have the same operand size Actually they don't. They can have 32-bit arguments, with the EXTENDED_ARG opcode. EXTENDED_ARG loads the high 16 bits of the argument in the opcode that immediately follows. >(and 16 bits is more than enough to address most bytecode arrays). Ah, but not *all* bytecode arrays. Apparently some (automatically generated) code at LucasFilm (if memory serves) exceeded some of the 16-bit limits for bytecode, so the EXTENDED_ARG opcode was added to fix this. >- Why are relative jumps unsigned? This means they can only jump >forward, and as >soon as you want to jump backward you have to switch to an absolute jump... With a backward jump, you already know the exact offset, so you know if you need a 16-bit or 32-bit operand. >(in that regard, I don't understand what JUMP_FORWARD can possibly bring over >JUMP_ABSOLUTE) It means you don't have to guess whether your jump target is going to cross the 64K boundary, thereby requiring you to have used a 32-bit operand. Of course, it does limit your forward jumping to skipping no more than a 64K block, but apparently nobody has exceeded that limit yet. :) Merely having 64K of total bytecode is presumably an easier limit to reach than *jumping over* 64K worth of bytecode. :) In truth, I don't know if that's really the reason why things were originally set up this way, but these are certainly among the reasons thing will probably stay this way. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How we can get rid of eggs for 2.6 and beyond
At 04:29 PM 3/22/2008 +0100, Martin v. Löwis wrote: >>>For those without the read-only flag, the specification should >>>explicitly say what manipulation is allowed. >>Since a distribution isn't really "mutable", I would think that >>uninstallation and reinstallation would be the only manipulation >>available. (As distinct from inspection, verification, and other >>read-only activities.) > >Sure, but what is precisely the semantics of uninstallation, in >terms of changes to the system state? > >I think any model where uninstallation is merely the removal >of files is too limited to be practical. The distutils only support the *addition* of files, so I'm not sure how only removing files is a limit here. Could you explain? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How we can get rid of eggs for 2.6 and beyond
At 11:19 AM 3/22/2008 -0400, Phillip J. Eby wrote: >Not exactly. More like, "package management tool X claims exclusive >rights to this package". Python tools would always defer this right >to the system packager, i.e. a system packager is not obliged to >respect a Python tool's claim to a file, but not the other way around. > >That way, system packaging tools don't need to do anything but mark >the installed files as belonging to them. This probably needs to be refined a little. Exclusive right is too strong, and it goes against Paul Moore's desire for using a single tool. Perhaps instead what it should be is an "uninstall warning" field that must be displayed to a user if an interactive program is doing uninstallation, and that a non-interactive program must refuse to uninstall unless explicitly requested to go ahead. Unfortunately, a warning message might then need to be localized. So this idea still needs some work. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] How we can get rid of eggs for 2.6 and beyond
At 02:14 PM 3/22/2008 +, Paul Moore wrote: >For the system Python, I need: >- a single way to list what's installed (including version) >- a single way to uninstall items as needed >- a way (or more than one) to install 3rd party software *which ties >into the above* Right, and the PEP effort is devoted to having one way to store the information required, to tie these things together. If there is a standard way to store that info on your system, then it doesn't matter how many tools there are, you still have your "one way" to list what's installed or to uninstall things, because you just pick the one lister and/or uninstaller whose UI you prefer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How we can get rid of eggs for 2.6 and beyond
At 12:33 PM 3/22/2008 +0100, Martin v. Löwis wrote: >>I probably should have brought this up, in fact, I think I >>mentioned it in a previous thread, but I would like to see PEP 262 >>add a way to say "this is a system-installed package, *don't >>touch*". The idea again is not to do the job of the native >>packaging system, but rather to ensure that Python-specific tools >>(e.g. easy_install and friends) do not interfere or conflict with it. > >Something like a read-only flag? Not exactly. More like, "package management tool X claims exclusive rights to this package". Python tools would always defer this right to the system packager, i.e. a system packager is not obliged to respect a Python tool's claim to a file, but not the other way around. That way, system packaging tools don't need to do anything but mark the installed files as belonging to them. Since most vendors at least *begin* with a "setup.py install", we could provide a way to indicate that the installation is being done on behalf of a system packaging tool, so that it can provide that indication. >For those without the read-only flag, the specification should >explicitly say what manipulation is allowed. Since a distribution isn't really "mutable", I would think that uninstallation and reinstallation would be the only manipulation available. (As distinct from inspection, verification, and other read-only activities.) It's possible, though, that there might also be actions such as restoring or relocating scripts or data in shared locations outside of the sys.path directory. That will get clearer as the spec gets defined. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] How we can get rid of eggs for 2.6 and beyond
At 11:00 AM 3/22/2008 +, Floris Bruynooghe wrote: >As long as systems (dpkg, rpm, ...) install the .egg-info files they >do communicate which modules/distributions are installed. The >installdb would just duplicate this information (according to the >current PEP). .egg-info/PKG-INFO don't list the specific files, though. >There is a way of telling if you have to keep you hands off a package >(sorry to bring this up again): installation paths. /usr/lib is the >system path, the local admin (and hence setuptools) should keep their >hands off it at all times (unless requested with a --prefix or so for >building the debs or rpms but an uninstall in those cases won't be >required from setuptools). As I mentioned previously, if the spec says anything about specific paths, it will be full of fail. The spec MUST be able to work with *any* local policy about where Python packages are to be installed. Otherwise, any tool that wants to work with install-dbs will end up accumulating a long list of paths to be handled specially for each OS vendor and version... and still not handle everything. No can do. This has to be a mechanism, not a policy. Vendors and admins should be able to enforce reasonable policies, without requiring that every tool have those policies built in. For one thing, it's an entry barrier to tools. Basically, what I'm proposing here is like WSGI for package management tools -- and building anything about paths into the spec would be like WSGI spelling out what pages should be at what URLs! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How we can get rid of eggs for 2.6 and beyond
At 09:44 PM 3/21/2008 -0400, A.M. Kuchling wrote: >On Fri, Mar 21, 2008 at 06:41:00PM -0400, Phillip J. Eby wrote: > > I'm making the assumption that the author(s) of PEP 262 had good > > reason for including what they did, rather than assuming that we > > should start the entire process over from scratch. > >The goal *was* originally to provide for RPM-like verification of file >content, but I don't know that the verification feature really matters >that much; OSes with packaging systems already support such a feature, >probably, and it probably isn't particularly useful for systems >without a packaging system. Actually, it's the places where there's no packaging system that it's most useful. For example, an application that installs plugins to itself. A development environment with multiple virtual pythons. Users installing stuff to their PYTHONPATH, etc. In these cases, having the Python-specific tools be able to verify content signatures is useful, to make sure that you know what you're updating or removing. This is particularly important if one installs anything just by unpacking it into the target directory; you could overwrite something and then have only size/signature info to sort out whose version of the file is actually there. I more question the permissions and uid/gid stuff; I'm not really clear on what I'd use that stuff for in easy_install/uninstall/etc. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How we can get rid of eggs for 2.6 and beyond
At 02:31 AM 3/22/2008 +0100, Martin v. Löwis wrote: >>I'm making the assumption that the author(s) of PEP 262 had good >>reason for including what they did, rather than assuming that we >>should start the entire process over from scratch. > >The objections to the PEP remain the same as they were then, >though: In the requirements, it says "we need", without saying >why we need. It then goes on saying "we want" (rephrased) >"to duplicate APT and RPM", without saying why we want that, >or why that brings us closer to what we need. > >IOW, the PEP is lacking a rationale. Ok, well, I have a rationale for it: make it possible to get rid of eggs as a mechanism for supporting easy_install. Many people (yourself included) have criticized eggs as an installation mechanism, and this is an alternative that gets rid of .egg files and directories in that case, and most of the need for .pth file usage as well. >If there was a chance that the infrastructure being developed >actually helps these tools, *that* would be a reasonable goal, >IMO. Yes, I'm of course primarily interested in Python-specific tools such as virtualenv, easy_install, buildout, and the as-yet-unwritten uninstallers, package listers, etc., that can usefully read or write such data. >However, I'm extremely skeptical that this can ever succeed >to the degree that whoever provides RPMs, .debs, or MSI >files will actually use such data, as they will find that >the data are incomplete, and they have to redo all of it, >anyway. The data isn't for them to use to meet their use cases, it's for them to *provide* so that Python tools don't stomp on, uninstall, or otherwise interfere with files installed by the system. In other words, for system packagers, it's a communication from the system to Python, rather than the other way around. Even though the distutils will build the file in the bdist, the system packaging tools would be free to generate their own file listing and signatures and such. >Do you also envision the objective of PEP 262, then? I.e. >to provide a database of installed packages, in .../install-db? In each directory relative to a given sys.path directory, yes. That is, installing a distutils distribution to any directory would result in a file being added to an install-db within that directory. (Assuming we use the proposed implementation model of PEP 262, which at the moment I don't see any substantial obstacle to.) >>And as I said, I'll be happy if all we do is get the distutils to >>abide by the spec for 2.6, even if it means we don't get an >>uninstall tool. That can always be installed later using Guido's >>bootstrap tool. :) > >I'm even more skeptical here. If the assumption is that the package >database allows for uninstallation, I'm -1. IOW, RPM, deb, MSI >should then *not* write to that package database, as they also write >to a different database, out of the scope of the PEP, and this is >what uninstallation should use. I probably should have brought this up, in fact, I think I mentioned it in a previous thread, but I would like to see PEP 262 add a way to say "this is a system-installed package, *don't touch*". The idea again is not to do the job of the native packaging system, but rather to ensure that Python-specific tools (e.g. easy_install and friends) do not interfere or conflict with it. A big problem in the early development of easy_install, even using eggs, was that there was no way to tell whether it was safe to delete or overwrite an existing file or directory that was already installed on the system. A mechanism like this would allow tools like easy_install to say, "oh, your system packager has a conflicting package here, you need to use that tool to sort this out if you really want to do something here. I'm not going to touch that." Without something like this, there is no way to tell the difference on many systems between a system package and something the user has put there with "sudo python setup.py install". ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How we can get rid of eggs for 2.6 and beyond
At 11:13 PM 3/21/2008 +0100, M.-A. Lemburg wrote: >On 2008-03-21 22:21, Phillip J. Eby wrote: > > At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote: > >> I guess the only way to support all of these variants is > >> to use a filesystem based approach, e.g. by placing a file > >> with a special extension into some dir on sys.path. > >> The "database" logic could then scan sys.path for these > >> files, read the data and provide an interface to it. > >> > >> All bdist formats would then have to include these files. > > > > That's the idea behind the current version of PEP 262, yes, and I think > > it should be kept. > > > >> A separate FILES section also doesn't seem to be necessary - > >> we could just add one or more entries or the format: > >> > >> CreatesDir abc/ > >> CreatesFile abc/xyz1.py > >> CreatesDir abc/def/ > >> CreatesFile abc/def/xyz2.py > >> CreatesFile abc/def/xyz3.py > >> CreatesFile abc/def/xyz4.ini > > > > I actually think the size and hash information is good, in order to be > > able to tell if you're looking at an original file. I'm not sure how > > useful the permissions and uid/gid info is. I'm hoping we'll hear from > > anybody who has a use case for that. > >You're heading off in the wrong direction: we should not be trying >to rewrite RPM or InnoSetup in Python. I'm making the assumption that the author(s) of PEP 262 had good reason for including what they did, rather than assuming that we should start the entire process over from scratch. >Anything more complicated should be left to tools which are >specifically written to manage complex software setups. Tools which will need this data, in order to do their work. Hence, the reason for standardizing the data, instead of the tool(s). >[snip long list of features, both desired and undesired] Actually, *all* of these features are out of scope for stdlib development, because I'm not proposing including *any* tools for this in the stdlib, apart from distutils install and bdist_* support. I'm proposing, rather, that we finish the vision of PEP 262, of having a standard specification that *all* tools will abide by -- including rpm, dpkg, and what-have-you. Since *all* of these tools need to abide by that specification, their requirements will need to be considered in the formulation of the spec. And as I said, I'll be happy if all we do is get the distutils to abide by the spec for 2.6, even if it means we don't get an uninstall tool. That can always be installed later using Guido's bootstrap tool. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How we can get rid of eggs for 2.6 and beyond
At 05:59 PM 3/21/2008 +0100, Christian Heimes wrote: >Phillip J. Eby schrieb: > > Questions, comments... volunteers? :) > >I've yet to read the monster package utils thread so I can't comment on >it. However I like to draw some attention to my PEP 370 >http://python.org/dev/peps/pep-0370/. It's about a site packages >directory in the users home directory. I think it quite related to the >discussion. Actually, it's 100% orthogonal, if done correctly. If anything, this slightly reduces the need for per-user site-packages, since in the new world, .pth files would generally only be needed for "develop" installs. (Assuming we find a way to support namespace packages without using .pth files, that is.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How we can get rid of eggs for 2.6 and beyond
At 08:06 PM 3/21/2008 +0100, M.-A. Lemburg wrote: >I guess the only way to support all of these variants is >to use a filesystem based approach, e.g. by placing a file >with a special extension into some dir on sys.path. >The "database" logic could then scan sys.path for these >files, read the data and provide an interface to it. > >All bdist formats would then have to include these files. That's the idea behind the current version of PEP 262, yes, and I think it should be kept. >A separate FILES section also doesn't seem to be necessary - >we could just add one or more entries or the format: > >CreatesDir abc/ >CreatesFile abc/xyz1.py >CreatesDir abc/def/ >CreatesFile abc/def/xyz2.py >CreatesFile abc/def/xyz3.py >CreatesFile abc/def/xyz4.ini I actually think the size and hash information is good, in order to be able to tell if you're looking at an original file. I'm not sure how useful the permissions and uid/gid info is. I'm hoping we'll hear from anybody who has a use case for that. And of course, there are still some issues to be resolved regarding requirements, package name/version stuff, etc. But we can hash those out once we reach a quorum on the Distutils-SIG. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How we can get rid of eggs for 2.6 and beyond
At 11:21 AM 3/21/2008 -0500, [EMAIL PROTECTED] wrote: > Joachim> I think, the uninstall should _not_ 'rm -rf' but only 'rm' the > Joachim> files (and 'rmdir' directories, but not recursively) that it > Joachim> created, and that have not been modified in the meantime (after > Joachim> the installation). > >That's not sufficient. Suppose file C (e.g. /usr/local/etc/mime.types) is >in both packages A and B. > > Install A - this will create C > Install B - this might overwrite C, saving a copy, or it might retain > A's copy. > Uninstall B - this has to know that C is used by A and not touch it Correct. However, in practice, B should not touch C, unless the file is shared between them. This is a key issue for support of namespace packages, at least if we want to avoid using .pth files. (Which is what setuptools-built system packages do for namespace packages currently.) Of course, one possible solution is for both A and B to depend on a "virtual package" that contains C, such that both A and B can install it if it's not there, and list it in their dependencies. But this is one of the handful of open issues that needs to be resolved with Real Life Package Management people, such as Debian, Fedora, etc. Neither overwriting, refusing to install, nor backups will properly address this issue. However, this is properly a topic for the Distutils-SIG or whatever SIG the actual spec goes to. On Python-Dev I'm only looking for a go/no-go on the overall approach. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 09:53 AM 3/21/2008 -0600, zooko wrote: >Um, isn't this tool called "unzip"? I have done this -- accessed the >source code -- many times, and unzip suffices. > >I don't know what else would be required in order to make an egg into >"a standard distutils-style installation". You also have to rename the EGG-INFO directory to a .egg-info file of the same basename as the original .egg; otherwise, pkg_resources and other runtime access to the egg won't know it's installed. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] How we can get rid of eggs for 2.6 and beyond
So, after having some time to absorb the Python-Dev threads about
setuptools, bootstrap, and all the rest, I think I see an opportunity
to let people route around the "damage" of eggs, while still making
it possible for the people who want to use easy_install or to put
dependencies in their setup.py to get what they want, too. (And
without them needing to install eggs, either.) At the same time, we
can address the issues that remain around uninstalling packages,
system vs. user packages, PYTHONPATH and site.py woes, and really
pretty much every other complaint I've heard in the last few days
about setuptools stomping on other people's stuff. (Even Paul's
Windows issues, hopefully.)
Now, you might be asking, "Okay, so why are you telling me about
this? Why not just go fix setuptools?" Well, I *can't*. Not
without some help from Python-Dev and the Distutils-SIG, to create an
updated standard for installed package metadata, by updating PEP 262
("A Database of Installed Python Packages") to include input from the
system packaging folks, support for namespace packages, and support
for setuptools-compatible dependency information.
What's that got to do with anything? Well, without it, setuptools
can't support uninstall or conflict management without using eggs to
compartmentalize the installed files. And because it has to use eggs
to do *that*, it has to munge .pth files and install its own site.py
when installing to PYTHONPATH. All of this ugliness follows directly
from the absence of a PEP 262-style installation database.
Sure, setuptools could create its own version of this, and I almost
did that four years ago. (If you look at the "open issues" part of
PEP 262, you'll see my comments from back then.) I decided not to
for two reasons: first, the distutils didn't support it yet, so it
didn't help for conflict detection and avoidance in the real world at
that point.
Second, there were no uninstall tools for it, so I'd have had to
write one myself. (Zed's "easy_f'ing_uninstall" to the contrary, it
ain't easy, and I have an aversion to deleting stuff on people's
systems without knowing what will break. There's a big difference
between them typing 'rm -rf' themselves, and me doing it.)
However, if tools exist and are distributed for such a "database",
and *everybody* agrees to use it as an officially-blessed standard,
then it should be possible for setuptools to co-exist with that
framework, and we're all happy campers.
In particular, the "installing eggs sucks" camp should be happy,
because it'll be possible for me (or anyone else) to write a version
of easy_install that doesn't install eggs any more, and
setuptools-based packages can go back to having "setup.py install"
install things the old way by default.
So, to accomplish this, we (for some value of "we") need to:
1. Hash out consensus around what changes or enhancements are needed
to PEP 262, to resolve the previously-listed open issues, those that
have come up since (namespace packages, dependency specifications,
canonical name/version forms), and anything else that comes up.
2. Update or replace the implementation as appropriate, and modify
the distutils to support it in Python 2.6 and beyond. And "support
it" means, "ensure that 'install' and *all* bdist commands update the
database". The bdist_rpm, bdist_wininst, and bdist_msi commands,
even bdist_dumb. (This should probably also include the add/remove
programs stuff in the Windows case.)
3. Create a document for system packagers referencing the PEP and
introducing them to what/why/how of the standard, in case they
weren't one of the original participants in creating this.
It will probably take some non-trivial work to do all this for Python
2.6, but it's probably possible, if we start now. I don't think it's
critical to have an uninstall tool distributed with 2.6, as long as
there's a reasonable way to bootstrap its installation later.
Questions, comments... volunteers? :)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:33 PM 3/21/2008 +, Paul Moore wrote: >On 21/03/2008, Terry Reedy <[EMAIL PROTECTED]> wrote: > > The standard (and to me, preferable) way of dealing with such > things is to > > have an 'installation manager' that can reinstall as well as delete and > > that has a check box for various things to delete. This is what Python > > needs. > >I'd dispute strongly that this is a "standard". It may be preferable, >but I'm not sure where you see evidence of it being a standard. I presume he means that there are a lot of entries in his Add/Remove Programs that work like that, and that it's an emerging standard for Windows. (Certainly I've seen quite a few entries like that in mine, although more often than not they only have one checkbox!) >Could I also point out that *if* such a standard is set up for Python, >bdist_wininst and bdist_msi should be modified to follow it. >Otherwise, it's not a standard, more of competing approach. The best thing to do would be to get a standard (ala PEP 262, but modified by the benefit of experience now) for tracking installed Python package distributions. Then we can standardize on platform tools for managing this data, and include them in the relevant platform distributions. (And that would include making bdist_wininst and bdist_msi follow this installation DB standard.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Wow, I think I actually *get* it now!
At 10:08 PM 3/20/2008 +, [EMAIL PROTECTED] wrote (off-list): >No, but in no situation, except one (where I was extremely pressed >for time) was I actually attempting to use setuptools to use any of >its features. My experience of it is: "If a project uses distutils >or apt, installation probably works. If it uses setuptools, it >probably throws a traceback or a wall of text explaining why my >environment is inadequate to perform the installation." Other >people chose to use it and in so doing broke my setup. Manually >copying a few files in these cases was a _lot_ easier than >attempting to diagnose and repair software that I didn't even want to use. > >I am not interesting in packaging or distribution. Far from it: I >run all of my software out of an SVN checkout and I _detest_ being >involved in discussions of deployment or installation. ... >However, the general message of the negative subjective experience I >have had while using setuptools is not FUD. It's an accurate >portrayal of a great deal of frustration. setuptools has, to this >date, not solved a single problem for *me*, personally or >professionally, but it has caused many. distutils, despite its many >flaws, has actually solved quite a few. Actually, this information is VERY helpful. It makes it blindingly obvious to me now that the difference between loving and hating setuptools is whether you're *intentionally* using it, or whether it shows up in your ecosystem uninvited. It also makes the difference in whether you get involved: with no investment in the tool itself, you have minimal motivation to RTFM, ask questions, or fix bugs. And when people in this scenario *do* communicate to me or the distutils-sig, they are much more likely to be impatient and hostile, and more likely to view the system as "fundamentally broken". This makes total sense to me now. I don't have any *solutions* to the problem, mind you, but at least now I understand what before seemed like some sort of bizarre anomaly where literally thousands of people use setuptools and many dozens actually express their happiness with or even love for the system, and then others hate it like they hate Microsoft, or worse. ;-) Meanwhile, from the "outsiders" point of view, setuptools looks like the Matrix or the Borg, happily assimilating the masses, who then start coming to you and say, "But you'll be so much happier once you join us..." ...and off in the distance, you hear a quiet rumbling of zombies chanting "ggs s mussst havve e!" :) Hm. So it seems to me that maybe one thing that would help is a "Setuptools Haters' Guide To Setuptools" -- that is, *short* documentation specifically written for people who don't want to use setuptools and want to minimize its impact on their systems. I could probably write something like that fairly easily, now that I have some idea of what to go in it, more than, "the existing documentation sucks". :) Can I count on some non-assimilated persons' help in critiquing such a document and suggesting any topics I miss? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 08:34 PM 3/20/2008 +, Paul Moore wrote: >I then went on to say that putting dependency information in setup.exe >and expecting users to use automatic dependency resolution encourages >developers to omit dependency details from documentation (to an extent >I can't quantify, but I believe is non-zero). That lack of >documentation "forces" me to rely on the automatic process. THAT is >the thing that removes my choice, not easy_install's ability to skip >dependency checking. Ah. Fair enough. So, if we get PyPI to display that information, that should fix this problem for you? >People are starting to omit distributing >bdist_wininst installers in favour of eggs only. You mean, they're shipping a .win32.egg, but not an .exe? > And you cannot (to my >knowledge) convert an egg into a bdist_wininst installer, Not at the moment, no. It seems like it ought to be *possible*, though, since the reverse translation can be done. Eggs are more restrictive in what they can include, so the reverse step actually ought to be relatively easy. Indeed, I would think that it could be done by a standalone tool without even using setuptools. All that really needs to happen (I believe) is that the zipfile directory needs all its names prepended with PURELIB or PLATLIB, and then add the appropriate prefix .exe and bdist_wininst extra data on the front of the restructured zip file. In fact, it should probably be possible to write such a tool by subclassing the distutils bdist_wininst command and overriding the run() and get_inidata() methods, using the existing create_exe() method to do that part of the magic. The other tool that would be handy to have, would be one that unpacks eggs into standard distutils-style installation. > > Personally, I'm not very thrilled with the number of complaints on > > this thread that could be resolved by RTFMing. >... >Honestly, I'm trying to help improve (by my measure of improvement, >certainly) setuptools. I've done as much (more!) homework as I feel is >appropriate (no, I haven't studied the whole manual all the way >through). Being treated as if it's my fault, and I haven't done >enough, is both discouraging and to be honest, somewhat offensive. My comment wasn't aimed specifically at you; you're only one of many people today who have appeared to state that something or other wasn't possible or documented, described optional behavior as required, etc. Addressing each and every one point by point looks petty, but then lumping them together like that makes it look like I'm picking on you specifically. Sorry about that. In any event, I'm not saying that anyone hasn't done enough or that it's their fault. The fact that I'm not thrilled about some of the things said in the thread doesn't somehow magically invalidate other people's frustrations, nor was it my intent to accuse you (or anyone) of making up their problems. I'm just expressing *my* frustration. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 05:55 PM 3/20/2008 +, Paul Moore wrote: >It's not that I object to the existence of automatic dependency >management, I object to being given no choice, as if my preference for >handling things manually is unacceptable. Note that easy_install has a --no-deps option, and you can make it the default in your distutils.cfg file. Also, setuptools-based packages *can* build bdist_wininst installers. (In fact, if memory serves, I added that feature at your request.) Personally, I'm not very thrilled with the number of complaints on this thread that could be resolved by RTFMing. There are extensive manuals, and they do contain the information that some people are saying isn't there. In several cases that I've seen here today alone, there are actually *entries in the tables of contents* that name the precise thing people here are characterizing as undocumented or even *impossible*, like: * Making your package available for EasyInstall * Installing on Un-networked Machines * Custom Installation Locations * Restricting Downloads with --allow-hosts It's easy to get the impression that people not only didn't RTFM, they didn't even Read The Friendly Table Of Contents of the said M. Nor, when, they found something in the manual that they didn't understand, write to the distutils-sig to ask anybody to explain, and perhaps suggest ways the FM's could be improved. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] The "autoinstall" package just uploaded to PyPI
I just wanted to throw in a quick note that this package: http://pypi.python.org/pypi/autoinstall which was just uploaded by Daniel Krech, is a lot closer in spirit to what I was trying to accomplish with PEP 365 than Guido's bootstrap proposal. Perhaps there's room for both in the stdlib? (And note that even though the examples use eggs, it does not do anything egg-specific; any zipfile importable by Python works with autoinstall.) There are a number of changes I would suggest making to autoinstall, like making it possible to access information about files in the cache, supporting non-toplevel modules, programmatic and environment-level control of the cache directory, that sort of thing. Heck, it'd be nice (although not essential) for it to support finding the right URL from PyPI. I also suspect that users might want to have some way to disable it or restrict it to certain hosts, etc., perhaps through a configuration file. It should probably also default the cache to a temporary directory, in the absence of other input. (Experience with pkg_resources' caching approach suggests that using the current directory or a home-directory-based location by default was a bad idea, at least without a fallback to a tempdir on write failure.) Any thoughts? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 09:44 AM 3/20/2008 -0400, Tres Seaver wrote: >I don't know how to make this requirement compatible with using shared >dependencies, except to make it easier for folks to download *all* the >requirements, and later install from the local "distribution cache" (a >directory full of .zip / .egg / .tgs files). It does turn out to be >quite easy to build a PyPI-style "simple" index for such a cache. Your >use case would then require: > > 1. Run some command to fetch the desired package and the transitive > closure of its dependencies into a working directory (the cache). > > 2. Run another command to build an index for that directory. > > 3. Run 'easy_install', pointing to the local index. Actually, if someone were to develop a patch for PyPI to do this, we could perhaps have a "display download dependencies" link for eggs shown on PyPI. That way, someone who wants to do a manual download could get a page with links for all the required eggs, and manually download them. (Of course, the other alternative would be for someone to provide an IE-controlling extension to urllib2 so that easy_install wouldn't be proxy-bound on such machines.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 09:33 AM 3/20/2008 +, Paul Moore wrote: >1. No integration with the system packager (Windows, in my case). If I >do easy_install nose, then nose does not show up in add/remove >programs. That significantly affects the way I manage my PC. The long-term fix here is probably to have a platform-specific installer capable of either turning eggs into .msi or .exe installers, or of doing the add/remove programs integration directly. (Someone, of course, will have to step up to create such a tool.) >5. Auto-discovery doesn't always work. I'm sorry, I really can't >recall the example at the moment, but sometimes easy_install says it >can't find a package I *know* is available. Sometimes it does that to me, too. But then I look at the project's page in PyPI, and they don't have a link to a download page. Usually, they've got a link to a page on their site with instructions about downloading, but that doesn't directly link to any tarballs or anything. So I grab the link of the real download page and paste it into a -f option to easy_install, so it knows where to find the link from. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [Distutils-SIG] PYTHONPATH installation (was Re: PEP 365 (Adding the pkg_resources module))
At 10:18 PM 3/19/2008 -0600, zooko wrote: >The fact that easy_install creates a site.py that changes the >semantics of PYTHONPATH is probably the most widely and deservedly >hated example of this kind of thing [2]. Yep, this was an unfortunate side effect of eggs growing outside their original ecological niche. Without the 'site' hack, it was impossible to install eggs to user directories and avoid installation conflicts. Specifically, if someone installed a package to PYTHONPATH with the distutils, and then installed a later version using setuptools, the setuptools-installed version would always end up on sys.path *after* the distutils-installed version. Detecting this condition and handling it properly was a major problem for users of easy_install, who wanted it to "just work". Standardization of a PEP 262-style installation database is still needed to address these problems, not to mention uninstallation. Maybe now with some package manager folks paying some attention here, we can do something about that. >[2] http://www.rittau.org/blog/20070726-02 >And no, PJE's suggested "trivial fix" does not satisfy the >objectors, as it can't support the use case of "cd somepkg ; python >./ setup.py install ; cd .. ; python -c 'import somepkg'". Well, it replaces the hack being complained about, with the problem that the hack was introduced to fix. :) Again, to properly fix this, we need a metadata standard for who owns what packages -- and it should probably include information about the *tool* that did the installation, so that system packagers can either tell Python-level tools to keep their hands off, or tell Python how to run the tool in question. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:58 AM 3/20/2008 -0400, Tres Seaver wrote: >A lot of setuptools warts are driven by related design problems in the >distutils, such as the choice to use imperative / procedural code for >everything: a declarative approach, with hooks for cases which actually >need them (likely 5% of existing packages) would have made writing tools >on top of the framework much simpler. It is ironic that Python is *too >powerful* a tool for the tasks normally done by distutils / setuptools: > a more restricted, and thererfore introspectable, configuration-driven >approoach seems much cleaner. +1 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
At 05:15 PM 3/19/2008 -0500, Jeff Rush wrote: >Phillip J. Eby wrote: > > At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: > >> Are you open to giving certain others patch view/commit privileges to > >> setuptools? > > > > Jim Fulton has such already. I'm open to extending that to others who > > have a good grasp of the subtleties involved. > > > > Truthfully, if we can just get 0.6 put to bed, I could probably open up > > the trunk a lot wider. > >What is needed to put 0.6 to bed? How can we help accelerate this? Get a tracker set up. I'm already in the main Python one, might as well use that. >It certainly is possible for someone to create a parallel packaging moduleset >that uses the existing eggs format and PyPI but without the currently >codebase, and then, once proven to work, lobby for it as distutils 3000. Yep. And I believe that something will look rather more like zc.buildout than setuptools, actually. Specifically in being data-driven rather than script-driven, and in the flexibility of what sort of parts get build and by what methods. Setuptools is still too rooted in distutils' world, the world where you can't depend on any other components being around to build things with. >Frankly I'd like to see setuptools exploded, with those parts of general use >folded back into the standard library, the creation of a set of >non-implementation-specific documents of the distribution formats and >behavior, leaving a small core of one implementation of how to do it and the >door open for others to compete with their own implementation. Apart from the exploding part, there are already documents. The only thing that makes them implementation-specific is that they haven't passed through any magic blessing process to make them standards. >You should document those ideas someplace and start getting community input. >There are a lot of diverse opinions on the right way to do this and the way >ahead is quite unclear. We might be talking about different things, as I'm more concerned with replacing setuptools and distutils on the build-and-distribute side. What's needed there is more the weeding out of too many ways to do simple things, and fixing the complete absence of ways to do complex things. :) For simple things the distutils are too hard, and for slightly-more-complex things, the entry barrier encourages people to abandon and replace them. On the package management side, I'm somewhat more inclined to agree with the need for a community approach, though. > > btw, offtopic question: are you by any chance the same Jeff Rush who > > invented EchoMail? > >Yep, that's me. Not many remember the Fidonet days. I designed >EchoMail on a >napkin during a DFW Sysop pizza party during a conversation on what >to do with >the unused capability of inter-BBS private file transfers. It too >escaped its >ecosystem and spread like wildfire, almost getting banned from Fidonet. ;-) Ah, so you *do* know what it's like to develop setuptools, then. I might even have met you at the one DFW sysop pizza party I ever attended. Back then, I ran the FreeZone, and before that, "Ferris Bueller's Fine Arts Forum", back in the late 80's and early 90's. My wife met me through the D/FW BBS list in the back of Computer Shopper, with a modem she bought at Software, Etc., up in Allen or wherever that place was. Not the chain store, the little consignment shop. Those were the days. But now we're *really* getting off-topic. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 10:48 AM 3/19/2008 -0700, Guido van Rossum wrote: >I don't understand PyPI all that well; it seems poor design that the >browsing via keywords is emphasized but there is no easy way to >*search* for a keyword (the list of all packages is not emphasized >enough on the main page -- it occurs in the side bar but not in the >main text). I assume there's a programmatic API (XML-RPC?) but I >haven't found it yet. http://wiki.python.org/moin/CheeseShopXmlRpc There's also a REST API that setuptools uses: http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api The API was originally designed for screen-scraping an older version of PyPI, but that has been replaced with a "lite" version served from: http://pypi.python.org/simple/ The "lite" version is intended for tools such as easy_install to process, as it consists strictly of links and can be statically cached. Zope Corp., for example, maintains a static mirror of this API, to guard themselves against PyPI outages and slowdowns, since their buildouts can involve huge numbers of eggs, both their own and external dependencies. >I'd love it if you could write or point me to code that takes a >package name and optional version and returns the URL for the source >archive, and the type (in case it can't be guessed from the filename >or the Content-type header). You can probably do that with the XML-RPC API. There's a function to get the versions of a package, given a (case-sensitive) name, and there's a function to get information for uploaded archives, given a name and a version. I originally intended to use it for the PEP 365 approach, but you can get the necessary information in just one static roundtrip using the REST (/simple) HTML API, if you're willing to parse the URLs for version information. (The catch of course being that distutils source distributions don't have unambiguously parseable filenames.) >Hm. Why not just use the existing convention for running setup.py >after unpacking? This works great in my experience, and has the >advantage of having an easy fallback if you end up having to do this >manually for whatever reason. Because I want bootstrap-ees to be able to use the bootstrap mechanism. For example, I expect at some point that setuptools will use other, non-self-contained packages, and other package managers such as zc.buildout et al also want to depend on setuptools without bundling it. > > * calling the bootstrap module 'bootstrap', as in 'python -m > > bootstrap projectname optionalversion'. The module would expose an > > API to allow it to be used programmatically as well as the command > > line, so that bootstrapped packages can use the bootstrap process to > > locate dependencies if they so desire. (Today's package management > > tools, at least, are all based on setuptools, so if it's not present > > they'll need to download that before beginning their own > > bootstrapping process.) > >This sounds like going beyond bootstrapping. My vision is that you use >the bootstrap module (with the command line you suggest above) once to >install setuptools or the alternate package manager of your choice, >and then you can use easy_install (or whatever alternative) to install >the rest. Well, I noticed that the other package managers were writing bootstrap scripts that then download setuptools' bootstrap script and run it as part of *their* bootstrap process... and then I got to thinking that it sure would be nice for setuptools to not have to be a giant monolithic download if I wanted to start using other packages in it... and that it sure would be nice to get rid of all these bootstrap scripts downloading other bootstrap scripts... and then I wrote PEP 365. :) One other thing that PEP 365 does for these use cases that your approach doesn't, is that pkg_resources could detect whether a desired package of a usable version was *already* installed, and skip it if so. So, we've already scaled back the intended use cases quite a bit, as people will have to write their own "is it already there?" and "is it the right version?" checks. > > Without one or the other, the bootstrap tool would have to grow a > > version parsing scheme of some type, and play guessing games with > > file extensions. (Which is one reason I limited PEP 365's scope to > > downloading eggs actually *uploaded* to PyPI, rather than arbitrary > > packages *linked* from PyPI.) > >There are two version parsers in distutils, referenced by PEP 345, the >PyPI 1.2 metadata standard. Yes, and StrictVersion doesn't parse release candidates. And neither LooseVersion nor StrictVersion supports handling multiple pre/post-release tags correctly. (E.g. "1.1a1dev-r2753") > > So, if I had to propose something right now, I would be inclined > to propose: > > > > * using setuptools' version parsing semantics for interpretation of > > alpha/beta/dev/etc. releases > >Can you point me to the code fo
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: >Are you open to giving certain others patch view/commit privileges >to setuptools? Jim Fulton has such already. I'm open to extending that to others who have a good grasp of the subtleties involved. Truthfully, if we can just get 0.6 put to bed, I could probably open up the trunk a lot wider. One of the things that slows me down is that patches usually don't come with tests, so I usually have to manually smoke-test them for scenarios I think they'll effect. There isn't really any automated procedure. Probably the most frustrating thing (or "chief amongst the most frustrating things") about setuptools development is that it's a black hole. By which I mean that backward compatibility and cruft accretion make it difficult to get out of. In the beginning, there was the distutils. Distutils begat setuptools, and setuptools begat virtualenv and zc.buildout and source control plugins. Etc., etc. What I think is really needed in the long run is to keep eggs, but get rid of setuptools and the distutils in their current form. There's a lot of brokenness there, and also a lot of accumulated cruft. We really need a distutils 3000, and it needs to be built on a better approach. In truth, my *real* motivation for PEP 365's bootstrap tool isn't so much to support the package management tools we have today, as it is to support a new one tomorrow. I have a few ideas for ways to shift the paradigm of how individual projects get built, to incorporate many scenarios that don't work well now. But to implement those things in such a next-generation tool, I will not want to be restricted to just what's in the stdlib or what can be bundled in the tool. (Btw, by "real" motivation, I don't mean I've been deceptive about my intentions, I mean that my strong intuition that such a bootstrap facility is needed, is probably being fueled by the long term desire to replace the entire distutils-based infrastructure with something better.) > I'd be willing to help out, and keep a carefully balanced hand in > what is accepted. And I think it's probably getting close to time I stepped down from day-to-day management of the codebase (which is more like month-to-month or quarter-to-quarter for me lately). It will probably be a lot easier for me to step back and critique stuff that goes in, after the fact, than to go over the stuff beforehand. :) I'm not sure exactly how to go about such a handoff though. My guess is that we need a bug/patch tracker, and a few people to review, test, and apply. Maybe a transitional period during which I just say yea or nay and let others do the test and apply, before opening it up entirely. That way, we can perhaps solidify a few principles that I'd like to have stay in place. (Like no arbitrary post-install code hooks.) btw, offtopic question: are you by any chance the same Jeff Rush who invented EchoMail? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] Capsule Summary of Some Packaging/Deployment Technology Concerns
We should probably move this off of Python-Dev, as we're getting into deep details now... At 07:27 PM 3/18/2008 -0500, Dave Peterson wrote: >If you really wanted to do a full-tree intersection, it seems to me >that the problem is detecting all the dependencies without having to >spend significant time downloading/building in order to find them >out. This could be solved by simply extending the cheeseshop >interface to export the set of requirements outside of the egg / >tarball / etc. We've done this for our own egg repository by >extracting the appropriate meta-data files out of EGG-INFO and >putting it into a separate file. This info is also useful for users >as it gives them an idea of how much *new* stuff is going to be >installed (a la yum, apt-get, etc.) ...and now we're more directly competing with them, too. The original idea Bob and I had was to do XML files ala Eclipse feature repositories, but then later I realized that for what we were doing, HTML was both adequate and already available. However, I don't see a problem in principle with having "header" files available for this sort of thing. >With our ETS projects, we've run into problems with the current >heuristic. Perhaps we just don't know how to make it work like we want? > >We have a set of projects that we want to be individually >installable (to the extent that we limit cross-project dependencies) >but we also want to make it easy to install the complete set. We >use a meta-egg for the latter. It's purpose is only to specify the >exact versions of each project that have been explicitly tested to >work together -- you could almost think of it as a source control system tag. I would think that as long as that meta-egg specifies *all* the required versions (right down to recursive dependencies), then there shouldn't be any problem. Maybe it's me who's not understanding something? I would think that you could get the appropriate data by running the tl.eggdeps tool. >A number of projects want to provide various types of files besides >code in their distributable, and they'd like these to end up in >standard locations for that type of file. Think documentation, >sample data, web templates, configuration settings, etc. Each of >these should be treated differently at installation time depending >on platform. On *nix, docs should go in /usr/share/doc whereas we >might need to create a C:\Python2.5\docs on Windows. With sample >data and templates, you probably just want it accessible outside of >the zipped egg so users can easily look at it, add to it, edit it, >etc. Configuration settings should be installed with some defaults >into a standard configuration directory like /etc on *nix, etc. > >Basically the issue is that it needs to be easier to include >different sets of files into an egg for different actions to be >taken during installation or packaging into an OS-specific distribution format. Yes, it would be nice to define a metadata standard for including installable "datasets" either through copying or symlinking, optionally with entry points for running some code, too. When you install an egg, these things could get added to a "post-install to-do" list, that you could then read to find out what steps to do, or invoke a tool on to actually do some of those steps. >But the docs for easy_install claim that the list of active eggs is >maintained in easy-install.pth. Also, if I create my own .pth file, >and the user tries to update my version to a new one, will the >easy_install tool modify my .pth file to remove the mention of the >old version from my sys.path and put the new version in the same >.pth file? Or will it now be listed in both places? Or will it >only in easy-install.pth? My understanding of the context of the question was that it applied to *system* packaging tools, which would be exclusively maintaining the .pth entries for the packages they installed. i.e., a scenario with *no* easy-install.pth. Setuptools will still detect the presence of their eggs, regardless of the means by which they're added to sys.path. But it would not *maintain* those .pth files. >Yes, but as you've already pointed out, they've escaped into a >larger ecosystem and this restriction is a severe limitation -- >leading to significant frustration. Especially as projects evolve >and want to do something more complex than simply install pure >Python code. Here at Enthought, we use and ship a number of >projects that have extensions and thus dynamic libraries that need >to either be modified during installation to work from the user's >installed location, or copied elsewhere on the system to avoid the >need to modify (which we also can't do via an egg install) env >variables, registries, etc. By the way, there *is* experimental shared library building support in setuptools, and I recently heard from Andi Vajda that he was successful in using it in his JCC p
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 03:43 PM 3/18/2008 -0500, Guido van Rossum wrote: >Only very few people would care about writing a setup >script that works with this bootstrap module; basically only package >manager implementers. That's true today, sure, but as soon as it is widely available, others are sure to want to use it too. I just want a bright-line distinction between what is and isn't bootstrappable, rather than a murky region of "maybe, if you're not doing anything too complicated". >There seems to be a misunderstanding about what I am proposing we do >instead. The boostrap installer should only be powerful enough to >allow it to be used to install a real package manager like setuptools. Which is why PEP 365 proposed only downloading an archive to a cache directory, and optionally running something from it. It explicitly disavows "installation" of anything, since the downloaded archive wouldn't have been added to sys.path except for the duration of the bootstrap process, and no scripts were to be installed. (Indeed, apart from the methods it would have used to locate the archive on PyPI, and to determine what to run from inside it, there was nothing particularly egg-specific about the proposed bootstrapping process.) So, to fully egg-neutralize the bootstrapping approach, we need only know how to locate an appropriate archive, and how to determine what to run from it. For the latter, we could use the already-in-2.6 convention of running __main__ from a zipfile or directory. (Too bad distutils source distributions have an extra directory name embedded in them, so one can't just execute them directly. Otherwise, we could've just let people drop in a __main__.py next to setup.py. OTOH, maybe it would be enough to use setuptools' algorithm for finding setup.py to locate __main__.py, and I'm fairly sure *that* can be briefly expressed in the PEP.) The other open question is a naming convention and version detection, so that the bootstrap tool can identify which of the files listed on PyPI is suitable for its use. (Both with regard to the version selection, and file type.) However, if PyPI were to grow support for designating the appropriate files and/or versions in some other way, we wouldn't need a naming convention as such. Without one or the other, the bootstrap tool would have to grow a version parsing scheme of some type, and play guessing games with file extensions. (Which is one reason I limited PEP 365's scope to downloading eggs actually *uploaded* to PyPI, rather than arbitrary packages *linked* from PyPI.) So, if I had to propose something right now, I would be inclined to propose: * using setuptools' version parsing semantics for interpretation of alpha/beta/dev/etc. releases * having a bdist_bootstrap format that's essentially a bdist_dumb .zip file with the internal path prefixes stripped off, making it an importable .zip with a different file extension. (Or maybe just .pyboot.zip?) The filename convention would use setuptools' canonicalization and escaping of names and version numbers, to allow unambiguous machine parsing of the filename. A __main__ module would have to be present for the archive to be run, as opposed to just being downloaded to a temporary directory. * calling the bootstrap module 'bootstrap', as in 'python -m bootstrap projectname optionalversion'. The module would expose an API to allow it to be used programmatically as well as the command line, so that bootstrapped packages can use the bootstrap process to locate dependencies if they so desire. (Today's package management tools, at least, are all based on setuptools, so if it's not present they'll need to download that before beginning their own bootstrapping process.) Apart from keeping the PEP self-contained and short, is there anything in this that you think you would object to? (You may reserve the right, of course, to later not like something in the details of setuptools' version/filename rules, after I've put them into the PEP, or really, anything else. I'm just asking if there's anything that's obviously offensive at this point, before I spend time on a new PEP.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:31 AM 3/18/2008 -0500, Guido van Rossum wrote: >I am hoping that someone will create a simpler bootstrap module that >is able to download a file of pure Python code and install it, perhaps >by running its setup.py, assuming that it only depends on distutils >(or other things previously installed). I will welcome such a module >into the stdlib. I'm not sure a PEP is even needed, though interested >parties are certainly welcome to write a PEP specifying the behavior >first. With 2.6 and 3.0 slated for release in September, there should >be enough time to get this done before then. Unfortunately, as I've already tried to explain, "download a file ... and install it" is not a sufficiently well-specified requirement to implement a robust tool. Even if it is not to support arbitrary existing distutils sources, there still needs to be a way to document precisely what the tool does and does not support installing, so that users can produce correct files for it to consume, register them properly with PyPI, etc. And as I said before (perhaps not very well) the distutils documentation has already proven to be inadequate as such a specification, both for users to create these files -- and even more important -- for programs to consume them. (For example, distutils source distribution tarball filenames are not unambiguously machine-parseable.) That's why I think some specific "format" (i.e. conventions) have to be defined for this to work, even if it's merely a set of documented restrictions on a distutils-based layout, file naming conventions, versioning, etc. In other words, you can't have your cake and eat it, too. If there's to be a bootstrap tool, you must bless *some* set of packaging conventions, including file naming, version parsing, and so on. Can we use setuptools' version parsing scheme to identify the "latest stable version", for example? What about setuptools' filename component canonicalization and escaping rules? Frankly, I don't care what the conventions are, only that they be unambiguously defined and reasonably implementable for producers and consumers alike. I just want there to be *some* sort of robust, documented, standard installation bootstrap vector in the stdlib, so that setuptools, zc.buildout, and virtualenv don't have to maintain their own (or depend on setuptools maintaining its own). But not only have you rejected the *only* existing robust and well-documented conventions for automated processing of Python libraries, you say you "have no time for this part of the thread" when I ask what conventions you want to bless *instead*. So I'm at a bit of a loss for what we're supposed to do now. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns
At 05:10 PM 3/17/2008 -0500, Jeff Rush wrote: >I was in a Packaging BoF yesterday and, although not very relevant to the >packager bootstrap thread, Guido has asked me to post some of the concerns. > >The BoF drew about 15 people, many of whom were packagers for Red Hat, Ubuntu >and such. Everyone had strong expressions of frustration with the status quo >and most had tried to resolve their issues but had their patches rejected. I >am not taking either side and whether those rejections were >justified I cannot >say, but the general feeling of their concerns intentionally not being >addressed isn't healthy. Several had abandoned setuptools, deeming it a >failed solution and others called for a fork. > >To start, I am not a leader of the group nor do I claim I accurately captured >and expressed all their concerns. I apologize to those in the BoF for any >misrepresentations. I'm actually happy to hear that there's this much energy available -- hopefully some of it can be harnessed towards positive solutions. When I began developing setuptools, I often asked for the input of packagers, developers, etc., through the distutils-sig... and was met with overwhelming silence. So the fact that there is now a group of people who are ready to work for some solutions seems like a positive change, to me. It's hard to make design decisions regarding itches you don't personally have, and which other people won't help scratch. Unfortunately, a lot of the proposals from packaging system people have been of the form of, "fix this for us by breaking things for other people". Not all of them, though. Many have been very helpful, contributing troubleshooting help and good patches. That some of those good patches took nearly a year to get into setuptools (some from Fedora just got into 0.6c8 that were sent to me almost a year ago) is because I'm the only person reviewing setuptools patches, and I've spent only a few days in the last year doing focused development work on setuptools (as opposed to answering questions about it on the SIG). It's never a good thing when people's patches sit around, regardless of where they come from. But that's not the same thing as *rejecting* the patches. >1. Many felt the existing dependency resolver was not correct. They wanted a > full tree traversal resulting in an intersection of all restrictions, > instead of a first-acceptable-solution approach taking now, which can > result in top-level dependencies not being enforced upon lower > levels. The > latter is faster however. One solution would be to make the resolver > pluggable. Patches welcome, on both counts. Personally, Bob and I originally wanted a full-tree intersection, too, but it turned out to be hairier to implement than it seems at first. My guess is that none of the people who want it, have actually tried to implement it without a factorial or exponential O(). But that doesn't mean I'll be unhappy if somebody succeeds. :) Intuitively, it seems easy, just gather the requirements and intersect. In practice, different versions of a package may have different dependencies, so the intersection is not nearly as simple as that. We ended up just going for a depth-first version of the current algorithm (switched to breadth-first later, after field tests showed some problems with that), being greedy by testing latest-version-first, on the assumption that more recent versions would be likely to have the most-restrictive version requirements. In other words, we attempt to achieve heuristically what's being proposed to do algorithmically. And my guess is that whatever cases the heuristic is failing at, would probably not be helped by an algorithmic approach either. But I would welcome some actual data, either way. Again, though, patches are welcome. :) (Specifically, for the trunk; I don't see a resolver overhaul as being suitable for the 0.6 stable branch.) >2. People want a solution for the handling of documentation. The distutils > module has had commented out sections related to this for several years. As with so many other things, this gets tossed around the distutils-sig every now and then. A couple of times I've thrown out some options for how this might be done, but then the conversation peters out around the time anybody would have to actually do some work on it. (Me included, since I don't have an itch that needs scratching in this area.) In particular, if somebody wants to come up with a metadata standard for including documentation in eggs, we've got a boatload of hooks by which it could be done. Nothing's stopping anybody from proposing a standard and building a tool, here. (e.g. using the setuptools command hook, .egg-info writer hook, etc.) >3. A more flexible internal handing of the different types of files is needed. > Currently the code, data, lib, etc. files are aggregated at > build time and > people woul
Re: [Python-Dev] [Distutils] PEP 365 (Adding the pkg_resources module)
At 02:44 PM 3/17/2008 -0500, Jeff Rush wrote: >Guido van Rossum wrote: > > On Mon, Mar 17, 2008 at 11:35 AM, Paul Moore <[EMAIL PROTECTED]> wrote: > >> > >> I'm +lots on someone giving a clear explanation of the meaning and > >> interrelationship of the various terms involved in this discussion > >> (setuptools, easy_install, pkg_resources, eggs, "package managers" as > >> distinct from setuptools, etc etc) so that the discussion gets some > >> much-needed clarity :-( > > > > Right. But finding someone who can explain all this is apparently > > hard. All the owners of package managers seem busy... > >In preparing for my PyCon 2008 tutorial on eggs and buildout, I spent three >full-time weeks carefully going over sources for distutils, setuptools and >buildout to discover those aspects not documented. I can explain how they >work, although I'm not sure this is the correct forum. I'd like to first >offer my slides from my tutorial, 150 of them with detailed handout notes on >many of them. > >http://wiki.python.org/moin/buildout/pycon2008_tutorial Wow. I am skimming over the 44-page one on setuptools, and that is definitely the most comprehensive doc anyone has produced on it, aside from the official docs. Thank you! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 01:59 PM 3/17/2008 -0500, Guido van Rossum wrote: >I have certainly personally encountered plenty of situations where I >wasn't able to complete an egg-based install because some dependency >was broken (e.g. not available for the Python version I was using). That's odd -- setuptools-based installs should be able to find and install packages from source. I have noticed a recent phenomenon where new developers upload *only* an egg to PyPI, without the source, but that's usually short-lived until someone points it out to them. Do you happen to know what packages you had this problem with? >I'm okay if setuptools, once it's been installed, runs some setup code >that creates the .egg-info directory and whatever else. This means I'm >also okay with the bootstrap module finding and invoking that setup >code. But I'm *not* okay with building any kind of egg management into >the bootstrap module. The bootstrap module must be be neutral w.r.t. >the package management style. Ok, well then we'll have to invent a new kind of binary package, whose name isn't 'egg'. Supporting distutils source packages is almost certainly a non-starter, if you want to avoid bringing the rest of setuptools into play. The only way to correctly determine what a source package contains is to run its setup script... and running unboxed setup scripts isn't safe because there are people who hardcode paths (or more precisely, use bad ways of computing them) in their setup scripts. I'm not saying the tool needs to guard against *malicious* scripts, just badly-written ones. (Setuptools does this with its "sandboxing" module, when running source packages' setup scripts.) So, if source is out, then some binary format is needed, which means defining the conventions for said format... i.e. "eggs lite" or "egg substitutes". :) > > So, it might be simpler all around to just clear up the > > "controversy". To the best of my recollection, only MAL and MvL have > > ever objected on Python-Dev to the idea of supporting eggs. > >You can add my name to the list. I've heard plenty of people speak >highly of eggs, but I've *also* heard from plenty of people (besides >MAL and MvL) who have serious difficulties with the concept of eggs. I did say "on Python-Dev", and you implied that it was not controversial with you, except for the maintenance-related concerns. I'm not fighting about this, but I would rather you were straight-up with your objections rather than deferring it to a controversy that "might go away in a few years". That way, I could at least attempt to do something about the concerns. OTOH, if your objections were non-specific and likely to stay that way, then I could have at least not wasted your time with any of this. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote: >On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby ><[EMAIL PROTECTED]> wrote: > > At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote: > > >There will be no egg support in the standard library. > > > > Are there any qualifications on that statement, or is this in the > > same category as "from __future__ import braces"? > >IIUC eggs are a method of package management that includes support for >dependencies, multiple versions, and C extensions in zip files, as >well as conventions for naming these and for encoding metadata (e.g. >how to find out the version or the dependencies). This whole set of >conventions is IMO too much to include into the stdlib ATM -- if only >because it has proved controversial in the past. Maybe a few years >from now it will be no longer controversial and then my objections >will disappear. So, does this mean that the bootstrap tool must not use eggs? That seems a little bit odd, in that setuptools will at least need its .egg-info directory to get installed, and all of the people who'll be using this initially will be using it precisely in order to have support for eggs... So, it might be simpler all around to just clear up the "controversy". To the best of my recollection, only MAL and MvL have ever objected on Python-Dev to the idea of supporting eggs. Note: I'm specifically segregating "egg support" from the topic of including setuptools or easy_install in the stdlib directly. There are many legitimate reservations and open questions about the latter, including availability of volunteer support, choice of defaults, whether to replace distutils with setuptools, etc. etc. I recognize and respect the validity of those issues, which is precisely why I withdrew setuptools from inclusion in Python 2.5. However, regarding support for eggs, my understanding is that there were only two objections to eggs -- even at the time of the 2.5 setuptools discussions. And even though MvL objects to the idea of eggs in *principle*, I didn't read his recent posts as objecting to having the bootstrap tool download and install eggs in *practice*. (Although I hope he will clarify that stance one way or the other.) That leaves MAL, whose objections to PEP 365 centered on the API (he said he was "+1 on the concepts being added to the stdlib, -1 on adding the module in its current state"). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new "egglib" module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.) I would guess, however, that MAL's issues with the pkg_resources API would not apply to a bootstrap module whose sole purpose was to download eggs and put them on sys.path. Or, perhaps he would object *more*, I don't know. We could certainly ask him, though. :) So, was there anyone else you were counting towards "controversy"? The only other person I recall objecting to setuptools in any way on Python-Dev was effbot, and IIUC his objections were practical/administrative re: supporting easy_install and setuptools, not to the idea of .egg support in general. In summary, I think the controversy on Python-Dev regarding .egg support has actually been over for some time now. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote: >There will be no egg support in the standard library. Are there any qualifications on that statement, or is this in the same category as "from __future__ import braces"? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote: >I don't think this should play games with scripts being overridden or >whatever. If a bootstrap script is to be installed it should have a >separate name. I'm not sure what the advantage is of a bootstrap >script over "python -m bootstrap_module ..." though. And -m also makes explicit: 1. that it's a Python-specific tool 2. which Python version it will apply to >The PEP suggests that other package managers also benefit. How do they >benefit if the bootstrap script installs setuptools? Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :) >I'd also like to avoid the specific name "easy_install" for any of >this. That's a "brand name" (and a misleading one if you ask me, but >that's politics again :-). Ok, so if someone will propose a name and API for the thing, I'll implement it. (Assuming the proposed API is sane and reasonably implementable, of course.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 09:45 AM 3/17/2008 -0500, Martin v. Löwis wrote: > > Well, it might be replaced by a protracted discussion of how the > > module should work and what its API should be, but perhaps that would > > be a better one to have. :) > >Indeed, that's likely to happen :-) > > > So, the original proposal (from the previous thread about this) was > > that the module be named easy_install, and that it simply downloads > > setuptools and delegates to the "real" easy_install. That way, > > people can simply use "python -m easy_install ...", without worrying > > about whether setuptools has been installed yet. > >I thought the original proposal was to install a *binary* easy_install >that takes that function. What do you mean by "binary"? I thought we were talking about a module. Do you mean a script to be installed alongside Python itself in e.g. /usr/bin? In the original discussion, it was a module to be added alongside pkg_resources, which would use pkg_resources to find and/or install setuptools. I also personally like the use of -m instead of a script because it makes it quite clear that this is a Python-specific installation tool, and *which* version of Python, as well, without having to have easy_install-2.5, easy_install-2.6, etc. > > IIRC, other package management tools such as zc.buildout and > > workingenv can then be installed using easy_install. > > > > Any objections? Should I revise the PEP? > >I'm fine with the module, but would really like to see a command >line utility in addition. > >This, of course, would raise the issue who "owns" the easy_install >script name; ideally, the script would not have to be overwritten >when setuptools gets installed. It won't have to. The module will attempt to import the setuptools-supplied version of easy_install, and delegate to it if possible, before trying to do any download. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 08:48 AM 3/17/2008 -0500, Guido van Rossum wrote: >On Sun, Mar 16, 2008 at 7:06 PM, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > > So, if the consensus is that it would be better to have a module that > > only does bootstrap installs of pure-Python eggs from PyPI, I'm > > totally fine with that. > >Let's just do this; it will avoid a protracted discussion of the >merits of eggs, pkg_resources, and setuptools. Well, it might be replaced by a protracted discussion of how the module should work and what its API should be, but perhaps that would be a better one to have. :) So, the original proposal (from the previous thread about this) was that the module be named easy_install, and that it simply downloads setuptools and delegates to the "real" easy_install. That way, people can simply use "python -m easy_install ...", without worrying about whether setuptools has been installed yet. IIRC, other package management tools such as zc.buildout and workingenv can then be installed using easy_install. Any objections? Should I revise the PEP? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Quick summary of the below: I'm definitely fine with doing a simpler, pure-bootstrap module, if there's some consensus on what should go in it. I just wish we could've had this discussion last year, when OSAF was still able to fund the work... ;-) At 06:13 PM 3/16/2008 -0500, Guido van Rossum wrote: >Phillip asked me to give an opinion on his pkg_resources PEP. While >the PEP is short and sweet, the pkg_resources module itself is huge >(1800 non-blank lines; 16 classes plus 5 exceptions; it exports 67 >names in total according to __all__). And pkg_resources.txt is another >1700 lines of documentation. I find that hard to swallow. Is there >anyone besides Phillip who can claim he understands this module? Bob Ippolito actually wrote the very first version of pkg_resources. Others, such as Philip Jenvey of the Jython project, have provided patches. From previous discussions on the distutils-sig, I know that Jim Fulton has in-depth knowledge of both pkg_resources and easy_install. Of course, that's not the same as any of these guys volunteering to be maintainers. :) >If its inclusion is really meant just as a bootstrap to simplify >installing other package management solutions, as the PEP claims, I >would prefer to see something with a much smaller footprint. Actually, the PEP says: "pkg_resources is a module used to find and manage Python package/version dependencies and access bundled files and resources, including those inside of zipped .egg files. Currently, pkg_resources is only available through installing the entire setuptools distribution, but it does not depend on any other part of setuptools; in effect, it comprises the entire runtime support library for Python Eggs, and is independently useful." This kind of glosses over the part where this is also for runtime support of projects that use eggs. Which, these days, is, well, almost any large Python project, from Chandler to Enthought to Zope. > Surely >there is no need for example to have support for C extensions inside >zip files *as part of the bootstrap module*? It's a runtime; the PEP actually merely proposes that a further addition to be made to support bootstrapping, *also*. Otherwise, the PEP would be even shorter. :) The reason I proposed it this way was for simplicity -- and politics. Currently, people using setuptools in their setup.py have to include a similar bootstrap module to download setuptools if it's not available, and pkg_resources already has version checking logic and everything needed to find dependencies and download them. (Plus, I figured it'd be easier to just use what was already there and stable, rather than creating something different.) That was the simplicity part. The politics part was that: 1. I thought it would be less controversial to include the "runtime for eggs" than to include something that's just a bootstrapper for setuptools. However, MvL surprised me by actually being in *favor* of including a setuptools bootstrapper. 2. I thought that it would have broader acceptance if it was oriented towards bootstrapping *any* package, not just setuptools. So, if the consensus is that it would be better to have a module that only does bootstrap installs of pure-Python eggs from PyPI, I'm totally fine with that. >Unless I find someone besides Phillip who is interested in having this >included and is willing to help maintain it, I don't really think it >would be wise to accept this into the standard library. > >Phillip, in the PEP you mention that there are several other package >management tools that also like to use pkg_resources. Maybe you can >get some folks from those tools to speak up and explain what >pkg_resources means to them, and maybe even volunteer to co-own it >once it's in the standard library? The distutils-sig is the de facto place for discussions regarding those tools, so I've cc'd this there. Hopefully, one or more volunteers will step up if they want this. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Equality on method objects
At 12:26 PM 3/10/2008 +0100, Armin Rigo wrote: >Hi Phillip, > >On Sun, Mar 09, 2008 at 07:05:12PM -0400, Phillip J. Eby wrote: > > I did not, however, need the equality of bound methods to be based on > > object value equality, just value identity. > > > > ...at least until recently, anyway. I do have one library that wants > > to have equality-based comparison of im_self. What I ended up doing > > is writing code that tests what the current Python interpreter is > > doing, and if necessary implements a special method type, just for > > purposes of working around the absence of im_self equality > > testing. However, it's a pretty specialized case (...) > >I found myself in exactly the same case: a pretty specialized example >where I wanted bound methods to use im_self equality rather than >identity, solved by writing my own bound-method-like object. But that's >not really hard to do, and the general tendency (which matches my own >opinion too) seems to be that using im_self identity is less surprizing. > >In general, "x.append" is interchangeable with "x.append" even if >"x.append is not x.append", so let's go for the least surprizing >behavior: "m1.im_self is m2.im_self and m1.im_func==m2.im_func". >Objection? Nope; that's exactly what I proposed at the end of the email quoted above. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Equality on method objects
At 01:59 PM 3/9/2008 -0800, Guido van Rossum wrote: >Do we have much of a use case for this? I've often had APIs that take a callback that promise to only invoke the callback once, even if it's added more than once. And I've used dicts, lists, and sets for same. I did not, however, need the equality of bound methods to be based on object value equality, just value identity. ...at least until recently, anyway. I do have one library that wants to have equality-based comparison of im_self. What I ended up doing is writing code that tests what the current Python interpreter is doing, and if necessary implements a special method type, just for purposes of working around the absence of im_self equality testing. However, it's a pretty specialized case, and if I didn't have to support older Python versions I'd just use partial() -- assuming that partial() supports hashing and equality comparisons, that is, which I haven't checked. I imagine hashing a partial() might be at least as tricky as getting bound methods "right". :) >That said, if there's a use case, I agree that it would be okay with >basing the equality of x.foo and y.foo on whether x and y are the same >object, not on whether x==y (consider 0 .__add__ == 0.0 .__add__). +1 for making two bound methods m1 and m2 equal if and only if "m1.im_self is m2.im_self and m1.im_func==m2.im_func", and making the hash based on im_func and id(im_self). I don't think that the im_func comparison should be identity-based by default, however. (The im_func could be another bound method, for example.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documentation for ability to execute zipfiles & directories
At 05:40 PM 3/4/2008 +0300, Oleg Broytmann wrote: >On Wed, Mar 05, 2008 at 12:14:04AM +1000, Nick Coghlan wrote: > > As a more helpful answer, the ZIP spec allows additional data to be > > included in the file before the ZIP header. A more common way of using > > this is to add a zip file on to the end of an ELF executable while still > > using normal zipfile utilities to read the data in the zip file section > > and ignore the executable part. > > > > It turns out you can actually use the same trick to prepend a shebang > > line like "/usr/bin/env python" and a newline character > >That's what I thought, too. > > > - the whole zip > > file is still a binary file, but that doesn't prevent the shell from > > reading that first line of text and handing the file over to Python for > > execution. > >Unix doesn't distinguish text and binary files. (-: > > > The fact that this actually works was also news to me when the issue I > > linked in my previous post was first brought to my attention :) > >So it really works? Amazing! Setuptools has been distributed this way for some time: http://pypi.python.org/pypi/setuptools#cygwin-mac-os-x-linux-other It actually contains an entire shell script prefix that launches Python and invokes an entry point inside the egg. With the new interpreter capability, this would've been a *lot* simpler to implement. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] refleaks and caches
At 05:05 PM 1/26/2008 -0800, Neal Norwitz wrote: >Around Jan 13, the refleak hunting test that is reported on >python-checkins started to report refleaks on virtually every run. I >suspect this is due to r59944 (at 2008-01-13 16:29:41) which was from >patch #1700288 to cache methods. With this patch it makes it much >harder to spot refleaks. Does anyone have ideas how to fix it? The >only one I have is to disable the cache with a special flag, env't >variable, sys variable/function or the like. We could make this >available only in debug mode. The cache should normally be enabled, >but could be disabled solely on the refleak runs. > >Suggestions? Expose an API to clear the cache, and clear it at shutdown? It should probably be part of interpreter shutdown anyway. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: per user site-packages directory
At 04:42 PM 1/22/2008 +0100, M.-A. Lemburg wrote: >I don't really understand what all this has to do with per user >site-packages. > >Note that the motivation for having per user site-packages >was to: > > * address a common request by Python extension package users, > > * get rid off the hackery done by setuptools in order >to provide this. Setuptools doesn't do any hackery for per-user site-packages, although its documentation does explain how to set up such a thing if you want it: http://peak.telecommunity.com/DevCenter/EasyInstall#administrator-installation http://peak.telecommunity.com/DevCenter/EasyInstall#mac-os-x-user-installation Meanwhile, note that having per-user site-packages directories doesn't eliminate the need to be able to have PYTHONPATH directories treated as "site" directories, which is hasn't been discussed at all. >As such the PEP can also be seen as an effort to enable code >cleanup *before* adding e.g. pkg_resources to the stdlib. Code cleanup of what? There's nothing in pkg_resources that would change for per-user site package directories, since pkg_resources doesn't do any installation work. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (was Re: PEP: per user site-packages directory)
At 10:48 AM 1/21/2008 -0500, Steve Holden wrote: >Phillip J. Eby wrote: > > (Heck, if what you really want is to have easy_install support in > > 2.6, we could just as easily bundle an easy_install.py that asks for > > an install of setuptools if it's not already present.) > > >Would the easiest way to do this be to insert a default dependency on >setuptools? Sorry, I don't understand the question. What do you mean by "default dependency" and to what are you proposing it be inserted? :) What I meant was that we could include an easy_install.py whose sole function is to ensure that setuptools is installed and then invoke the "real" easy_install. Thus, the first time you ran easy_install, a current version would be downloaded, and thereafter the real one would be runnable. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 365 (was Re: PEP: per user site-packages directory)
At 01:06 AM 1/22/2008 +1000, Nick Coghlan wrote: >Steve Holden wrote: > > Christian Heimes wrote: > >> Steve Holden wrote: > >>> Maybe once we get easy_install as a part of the core (so there's no need > >>> to find and run ez_setup.py to start with) things will start to improve. > >>> This is an issue the whole developer community needs to take seriously > >>> if we are interested in increasing take-up. > >> setuptools and easy_install won't be included in Python 2.6 and 3.0: > >> http://www.python.org/dev/peps/pep-0365/ > >> > > Yes, and yet another release (two releases) will go out without easy > > access to the functionality in Pypi. PEP 365 is a good start, but Pypi > > loses much of its point until new Python users get access to it "out of > > the box". I also appreciate that resource limitations are standing in > > the way of setuptools' inclusion (is there something I can do about > > that?) Just to hammer the point home, however ... > >Have another look at the rationale given in PEP 365 - it isn't the >resourcing to do the work that's a problem, but the relatively slow >release cycle of the core. > >By including pkg_resources in the core (with the addition of access to >pure Python modules and packages on PyPI), we would get a simple, stable >base for Python packaging to work from, and put users a single standard >command away from the more advanced (but also more volatile) features of >easy_install and friends. By the way, if we're actually going to get that into 2.6, it would be good for the PEP to actually be approved before then. :) With respect to Steve's comments about out-of-the-box usability, it should be noted that when you bootstrap a package with pkg_resources, it should be possible to include other command-line arguments after the package specifier. So for example: python -m pkg_resources setuptools SomePackage==1.2 would download and install setuptools, and run its "bootstrap script" with "SomePackage==1.2" as a command-line argument. And setuptools' bootstrap script is basically easy_install with some extra code to make sure the setuptools egg gets installed too. In other words, with PEP 365 in place, "python -m pkg_resources setuptools" is basically a way to say "easy_install" without needing setuptools installed. (Heck, if what you really want is to have easy_install support in 2.6, we could just as easily bundle an easy_install.py that asks for an install of setuptools if it's not already present.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 04:40 AM 1/16/2008 +0100, Christian Heimes wrote: >Phillip J. Eby wrote: > > I guess it's not right then. ;-) Though I shouldn't make fun, since it > > turns out that my code sketch was not a correct translation of > > peak.util.imports. (See below.) > >*gr* I spent more than hour to find my error ... Sorry about that - as I said, __notified__ is very much an implicit thing in peak.util.imports. And I believe I've also mentioned a lot of times how hard it is to get this stuff right... :) > > That is, module.__notified__ has to be set *before* the recursive > > notification call. This effectively happens in peak.util.imports now, > > except that __notified__ isn't an explicit attribute, just a side effect > > of other module state changes. > >It's done. Your proposed test cases passes together with my tests. The >ref leak tests don't show a single missing reference. Congrats! Now all we need to do is get the authors of other lazy import/export/whatever systems to chime in with whatever additional invariants *they* might need... ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 02:28 AM 1/16/2008 +0100, Christian Heimes wrote:
>Phillip J. Eby wrote:
> > At 10:14 PM 1/15/2008 +0100, Christian Heimes wrote:
> >> My code queues up new hooks while a sequence of hooks is processed. It
> >> makes sure that hooks for a parent aren't called in the middle of a
> >> child's hook chain.
> >
> > Notice that that's not necessary with the notification algorithm I gave,
> > since the list in post_import_hooks suffices as a queue. So, just as in
> > peak.util.imports, the registration code doesn't need to know whether
> > callbacks are being run; it only needs to know whether they're *finished*.
>
>Are you sure your proposed algorithm and output match for the test case?
>I'm confident I got it right in C but I'm getting a different output.
I guess it's not right then. ;-) Though I shouldn't make fun, since
it turns out that my code sketch was not a correct translation of
peak.util.imports. (See below.)
>Without the extra imp.notify_module_loaded('a.b') in func_a1(mod)::
>
>['func_a1', 'func_a2', 'func_ab1', 'func_ab2', 'func_ab3']
>
>
>With the extra imp.notify_module_loaded('a.b') in func_a1(mod)::
>
>['func_a1', 'func_ab1', 'func_ab2', 'func_ab3', 'func_a2']
Right - that's why I put it in there, to foil trivial implementations
that don't really satisfy the invariant.
>I can't see how your implementation results in the first output when
>func_a1() calls the notification method.
Hm, you're right, my implementation sketch waits too long to set the
__notified__ flag. It should have read:
def notify(name):
try:
module = sys.modules[name]
except KeyError:
raise ImportError("Module %s has not been imported" % (name,))
if module.__notified__:
return
try:
module.__notified__ = True
if '.' in name:
notify(name[:name.rfind('.')])
for callback in post_import_hooks[name]:
callback(module)
finally:
post_import_hooks[name] = None
That is, module.__notified__ has to be set *before* the recursive
notification call. This effectively happens in peak.util.imports
now, except that __notified__ isn't an explicit attribute, just a
side effect of other module state changes.
>I'm aware of the implications and my code already uses the lock. The
>PyImport_NotifyLoaded() method excepts to be called with the importer
>lock acquired. So I'm locking the importer lock in
>imp_notify_module_loaded(). The PyImport_RegisterPostImportHook() method
>does the locking before it accesses sys.modules and sys.post_import_hooks.
Great!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Extending generic functions
At 02:19 PM 1/15/2008 -0800, Guido van Rossum wrote: >While I have you, I've come across a need that I don't know how to do >with GFs. Suppose I have a GF that implements some recursive function >over container types, e.g. serialization or flattening. Now suppose >I'd like to create *another* GF that implements the same algorithm >except it does something different for one particular type; as a >concrete example, suppose we want to treat tuples atomically when >flattening. Is there a way to reuse the work of the first GF? Yes. RuleDispatch actually has a 'clone()' feature for single-dispatch generics that does exactly what you're looking for: http://peak.telecommunity.com/DevCenter/VisitorRevisited (see the heading "Extension and Reuse"). It's probably not a bad idea to put a cloning feature on my extended to-do list for PEAK-Rules. In PEAK-Rules (the system after which PEP 3124 was modelled), a generic function has a RuleSet that contains its rules, and RuleSets can be subscribed to. So, you could create a listener that automatically takes the rules added to one function and adds them to others. It's not packaged as a convenient decorator or anything, but one certainly could make one. It'd also need to have some way to ensure that the rules from the original function were treated at a lower combination precedence than anything else, but that could be handled by with a custom method type pretty easily, I think. All in all, a cloning feature might be somewhere around 20-50 lines of code to add in -- and a third party could probably roll their own without changing PEAK-Rules' source. > It doesn't work to create a new GF that calls on the first GF for types >it doesn't understand; e.g. a list could contain a tuple. Does your GF >machinery let me do this in a relatively clean way? It's relatively clean. One of my goals for the changed architecture in PEAK-Rules vs. RuleDispatch was to make it possible to do all sorts of things like this, by opening up the whole thing to extensibility. Btw, a lot of the credit for PEAK-Rules' design goes to you, in a roundabout way. Your tuple-of-types prototype made me see that it could be practical to implement generic functions using generic functions as a base -- getting rid of interfaces and adaptation altogether. I just needed to come up with a design that allowed separating the genericness of a function (e.g. the rules to be applied) from the implementation of genericness (the "engine" that turns rules into executability. In this way, a generic function can start out using just tuples of types, but then graduate to a full expression-based system, just by changing the engine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Monkeypatching idioms -- elegant or ugly?
At 01:51 PM 1/15/2008 -0800, Guido van Rossum wrote: >On Jan 15, 2008 1:27 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > > Second, a "metaclass" to add a number of methods (or other attributes) > > > to an existing class, using a convenient class notation: > > > > I think this is similar to my "partial" classes: > > > > http://pypi.python.org/pypi/partial > >Indeed it is. I guess my only innovation is realizing that you don't >have to create a real metaclass -- you can set __metaclass__ to a >function that does the magic. I like your feature of refusing >overrides unless flagged with @replace. > >I think that despite the objection that monkeypatching shoudn't be >made too easy, it's worth at looking into a unification of the API, >features, and implementation. I'm curious: has this affected your thoughts re: overloading existing functions? Note that overloading-in-place would provide the next_method idiom for calling the original function. (I'm assuming you still don't like the idea of changing a function's code to do it, just wondering about the non-implementation aspect. :) ) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 10:14 PM 1/15/2008 +0100, Christian Heimes wrote: >My code queues up new hooks while a sequence of hooks is processed. It >makes sure that hooks for a parent aren't called in the middle of a >child's hook chain. Notice that that's not necessary with the notification algorithm I gave, since the list in post_import_hooks suffices as a queue. So, just as in peak.util.imports, the registration code doesn't need to know whether callbacks are being run; it only needs to know whether they're *finished*. Of course, both the notification and registration functions must hold the import lock to prevent a race condition where one thread adds a hook to the list after another thread has just finished iterating over it and is about to replace the list with None. At least, they have to if they're executing any Python code that might cause the GIL to be released. The callbacks will release the GIL, of course, but the registration code probably doesn't... well, it will if it calls the hook, and ISTM that the hooks should always execute with the import lock held, even if they're fired at registration. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 10:10 PM 1/11/2008 +0100, Christian Heimes wrote:
>Phillip J. Eby wrote:
> > *sigh*. We seem to be getting further and further off course,
> > here. The more different you make the semantics of the system, the
> > harder it will be to verify that it's doing the right thing without
> > having real field experience with the new approach.
>
>*sigh*, too. :/
>
>This discussion has neither helping me nor you. Could you please write
>an unit test or two to show me exactly what my implementation is doing
>wrong and how you think the callbacks should be called. I know a simple
>test won't proof the correctness of the implementation but a failing
>test would at least show the incorrectness.
when_imported('a.b')(func_ab1)
when_imported('a.b')(func_ab2)
@when_imported('a')
def func_a1(module_a):
when_imported('a.b')(func_ab3)
notify_module('a.b') # <- this is here to foil trivial implementations
when_imported('a')(func_a2)
notify_module('a.b')
This should produce the calling sequence:
func_a1, func_a2, func_ab1, func_ab2, func_ab3.
>I'm still not sure which way is the correct way in your opinion and I
>hate guessing what you are trying to explain to me.
The invariants to ensure are:
1. notification is only done once for a given module, ever, even if
the notification function is called more than once, even if it's
called during notifications for that module
2. notifications for a child module/package may not begin until the
notifications for the parent package have begun
3. two registrations for the same module must always be invoked in
the same order as they were registered, even if some of the
registrations are done during notification.
In order to implement these invariants, you will have to have a way
to know whether notifications have been begun for a given module. In
peak.util.imports, the module objects effectively keep track of this,
although they don't have a specific flag. For the Python
implementation, you could add a __notified__ field to module objects,
and implement the notify function thus:
def notify(name):
try:
module = sys.modules[name]
except KeyError:
raise ImportError("Module %s has not been imported" % (name,))
if module.__notified__:
return
if '.' in name:
notify(name[:name.rfind('.')])
try:
module.__notified__ = True
for callback in post_import_hooks[name]:
callback(module)
finally:
post_import_hooks[name] = None
Of course, __notified__ would actually be a structure slot, rather
than an attribute, so as to avoid any attribute lookup issues with
module subtypes (e.g. lazy modules).
The register function would simply append a hook to the entry in
post_import_hooks if it's not None, or call the hook otherwise.
With this implementation, I could make a version of peak.util.imports
that did its own lazy modules, but used the base system for all the hooks.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 12:08 AM 1/11/2008 +0100, Christian Heimes wrote: >Phillip J. Eby wrote: > > Yes, that's the general idea. But what happens if you are in the middle > > of firing hooks for 'a', and a new hook for 'a.b.c' is added? What > > about a new hook for 'a'? > >If 'a' registers a new hook for a child of 'a' (e.g. 'a.b.c' or 'a.f') >then the new hooks are called with the remaining hooks for 'a.b.c': > >import a.b.c >* hook_a1 >* hook_a1 -> registers the hook_ab2 for 'a.b' >* hook_ab1 -> registers a hook_aX for 'a' >hook_aX is fired immediately >* hook_ab2 -- the hook registered by hook_a1 This scenario isn't specific enough to produce/deduce the problem. > > Well, it certainly can (and should) do the same if a module object is > > provided, since the module has a __name__. > >Maybe I should add two methods to imp. One that calls the parent hooks >of a module automatically but relies on the existence of the parents and >the module in sys.modules. >And a second method which calls the hooks for a module object w/o >inspecting sys.modules. *sigh*. We seem to be getting further and further off course, here. The more different you make the semantics of the system, the harder it will be to verify that it's doing the right thing without having real field experience with the new approach. > > Only if you can guarantee that no hook for a submodule is run until all > > the parent hooks are finished being called *and* that adding new > > callbacks while callbacks are being run will still call them... *after* > > any already-added callbacks. > >Uhm, now it starts to become a mind bending problem. I may have to queue >and delay the registration of new hooks while other hooks are called by >the system. If an user registers a callback for 'a' while the callbacks >for 'a' are called than the registration is queued and the called after >the remaining hooks for 'a' are called. This could probably be >implemented by *not* setting the entry to None after I get hold of the >iterator but after the end of the iteration. iter() notices when a new >element is appended. Yep - that's precisely what peak.util.imports does, and is why it does it that way. However, it has other ways of guaranteeing that the notification callback occurs only once, because the module object keeps track of that. >In the above sample can hook_aX be called after hook_ab1 or should it be >called after hook_ab2? That question can't be answered from the limited information you supplied about the scenario. What must not happen is that hook_aX must not be called before any *already-registered* hooks for 'a'. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 01:47 AM 1/11/2008 +0100, Christian Heimes wrote: >Phillip J. Eby wrote: > > At 11:45 PM 1/10/2008 +0100, Christian Heimes wrote: > >> In my version a hook is immediately called when the the registry value > >> is set to None. When a hook is registered for a module during the > >> execution of the callback then the hook is fired directly and not after > >> the existing hooks are called. Is this a problem for you? > > > > Yes, because it violates the invariant that hooks for a given module > > are called in the same order that they were registered in. > >Please check the changes and the new unit test in r59902 ( >http://svn.python.org/view?rev=59902&view=rev ). Are you satisfied with >the ordering or do you think I should queue the hooks for already loaded >modules? As I said above, calling the callbacks immediately is a problem, because it leads to significantly less predictability (and therefore control) of callback order. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 11:45 PM 1/10/2008 +0100, Christian Heimes wrote: >In my version a hook is immediately called when the the registry value >is set to None. When a hook is registered for a module during the >execution of the callback then the hook is fired directly and not after >the existing hooks are called. Is this a problem for you? Yes, because it violates the invariant that hooks for a given module are called in the same order that they were registered in. In case you're wondering why it's a problem, it's because if a hook imports something that registers a hook, when previously that thing wasn't imported until later, all of a sudden your callback order is very different than what it was before. More succinctly: if making small changes to your program can cause large differences in the result, it's hard to debug. (Especially if you have no idea what 3rd party module changed its import order and messed things up.) Believe me, it's a lot easier to debug if there is a globally understandable hook order, even if it's still a partial ordering. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 09:40 PM 1/10/2008 +0100, Christian Heimes wrote:
>Phillip J. Eby wrote:
>[...]
>
> > There's also one twist that I haven't sorted out yet: "Importing"
> > guarantees that a parent module 'foo' will have a 'bar' attribute for
> > the 'foo.bar' module, if 'foo.bar' is lazy. It does this by
> > registering a callback, ideally *before* any other callback is
> > registered for 'foo' or 'foo.bar' that would look at 'foo.bar'. I
> > don't see how to maintain this condition in a world where import
> > callbacks can be registered independently.
>
>I've moved the PyImport_NotifyModuleLoaded() call to import_submodule().
>It (should) guarantee that the hooks for a parent package is called
>before the hooks for its children are called. I've analyzed the code
>carefully enough to be sure but all unit test results are on my side.
>
>On other words "import a.b.c" fires the hooks for "a", then "a.b" and at
>last "a.b.c".
Yes, that's the general idea. But what happens if you are in the
middle of firing hooks for 'a', and a new hook for 'a.b.c' is
added? What about a new hook for 'a'?
>I could also modify imp.notify_module_loaded to accepts the module name
>as string ("a.b.c."). If the module is provided by name (e.g. "a.b.c.")
>rather than by object it makes sure that the hooks for "a", "a.b" and
>"a.b.c" are called in the right order.
Well, it certainly can (and should) do the same if a module object is
provided, since the module has a __name__.
>Would the modification fulfill your needs if
>imp.notify_module_loaded("foo.bar.baz") call the hooks for "foo",
>"foo.bar" and "foo.bar.baz" in that order?
Only if you can guarantee that no hook for a submodule is run until
all the parent hooks are finished being called *and* that adding new
callbacks while callbacks are being run will still call them...
*after* any already-added callbacks.
>The initial design used to set the hooks to None *after* the hooks were
>called. I removed code yesterday because I thought it's not required.
>Today I've re-added the checks for Py_None.
In general, if you think something in peak.util.imports isn't
required, you're probably wrong. ;)
>I'm not setting the hooks to Py_None before the hook are called.
That's fine, but here's a different catch: are you iterating over the
hooks by taking the length before you start? If so, then hooks that
are added *while* the hooks are being called back, will never get
called, because they'll be added to the end of the list (and you'll
never reach the end). Make sure there's a test for that case.
peak.util.imports sets to None after callbacks, but it uses regular
list iteration, so hooks can be added to the end of the list while
the hooks are still being called.
An error while running the hooks should also set the hook list to
None and discard all the hooks. There isn't any sane way to recover
from an error in a post-import hook.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pkgutil, pkg_resource and Python 3.0 name space packages
At 10:47 AM 1/10/2008 +, Paul Moore wrote: >On 09/01/2008, Steve Holden <[EMAIL PROTECTED]> wrote: > > The idea that users would /program their own computers/ was totally > > alien to the Windows mindset. > >Actually, the alien idea is that more than one person would use the >same (Windows) computer. Not surprising as these were *personal* >computers. It's Windows as a server OS that's the bizarre idea... multiuser != server I've been in a couple of organizations where PCs were shared and thus had different users logging into them. Cash registers and call centers, for example. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 07:22 PM 1/10/2008 +1000, Nick Coghlan wrote: >Christian Heimes wrote: > > A module is successfully loaded > > ''' > > > > The import machinery checks if sys.post_import_hooks contains post import > > hooks for the newly loaded module. If hooks are found then the hooks are > > called in the order they were registered with the module instance as first > > argument. The processing of the hooks is stopped when a method raises an > > exception. At the end the entry for the module name is removed from > > sys.post_import_hooks, even when an error has occured. > >Doesn't the module remain in post_import_hooks, only mapped to None to >indicate that any hooks should be run immediately? It should be, yes. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Post import hooks
At 03:20 AM 1/10/2008 +0100, Christian Heimes wrote: >PyObject* PyImport_NotifyModuleLoaded(PyObject *module) >Notify the post import system that a module was requested. Returns the >module or NULL if an error has occured. The big problem here is that the interaction with laziness is actually pretty complex, when it comes to re-entrancy and initialization order. A package can actually import a submodule, and not yet be finished importing, for example. So you can actually have child hooks executing before parent hooks in this case. The "Importing" package prevents this by not registering child hooks until a parent is actually imported, thus guaranteeing a sane hook execution order. Relative order for hooks targeting the same module is maintained, but parent module hooks are guaranteed to execute before child hooks, even if the child finishes importing before the parent. This would be a bit trickier to implement with your C API, since "Importing" does this by registering a lot of lambdas. But, now that I've reviewed my own code and pieced back together the rationale for it doing things in this seemingly-odd way, it makes sense. There's also one twist that I haven't sorted out yet: "Importing" guarantees that a parent module 'foo' will have a 'bar' attribute for the 'foo.bar' module, if 'foo.bar' is lazy. It does this by registering a callback, ideally *before* any other callback is registered for 'foo' or 'foo.bar' that would look at 'foo.bar'. I don't see how to maintain this condition in a world where import callbacks can be registered independently. Bleah. All of the above isn't really a good explanation of the problem. Let me try to simplify it: * Lazy importing needs to guarantee that foo.bar = sys.modules['foo.bar'], when callbacks for 'foo.bar' execute (in case they refer to foo.bar) * To do this, it registers a callback that sets foo.bar = sys.modules['foo.bar'], and does not actually register any foo.bar callbacks until 'foo' is really imported (and thus foo.bar gets set by that callback) In the case of the PEP, it's harder for me to figure out what happens, because you might not have any lazy modules around, and the foo.bar issue would then not come up. You also have the possibility of a problem where a lazy import callback occurs in 3rd party code, while callbacks are occurring from the import machinery. (Which means that the notification API should probably set the hooks entry to None while it's running, so that if it's called from inside a hook, it will not double-run the hooks, and new hooks registered while hooks are running will get run immediately as they are encountered, instead of getting added to the list.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Lazy module imports and post import hook
At 09:20 PM 1/9/2008 +0100, Christian Heimes wrote: >Brett Cannon wrote: > > I agree with Nick and Nick. This should really be two separate PEPs. > >I'm fine with the proposal and I'm going to chop the PEP in two parts >tonight. > >Somehow I suspect that the lazy import PEP will be postponed or reject. Probably. After the split, I'll review things again, with a closer eye on the initialization order issues, especially with respect to ensuring that lazy imports set the corresponding attribute in the parent package at the right point in time. (It should happen *before* the submodule can actually be imported.) The big advantage to a stdlib implementation of lazy modules would be that it could be more vetted and "blessed" -- the downside is that it's a new and nontrivial implementation. :( ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
