Re: [Python-Dev] UserString
On 2005 Feb 21, at 04:42, Guido van Rossum wrote: Oh, bah. That's not what basestring was for. I can't blame you or your client, but my *intention* was that basestring would *only* be the base of the two *real* built-in string types (str and unicode). I think all this just reinforces the notion that LBYL is a bad idea! In this case, perhaps; but in general? (And I think there's a legitimate desire to sometimes special-case string-like things, e.g. consider a function that takes either a stream or a filename argument.) Anyway, can you explain why LBYL is bad? In the general case, it's bad because of a combination of issues. It may violate "once, and only once!" -- the operations one needs to check may basicaly duplicate the operations one then wants to perform. Apart from wasted effort, it may happen that the situation changes between the look and the leap (on an external file, or due perhaps to threading or other reentrancy). It's often hard in the look to cover exactly the set of prereq's you need for the leap -- e.g. I've often seen code such as if i < len(foo): foo[i] = 24 which breaks for i<-len(foo); the first time this happens the guard's changed to 0<=i Etc -- this is all very general and generic. I had convinced myself that strings were a special case worth singling out, via isinstance and basestring, just as (say) dictionaries are singled out quite differently by metods such as get... I may well have been too superficial in this conclusion. Then you would be able to test whether something is sequence-like by the presence of __getitem__ or __iter__ methods, without getting tripped up by strings. There would be other ways to get out of this dilemma; we could introduce a char type, for example. Also, strings might be recognizable by other means, e.g. the presence of a lower() method or some other characteristic method that doesn't apply to sequence in general. Sure, there would many possibilities. (To Alex: leaving transform() out of the string interface seems to me the simplest solution.) I guess you mean translate. Yes, that would probably be simplest. Alex ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Eliminating the block stack (was Re: Store x Load x --> DupStore)
> >>Eliminating the blockstack would be nice (esp. if it's enough to get > >>frames small enough that they get allocated by PyMalloc) but this > >>seemed to be tricky too (or at least Armin, Samuele and I spent a > >>cuple of hours yakking about it on IRC and didn't come up with a clear > >>approach). Dynamically allocating the blockstack would be simpler, > >>and might acheive a similar win. (This is all from memory, I haven't > >>thought about specifics in a while). I don't know if this helps, but since I invented the block stack around 1990, I believe I recall the main reason to make it dynamic was to simplify code generation, not because it is inherently dynamic. At the time an extra run-time data structure seemed to require less coding than an extra compile-time data structure. The same argument got me using dicts for locals; that was clearly a bottleneck and eliminated long ago, but I think we should be able to lose the block stack now, too. Somewhat ironically, eliminating the block stack will reduce the stack frame size, while eliminating the dict for locals added to it. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] UserString
> >> Oh, bah. That's not what basestring was for. I can't blame you or your > >> client, but my *intention* was that basestring would *only* be the > >> base of the two *real* built-in string types (str and unicode). > > I think all this just reinforces the notion that LBYL is > a bad idea! In this case, perhaps; but in general? (And I think there's a legitimate desire to sometimes special-case string-like things, e.g. consider a function that takes either a stream or a filename argument.) Anyway, can you explain why LBYL is bad? > > The need to check "is this thingy here string-like" is sort of frequent, > > because strings are sequences which, when iterated on, yield sequences > > (strings of length 1) which, when iterated on, yield sequences ad > > infinitum. > > Yes, this characteristic of strings is unfortunate because it > tends to make some degree of LBYLing unavoidable. I don't > think the right solution is to try to come up with safe ways > of doing LBYL on strings, though, at least not in the long > term. > > Maybe in Python 3000 this could be fixed by making strings *not* > be sequences. They would be sliceable, but *not* indexable or > iterable. If you wanted to iterate over their chars, you > would have to say 'for c in s.chars()' or something. > > Then you would be able to test whether something is sequence-like > by the presence of __getitem__ or __iter__ methods, without > getting tripped up by strings. There would be other ways to get out of this dilemma; we could introduce a char type, for example. Also, strings might be recognizable by other means, e.g. the presence of a lower() method or some other characteristic method that doesn't apply to sequence in general. (To Alex: leaving transform() out of the string interface seems to me the simplest solution.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] UserString
At 04:32 PM 2/21/05 +1300, Greg Ewing wrote: Alex Martelli wrote: The need to check "is this thingy here string-like" is sort of frequent, because strings are sequences which, when iterated on, yield sequences (strings of length 1) which, when iterated on, yield sequences ad infinitum. Yes, this characteristic of strings is unfortunate because it tends to make some degree of LBYLing unavoidable. FWIW, the trick I usually use to deal with this aspect of strings in recursive algorithms is to check whether the current item of an iteration is the same object I'm iterating over; if so, I know I've descended into a string. It doesn't catch it on the first recursion level of course (unless it was a 1-character string to start with), but it's a quick-and-dirty way to EAFP such algorithms. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
At 04:32 PM 2/21/05 +1300, Greg Ewing wrote: Phillip J. Eby wrote: Hm, actually I think I see the answer; in the case of module-level code there can be no "anonymous local variables" the way there can in functions. Why not? There's still a frame object associated with the call of the anonymous function holding the module's top-level code. The compiler can allocate locals in that frame, even if the user's code can't. That's a good point, but if you look at my "eliminating the block stack" post, you'll see that there's a simpler way to potentially get rid of the block stack, where "simpler" means "simpler changes in fewer places". ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] UserString
Alex Martelli wrote: On 2005 Feb 20, at 17:06, Guido van Rossum wrote: Oh, bah. That's not what basestring was for. I can't blame you or your client, but my *intention* was that basestring would *only* be the base of the two *real* built-in string types (str and unicode). I think all this just reinforces the notion that LBYL is a bad idea! The need to check "is this thingy here string-like" is sort of frequent, because strings are sequences which, when iterated on, yield sequences (strings of length 1) which, when iterated on, yield sequences ad infinitum. Yes, this characteristic of strings is unfortunate because it tends to make some degree of LBYLing unavoidable. I don't think the right solution is to try to come up with safe ways of doing LBYL on strings, though, at least not in the long term. Maybe in Python 3000 this could be fixed by making strings *not* be sequences. They would be sliceable, but *not* indexable or iterable. If you wanted to iterate over their chars, you would have to say 'for c in s.chars()' or something. Then you would be able to test whether something is sequence-like by the presence of __getitem__ or __iter__ methods, without getting tripped up by strings. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
Phillip J. Eby wrote: Hm, actually I think I see the answer; in the case of module-level code there can be no "anonymous local variables" the way there can in functions. Why not? There's still a frame object associated with the call of the anonymous function holding the module's top-level code. The compiler can allocate locals in that frame, even if the user's code can't. I guess you'd need to also have a "reset stack to level X" opcode, then, and both it and the set-handler opcode would have to be placed at every destination of a jump that crosses block boundaries. It's not clear how big a win that is, due to the added opcodes even on non-error paths. Only exceptions and break statements would require stack pointer adjustment, and they're relatively rare. I don't think an extra opcode in those cases would make much of a difference. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Eliminating the block stack (was Re: Store x Load x --> DupStore)
Phillip J. Eby wrote: At 03:56 PM 2/20/05 -0500, Phillip J. Eby wrote: At 07:00 PM 2/20/05 +, Michael Hudson wrote: Eliminating the blockstack would be nice (esp. if it's enough to get frames small enough that they get allocated by PyMalloc) Someone might like to take a look at the way Pyrex generates C code for try-except and try-finally blocks. It manages to get (what I hope is) the same effect using local variables and gotos. It doesn't have to deal with a stack pointer, but I think that should just be a compiler-determinable adjustment to be done when jumping to an outer block. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | [EMAIL PROTECTED] +--+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
"Phillip J. Eby" <[EMAIL PROTECTED]> writes: > At 07:00 PM 2/20/05 +, Michael Hudson wrote: >>"Phillip J. Eby" <[EMAIL PROTECTED]> writes: >> >> > At 08:17 AM 2/20/05 -0800, Guido van Rossum wrote: >> >>Where are the attempts to speed up function/method calls? That's an >> >>area where we could *really* use a breakthrough... >> > >> > Amen! >> > >> > So what happened to Armin's pre-allocated frame patch? Did that >> get into 2.4? >> >>No, because it slows down recursive function calls, or functions that >>happen to be called at the same time in different threads. Fixing >>*that* would require things like code specific frame free-lists and >>that's getting a bit convoluted and might waste quite a lot of memory. > > Ah. I thought it was just going to fall back to the normal case if > the pre-allocated frame wasn't available (i.e., didn't have a refcount > of 1). Well, I don't think that's the test, but that might work. Someone should try it :) (I'm trying something else currently). >>Eliminating the blockstack would be nice (esp. if it's enough to get >>frames small enough that they get allocated by PyMalloc) but this >>seemed to be tricky too (or at least Armin, Samuele and I spent a >>cuple of hours yakking about it on IRC and didn't come up with a clear >>approach). Dynamically allocating the blockstack would be simpler, >>and might acheive a similar win. (This is all from memory, I haven't >>thought about specifics in a while). > > I'm not very familiar with the operation of the block stack, but why > does it need to be a stack? Finally blocks are the problem, I think. > For exception handling purposes, wouldn't it suffice to know the > offset of the current handler, and have an opcode to set the current > handler location? And for "for" loops, couldn't an anonymous local > be used to hold the loop iterator instead of using a stack variable? > Hm, actually I think I see the answer; in the case of module-level > code there can be no "anonymous local variables" the way there can in > functions. Hmm. I don't think this is the killer blow. I can't remember the details and it's too late to think about them, so I'm going to wait and see if Samuele replies :) >>All of it, in easy cases. ISTR that the fast path could be a little >>wider -- it bails when the called function has default arguments, but >>I think this case could be handled easily enough. > > When it has *any* default arguments, or only when it doesn't have > values to supply for them? When it has *any*, I think. I also think this is easy to change. >>Why are frames so big? > > Because there are CO_MAXBLOCKS * 12 bytes in there for the block > stack. If there was no need for that, frames could perhaps be > allocated via pymalloc. They only have around 100 bytes or so in > them, apart from the blockstack and locals/value stack. What I'm trying is allocating the blockstack separately and see if two pymallocs are cheaper than one malloc. >> > Do we need a tp_callmethod that takes an argument array, length, and >> > keywords, so that we can skip instancemethod allocation in the >> > common case of calling a method directly? >> >>Hmm, didn't think of that, and I don't think it's how the CALL_ATTR >>attempt worked. I presume it would need to take a method name too :) > > Er, yeah, I thought that was obvious. :) Someone should try this too :) Cheers, mwh -- It is never worth a first class man's time to express a majority opinion. By definition, there are plenty of others to do that. -- G. H. Hardy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Eliminating the block stack (was Re: Store x Load x --> DupStore)
At 03:56 PM 2/20/05 -0500, Phillip J. Eby wrote: At 07:00 PM 2/20/05 +, Michael Hudson wrote: Eliminating the blockstack would be nice (esp. if it's enough to get frames small enough that they get allocated by PyMalloc) but this seemed to be tricky too (or at least Armin, Samuele and I spent a cuple of hours yakking about it on IRC and didn't come up with a clear approach). Dynamically allocating the blockstack would be simpler, and might acheive a similar win. (This is all from memory, I haven't thought about specifics in a while). I think I have an idea how to do it in a (relatively) simple fashion; see if you can find a hole in it: * Change the PyTryBlock struct to include an additional member, 'int b_prev', that refers to the previous block in a chain * Change the compiler's emission of SETUP_* opcodes, so that instead of a PyTryBlock being added to the blockstack at interpretation time, it's added to the end of a 'co_blktree' block array at compile time, with its 'b_prev' pointing to the current "top" of the block stack. Instead of the SETUP_* argument being the handler offset, have it be the index of the just-added blocktree entry. * Replace f_blockstack and f_iblock with 'int f_iblktree', and change PyFrame_BlockSetup() to set this equal to the SETUP_* argument, and PyFrame_BlockPop() to use this as an index into the code's co_blktree to retrieve the needed values. PyFrame_BlockPop() would then set f_iblktree equal to the "popped" block's 'b_prev' member, thus "popping" the block from this virtual stack. (Note, by the way, that the blocktree could actually be created as a post-processing step of the current compilation process, by a loop that scans the bytecode and tracks the current stack and blockstack levels, and then replaces the SETUP_* opcodes' arguments. This might be a simpler option than trying to change the compiler to do it along the way.) Can anybody see any flaws in this concept? As far as I can tell it just generates all possible block stack states at compile time, but doesn't change block semantics in the least, and it scarcely touches the eval loop. It seems like it could drop the size of frames enough to let them use pymalloc instead of the OS malloc, at the cost of a 16 bytes per block increase in the size of code objects. (And of course the necessary changes to 'marshal' and 'dis' as well as the compiler and eval loop.) (More precisely, frames whose f_nlocals + f_stacksize is 40 or less, would be 256 bytes or less, and therefore pymalloc-able. However, this should cover all but the most complex functions.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
At 07:00 PM 2/20/05 +, Michael Hudson wrote: "Phillip J. Eby" <[EMAIL PROTECTED]> writes: > At 08:17 AM 2/20/05 -0800, Guido van Rossum wrote: >>Where are the attempts to speed up function/method calls? That's an >>area where we could *really* use a breakthrough... > > Amen! > > So what happened to Armin's pre-allocated frame patch? Did that get into 2.4? No, because it slows down recursive function calls, or functions that happen to be called at the same time in different threads. Fixing *that* would require things like code specific frame free-lists and that's getting a bit convoluted and might waste quite a lot of memory. Ah. I thought it was just going to fall back to the normal case if the pre-allocated frame wasn't available (i.e., didn't have a refcount of 1). Eliminating the blockstack would be nice (esp. if it's enough to get frames small enough that they get allocated by PyMalloc) but this seemed to be tricky too (or at least Armin, Samuele and I spent a cuple of hours yakking about it on IRC and didn't come up with a clear approach). Dynamically allocating the blockstack would be simpler, and might acheive a similar win. (This is all from memory, I haven't thought about specifics in a while). I'm not very familiar with the operation of the block stack, but why does it need to be a stack? For exception handling purposes, wouldn't it suffice to know the offset of the current handler, and have an opcode to set the current handler location? And for "for" loops, couldn't an anonymous local be used to hold the loop iterator instead of using a stack variable? Hm, actually I think I see the answer; in the case of module-level code there can be no "anonymous local variables" the way there can in functions. Hmm. I guess you'd need to also have a "reset stack to level X" opcode, then, and both it and the set-handler opcode would have to be placed at every destination of a jump that crosses block boundaries. It's not clear how big a win that is, due to the added opcodes even on non-error paths. Hey, wait a minute... all the block stack data is static, isn't it? I mean, the contents of the block stack at any point in a code string could be determined statically, by examination of the bytecode, couldn't it? If that's the case, then perhaps we could design a pre-computed data structure similar to co_lnotab that would be used by the evaluator in place of the blockstack. Of course, I may be talking through my hat here, as I have very little experience with how the blockstack works. However, if this idea makes sense, then perhaps it could actually speed up non-error paths as well (except perhaps for the 'return' statement), at the cost of a larger code structure and compiler complexity. But, if it also means that frames can be allocated faster (e.g. via pymalloc), it might be worth it, just like getting rid of SET_LINENO turned out to be a net win. All of it, in easy cases. ISTR that the fast path could be a little wider -- it bails when the called function has default arguments, but I think this case could be handled easily enough. When it has *any* default arguments, or only when it doesn't have values to supply for them? Why are frames so big? Because there are CO_MAXBLOCKS * 12 bytes in there for the block stack. If there was no need for that, frames could perhaps be allocated via pymalloc. They only have around 100 bytes or so in them, apart from the blockstack and locals/value stack. > Do we need a tp_callmethod that takes an argument array, length, and > keywords, so that we can skip instancemethod allocation in the > common case of calling a method directly? Hmm, didn't think of that, and I don't think it's how the CALL_ATTR attempt worked. I presume it would need to take a method name too :) Er, yeah, I thought that was obvious. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
At 06:38 PM 2/20/05 +, Michael Hudson wrote:
>> It folds the two steps into a new opcode. In the case of
>> store_name/load_name, it saves one three byte instruction, a trip around
>> the eval-loop, two stack mutations, a incref/decref pair, a dictionary
>> lookup, and an error check (for the lookup). While it acts like a dup
>> followed by a store, it is implemented more simply as a store that
>> doesn't pop the stack. The transformation is broadly applicable and
>> occurs thousands of times in the standard library and test suite.
I'm still a little curious as to what code creates such opcodes...
A simple STORE+LOAD case:
>>> dis.dis(compile("x=1; y=x*2","?","exec"))
1 0 LOAD_CONST 0 (1)
3 STORE_NAME 0 (x)
6 LOAD_NAME0 (x)
9 LOAD_CONST 1 (2)
12 BINARY_MULTIPLY
13 STORE_NAME 1 (y)
16 LOAD_CONST 2 (None)
19 RETURN_VALUE
And a simple DUP+STORE case:
>>> dis.dis(compile("x=y=1","?","exec"))
1 0 LOAD_CONST 0 (1)
3 DUP_TOP
4 STORE_NAME 0 (x)
7 STORE_NAME 1 (y)
10 LOAD_CONST 1 (None)
13 RETURN_VALUE
Of course, I'm not sure how commonly this sort of code occurs in places
where it makes a difference to anything. Function call overhead continues
to be Python's most damaging performance issue, because it makes it
expensive to use abstraction.
Here's a thought. Suppose we split frames into an "object" part and a
"struct" part, with the object part being just a pointer to the struct
part, and a flag indicating whether the struct part is stack-allocated or
malloc'ed. This would let us stack-allocate the bulk of the frame
structure, but still have a frame "object" to pass around. On exit from
the C routine that stack-allocated the frame struct, we check to see if the
frame object has a refcount>1, and if so, malloc a permanent home for the
frame struct and update the frame object's struct pointer and flag.
In this way, frame allocation overhead could be reduced to the cost of an
alloca, or just incorporated into the stack frame setup of the C routine
itself, allowing the entire struct to be treated as "local variables" from
a C perspective (which might benefit performance on architectures that
reserve a register for local variable access).
Of course, this would slow down exception handling and other scenarios that
result in extra references to a frame object, but if the OS malloc is the
slow part of frame allocation (frame objects are too large for pymalloc),
then perhaps it would be a net win. On the other hand, this approach would
definitely use more stack space per calling level.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
Michael Hudson wrote: "Phillip J. Eby" <[EMAIL PROTECTED]> writes: [SNIP] And whatever happened to CALL_METHOD? It didn't work as an optimization, as far as I remember. I think the patch is on SF somewhere. Or is a branch in CVS? Oh, it's patch #709744. Do we need a tp_callmethod that takes an argument array, length, and keywords, so that we can skip instancemethod allocation in the common case of calling a method directly? Hmm, didn't think of that, and I don't think it's how the CALL_ATTR attempt worked. I presume it would need to take a method name too :) CALL_ATTR basically replaced ``LOAD_ATTR; CALL_FUNCTION`` with a single opcode. Idea was that the function creation by the LOAD_ATTR was a wasted step so might as well just skip it and call the method directly. Problem was the work required to support both classic and new-style classes. Now I have not looked at the code since it was written back at PyCon 2003 and I was a total newbie to the core's C code at that point and I think Thomas said it had been two years since he did any major core hacking. In other words it could possibly have been done better. =) -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
"Phillip J. Eby" <[EMAIL PROTECTED]> writes: > At 08:17 AM 2/20/05 -0800, Guido van Rossum wrote: >>Where are the attempts to speed up function/method calls? That's an >>area where we could *really* use a breakthrough... > > Amen! > > So what happened to Armin's pre-allocated frame patch? Did that get into 2.4? No, because it slows down recursive function calls, or functions that happen to be called at the same time in different threads. Fixing *that* would require things like code specific frame free-lists and that's getting a bit convoluted and might waste quite a lot of memory. Eliminating the blockstack would be nice (esp. if it's enough to get frames small enough that they get allocated by PyMalloc) but this seemed to be tricky too (or at least Armin, Samuele and I spent a cuple of hours yakking about it on IRC and didn't come up with a clear approach). Dynamically allocating the blockstack would be simpler, and might acheive a similar win. (This is all from memory, I haven't thought about specifics in a while). > Also, does anybody know where all the time goes in a function call, > anyway? I did once... > I assume that some of the pieces are: > > * tuple/dict allocation for arguments (but some of this is bypassed on > the fast branch for Python-to-Python calls, right?) All of it, in easy cases. ISTR that the fast path could be a little wider -- it bails when the called function has default arguments, but I think this case could be handled easily enough. > * frame allocation and setup (but Armin's patch was supposed to > eliminate most of this whenever a function isn't being used > re-entrantly) Ah, you remember the wart :) I think even with the patch, frame setup is a significant amount of work. Why are frames so big? > * argument "parsing" (check number of args, map kwargs to their > positions, etc.; but isn't some of this already fast-pathed for > Python-to-Python calls?) Yes. With some effort you could probably avoid a copy (and incref) of the arguments from the callers to the callees stack area. BFD. > I suppose the fast branch fixes don't help special methods like > __getitem__ et al, since those don't go through the fast branch, but I > don't think those are the majority of function calls. Indeed. I suspect this fails the effort/benefit test, but I could be wrong. > And whatever happened to CALL_METHOD? It didn't work as an optimization, as far as I remember. I think the patch is on SF somewhere. Or is a branch in CVS? Oh, it's patch #709744. > Do we need a tp_callmethod that takes an argument array, length, and > keywords, so that we can skip instancemethod allocation in the > common case of calling a method directly? Hmm, didn't think of that, and I don't think it's how the CALL_ATTR attempt worked. I presume it would need to take a method name too :) I already have a patch that does this for regular function calls (it's a rearrangement/refactoring not an optimization though). Cheers, mwh -- I think perhaps we should have electoral collages and construct our representatives entirely of little bits of cloth and papier mache. -- Owen Dunn, ucam.chat, from his review of the year ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
Guido van Rossum <[EMAIL PROTECTED]> writes: >> Any objections to new peephole transformation that merges a store/load >> pair into a single step? >> >> There is a tested patch at: www.python.org/sf/1144842 >> >> It folds the two steps into a new opcode. In the case of >> store_name/load_name, it saves one three byte instruction, a trip around >> the eval-loop, two stack mutations, a incref/decref pair, a dictionary >> lookup, and an error check (for the lookup). While it acts like a dup >> followed by a store, it is implemented more simply as a store that >> doesn't pop the stack. The transformation is broadly applicable and >> occurs thousands of times in the standard library and test suite. I'm still a little curious as to what code creates such opcodes... > What exactly are you trying to accomplish? Do you have examples of > code that would be sped up measurably by this transformation? Does > anybody care about those speedups even if they *are* measurable? > > I'm concerned that there's too much hacking of the VM going on with > too little benefit. The VM used to be relatively simple code that many > people could easily understand. The benefit of that was that new > language features could be implemented relatively easily even by > relatively inexperienced developers. All that seems to be lost, and I > fear that the end result is going to be a calcified VM that's only 10% > faster than the original, since we appear to have reached the land of > diminishing returns here. In the case of the bytecode optimizer, I'm not sure this is a fair accusation. Even if you don't understand it, you can ignore it and not have your understanding of the rest of the VM affected (I'm not sure that compile.c has ever been "easily understood" in any case :). > I don't see any concentrated efforts trying to figure out where the > biggest pain is and how to relieve it; rather, it looks as if the > easiest targets are being approached. Now, if these were low-hanging > fruit, I'd happily agree, but I'm not so sure that they are all that > valuable. I think some of the peepholer's work are pure wins -- x,y = y,x unpacking and the creation of constant tuples certainly spring to mind. If Raymond wants to spend his time on this stuff, that's his choice. I don't think the obfuscation cost is all that high. > Where are the attempts to speed up function/method calls? That's an > area where we could *really* use a breakthrough... The problem is that it's hard! > Eventually we'll need a radically different approach, maybe PyPy, > maybe Starkiller. Yup. Cheers, mwh -- Gevalia is undrinkable low-octane see-through only slightly roasted bilge water. Compared to .us coffee it is quite drinkable. -- Måns Nilsson, asr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
Guido van Rossum wrote: I'm concerned that there's too much hacking of the VM going on with too little benefit. I completely agree. It would be so much more useful if people tried to fix the bugs that have been reported. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Requesting that a class be a new-style class
At 09:15 AM 2/20/05 +0100, Alex Martelli wrote: This is because types.ClassType turns somersaults to enable this: in this latter construct, Python's mechanisms determine ClassType as the metaclass (it's the metaclass of the first base class), but then ClassType in turn sniffs around for another metaclass to delegate to, among the supplied bases, and having found one washes its hands of the whole business;-). To be pedantic, the actual algorithm in 2.2+ has nothing to do with the first base class; that's the pre-2.2 algorithm. The 2.2 algorithm looks for the most-derived metaclass of the base classes, and simply ignores classic bases altogether. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
At 08:17 AM 2/20/05 -0800, Guido van Rossum wrote: Where are the attempts to speed up function/method calls? That's an area where we could *really* use a breakthrough... Amen! So what happened to Armin's pre-allocated frame patch? Did that get into 2.4? Also, does anybody know where all the time goes in a function call, anyway? I assume that some of the pieces are: * tuple/dict allocation for arguments (but some of this is bypassed on the fast branch for Python-to-Python calls, right?) * frame allocation and setup (but Armin's patch was supposed to eliminate most of this whenever a function isn't being used re-entrantly) * argument "parsing" (check number of args, map kwargs to their positions, etc.; but isn't some of this already fast-pathed for Python-to-Python calls?) I suppose the fast branch fixes don't help special methods like __getitem__ et al, since those don't go through the fast branch, but I don't think those are the majority of function calls. And whatever happened to CALL_METHOD? Do we need a tp_callmethod that takes an argument array, length, and keywords, so that we can skip instancemethod allocation in the common case of calling a method directly? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] UserString
On 2005 Feb 20, at 17:06, Guido van Rossum wrote:
[Alex]
I did have some issues w/UserString at a client's, but that was
connected to some code doing type-checking (and was fixed by injecting
basestring as a base of the client's subclass of UserString and
ensuring the type-checking always used isinstance and basestring).
Oh, bah. That's not what basestring was for. I can't blame you or your
client, but my *intention* was that basestring would *only* be the
base of the two *real* built-in string types (str and unicode). The
reason for its existence was that some low-level built-in (or
extension) operations only accept those two *real* string types and
consequently some user code might want to validate ("look before you
leap") its own arguments if those eventually ended up being passed to
aforementioned low-level built-in code. My intention was always that
UserString and other string-like objects would explicitly *not*
inherit from basestring. Of course, my intention was lost, your client
used basestring to mean "any string-ish object", got away with it
because they weren't using any of those low-level built-ins, and you
had to comply rather than explain it to them.
I would gladly have explained, if I had understood your design intent
correctly at the time (whether the explanation would have done much
good is another issue); but I'm afraid I didn't. Now I do (thanks for
explaining!) though I'm not sure what can be done in retrospect to
communicate it more widely.
The need to check "is this thingy here string-like" is sort of
frequent, because strings are sequences which, when iterated on, yield
sequences (strings of length 1) which, when iterated on, yield
sequences ad infinitum. Strings are sequences but more often than not
one wants to treat them as "scalars" instead. isinstance and
basestring allow that frequently needed check so nicely, that, if
they're not intended for it, they're an "attractive nuisance"
legally;-).
The need to make stringlike thingies emerges both for bad reasons
(e.g., I never liked that client's "string cum re" perloidism) and good
ones (e.g., easing the interfacing with external frameworks that have
their own stringythings, such as Qt's QtString); and checking if
something is stringlike is also frequent, as per previous para.
Darn...
Sounds like a good reason to add interfaces to the language. :-)
If an interface must be usable to say "is this string-like?" it will
have to be untyped, I guess, and the .translate method will be a small
problem (one-argument for unicode, two-args for str, and very different
argument semantics) -- don't recall offhand if there are other such
nonpolymorphic methods there.
Alex
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Store x Load x --> DupStore
> Any objections to new peephole transformation that merges a store/load > pair into a single step? > > There is a tested patch at: www.python.org/sf/1144842 > > It folds the two steps into a new opcode. In the case of > store_name/load_name, it saves one three byte instruction, a trip around > the eval-loop, two stack mutations, a incref/decref pair, a dictionary > lookup, and an error check (for the lookup). While it acts like a dup > followed by a store, it is implemented more simply as a store that > doesn't pop the stack. The transformation is broadly applicable and > occurs thousands of times in the standard library and test suite. What exactly are you trying to accomplish? Do you have examples of code that would be sped up measurably by this transformation? Does anybody care about those speedups even if they *are* measurable? I'm concerned that there's too much hacking of the VM going on with too little benefit. The VM used to be relatively simple code that many people could easily understand. The benefit of that was that new language features could be implemented relatively easily even by relatively inexperienced developers. All that seems to be lost, and I fear that the end result is going to be a calcified VM that's only 10% faster than the original, since we appear to have reached the land of diminishing returns here. I don't see any concentrated efforts trying to figure out where the biggest pain is and how to relieve it; rather, it looks as if the easiest targets are being approached. Now, if these were low-hanging fruit, I'd happily agree, but I'm not so sure that they are all that valuable. Where are the attempts to speed up function/method calls? That's an area where we could *really* use a breakthrough... Eventually we'll need a radically different approach, maybe PyPy, maybe Starkiller. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] UserString
[Alex]
> I did have some issues w/UserString at a client's, but that was
> connected to some code doing type-checking (and was fixed by injecting
> basestring as a base of the client's subclass of UserString and
> ensuring the type-checking always used isinstance and basestring).
Oh, bah. That's not what basestring was for. I can't blame you or your
client, but my *intention* was that basestring would *only* be the
base of the two *real* built-in string types (str and unicode). The
reason for its existence was that some low-level built-in (or
extension) operations only accept those two *real* string types and
consequently some user code might want to validate ("look before you
leap") its own arguments if those eventually ended up being passed to
aforementioned low-level built-in code. My intention was always that
UserString and other string-like objects would explicitly *not*
inherit from basestring. Of course, my intention was lost, your client
used basestring to mean "any string-ish object", got away with it
because they weren't using any of those low-level built-ins, and you
had to comply rather than explain it to them.
Sounds like a good reason to add interfaces to the language. :-)
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Store x Load x --> DupStore
Any objections to new peephole transformation that merges a store/load pair into a single step? There is a tested patch at: www.python.org/sf/1144842 It folds the two steps into a new opcode. In the case of store_name/load_name, it saves one three byte instruction, a trip around the eval-loop, two stack mutations, a incref/decref pair, a dictionary lookup, and an error check (for the lookup). While it acts like a dup followed by a store, it is implemented more simply as a store that doesn't pop the stack. The transformation is broadly applicable and occurs thousands of times in the standard library and test suite. Raymond Hettinger ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Re: Prospective Peephole Transformation
Martin v. Löwis wrote:
>> I'd say that this explains why it would still make sense to let the code
>> generator change
>> "x in (a, b, c)" to "x == a or x == b or x == c", as long as a, b, and c are
>> all integers.
>
> How often does that happen in real code?
don't know, but it happens:
[EMAIL PROTECTED] Python-2.4]$ grep "if.*in *([0-9]" Lib/*.py
Lib/BaseHTTPServer.py:if self.command != 'HEAD' and code >= 200 and
code not in (204, 304):
Lib/asyncore.py:if err in (0, EISCONN):
Lib/mimify.py:if len(args) not in (0, 1, 2):
Lib/sunau.py:if nchannels not in (1, 2, 4):
Lib/sunau.py:if sampwidth not in (1, 2, 4):
Lib/urllib2.py:if code not in (200, 206):
Lib/urllib2.py:if (code in (301, 302, 303, 307) and m in ("GET", "HEAD")
Lib/whichdb.py:if magic in (0x00061561, 0x61150600):
Lib/whichdb.py:if magic in (0x00061561, 0x61150600):
[EMAIL PROTECTED] Python-2.4]$ grep "if.*in *\[[0-9]" Lib/*.py
Lib/decimal.py:if value[0] not in [0,1]:
Lib/smtplib.py:if code not in [235, 503]:
judging from the standard library, "string in string tuple/list" is a lot more
common.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Requesting that a class be a new-style class
Alex Martelli <[EMAIL PROTECTED]> writes: > On 2005 Feb 20, at 04:35, Jack Diederich wrote: > >> I didn't dig into the C but does having 'type' >> as metaclass guarantee the same behavior as inheriting 'object' or >> does object >> provide something type doesn't? *wince* > > I believe the former holds, since for example: I was going to say that 'type(object) is type' is everything you need to know, but you also need the bit of code in type_new that replaces an empty bases tuple with (object,) -- but class C: __metaclass__ = Type and class C(object): pass produce identical classes. > This is because types.ClassType turns somersaults to enable this: in > this latter construct, Python's mechanisms determine ClassType as the > metaclass (it's the metaclass of the first base class), but then > ClassType in turn sniffs around for another metaclass to delegate to, > among the supplied bases, and having found one washes its hands of the > whole business;-). It's also notable that type_new does exactly the same thing! Cheers, mwh -- Jokes around here tend to get followed by implementations. -- from Twisted.Quotes ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] UserString
On 2005 Feb 20, at 05:20, Raymond Hettinger wrote: ... This sort of thing is easy to test for and easy to fix. The question is whether we care about updating this module anymore or is it a relic. Also, is the use case one that we care about. AFAICT, this has never come up before. I did have some issues w/UserString at a client's, but that was connected to some code doing type-checking (and was fixed by injecting basestring as a base of the client's subclass of UserString and ensuring the type-checking always used isinstance and basestring). My two cents: a *mixin* to make it easy to emulate full-fledged strings would be almost as precious as your DictMixin (ones to emulate lists, sets, files [w/buffering], ..., might be even more useful). The point is all of these rich interfaces have a lot of redundancy and a mixin can provide all methods generically based on a few fundamental methods, which can be quite useful, just like DictMixin. But a complete emulation of strings (etc) is mostly of "didactical" use, a sort of checklist to help ensure one implements all methods, not really useful for new code "in production"; at least, I haven't found such uses recently. The above-mentioned client's class was an attempt to join RE functionality to strings and was a rather messy hack anyway, for example (perhaps prompted by client's previous familiarity with Perl, I'm not sure); at any rate, the client should probably have subclassed str or unicode if he really wanted that hack. I can't think of a GOOD use for UserString (etc) since subclassing str (etc) was allowed in 2.2 or at least since a few loose ends about newstyle classes were neatly tied up in 2.3. If we do decide "it is a relic, no more updates" perhaps some indication of deprecation would be warranted. ((In any case, I do think the mixins would be useful)). Alex ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Requesting that a class be a new-style class
On 2005 Feb 20, at 04:35, Jack Diederich wrote: I always use new style classes so I only have to remember one set of behaviors. I agree: that's reason #1 I recommend always using new-style whenever I teach / tutor / mentor in Python nowadays. "__metaclass__ = type" is warty, it has the "action at a distance" problem that decorators solve for functions. I disagree. I view it as akin to a "from __future__ import" except that -- since the compiler doesn't need-to-know, as typeclass-picking happens at runtime -- it was accomplished by less magical and more flexible means. I didn't dig into the C but does having 'type' as metaclass guarantee the same behavior as inheriting 'object' or does object provide something type doesn't? *wince* I believe the former holds, since for example: >>> class X: __metaclass__ = type ... >>> X.__bases__ (,) If you're making a newstyle class with an oldstyle base, it's different: >>> class Y: pass ... >>> class X(Y): __metaclass__ = type ... Traceback (most recent call last): File "", line 1, in ? TypeError: Error when calling the metaclass bases a new-style class can't have only classic bases in this case, you do need to inherit object explicitly: >>> class X(Y, object): pass ... >>> X.__bases__ (, ) >>> type(X) This is because types.ClassType turns somersaults to enable this: in this latter construct, Python's mechanisms determine ClassType as the metaclass (it's the metaclass of the first base class), but then ClassType in turn sniffs around for another metaclass to delegate to, among the supplied bases, and having found one washes its hands of the whole business;-). Alex ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
