Re: [Python-Dev] PATCH submitted: Speed up + for string Re: PATCH submitted: Speed up + for string concatenation, now as fast as "".join(x) idiom

2006-10-18 Thread Chetan Pandya
The discussion on this topic seems to have died down. However, I had a look at the patch and here are some comments:This has the potential to speed up simple strings expressions likes = '1' + '2' + '3' + '4' + '5' + '6' + '7' + '8'
However, if this is followed bys += '9' this (the 9th string) will cause rendering of the existing value of s and then create another concatenated string. This can, however, be changed, but I have not checked to see if it is worth it.
The deallocation code needs to be robust for a complex tree - it is currently not recursive, but needs to be, like the concatenation code.Construct like s = a + b + c + d + e , where a, b etc. have been assigned string values earlier will not benefit from the patch.
If the values are generated and concatenated in a single _expression_, that is another type of construct that will benefit.There are some other changes needed that I can write up if needed.-Chetan
On 10/13/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]
> wrote:Date: Fri, 13 Oct 2006 12:02:06 -0700From: Josiah Carlson <
[EMAIL PROTECTED]>Subject: Re: [Python-Dev] PATCH submitted: Speed up + for   stringconcatenation, now as fast as "".join(x) idiomTo: Larry Hastings <
[EMAIL PROTECTED]>, python-dev@python.orgMessage-ID: <[EMAIL PROTECTED]
>Content-Type: text/plain; charset="US-ASCII"Larry Hastings <[EMAIL PROTECTED]> wrote:[snip]> The machine is dual-core, and was quiescent at the time.  XP's scheduler
> is hopefully good enough to just leave the process running on one core.It's not.  Go into the task manager (accessable via Ctrl+Alt+Del bydefault) and change the process' affinity to the second core.  In my
experience, running on the second core (in both 2k and XP) tends toproduce slightly faster results.  Linux tends to keep processes on asingle core for a few seconds at a time. - Josiah

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PATCH submitted: Speed up + for string Re: PATCHsubmitted: Speed up + for string concatenation, now as fast as "".join(x) idiom

2006-10-18 Thread Kristján V . Jónsson



Doesn't it end up in a call to PyString_Concat()?  
That should return a PyStringConcatenationObject too, right?
K

  
  
  Construct like s = a + b + c + d + e , where a, b etc. have been 
  assigned string values earlier will not benefit from the patch. 

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PATCH submitted: Speed up + for string Re: PATCHsubmitted: Speed up + for string concatenation, now as fast as "".join(x) idiom

2006-10-18 Thread Chetan Pandya
My statement wasn't clear enough.Rendering occurs if the string being concatenated is already a concatenation object created by an earlier assignment. In s = a + b + c + d + e + f , there would be rendering of the source string if it is already a concatenation.
Here is an example that would make it clear:a = "Value a ="a += "anything"  # creates a concatenationc = a + b #This would cause rendering of a and then c will become concatenation between a and b.
c += "Something"# This will not append to the concatenation object, but cause rendering of c and then it will create a concatenation between c and "Something", which will be assigned to c.

Now if there are a series of assignments,(1) s = c + "something" # causes rendering of c(2) s += a   # causes rendering of s and creates a new concatenation(3) s += b  # causes rendering of s and creates a new concatenation
(4) s += c  # causes rendering of s and creates a new concatenation(5) print s   # causes rendering of sIf there is list of strings created and then they are concatenated with +=, I would expect it to be slower because of the additional allocations involved in rendering.
-ChetanOn 10/18/06, Kristján V. Jónsson <
[EMAIL PROTECTED]> wrote:





Doesn't it end up in a call to PyString_Concat()?  
That should return a PyStringConcatenationObject too, right?
K

  
  
  Construct like s = a + b + c + d + e , where a, b etc. have been 
  assigned string values earlier will not benefit from the patch. 





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PATCH submitted: Speed up + for string Re: PATCHsubmitted: Speed up + for string concatenation, now as fast as "".join(x) idiom

2006-10-18 Thread Scott Dial
Chetan Pandya wrote:
> My statement wasn't clear enough.
> 
> Rendering occurs if the string being concatenated is already a 
> concatenation object created by an earlier assignment.
> 

I'm not sure how you came to that conclusion. My reading of the patch 
doesn't suggest that at all. The operation of string_concat would not 
produce what you suggest. I can't find anything anywhere that would 
cause the behavior you suggest. The only case where this is true is if 
the depth of the tree is too great.

To revisit your example, I will notate concats as string pairs:
a = "Value a ="  # new string
a += "anything"  # ("Value a =", "anything")
c = a + b# (("Value a =", "anything"), b)
c += "Something" # ((("Value a =", "anything"), b), "Something"

So again, for your other example of repeated right-hand concatenation, 
you do not continually render the concat object, you merely create new 
one and attach to the leaves. Once the print is executed, you will force 
the rendering of the object, but only once that happens.

So in contrast to your statement, there are actually there are fewer 
allocations of strings and smaller objects being allocated than the 
current trunk uses.

-- 
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]

-- 
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] PATCH submitted: Speed up + for string Re: PATCHsubmitted: Speed up + for string concatenation, now as fast as "".join(x) idiom

2006-10-18 Thread Larry Hastings
Chetan Pandya wrote:
> The deallocation code needs to be robust for a complex tree - it is 
> currently not recursive, but needs to be, like the concatenation code.
It is already both those things.

Deallocation is definitely recursive.  See Objects/stringobject.c, 
function (*ahem*) recursive_dealloc.  That Py_DECREF() line is where it 
recurses into child string concatenation objects.

You might have been confused because it is *optimized* for the general 
case, where the tree only recurses down the left-hand side.  For the 
left-hand side it iterates, instead of recursing, which is both slightly 
faster and much more robust (unlikely to blow the stack).


> Rendering occurs if the string being concatenated is already a 
> concatenation object created by an earlier assignment.
Nope.  Rendering only occurs when somebody asks for the string's value, 
not when merely concatenating.  If you add nine strings together, the 
ninth one fails the "left side has room" test and creates a second object.

Try stepping through it.  Run Python interactively under the debugger.  
Let it get to the prompt.  Execute some expression like "print 3", just 
so the interpreter creates its concatenated encoding object (I get 
"encodings.cp437").  Now, in the debugger, put a breakpoint in the 
rendering code in recursiveConcatenate(), and another on the "op = 
(PyStringConcatenationObject *)PyObject_MALLOC()" line in 
string_concat.  Finally, go back to the Python console and concatenate 
nine strings with this code:
  x = ""
  for i in xrange(9):
x += "a"
You won't hit any breakpoints for rendering, and you'll hit the string 
concatenation object malloc line twice.  (Note that for demonstration 
purposes, this code is more illustrative than running x = "a" + "b" ... 
+ "i" because the peephole optimizer makes a constant folding pass.  
It's mostly harmless, but for my code it does mean I create 
concatenation objects more often.)


In the interests of full disclosure, there is *one* scenario where pure 
string concatenation will cause it to render.  Rendering or deallocating 
a recursive object that's too deep would blow the program stack, so I 
limit recursion depth on the right seven slots of the recursion object.  
That's what the "right recursion depth" field is used for.  If you 
attempt to concatenate a string concatenation object that's already at 
the depth limit, it renders the deep object first.  The depth limit is 
2**14 right now.

You can force this to happen by prepending like crazy:
  x = ""
  for i in xrange(2**15):
x = "a" + x

Since my code is careful to be only iterative when rendering and 
deallocating down the left-hand side of the tree, there is no depth 
limit for the left-hand side.


Step before you leap,


/larry/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] Segfault in python 2.5

2006-10-18 Thread Mike Klaas
[http://sourceforge.net/tracker/index.php?func=detail&aid=1579370&group_id=5470&atid=105470]

Hello,

I'm managed to provoke a segfault in python2.5 (occasionally it just a
"invalid argument to internal function" error).  I've posted a
traceback and a general idea of what the code consists of in the
sourceforge entry.  Unfortunately, I've been attempting for hours to
reduce the problem to a completely self-contained script, but it is
resisting my efforts due to timing problems.

Should I continue in that vein, or is it more useful to provide more
detailed results from gdb?

Thanks,
-Mike
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Segfault in python 2.5

2006-10-18 Thread Michael Hudson
"Mike Klaas" <[EMAIL PROTECTED]> writes:

> [http://sourceforge.net/tracker/index.php?func=detail&aid=1579370&group_id=5470&atid=105470]
>
> Hello,
>
> I'm managed to provoke a segfault in python2.5 (occasionally it just a
> "invalid argument to internal function" error).  I've posted a
> traceback and a general idea of what the code consists of in the
> sourceforge entry. 

I've been reading the bug report with interest, but unless I can
reproduce it it's mighty hard for me to debug, as I'm sure you know.

> Unfortunately, I've been attempting for hours to
> reduce the problem to a completely self-contained script, but it is
> resisting my efforts due to timing problems.
>
> Should I continue in that vein, or is it more useful to provide more
> detailed results from gdb?

Well, I don't think that there's much point in posting masses of
details from gdb.  You might want to try trying to fix the bug
yourself I guess, trying to figure out where the bad pointers come
from, etc.

Are you absolutely sure that the fault does not lie with any extension
modules you may be using?  Memory scribbling bugs have been known to
cause arbitrarily confusing problems...

Cheers,
mwh

-- 
  I'm not sure that the ability to create routing diagrams 
  similar to pretzels with mad cow disease is actually a 
  marketable skill. -- Steve Levin
   -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Segfault in python 2.5

2006-10-18 Thread Jack Jansen

On 18-Oct-2006, at 22:08 , Michael Hudson wrote:
>> Unfortunately, I've been attempting for hours to
>> reduce the problem to a completely self-contained script, but it is
>> resisting my efforts due to timing problems.

Has anyone ever tried to use helgrind (the valgrind module, not the  
heavy metal band:-) on Python?
--
Jack Jansen, <[EMAIL PROTECTED]>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma  
Goldman


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Segfault in python 2.5

2006-10-18 Thread Mike Klaas
On 10/18/06, Michael Hudson <[EMAIL PROTECTED]> wrote:
> "Mike Klaas" <[EMAIL PROTECTED]> writes:

> I've been reading the bug report with interest, but unless I can
> reproduce it it's mighty hard for me to debug, as I'm sure you know.

Indeed.

> > Unfortunately, I've been attempting for hours to
> > reduce the problem to a completely self-contained script, but it is
> > resisting my efforts due to timing problems.
> >
> > Should I continue in that vein, or is it more useful to provide more
> > detailed results from gdb?
>
> Well, I don't think that there's much point in posting masses of
> details from gdb.  You might want to try trying to fix the bug
> yourself I guess, trying to figure out where the bad pointers come
> from, etc.

I've peered at the code, but my knowledge of the python core is
superficial at best.  The fact that it is occuring as a result of a
long string of garbage collection/dealloc/etc. and involves threading
lowers my confidence further.   That said, I'm beginning to think that
to reproduce this in a standalone script will require understanding
the problem in greater depth regardless...

> Are you absolutely sure that the fault does not lie with any extension
> modules you may be using?  Memory scribbling bugs have been known to
> cause arbitrarily confusing problems...

I've had sufficient experience being arbitrarily confused to never be
sure about such things, but I am quite confident.  The script I posted
in the bug report is all stock python save for the operation in <>'s.
That operation is pickling and unpickling (using pickle, not cPickle)
a somewhat complicated pure-python instance several times.  It's doing
nothing with the actual instance--it just happens to take the right
amount of time to trigger the segfault.  It's still not perfect--this
trimmed-down version segfaults only sporatically, while the original
python script segfaults reliably.

-Mike
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Python-Dev Digest, Vol 39, Issue 54

2006-10-18 Thread Chetan Pandya
I got up in the middle of the night and wrote the email - and it shows.Apologies for creating confusion. My comments below.-ChetanOn 10/18/06, 
[EMAIL PROTECTED] 
Date: Wed, 18 Oct 2006 13:04:14 -0700From: Larry Hastings <[EMAIL PROTECTED]>Subject: Re: [Python-Dev] PATCH submitted: Speed up + for string Re:PATCHsubmitted: Speed up + for string concatenation, now as fast as
"".join(x) idiomTo: python-dev@python.orgMessage-ID: <[EMAIL PROTECTED]>Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Chetan Pandya wrote:> The deallocation code needs to be robust for a complex tree - it is> currently not recursive, but needs to be, like the concatenation code.It is already both those things.
Deallocation is definitely recursive.  See Objects/stringobject.c,function (*ahem*) recursive_dealloc.  That Py_DECREF() line is where itrecurses into child string concatenation objects.You might have been confused because it is *optimized* for the general
case, where the tree only recurses down the left-hand side.  For theleft-hand side it iterates, instead of recursing, which is both slightlyfaster and much more robust (unlikely to blow the stack).
Actually I looked at the setting of ob_sstrings to  NULL in recursive_dealloc and thought  none of the strings will get destroyed as the list is destroyed. However it is only setting the first element to NULL, which is fine.
> Rendering occurs if the string being concatenated is already a> concatenation object created by an earlier assignment.
Nope.  Rendering only occurs when somebody asks for the string's value,not when merely concatenating.  If you add nine strings together, theninth one fails the "left side has room" test and creates a second object.
I don't know what I was thinking. In the whole of string_concat() there is no call to render the string, except for the right recursion case. 
Try stepping through it.  Run Python interactively under the debugger.Let it get to the prompt.  Execute some _expression_ like "print 3", justso the interpreter creates its concatenated encoding object (I get
"encodings.cp437").  Now, in the debugger, put a breakpoint in therendering code in recursiveConcatenate(), and another on the "op =(PyStringConcatenationObject *)PyObject_MALLOC()" line in
string_concat.  Finally, go back to the Python console and concatenatenine strings with this code:  x = ""  for i in xrange(9):x += "a"You won't hit any breakpoints for rendering, and you'll hit the string
concatenation object malloc line twice.  (Note that for demonstrationpurposes, this code is more illustrative than running x = "a" + "b" ...+ "i" because the peephole optimizer makes a constant folding pass.
It's mostly harmless, but for my code it does mean I createconcatenation objects more often.)I don't have a patch build, since I didn't download the revision used by the patch. However, I did look at values in the debugger and it looked like x in your example above had a reference count of 2 or more within string_concat even when there were no other assignments that would account for it. My idea was to investibate this, but this was the whole reason for saying that the concatenation will create new objects. However, I ran on another machine under debugger and I get the reference count as 1,  which is what I would expect.  I need to find out what has happened to my work machine.
In the interests of full disclosure, there is *one* scenario where purestring concatenation will cause it to render.  Rendering or deallocating
a recursive object that's too deep would blow the program stack, so Ilimit recursion depth on the right seven slots of the recursion object.That's what the "right recursion depth" field is used for.  If you
attempt to concatenate a string concatenation object that's already atthe depth limit, it renders the deep object first.  The depth limit is2**14 right now.
You can force this to happen by prepending like crazy:  x = ""  for i in xrange(2**15):x = "a" + xSince my code is careful to be only iterative when rendering anddeallocating down the left-hand side of the tree, there is no depth
limit for the left-hand side.The recursion limit seems to be optimistic, given the default stack limit, but of course, I haven't tried it. There is probably a depth limit on the left hand side as well, since recursiveConcatenate is recursive even on the left side. 
Step before you leap,/larry/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Segfault in python 2.5

2006-10-18 Thread Tim Peters
[Michael Hudson]
>> I've been reading the bug report with interest, but unless I can
>> reproduce it it's mighty hard for me to debug, as I'm sure you know.

[Mike Klaas]
> Indeed.

Note that I just attached a much simpler pure-Python script that fails
very quickly, on Windows, using a debug build.  Read the new comment
to learn why both "Windows" and "debug build" are essential to it
failing reliably and quickly ;-)

>>> Unfortunately, I've been attempting for hours to reduce the problem to a
>>> completely self-contained script, but it is resisting my efforts
due to timing
>>> problems.

Yes, but you did good!  This is still just an educated guess on my
part, but my education here is hard to match ;-):  this new business
of generators deciding to "clean up after themselves" if they're left
hanging appears to have made it possible for a generator to hold on to
a frame whose thread state has been free()'d, after the thread that
created the generator has gone away.  Then when the generator gets
collected as trash, the new exception-based "clean up abandoned
generator" gimmick tries to access the generator's frame's thread
state, but that's just a raw C struct (not a Python object with
reachability-based lifetime), and the thread free()'d that struct when
the thread went away.  The important timing-based vagary here is
whether dead-thread cleanup gets performed before the main thread
tries to clean up the trash generator.

> I've peered at the code, but my knowledge of the python core is
> superficial at best.  The fact that it is occuring as a result of a
> long string of garbage collection/dealloc/etc. and involves threading
> lowers my confidence further.   That said, I'm beginning to think that
> to reproduce this in a standalone script will require understanding
> the problem in greater depth regardless...

Or upgrade to Windows ;-)

>> Are you absolutely sure that the fault does not lie with any extension
>> modules you may be using?  Memory scribbling bugs have been known to
>> cause arbitrarily confusing problems...

Unless I've changed the symptom, it's been reduced to minimal pure
Python.  It does require a thread T, and creating a generator in T,
where the generator object's lifetime is controlled by the main
thread, and where T vanishes before the generator has exited of its
own accord.

Offhand I don't know how to repair it.  Thread states /aren't/ Python
objects, and there's no provision for a thread state to outlive the
thread it represents.

> I've had sufficient experience being arbitrarily confused to never be
> sure about such things, but I am quite confident.  The script I posted
> in the bug report is all stock python save for the operation in <>'s.
> That operation is pickling and unpickling (using pickle, not cPickle)
> a somewhat complicated pure-python instance several times.

FYI, in my whittled script, your `getdocs()` became simply:

def getdocs():
while True:
yield None

and it's called only once, via self.docIter.next().  In fact, the
"while True:" isn't needed there either (given that it's only resumed
once now).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Segfault in python 2.5

2006-10-18 Thread Mike Klaas
On 10/18/06, Tim Peters <[EMAIL PROTECTED]> wrote:
> [Mike Klaas]
> > Indeed.
>
> Note that I just attached a much simpler pure-Python script that fails
> very quickly, on Windows, using a debug build.  Read the new comment
> to learn why both "Windows" and "debug build" are essential to it
> failing reliably and quickly ;-)

Thanks!  Next time I find a bug, installing Windows will  certainly be
my first step .

<>
> Yes, but you did good!  This is still just an educated guess on my
> part, but my education here is hard to match ;-):  this new business
> of generators deciding to "clean up after themselves" if they're left
> hanging appears to have made it possible for a generator to hold on to
> a frame whose thread state has been free()'d, after the thread that
> created the generator has gone away.  Then when the generator gets
> collected as trash, the new exception-based "clean up abandoned
> generator" gimmick tries to access the generator's frame's thread
> state, but that's just a raw C struct (not a Python object with
> reachability-based lifetime), and the thread free()'d that struct when
> the thread went away.  The important timing-based vagary here is
> whether dead-thread cleanup gets performed before the main thread
> tries to clean up the trash generator.

Indeed--and normally it doesn't happen that way.  My/your script never
crashes on the first iteration because the thread's target is the
generator and thus it gets DECREF'd before the thread terminates.  But
the exception from the first iteration holds on to a reference to the
frame/generator so when it gets cleaned up (in the second iteration,
due to a new exception overwriting it) the generator is freed after
the thread is destroyed.  At least, I think...

<>
> Offhand I don't know how to repair it.  Thread states /aren't/ Python
> objects, and there's no provision for a thread state to outlive the
> thread it represents.

Take this with a grain of salt, but ISTM that the problem can be
repaired by resetting the generator's frame threadstate to the current
threadstate:

(in genobject.c:gen_send_ex():80)
Py_XINCREF(tstate->frame);
assert(f->f_back == NULL);
f->f_back = tstate->frame;
+f->f_tstate = tstate;

gen->gi_running = 1;
result = PyEval_EvalFrameEx(f, exc);
gen->gi_running = 0;

Shouldn't the thread state generally be the same anyway? (I seem to
recall some gloomy warning against resuming generators in separate
threads).

This solution is surely wrong--if f_tstate != tstate, then the
generator _is_ being resumed in another thread and so the generated
traceback will be wrong (among other issues which surely occur by
fudging a frame's threadstate).  Perhaps it could be set conditionally
by gen_close before signalling the exception?  A lie, but a smaller
lie than a segfault.  We could advertise that the exception ocurring
from generator .close() isn't guaranteed to have an accurate traceback
in this case.

Take all this with a grain of un-core-savvy salt.

Thanks again for investigating this, Tim,
-Mike
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Python-Dev Digest, Vol 39, Issue 54

2006-10-18 Thread Larry Hastings




Chetan Pandya wrote:

  
  I don't have a patch build, since I didn't download the revision
used by the patch. 
However, I did look at values in the debugger and it looked like x in
your example above had a reference count of 2 or more within
string_concat even when there were no other assignments that would
account for it.

It could be the optimizer.  If you concatenate hard-coded strings, the
peephole optimizer does constant folding.  It says "hey, look, this
binary operator is performed on two constant objects".  So it evaluates
the _expression_ itself and substitutes the result, in this case swapping
(pseudotokens here) [PUSH "a" PUSH "b" PLUS] for [PUSH "ab"].

Oddly, it didn't seem to optimize away the whole _expression_.  If you
say "a" + "b" + "c" + "d" + "e", I would have expected the peephole
optimizer to turn that whole shebang into [PUSH "abcde"].  But when I
gave it a cursory glance it seemed to skip every-other; it
constant-folded "a" + "b", then  + "c" and optimized ("a" + "b" + "c")
+ "d", resulting ultimately I believe in [PUSH "ab" PUSH "cd" PLUS PUSH
"e" PLUS].  But I suspect I missed something; it bears further
investigation.

But this is all academic, as real-world performance of my patch is not
contingent on what the peephole optimizer does to short runs of
hard-coded strings in simple test cases.


  
  The recursion limit seems to be optimistic, given the default
stack limit, but of course, I haven't tried it.
  

I've tried it, on exactly one computer (running Windows XP).  The depth
limit was arrived at experimentally.  But it is probably too optimistic
and should be winched down.

On the other hand, right
now when you do x = "a" + x ten zillion times there are always two
references to the concatenation object stored in x: the interpreter
holds one, and x itself holds the other.  That means I have to build
a new concatenation object each time, so it becomes a degenerate tree
(one leaf and one
subtree) recursing down the right-hand side.

I plan to fix that in
my next patch.  There's already code that says "if the next instruction
is a store, and the location we're storing to holds a reference to the
left-hand side of the concatenation, make the location drop its
reference".  That was an optimization for the old-style concat code;
when the left side only had one reference it would simply resize it and
memcpy() in the right side.  I plan to add support
for dropping the reference when it's the *right*-hand side of the
concatenation, as that would help prepending immensely.  Once that's
done, I believe it'll prepend ((depth limit) * (number of items in
ob_sstrings - 1)) + 1 strings before needing to render.


  
  There is probably a depth limit on the left hand side as well,
since recursiveConcatenate is recursive even on the left side.
  

Let me again stress that recursiveConcatenate is *iterative* down the
left side; it is *not* not *not* recursive.  The outer loop iterates
over "s->ob_sstrings[0]"s.  The nested "for" loop iterates
backwards, from the highest string used down to "s->ob_sstrings +
1", aka "&s->ob_sstrings[1]", recursing into them.  It then sets
"s" to "*s->ob_sstrings", aka "s->ob_sstrings[0]" and the outer
loop repeats.  This is iterative.

As a personal favor to me, please step through my code before you tell
me again how my code is recursive down the left-hand side.

Passing the dutchie,


larry


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com