Re: [Zope-dev] zope.interface memory optimization
On Wed, Nov 10, 2010 at 12:07:00PM +0100, Charlie Clark wrote: Am 10.11.2010, 01:40 Uhr, schrieb Brian Sutherland br...@vanguardistas.net: On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most ... ? Er, somehow my thoughts didn't make it into text: attributes are zope.schema fields (i.e. subclasses of Attribute). To really make that work, you'd have to add __slots__ to most of the classes in zope.schema as well. It might be purely academic but if this does significantly reduce memory use then it might be worth pursuing. Turns out it's not that academic. I tried a bit harder and managed to apply __slots__ to the Element, Attribute and Method classes in zope.interface. That saved an additional 0.5 % of memory (bytes). I have a small patch to zope.schema that fixes the breakage there. It looks like there is an additional 0.5% available if zope.schema is refactored to use slots as well. Slots are supposed to be beneficial when you've got lots of objects around. Zope schema fields are pretty static anyway so I don't see a great penalty in having to add slots to them, although obviously a migration of a larger project would probably be a pain. Searching for __dict__ here: http://zope3.pov.lt/trac/browser/zope.schema/trunk/src/zope/schema/_bootstrapfields.py Will give you a good idea of the problems. Field.bind and ValidatedProperty both depend heavily on __dict__. For ValidatedProperty in particular I have no idea how that would work when using slots. Back to the micro-optimisation. At Jeff Rush's talk at PyCon this year he covered the penalties incurred by . lookups so that might be something that could be investigated but might it also be possible to memoize the method calls? I keep forgetting when you can't do that. I had a look at http://en.wikipedia.org/wiki/Memoization, but I don't see how it could be applied in this case. Charlie -- Charlie Clark Managing Director Clark Consulting Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope ) -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
Am 10.11.2010, 01:40 Uhr, schrieb Brian Sutherland br...@vanguardistas.net: On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most ... ? Er, somehow my thoughts didn't make it into text: attributes are zope.schema fields (i.e. subclasses of Attribute). To really make that work, you'd have to add __slots__ to most of the classes in zope.schema as well. It might be purely academic but if this does significantly reduce memory use then it might be worth pursuing. Slots are supposed to be beneficial when you've got lots of objects around. Zope schema fields are pretty static anyway so I don't see a great penalty in having to add slots to them, although obviously a migration of a larger project would probably be a pain. Back to the micro-optimisation. At Jeff Rush's talk at PyCon this year he covered the penalties incurred by . lookups so that might be something that could be investigated but might it also be possible to memoize the method calls? I keep forgetting when you can't do that. Charlie -- Charlie Clark Managing Director Clark Consulting Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
Am 08.11.2010, 15:35 Uhr, schrieb Brian Sutherland
br...@vanguardistas.net:
If no-one replies, I'll assume that 3% is just not enough to be
interesting and do nothing;)
Hi Brian,
thanks for sharing this but it looks to me like a micro-optimisation that
isn't really worth going against best practice for - by using explicit
name-mangling.
-return self.__tagged_values.get(tag, default)
+return getattr(self, '_Element__tagged_values', {}).get(tag,
default)
Any improvements higher up the stack (using slots, different Python
version or compilers like LLVM) are likely to have a more significant
effect. Might be worth timing a slots-based implementation.
Charlie
--
Charlie Clark
Managing Director
Clark Consulting Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
On Tue, Nov 09, 2010 at 01:21:00AM +0200, Marius Gedminas wrote:
On Mon, Nov 08, 2010 at 03:35:09PM +0100, Brian Sutherland wrote:
I've committed 2 patches to a jinty-mem branch of zope.interface.
Together these patches reduce the startup memory use of my ZTK based
application by 3%. Is that enough to justify their risk/complexity?
https://mail.zope.org/pipermail/checkins/2010-November/052516.html
https://mail.zope.org/pipermail/checkins/2010-November/052517.html
If no-one replies, I'll assume that 3% is just not enough to be
interesting and do nothing;)
It's so interesting I'm going to ask for even more numbers: how much is
that 3% in kilobytes, or objects?
Ok, you asked for it ;)
The 3% was in kilobytes as measured using /bin/ps on OSX leopard with
python2.6 from MacPorts. The app I measured is a ZTK 1.0 based web
application.
My application without the changes:
Objects:245614
RSS memory (kb):59116
My application with the changes:
Objects:227147
RSS memory (kb):57180
I also did a micro-benchmark (more below) which indicate that when
creating an interface with 5 attributes and 5 methods you save 41% in
kilobytes and 34% in objects.
Is there any measurable speed impact
for application startup or test runs (faster/slower)?
My app doesn't startup measurably faster or slower. The ZTK tests seem
to run a little faster, but the error is high on my laptop:
Without patches:
./bin/test-ztk 71.93s
./bin/test-ztk 70.49s
./bin/test-ztk 70.31s
With patches:
./bin/test-ztk 70.28s
./bin/test-ztk 70.00s
./bin/test-ztk 69.98s
I also did some micro-benchmarking (script is attached). There were
repeatable results which indicate that startup will be faster and
runtime only probably faster:
Without patches:
Performance bench
-
one interface, 5 methods, 5 functions: 181.00 usec/pass
five inheriting interfaces: 355.76 usec/pass
one interface: 59.79 usec/pass
query uninitialized tagged value: 1.26 usec/pass
query initialized tagged value: 1.27 usec/pass
Memory bench
Memory before (kb):
RSS VSZ
33684 109780
objects created: 150001
Memory after (kb):
RSS VSZ
47856 124116
With patches:
Performance bench
-
one interface, 5 methods, 5 functions: 165.97 usec/pass
five inheriting interfaces: 341.22 usec/pass
one interface: 53.03 usec/pass
query uninitialized tagged value: 0.75 usec/pass
query initialized tagged value: 1.37 usec/pass
Memory bench
Memory before (kb):
RSS VSZ
33328 109524
objects created: 99001
Memory after (kb):
RSS VSZ
41728 117972
Otherwise I'm +1, assuming this doesn't somehow make the code 10%
slower.
Marius Gedminas
--
http://pov.lt/ -- Zope 3/BlueBream consulting and development
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
--
Brian Sutherland
#
# Perdormance Benchmark
#
import os
import timeit
import subprocess
from zope.interface import Interface, Attribute
print Performance bench
print -
s1 = \
class I1(Interface):
a1 = Attribute('one')
a2 = Attribute('two')
a3 = Attribute('thee')
a4 = Attribute('four')
a5 = Attribute('five')
def f1(arg):
pass
def f2(arg):
pass
def f3(arg):
pass
def f4(arg):
pass
def f5(arg):
pass
t = timeit.Timer(stmt=s1, setup=from zope.interface import Interface,
Attribute)
print one interface, 5 methods, 5 functions: %.2f usec/pass % (100 *
t.timeit(number=1)/1)
s1 = \
class I1(Interface):
pass
class I2(I1):
pass
class I3(I2):
pass
class I4(I3):
pass
class I5(I4):
pass
t = timeit.Timer(stmt=s1, setup=from zope.interface import Interface,
Attribute)
print five inheriting interfaces: %.2f usec/pass % (100 *
t.timeit(number=1)/1)
s1 = \
class I1(Interface):
pass
t = timeit.Timer(stmt=s1, setup=from zope.interface import Interface,
Attribute)
print one interface: %.2f usec/pass % (100 * t.timeit(number=1)/1)
setup = \
from zope.interface import Interface, Attribute
class I1(Interface):
pass
s1 = \
I1.queryTaggedValue('bob')
t = timeit.Timer(stmt=s1, setup=setup)
print query uninitialized tagged value: %.2f usec/pass % (100 *
t.timeit(number=5)/5)
setup = \
from zope.interface import Interface, Attribute
class I1(Interface):
pass
I1.setTaggedValue('bob', 'bobby')
s1 = \
I1.queryTaggedValue('bob')
t = timeit.Timer(stmt=s1, setup=setup)
print
Re: [Zope-dev] zope.interface memory optimization
On Tue, Nov 09, 2010 at 03:49:38PM +0100, Charlie Clark wrote:
Am 08.11.2010, 15:35 Uhr, schrieb Brian Sutherland
br...@vanguardistas.net:
If no-one replies, I'll assume that 3% is just not enough to be
interesting and do nothing;)
Hi Brian,
thanks for sharing this but it looks to me like a micro-optimisation that
isn't really worth going against best practice for - by using explicit
name-mangling.
-return self.__tagged_values.get(tag, default)
+return getattr(self, '_Element__tagged_values', {}).get(tag,
default)
In response to Marius' comments I changed the patch a bit earlier to
this:
def queryTaggedValue(self, tag, default=None):
Returns the value associated with 'tag'.
tv = self.__tagged_values
if tv is None:
return default
return tv.get(tag, default)
Which is is faster as well as not doing the explicit name mangling you
object to.
Any improvements higher up the stack (using slots, different Python
version or compilers like LLVM) are likely to have a more significant
effect.
That would be great, when it arrives.
Actually, in my case, memory is the issue. So the only optimization
that'll really help in a big way is if multiple python processes running
similar code start sharing data.
Some people have poked at that problem, but I havn't seen anything
significant yet.
Might be worth timing a slots-based implementation.
I just tried to add slots to Element, Attribute and Method classes, but
it there's too many tests that break to benchmark it running the ZTK
tests.
On the micro benchmarks, the only difference I see is a memory
improvement which I think will not survive in practice because most
Charlie
--
Charlie Clark
Managing Director
Clark Consulting Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
--
Brian Sutherland
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
On Tue, Nov 09, 2010 at 05:04:11PM +0100, Brian Sutherland wrote: Might be worth timing a slots-based implementation. I just tried to add slots to Element, Attribute and Method classes, but it there's too many tests that break to benchmark it running the ZTK tests. On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most ... ? Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
On Wed, Nov 10, 2010 at 12:14:54AM +0200, Marius Gedminas wrote: On Tue, Nov 09, 2010 at 05:04:11PM +0100, Brian Sutherland wrote: Might be worth timing a slots-based implementation. I just tried to add slots to Element, Attribute and Method classes, but it there's too many tests that break to benchmark it running the ZTK tests. On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most ... ? Er, somehow my thoughts didn't make it into text: attributes are zope.schema fields (i.e. subclasses of Attribute). To really make that work, you'd have to add __slots__ to most of the classes in zope.schema as well. Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope ) -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] zope.interface memory optimization
I've committed 2 patches to a jinty-mem branch of zope.interface. Together these patches reduce the startup memory use of my ZTK based application by 3%. Is that enough to justify their risk/complexity? https://mail.zope.org/pipermail/checkins/2010-November/052516.html https://mail.zope.org/pipermail/checkins/2010-November/052517.html If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;) -- Brian Sutherland ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
Hi. On Mon, Nov 8, 2010 at 3:35 PM, Brian Sutherland br...@vanguardistas.net wrote: I've committed 2 patches to a jinty-mem branch of zope.interface. Together these patches reduce the startup memory use of my ZTK based application by 3%. Is that enough to justify their risk/complexity? https://mail.zope.org/pipermail/checkins/2010-November/052516.html https://mail.zope.org/pipermail/checkins/2010-November/052517.html If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;) The changes looked all low-risk to me. I see no reason, why we shouldn't put them in. There's some quite complex applications built with zope.interface that might benefit even more from this. Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
On Mon, Nov 08, 2010 at 03:35:09PM +0100, Brian Sutherland wrote: I've committed 2 patches to a jinty-mem branch of zope.interface. Together these patches reduce the startup memory use of my ZTK based application by 3%. Is that enough to justify their risk/complexity? https://mail.zope.org/pipermail/checkins/2010-November/052516.html https://mail.zope.org/pipermail/checkins/2010-November/052517.html If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;) It's so interesting I'm going to ask for even more numbers: how much is that 3% in kilobytes, or objects? Is there any measurable speed impact for application startup or test runs (faster/slower)? Otherwise I'm +1, assuming this doesn't somehow make the code 10% slower. Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope.interface memory optimization
On Tue, 2010-11-09 at 01:21 +0200, Marius Gedminas wrote: On Mon, Nov 08, 2010 at 03:35:09PM +0100, Brian Sutherland wrote: I've committed 2 patches to a jinty-mem branch of zope.interface. Together these patches reduce the startup memory use of my ZTK based application by 3%. Is that enough to justify their risk/complexity? https://mail.zope.org/pipermail/checkins/2010-November/052516.html https://mail.zope.org/pipermail/checkins/2010-November/052517.html If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;) I'm -1 on it for only 3%. ;-) - C It's so interesting I'm going to ask for even more numbers: how much is that 3% in kilobytes, or objects? Is there any measurable speed impact for application startup or test runs (faster/slower)? Otherwise I'm +1, assuming this doesn't somehow make the code 10% slower. Marius Gedminas ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
