Re: [Zope-dev] SVN: Zope/trunk/ Optimized the `OFS.Traversable.getPhysicalPath` method to avoid excessive amounts of method calls. Thx to Nikolay Kim from Enfold
On Thu, Jul 14, 2011 at 7:03 PM, Tres Seaver wrote: > A further micro-optimization is to pre-allocate a big list in a thread > local. Running the attached script produces:: > > $ python perftest.py > Via insert_0: 24.8401839733 > Via append: 15.6596238613 > speedup: 37.0% > Via tuple_add: 21.9555268288 > speedup: 11.6% > Via prealloc: 10.5278339386 > speedup: 57.6% Thanks for that test! I've run it with a much more reasonable test data set from one of our live databases, for example: letters = ['', 'nordic', 'en', 'nordic-council', 'organisation-and-structure', 'committees', 'welfare-committee', 'meetings-and-meeting-documents', 'meeting-of-the-welfare-committee-31-march-2011-stockholm'] which produces quite a different result: Via insert_0: 4.35216093063 Via append: 4.80827713013 speedup: -10.5% Via tuple_add: 1.73557782173 speedup: 60.1% Via prealloc: 2.17882084846 speedup: 49.9% If I run it with a smaller segment, like: letters = ['', 'nordic', 'en', 'nordic-council'] the effect is even clearer: Via insert_0: 1.88715004921 Via append: 2.91810202599 speedup: -54.6% Via tuple_add: 0.809303998947 speedup: 57.1% Via prealloc: 1.56584882736 speedup: 17.0% Looks like the tuple add is faster until you hit very long sequences. But even if you had such a nested ZODB structure, the majority of lookups would still happen for the much shorter sequences. So I think the tuple add is the winner for this case. Hanno ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] SVN: Zope/trunk/ Optimized the `OFS.Traversable.getPhysicalPath` method to avoid excessive amounts of method calls. Thx to Nikolay Kim from Enfold
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/14/2011 11:47 AM, Tres Seaver wrote: > While we're at it, 'list.insert(0,foo)' is known to be slower in a > tight loop than 'list.append(foo)', with a 'reversed' at the end of > the loop:: > > $ python -m timeit -s "from string import letters" "path = []" \ "for > letter in letters: path.append(letter)" \ "result = > tuple(reversed(path))" 10 loops, best of 3: 15.9 usec per loop > > $ python -m timeit -s "from string import letters" "path = []" \ "for > letter in letters: path.insert(0, letter)" \ "result = tuple(path)" > 1 loops, best of 3: 25.3 usec per loop > > > For the sake of comparison, the original tuple addition is actually > between the two: > > $ python -m timeit -s "from string import letters" "result = ()" \ > "for letter in letters: result += (letter,)" 1 loops, best of 3: > 21.6 usec per loop A further micro-optimization is to pre-allocate a big list in a thread local. Running the attached script produces:: $ python perftest.py Via insert_0: 24.8401839733 Via append: 15.6596238613 speedup: 37.0% Via tuple_add: 21.9555268288 speedup: 11.6% Via prealloc: 10.5278339386 speedup: 57.6% Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4fIVAACgkQ+gerLs4ltQ6BlACguSHcHLWGjVIVJlQYIpL42RNc mqMAn1Y16tyVmUkxd2ipNrZSeZpSFVEA =oJJY -END PGP SIGNATURE- perftest.py Description: application/httpd-cgi ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] SVN: Zope/trunk/ Optimized the `OFS.Traversable.getPhysicalPath` method to avoid excessive amounts of method calls. Thx to Nikolay Kim from Enfold
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 07/14/2011 04:16 AM, Hanno Schlichting wrote:
> Log message for revision 122213: Optimized the
> `OFS.Traversable.getPhysicalPath` method to avoid excessive amounts
> of method calls. Thx to Nikolay Kim from Enfold
>
>
> Changed: U Zope/trunk/doc/CHANGES.rst U
> Zope/trunk/src/OFS/Traversable.py
>
> -=- Modified: Zope/trunk/doc/CHANGES.rst
> ===
> --- Zope/trunk/doc/CHANGES.rst2011-07-14 07:16:04 UTC (rev 122212)
> +++ Zope/trunk/doc/CHANGES.rst2011-07-14 08:16:08 UTC (rev 122213)
> @@ -19,6 +19,9 @@ Features Added ++
>
> +- Optimized the `OFS.Traversable.getPhysicalPath` method to avoid
> excessive + amounts of method calls. + - During startup open a
> connection to every configured database, to ensure all of them can
> indeed be accessed. This avoids surprises during runtime when
> traversal to some database mountpoint could fail as the underlying
> storage
>
> Modified: Zope/trunk/src/OFS/Traversable.py
> ===
> --- Zope/trunk/src/OFS/Traversable.py 2011-07-14 07:16:04 UTC (rev
> 122212) +++ Zope/trunk/src/OFS/Traversable.py 2011-07-14 08:16:08 UTC
> (rev 122213) @@ -114,14 +114,47 @@ access this object again later,
> for example in a copy/paste operation. getPhysicalRoot() and
> getPhysicalPath() are designed to operate together. + +This
> implementation is optimized to avoid excessive amounts of function +
> calls while walking up from an object on a deep level. """ -
> path = (self.getId(),) +try: +id = self.id +
> except AttributeError: +id = self.getId() +else:
> +if id is None: +id = self.getId()
>
> p = aq_parent(aq_inner(self)) +if p is None: +
> return (id, )
>
> -if p is not None: -path = p.getPhysicalPath() +
> path +path = [id] +func =
> self.getPhysicalPath.im_func +while p is not None: +
> if func is p.getPhysicalPath.im_func: +try: +
> pid = p.id +except AttributeError: +
> pid = p.getId() +else: +if pid is
> None: +pid = p.getId()
>
> +path.insert(0, pid) +try: +
> p = p.__parent__ +except AttributeError: +
> p = None +else: +if
> IApplication.providedBy(p): +path.insert(0, '') +
> path = tuple(path) +else: +path =
> p.getPhysicalPath() + tuple(path) +break + return
> path
>
> security.declarePrivate('unrestrictedTraverse')
While we're at it, 'list.insert(0,foo)' is known to be slower in a tight
loop than 'list.append(foo)', with a 'reversed' at the end of the loop::
$ python -m timeit -s "from string import letters" "path = []" \
"for letter in letters: path.append(letter)" \
"result = tuple(reversed(path))"
10 loops, best of 3: 15.9 usec per loop
$ python -m timeit -s "from string import letters" "path = []" \
"for letter in letters: path.insert(0, letter)" \
"result = tuple(path)"
1 loops, best of 3: 25.3 usec per loop
For the sake of comparison, the original tuple addition is actually
between the two:
$ python -m timeit -s "from string import letters" "result = ()" \
"for letter in letters: result += (letter,)"
1 loops, best of 3: 21.6 usec per loop
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk4fD4gACgkQ+gerLs4ltQ7b3QCg1z+oZKKDZ5IhuMvAockJ9mvx
SO4AoIr3IEMnq78m70DZAOc0yV9+++y4
=8vOY
-END PGP SIGNATURE-
___
Zope-Dev maillist - Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )
