Author: Armin Rigo <ar...@tunes.org> Branch: extradoc Changeset: r4202:7f5d0a8f1a66 Date: 2012-04-27 09:46 +0200 http://bitbucket.org/pypy/extradoc/changeset/7f5d0a8f1a66/
Log: Expand here and there. diff --git a/blog/draft/stm-apr2012.rst b/blog/draft/stm-apr2012.rst --- a/blog/draft/stm-apr2012.rst +++ b/blog/draft/stm-apr2012.rst @@ -1,47 +1,75 @@ STM update (and thanks everybody) ================================= -A short update on the Software Transactional Memory (STM) side. I have -now reached the point where the basics seem to work, and the integration -with the Garbage Collection subsystem is --- not done by far, but at -least "not crashing in my simple tests and not leaking memory too -quickly". (It is leaking a bit of memory though, and it is never -calling ``__del__`` so far.) +A short update on the Software Transactional Memory (STM) side. Let me +remind you that the work is to add STM internally into PyPy, with the +goal of letting the user's programs run on multiple cores after a minor +adaptation. (The goal is not to expose STM to the user's program.) I +will soon write some official documentation that explains in more +details exactly what you get. For now you can read the previous blog__ +posts__, and you can also find technical details in the `call for +donation`_ itself; or directly look at how I adapted the examples linked +to later in this post. + +.. _`call for donation`: http://pypy.org/tmdonate.html +.. __: http://morepypy.blogspot.com/2012/03/call-for-donations-for-software.html +.. __: http://morepypy.blogspot.com/2012/01/transactional-memory-ii.html + +I have now reached the point where the basics seem to work. There is no +integration with the JIT so far; moreover the integration with the +Garbage Collection subsystem is not finished right now, but at least +"not crashing in my simple tests and not leaking memory too quickly". +(It means that it is never calling ``__del__`` so far, although it +releases memory; and when entering transactional mode or when going to +the next transaction, all live objects become immortal. This should +still let most not-too-long-running programs work.) If you want to play with it, you can download `this binary`_ (you need to put it in a place with the paths ``lib-python`` and ``lib_pypy``, for example inside the main directory from a regular `nightly tarball`_ -or from a full checkout). It -is for Linux x86 32-bit. -This version was compiled from the `stm-gc`_ branch on the 25th of April. -It runs e.g. the `modified version of richards`_. -This branch could also be translated for Linux x86-64, -but I didn't do fixes for other platforms for now. +or from a full checkout). +This version was compiled for Linux x86 32-bit from the `stm-gc`_ branch +on the 25th of April. It runs e.g. the modified version of richards_. +This branch could also be translated for Linux x86-64, but not for +other OSes nor other CPUs for now. .. _`this binary`: http://wyvern.cs.uni-duesseldorf.de/~arigo/pypy-stm-22fccf3c9b5e.tar.bz2 .. _`nightly tarball`: http://buildbot.pypy.org/nightly/trunk/ .. _`stm-gc`: https://bitbucket.org/pypy/pypy/src/stm-gc -.. _`modified version of richards`: https://bitbucket.org/pypy/pypy/raw/stm-gc/pypy/translator/stm/test/richards.py +.. _richards: https://bitbucket.org/pypy/pypy/raw/stm-gc/pypy/translator/stm/test/richards.py -It exposes the same interface as the pure Python transaction_ module -(except of course that it's not using the naive implementation linked -above, but its own). A difference is that it -doesn't support epoll right now, so it cannot be used to play with `a -branch of Twisted`_; but that's coming soon. For now you can use it to -get multi-core usage on purely computational programs, like PyPy's own -``translate.py`` --- for that I did a tweak `in rpython/rtyper.py`_ (lines -273-281 are all that I needed to add), but there are a few more places -in the whole ``translate.py`` that could be similarly enhanced. +The resulting ``pypy-stm`` exposes the same interface as the pure Python +transaction_ module, which is an emulator (running on CPython or any +version of PyPy) which can be used to play around and prepare your +programs. See the comments in there. A difference is that the real +``pypy-stm`` doesn't support epoll right now, so it cannot be used yet +to play with `a branch of Twisted`_ that was already adapted (thanks +Jean-Paul Calderone); but that's coming soon. For now you can use it to +get multi-core usage on purely computational programs. + +I did for example adapt PyPy's own ``translate.py``: see the tweak `in +rpython/rtyper.py`_. Lines 273-281 are all that I needed to add, and +they are mostly a "simplification and parallelization" of the lines +above. There are a few more places in the whole ``translate.py`` that +could be similarly modified, but overall it is just that: a few places. +I did not measure performance, but I checked that it is capable of using +multiple cores in the RTyping step of translation, with --- as expected +--- some still-reasonable number of conflicts, particularly at the +beginning when shared data structures are still being built. .. _transaction: https://bitbucket.org/pypy/pypy/raw/stm-gc/lib_pypy/transaction.py .. _`a branch of Twisted`: svn://svn.twistedmatrix.com/svn/Twisted/branches/stm-5526 .. _`in rpython/rtyper.py`: https://bitbucket.org/pypy/pypy/src/stm-gc/pypy/rpython/rtyper.py#cl-249 -The performance is not great, even taking into account that it has no -JIT so far, but it seems to scale; at least, it does scale on my -2-real-cores, 4-hyperthreaded-cores laptop, roughly as expected. +On a few smaller, more regular examples like richards_, I did measure +the performance. It is not great, even taking into account that it has +no JIT so far; but it is well within the 2-to-5-times slower goal, and +it does seem to scale. At least, it scales roughly as expected on my +2-real-cores, 4-hyperthreaded-cores laptop (which means that using two +cores is not quite twice as fast as using only one, and using all four +cores is again not twice faster but only 20-30%). -And... +And finally... ...a big thank you to everyone who contributed some money to support this! As you see on the PyPy_ site, we got more than 6700$ so far in _______________________________________________ pypy-commit mailing list pypy-commit@python.org http://mail.python.org/mailman/listinfo/pypy-commit