Author: Matt Jackson <[email protected]> Branch: Changeset: r94508:b84ccf3ff0d6 Date: 2018-04-21 19:30 +0100 http://bitbucket.org/pypy/pypy/changeset/b84ccf3ff0d6/
Log: coalesced the 3 existing contributing docs into a single doc diff --git a/pypy/doc/contributing.rst b/pypy/doc/contributing.rst new file mode 100644 --- /dev/null +++ b/pypy/doc/contributing.rst @@ -0,0 +1,490 @@ +Contributing Guidelines +=========================== + +.. contents:: + +PyPy is a very large project that has a reputation of being hard to dive into. +Some of this fame is warranted, some of it is purely accidental. There are three +important lessons that everyone willing to contribute should learn: + +* PyPy has layers. There are many pieces of architecture that are very well + separated from each other. More about this below, but often the manifestation + of this is that things are at a different layer than you would expect them + to be. For example if you are looking for the JIT implementation, you will + not find it in the implementation of the Python programming language. + +* Because of the above, we are very serious about Test Driven Development. + It's not only what we believe in, but also that PyPy's architecture is + working very well with TDD in mind and not so well without it. Often + development means progressing in an unrelated corner, one unittest + at a time; and then flipping a giant switch, bringing it all together. + (It generally works out of the box. If it doesn't, then we didn't + write enough unit tests.) It's worth repeating - PyPy's + approach is great if you do TDD, and not so great otherwise. + +* PyPy uses an entirely different set of tools - most of them included + in the PyPy repository. There is no Makefile, nor autoconf. More below. + +The first thing to remember is that PyPy project is very different than most projects out there. +It's also different from a classic compiler project, so academic courses +about compilers often don't apply or lead in the wrong direction. + +Getting involved +---------------- + +PyPy employs an open development process. You are invited to join our +`pypy-dev mailing list`_, our IRC channel (#pypy on freenode) or look at the +other :ref:`contact possibilities <contact>`. Usually we give out commit +rights fairly liberally, so if you want to do something with PyPy, you can +become a committer. We also run frequent coding sprints which are separately +announced and often happen around Python conferences such as EuroPython or +PyCon. Upcoming events are usually announced on `the blog`_. + +.. _the blog: http://morepypy.blogspot.com +.. _pypy-dev mailing list: http://mail.python.org/mailman/listinfo/pypy-dev + + +Hacking +-------- + +The first and most important rule how _not_ to contribute to PyPy is +"just hacking". This won't work. There are two major reasons why not +-- build times are large and PyPy has very thick layer separation which +make it harder to "just hack a feature". Instead, reach out on the dev mailing +list or the IRC channel, and we're more than happy to help! :) + +Source Control +============== + +Using Mercurial +--------------- + +PyPy development is based on Mercurial (hg). If you are not used to +version control, the cycle for a new PyPy contributor goes typically +like this: + +* Make an account on bitbucket_. + +* Go to https://bitbucket.org/pypy/pypy/ and click "fork" (left + icons). You get a fork of the repository, e.g. in + https://bitbucket.org/yourname/pypy/. + +* Clone this new repo (i.e. the fork) to your local machine with the command + ``hg clone ssh://[email protected]/yourname/pypy``. It is a very slow + operation but only ever needs to be done once. See also + http://pypy.org/download.html#building-from-source . + If you already cloned + ``https://bitbucket.org/pypy/pypy`` before, even if some time ago, + then you can reuse the same clone by editing the file ``.hg/hgrc`` in + your clone to contain the line ``default = + ssh://[email protected]/yourname/pypy``, and then do ``hg pull && hg + up``. If you already have such a clone but don't want to change it, + you can clone that copy with ``hg clone /path/to/other/copy``, and + then edit ``.hg/hgrc`` as above and do ``hg pull && hg up``. + +* Now you have a complete copy of the PyPy repo. Make a branch + with a command like ``hg branch name_of_your_branch``. + +* Edit things. Use ``hg diff`` to see what you changed. Use ``hg add`` + to make Mercurial aware of new files you added, e.g. new test files. + Use ``hg status`` to see if there are such files. Write and run tests! + (See the rest of this page.) + +* Commit regularly with ``hg commit``. A one-line commit message is + fine. We love to have tons of commits; make one as soon as you have + some progress, even if it is only some new test that doesn't pass yet, + or fixing things even if not all tests pass. Step by step, you are + building the history of your changes, which is the point of a version + control system. (There are commands like ``hg log`` and ``hg up`` + that you should read about later, to learn how to navigate this + history.) + +* The commits stay on your machine until you do ``hg push`` to "push" + them back to the repo named in the file ``.hg/hgrc``. Repos are + basically just collections of commits (a commit is also called a + changeset): there is one repo per url, plus one for each local copy on + each local machine. The commands ``hg push`` and ``hg pull`` copy + commits around, with the goal that all repos in question end up with + the exact same set of commits. By opposition, ``hg up`` only updates + the "working copy" by reading the local repository, i.e. it makes the + files that you see correspond to the latest (or any other) commit + locally present. + +* You should push often; there is no real reason not to. Remember that + even if they are pushed, with the setup above, the commits are (1) + only in ``bitbucket.org/yourname/pypy``, and (2) in the branch you + named. Yes, they are publicly visible, but don't worry about someone + walking around the thousands of repos on bitbucket saying "hah, look + at the bad coding style of that guy". Try to get into the mindset + that your work is not secret and it's fine that way. We might not + accept it as is for PyPy, asking you instead to improve some things, + but we are not going to judge you. + +* The final step is to open a pull request, so that we know that you'd + like to merge that branch back to the original ``pypy/pypy`` repo. + This can also be done several times if you have interesting + intermediate states, but if you get there, then we're likely to + proceed to the next stage, which is... + +* Get a regular account for pushing directly to + ``bitbucket.org/pypy/pypy`` (just ask and you'll get it, basically). + Once you have it you can rewrite your file ``.hg/hgrc`` to contain + ``default = ssh://[email protected]/pypy/pypy``. Your changes will + then be pushed directly to the official repo, but (if you follow these + rules) they are still on a branch, and we can still review the + branches you want to merge. + +* If you get closer to the regular day-to-day development, you'll notice + that we generally push small changes as one or a few commits directly + to the branch ``default``. Also, we often collaborate even if we are + on other branches, which do not really "belong" to anyone. At this + point you'll need ``hg merge`` and learn how to resolve conflicts that + sometimes occur when two people try to push different commits in + parallel on the same branch. But it is likely an issue for later ``:-)`` + +.. _bitbucket: https://bitbucket.org/ + + +Architecture +============ + +PyPy has layers. The 100 miles view: + +* :ref:`RPython <rpython:language>` is the language in which we write interpreters. Not the entire + PyPy project is written in RPython, only the parts that are compiled in + the translation process. The interesting point is that RPython has no parser, + it's compiled from the live python objects, which makes it possible to do + all kinds of metaprogramming during import time. In short, Python is a meta + programming language for RPython. + + The RPython standard library is to be found in the ``rlib`` subdirectory. + +* The translation toolchain - this is the part that takes care of translating + RPython to flow graphs and then to C. There is more in the :doc:`architecture <architecture>` + document written about it. + + It lives in the ``rpython`` directory: ``flowspace``, ``annotator`` + and ``rtyper``. + +* Python Interpreter and modules + + This is in the ``pypy`` directory. ``pypy/interpreter`` is a standard + interpreter for Python written in RPython. The fact that it is + RPython is not apparent at first. Built-in modules are written in + ``pypy/module/*``. Some modules that CPython implements in C are + simply written in pure Python; they are in the top-level ``lib_pypy`` + directory. The standard library of Python (with a few changes to + accomodate PyPy) is in ``lib-python``. + +* :ref:`Just-in-Time Compiler (JIT) <rpython:jit>`: we have a tracing JIT that traces the + interpreter written in RPython, rather than the user program that it + interprets. As a result it applies to any interpreter, i.e. any + language. But getting it to work correctly is not trivial: it + requires a small number of precise "hints" and possibly some small + refactorings of the interpreter. The JIT itself also has several + almost-independent parts: the tracer itself in ``rpython/jit/metainterp``, the + optimizer in ``rpython/jit/metainterp/optimizer`` that optimizes a list of + residual operations, and the backend in ``rpython/jit/backend/<machine-name>`` + that turns it into machine code. Writing a new backend is a + traditional way to get into the project. + +* Garbage Collectors (GC): as you may notice if you are used to CPython's + C code, there are no ``Py_INCREF/Py_DECREF`` equivalents in RPython code. + :ref:`rpython:garbage-collection` is inserted + during translation. Moreover, this is not reference counting; it is a real + GC written as more RPython code. The best one we have so far is in + ``rpython/memory/gc/incminimark.py``. + +Layers +------ + +PyPy has layers. Just like Ogres or onions. +Those layers help us keep the respective parts separated enough +to be worked on independently and make the complexity manageable. This is, +again, just a sanity requirement for such a complex project. For example writing +a new optimization for the JIT usually does **not** involve touching a Python +interpreter at all or the JIT assembler backend or the garbage collector. +Instead it requires writing small tests in +``rpython/jit/metainterp/optimizeopt/test/test_*`` and fixing files there. +After that, you can just compile PyPy and things should just work. + +The short list of layers for further reading. For each of those layers, a good +entry point is a test subdirectory in respective directories. It usually +describes (better or worse) the interfaces between the submodules. For the +``pypy`` subdirectory, most tests are small snippets of python programs that +check for correctness (calls ``AppTestXxx``) that will call the appropriate +part of the interpreter. For the ``rpython`` directory, most tests are small +RPython interpreters that perform certain tasks. To see how they translate +to low-level graphs, run them with ``--view``. To see small interpreters +with a JIT compiler, use ``--viewloops`` option. + +* **python interpreter** - it's the part implemented in the ``pypy/`` directory. + It's implemented in RPython, which is a high level static language with + classes, garbage collection, just-in-time compiler generation and the ability + to call C. A cool part about it is that it can be run untranslated, so all + the tests are runnable without translating PyPy. + + **interpreter** contains the interpreter core + + **objspace** contains implementations of various objects exported to + the Python layer + + **module** directory contains extension modules written in RPython + +* **rpython compiler** that resides in ``rpython/annotator`` and + ``rpython/rtyper`` directories. Consult `Getting Started with RPython`_ + for further reading + +* **JIT generator** lives in ``rpython/jit`` directory. optimizations live + in ``rpython/jit/metainterp/optimizeopt``, the main JIT in + ``rpython/jit/metainterp`` (runtime part) and + ``rpython/jit/codewriter`` (translation-time part). Backends live in + ``rpython/jit/backend``. + +* **garbage collection** lives in ``rpython/memory`` + +The rest of directories serve specific niche goal and are unlikely a good +entry point. + +.. _`Getting started with RPython`: http://rpython.readthedocs.org/en/latest/getting-started.html + +Where to start reading the sources +---------------------------------- + +PyPy is made from parts that are relatively independent of each other. +You should start looking at the part that attracts you most (all paths are +relative to the PyPy top level directory). You may look at our :doc:`directory reference <dir-reference>` +or start off at one of the following points: + +* :source:`pypy/interpreter` contains the bytecode interpreter: bytecode dispatcher + in :source:`pypy/interpreter/pyopcode.py`, frame and code objects in + :source:`pypy/interpreter/eval.py` and :source:`pypy/interpreter/pyframe.py`, + function objects and argument passing in :source:`pypy/interpreter/function.py` + and :source:`pypy/interpreter/argument.py`, the object space interface + definition in :source:`pypy/interpreter/baseobjspace.py`, modules in + :source:`pypy/interpreter/module.py` and :source:`pypy/interpreter/mixedmodule.py`. + Core types supporting the bytecode interpreter are defined in :source:`pypy/interpreter/typedef.py`. + +* :source:`pypy/interpreter/pyparser` contains a recursive descent parser, + and grammar files that allow it to parse the syntax of various Python + versions. Once the grammar has been processed, the parser can be + translated by the above machinery into efficient code. + +* :source:`pypy/interpreter/astcompiler` contains the compiler. This + contains a modified version of the compiler package from CPython + that fixes some bugs and is translatable. + +* :source:`pypy/objspace/std` contains the :ref:`Standard object space <standard-object-space>`. The main file + is :source:`pypy/objspace/std/objspace.py`. For each type, the file + ``xxxobject.py`` contains the implementation for objects of type ``xxx``, + as a first approximation. (Some types have multiple implementations.) + +Testing +======= + +Test driven development +----------------------- + +Instead, we practice a lot of test driven development. This is partly because +of very high quality requirements for compilers and partly because there is +simply no other way to get around such complex project, that will keep you sane. +There are probably people out there who are smart enough not to need it, we're +not one of those. You may consider familiarizing yourself with `pytest`_, +since this is a tool we use for tests. +This leads to the next issue: + +.. _pytest: http://pytest.org/ + +py.test and the py lib +---------------------- + +The `py.test testing tool`_ drives all our testing needs. + +We use the `py library`_ for filesystem path manipulations, terminal +writing, logging and some other support functionality. + +You don't necessarily need to install these two libraries because +we also ship them inlined in the PyPy source tree. + +.. _py library: http://pylib.readthedocs.org/ + +Running PyPy's unit tests +------------------------- + +PyPy development always was and is still thoroughly test-driven. +We use the flexible `py.test testing tool`_ which you can `install independently +<http://pytest.org/latest/getting-started.html#getstarted>`_ and use for other projects. + +The PyPy source tree comes with an inlined version of ``py.test`` +which you can invoke by typing:: + + python pytest.py -h + +This is usually equivalent to using an installed version:: + + py.test -h + +If you encounter problems with the installed version +make sure you have the correct version installed which +you can find out with the ``--version`` switch. + +You will need the `build requirements`_ to run tests successfully, since many of +them compile little pieces of PyPy and then run the tests inside that minimal +interpreter + +Now on to running some tests. PyPy has many different test directories +and you can use shell completion to point at directories or files:: + + py.test pypy/interpreter/test/test_pyframe.py + + # or for running tests of a whole subdirectory + py.test pypy/interpreter/ + +See `py.test usage and invocations`_ for some more generic info +on how you can run tests. + +Beware trying to run "all" pypy tests by pointing to the root +directory or even the top level subdirectory ``pypy``. It takes +hours and uses huge amounts of RAM and is not recommended. + +To run CPython regression tests you can point to the ``lib-python`` +directory:: + + py.test lib-python/2.7/test/test_datetime.py + +This will usually take a long time because this will run +the PyPy Python interpreter on top of CPython. On the plus +side, it's usually still faster than doing a full translation +and running the regression test with the translated PyPy Python +interpreter. + +.. _py.test testing tool: http://pytest.org +.. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage +.. _`build requirements`: build.html#install-build-time-dependencies + +Tooling & Utilities +=================== + +If you are interested in the inner workings of the PyPy Python interpreter, +there are some features of the untranslated Python interpreter that allow you +to introspect its internals. + + +Interpreter-level console +------------------------- + +To start interpreting Python with PyPy, install a C compiler that is +supported by distutils and use Python 2.7 or greater to run PyPy:: + + cd pypy + python bin/pyinteractive.py + +After a few seconds (remember: this is running on top of CPython), you should +be at the PyPy prompt, which is the same as the Python prompt, but with an +extra ">". + +If you press +<Ctrl-C> on the console you enter the interpreter-level console, a +usual CPython console. You can then access internal objects of PyPy +(e.g. the :ref:`object space <objspace>`) and any variables you have created on the PyPy +prompt with the prefix ``w_``:: + + >>>> a = 123 + >>>> <Ctrl-C> + *** Entering interpreter-level console *** + >>> w_a + W_IntObject(123) + +The mechanism works in both directions. If you define a variable with the ``w_`` prefix on the interpreter-level, you will see it on the app-level:: + + >>> w_l = space.newlist([space.wrap(1), space.wrap("abc")]) + >>> <Ctrl-D> + *** Leaving interpreter-level console *** + + KeyboardInterrupt + >>>> l + [1, 'abc'] + +Note that the prompt of the interpreter-level console is only '>>>' since +it runs on CPython level. If you want to return to PyPy, press <Ctrl-D> (under +Linux) or <Ctrl-Z>, <Enter> (under Windows). + +Also note that not all modules are available by default in this mode (for +example: ``_continuation`` needed by ``greenlet``) , you may need to use one of +``--withmod-...`` command line options. + +You may be interested in reading more about the distinction between +:ref:`interpreter-level and app-level <interpreter-level>`. + +pyinteractive.py options +------------------------ + +To list the PyPy interpreter command line options, type:: + + cd pypy + python bin/pyinteractive.py --help + +pyinteractive.py supports most of the options that CPython supports too (in addition to a +large amount of options that can be used to customize pyinteractive.py). +As an example of using PyPy from the command line, you could type:: + + python pyinteractive.py --withmod-time -c "from test import pystone; pystone.main(10)" + +Alternatively, as with regular Python, you can simply give a +script name on the command line:: + + python pyinteractive.py --withmod-time ../../lib-python/2.7/test/pystone.py 10 + +The ``--withmod-xxx`` option enables the built-in module ``xxx``. By +default almost none of them are, because initializing them takes time. +If you want anyway to enable all built-in modules, you can use +``--allworkingmodules``. + +See our :doc:`configuration sections <config/index>` for details about what all the commandline +options do. + + +.. _trace example: + +Tracing bytecode and operations on objects +------------------------------------------ + +You can use a simple tracing mode to monitor the interpretation of +bytecodes. To enable it, set ``__pytrace__ = 1`` on the interactive +PyPy console:: + + >>>> __pytrace__ = 1 + Tracing enabled + >>>> x = 5 + <module>: LOAD_CONST 0 (5) + <module>: STORE_NAME 0 (x) + <module>: LOAD_CONST 1 (None) + <module>: RETURN_VALUE 0 + >>>> x + <module>: LOAD_NAME 0 (x) + <module>: PRINT_EXPR 0 + 5 + <module>: LOAD_CONST 0 (None) + <module>: RETURN_VALUE 0 + >>>> + + +Demos +----- + +The `example-interpreter`_ repository contains an example interpreter +written using the RPython translation toolchain. + +.. _example-interpreter: https://bitbucket.org/pypy/example-interpreter + + +graphviz & pygame for flow graph viewing (highly recommended) +------------------------------------------------------------- + +graphviz and pygame are both necessary if you want to look at generated flow +graphs: + + graphviz: http://www.graphviz.org/Download.php + + pygame: http://www.pygame.org/download.shtml + diff --git a/pypy/doc/getting-started-dev.rst b/pypy/doc/getting-started-dev.rst deleted file mode 100644 --- a/pypy/doc/getting-started-dev.rst +++ /dev/null @@ -1,345 +0,0 @@ -Getting Started Developing With PyPy -==================================== - -.. contents:: - - -Using Mercurial ---------------- - -PyPy development is based on Mercurial (hg). If you are not used to -version control, the cycle for a new PyPy contributor goes typically -like this: - -* Make an account on bitbucket_. - -* Go to https://bitbucket.org/pypy/pypy/ and click "fork" (left - icons). You get a fork of the repository, e.g. in - https://bitbucket.org/yourname/pypy/. - -* Clone this new repo (i.e. the fork) to your local machine with the command - ``hg clone ssh://[email protected]/yourname/pypy``. It is a very slow - operation but only ever needs to be done once. See also - http://pypy.org/download.html#building-from-source . - If you already cloned - ``https://bitbucket.org/pypy/pypy`` before, even if some time ago, - then you can reuse the same clone by editing the file ``.hg/hgrc`` in - your clone to contain the line ``default = - ssh://[email protected]/yourname/pypy``, and then do ``hg pull && hg - up``. If you already have such a clone but don't want to change it, - you can clone that copy with ``hg clone /path/to/other/copy``, and - then edit ``.hg/hgrc`` as above and do ``hg pull && hg up``. - -* Now you have a complete copy of the PyPy repo. Make a branch - with a command like ``hg branch name_of_your_branch``. - -* Edit things. Use ``hg diff`` to see what you changed. Use ``hg add`` - to make Mercurial aware of new files you added, e.g. new test files. - Use ``hg status`` to see if there are such files. Write and run tests! - (See the rest of this page.) - -* Commit regularly with ``hg commit``. A one-line commit message is - fine. We love to have tons of commits; make one as soon as you have - some progress, even if it is only some new test that doesn't pass yet, - or fixing things even if not all tests pass. Step by step, you are - building the history of your changes, which is the point of a version - control system. (There are commands like ``hg log`` and ``hg up`` - that you should read about later, to learn how to navigate this - history.) - -* The commits stay on your machine until you do ``hg push`` to "push" - them back to the repo named in the file ``.hg/hgrc``. Repos are - basically just collections of commits (a commit is also called a - changeset): there is one repo per url, plus one for each local copy on - each local machine. The commands ``hg push`` and ``hg pull`` copy - commits around, with the goal that all repos in question end up with - the exact same set of commits. By opposition, ``hg up`` only updates - the "working copy" by reading the local repository, i.e. it makes the - files that you see correspond to the latest (or any other) commit - locally present. - -* You should push often; there is no real reason not to. Remember that - even if they are pushed, with the setup above, the commits are (1) - only in ``bitbucket.org/yourname/pypy``, and (2) in the branch you - named. Yes, they are publicly visible, but don't worry about someone - walking around the thousands of repos on bitbucket saying "hah, look - at the bad coding style of that guy". Try to get into the mindset - that your work is not secret and it's fine that way. We might not - accept it as is for PyPy, asking you instead to improve some things, - but we are not going to judge you. - -* The final step is to open a pull request, so that we know that you'd - like to merge that branch back to the original ``pypy/pypy`` repo. - This can also be done several times if you have interesting - intermediate states, but if you get there, then we're likely to - proceed to the next stage, which is... - -* Get a regular account for pushing directly to - ``bitbucket.org/pypy/pypy`` (just ask and you'll get it, basically). - Once you have it you can rewrite your file ``.hg/hgrc`` to contain - ``default = ssh://[email protected]/pypy/pypy``. Your changes will - then be pushed directly to the official repo, but (if you follow these - rules) they are still on a branch, and we can still review the - branches you want to merge. - -* If you get closer to the regular day-to-day development, you'll notice - that we generally push small changes as one or a few commits directly - to the branch ``default``. Also, we often collaborate even if we are - on other branches, which do not really "belong" to anyone. At this - point you'll need ``hg merge`` and learn how to resolve conflicts that - sometimes occur when two people try to push different commits in - parallel on the same branch. But it is likely an issue for later ``:-)`` - -.. _bitbucket: https://bitbucket.org/ - - -Running PyPy's unit tests -------------------------- - -PyPy development always was and is still thoroughly test-driven. -We use the flexible `py.test testing tool`_ which you can `install independently -<http://pytest.org/latest/getting-started.html#getstarted>`_ and use for other projects. - -The PyPy source tree comes with an inlined version of ``py.test`` -which you can invoke by typing:: - - python pytest.py -h - -This is usually equivalent to using an installed version:: - - py.test -h - -If you encounter problems with the installed version -make sure you have the correct version installed which -you can find out with the ``--version`` switch. - -You will need the `build requirements`_ to run tests successfully, since many of -them compile little pieces of PyPy and then run the tests inside that minimal -interpreter - -Now on to running some tests. PyPy has many different test directories -and you can use shell completion to point at directories or files:: - - py.test pypy/interpreter/test/test_pyframe.py - - # or for running tests of a whole subdirectory - py.test pypy/interpreter/ - -See `py.test usage and invocations`_ for some more generic info -on how you can run tests. - -Beware trying to run "all" pypy tests by pointing to the root -directory or even the top level subdirectory ``pypy``. It takes -hours and uses huge amounts of RAM and is not recommended. - -To run CPython regression tests you can point to the ``lib-python`` -directory:: - - py.test lib-python/2.7/test/test_datetime.py - -This will usually take a long time because this will run -the PyPy Python interpreter on top of CPython. On the plus -side, it's usually still faster than doing a full translation -and running the regression test with the translated PyPy Python -interpreter. - -.. _py.test testing tool: http://pytest.org -.. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage -.. _`build requirements`: build.html#install-build-time-dependencies - -Special Introspection Features of the Untranslated Python Interpreter ---------------------------------------------------------------------- - -If you are interested in the inner workings of the PyPy Python interpreter, -there are some features of the untranslated Python interpreter that allow you -to introspect its internals. - - -Interpreter-level console -~~~~~~~~~~~~~~~~~~~~~~~~~ - -To start interpreting Python with PyPy, install a C compiler that is -supported by distutils and use Python 2.7 or greater to run PyPy:: - - cd pypy - python bin/pyinteractive.py - -After a few seconds (remember: this is running on top of CPython), you should -be at the PyPy prompt, which is the same as the Python prompt, but with an -extra ">". - -If you press -<Ctrl-C> on the console you enter the interpreter-level console, a -usual CPython console. You can then access internal objects of PyPy -(e.g. the :ref:`object space <objspace>`) and any variables you have created on the PyPy -prompt with the prefix ``w_``:: - - >>>> a = 123 - >>>> <Ctrl-C> - *** Entering interpreter-level console *** - >>> w_a - W_IntObject(123) - -The mechanism works in both directions. If you define a variable with the ``w_`` prefix on the interpreter-level, you will see it on the app-level:: - - >>> w_l = space.newlist([space.wrap(1), space.wrap("abc")]) - >>> <Ctrl-D> - *** Leaving interpreter-level console *** - - KeyboardInterrupt - >>>> l - [1, 'abc'] - -Note that the prompt of the interpreter-level console is only '>>>' since -it runs on CPython level. If you want to return to PyPy, press <Ctrl-D> (under -Linux) or <Ctrl-Z>, <Enter> (under Windows). - -Also note that not all modules are available by default in this mode (for -example: ``_continuation`` needed by ``greenlet``) , you may need to use one of -``--withmod-...`` command line options. - -You may be interested in reading more about the distinction between -:ref:`interpreter-level and app-level <interpreter-level>`. - -pyinteractive.py options -~~~~~~~~~~~~~~~~~~~~~~~~ - -To list the PyPy interpreter command line options, type:: - - cd pypy - python bin/pyinteractive.py --help - -pyinteractive.py supports most of the options that CPython supports too (in addition to a -large amount of options that can be used to customize pyinteractive.py). -As an example of using PyPy from the command line, you could type:: - - python pyinteractive.py --withmod-time -c "from test import pystone; pystone.main(10)" - -Alternatively, as with regular Python, you can simply give a -script name on the command line:: - - python pyinteractive.py --withmod-time ../../lib-python/2.7/test/pystone.py 10 - -The ``--withmod-xxx`` option enables the built-in module ``xxx``. By -default almost none of them are, because initializing them takes time. -If you want anyway to enable all built-in modules, you can use -``--allworkingmodules``. - -See our :doc:`configuration sections <config/index>` for details about what all the commandline -options do. - - -.. _trace example: - -Tracing bytecode and operations on objects -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can use a simple tracing mode to monitor the interpretation of -bytecodes. To enable it, set ``__pytrace__ = 1`` on the interactive -PyPy console:: - - >>>> __pytrace__ = 1 - Tracing enabled - >>>> x = 5 - <module>: LOAD_CONST 0 (5) - <module>: STORE_NAME 0 (x) - <module>: LOAD_CONST 1 (None) - <module>: RETURN_VALUE 0 - >>>> x - <module>: LOAD_NAME 0 (x) - <module>: PRINT_EXPR 0 - 5 - <module>: LOAD_CONST 0 (None) - <module>: RETURN_VALUE 0 - >>>> - - -Demos ------ - -The `example-interpreter`_ repository contains an example interpreter -written using the RPython translation toolchain. - -.. _example-interpreter: https://bitbucket.org/pypy/example-interpreter - - -Additional Tools for running (and hacking) PyPy ------------------------------------------------ - -We use some optional tools for developing PyPy. They are not required to run -the basic tests or to get an interactive PyPy prompt but they help to -understand and debug PyPy especially for the translation process. - - -graphviz & pygame for flow graph viewing (highly recommended) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -graphviz and pygame are both necessary if you -want to look at generated flow graphs: - - graphviz: http://www.graphviz.org/Download.php - - pygame: http://www.pygame.org/download.shtml - - -py.test and the py lib -~~~~~~~~~~~~~~~~~~~~~~ - -The `py.test testing tool`_ drives all our testing needs. - -We use the `py library`_ for filesystem path manipulations, terminal -writing, logging and some other support functionality. - -You don't necessarily need to install these two libraries because -we also ship them inlined in the PyPy source tree. - -.. _py library: http://pylib.readthedocs.org/ - - -Getting involved ----------------- - -PyPy employs an open development process. You are invited to join our -`pypy-dev mailing list`_ or look at the other :ref:`contact -possibilities <contact>`. Usually we give out commit rights fairly liberally, so if you -want to do something with PyPy, you can become a committer. We also run frequent -coding sprints which are separately announced and often happen around Python -conferences such as EuroPython or PyCon. Upcoming events are usually announced -on `the blog`_. - -.. _the blog: http://morepypy.blogspot.com -.. _pypy-dev mailing list: http://mail.python.org/mailman/listinfo/pypy-dev - - -.. _start-reading-sources: - -Where to start reading the sources ----------------------------------- - -PyPy is made from parts that are relatively independent of each other. -You should start looking at the part that attracts you most (all paths are -relative to the PyPy top level directory). You may look at our :doc:`directory reference <dir-reference>` -or start off at one of the following points: - -* :source:`pypy/interpreter` contains the bytecode interpreter: bytecode dispatcher - in :source:`pypy/interpreter/pyopcode.py`, frame and code objects in - :source:`pypy/interpreter/eval.py` and :source:`pypy/interpreter/pyframe.py`, - function objects and argument passing in :source:`pypy/interpreter/function.py` - and :source:`pypy/interpreter/argument.py`, the object space interface - definition in :source:`pypy/interpreter/baseobjspace.py`, modules in - :source:`pypy/interpreter/module.py` and :source:`pypy/interpreter/mixedmodule.py`. - Core types supporting the bytecode interpreter are defined in :source:`pypy/interpreter/typedef.py`. - -* :source:`pypy/interpreter/pyparser` contains a recursive descent parser, - and grammar files that allow it to parse the syntax of various Python - versions. Once the grammar has been processed, the parser can be - translated by the above machinery into efficient code. - -* :source:`pypy/interpreter/astcompiler` contains the compiler. This - contains a modified version of the compiler package from CPython - that fixes some bugs and is translatable. - -* :source:`pypy/objspace/std` contains the :ref:`Standard object space <standard-object-space>`. The main file - is :source:`pypy/objspace/std/objspace.py`. For each type, the file - ``xxxobject.py`` contains the implementation for objects of type ``xxx``, - as a first approximation. (Some types have multiple implementations.) diff --git a/pypy/doc/how-to-contribute.rst b/pypy/doc/how-to-contribute.rst deleted file mode 100644 --- a/pypy/doc/how-to-contribute.rst +++ /dev/null @@ -1,93 +0,0 @@ -How to contribute to PyPy -========================= - -This page describes how to contribute to the PyPy project. The first thing -to remember is that PyPy project is very different than most projects out there. -It's also different from a classic compiler project, so academic courses -about compilers often don't apply or lead in the wrong direction. - - -Don't just hack ---------------- - -The first and most important rule how not to contribute to PyPy is -"just hacking". This won't work. There are two major reasons why not --- build times are large and PyPy has very thick layer separation which -make it harder to "just hack a feature". - - -Test driven development ------------------------ - -Instead, we practice a lot of test driven development. This is partly because -of very high quality requirements for compilers and partly because there is -simply no other way to get around such complex project, that will keep you sane. -There are probably people out there who are smart enough not to need it, we're -not one of those. You may consider familiarizing yourself with `pytest`_, -since this is a tool we use for tests. -This leads to the next issue: - -.. _pytest: http://pytest.org/ - - -Layers ------- - -PyPy has layers. Just like Ogres or onions. -Those layers help us keep the respective parts separated enough -to be worked on independently and make the complexity manageable. This is, -again, just a sanity requirement for such a complex project. For example writing -a new optimization for the JIT usually does **not** involve touching a Python -interpreter at all or the JIT assembler backend or the garbage collector. -Instead it requires writing small tests in -``rpython/jit/metainterp/optimizeopt/test/test_*`` and fixing files there. -After that, you can just compile PyPy and things should just work. - -The short list of layers for further reading. For each of those layers, a good -entry point is a test subdirectory in respective directories. It usually -describes (better or worse) the interfaces between the submodules. For the -``pypy`` subdirectory, most tests are small snippets of python programs that -check for correctness (calls ``AppTestXxx``) that will call the appropriate -part of the interpreter. For the ``rpython`` directory, most tests are small -RPython interpreters that perform certain tasks. To see how they translate -to low-level graphs, run them with ``--view``. To see small interpreters -with a JIT compiler, use ``--viewloops`` option. - -* **python interpreter** - it's the part implemented in the ``pypy/`` directory. - It's implemented in RPython, which is a high level static language with - classes, garbage collection, just-in-time compiler generation and the ability - to call C. A cool part about it is that it can be run untranslated, so all - the tests are runnable without translating PyPy. - - **interpreter** contains the interpreter core - - **objspace** contains implementations of various objects exported to - the Python layer - - **module** directory contains extension modules written in RPython - -* **rpython compiler** that resides in ``rpython/annotator`` and - ``rpython/rtyper`` directories. Consult `Getting Started with RPython`_ - for further reading - -* **JIT generator** lives in ``rpython/jit`` directory. optimizations live - in ``rpython/jit/metainterp/optimizeopt``, the main JIT in - ``rpython/jit/metainterp`` (runtime part) and - ``rpython/jit/codewriter`` (translation-time part). Backends live in - ``rpython/jit/backend``. - -* **garbage collection** lives in ``rpython/memory`` - -The rest of directories serve specific niche goal and are unlikely a good -entry point. - - -More documentation ------------------- - -* `Getting Started Developing With PyPy`_ - -* `Getting Started with RPython`_ - -.. _`Getting Started Developing With PyPy`: getting-started-dev.html -.. _`Getting started with RPython`: http://rpython.readthedocs.org/en/latest/getting-started.html diff --git a/pypy/doc/index.rst b/pypy/doc/index.rst --- a/pypy/doc/index.rst +++ b/pypy/doc/index.rst @@ -9,7 +9,7 @@ * If you're interested in trying PyPy out, check out the :doc:`installation instructions <install>`. -* If you want to help develop PyPy, please have a look at :doc:`how to contribute <how-to-contribute>` +* If you want to help develop PyPy, please have a look at :doc:`contributing <contributing>` and get in touch (:ref:`contact`)! All of the documentation and source code is available under the MIT license, @@ -63,9 +63,7 @@ .. toctree:: :maxdepth: 1 - getting-started-dev - how-to-contribute - you-want-to-help + contributing architecture configuration project-ideas diff --git a/pypy/doc/you-want-to-help.rst b/pypy/doc/you-want-to-help.rst deleted file mode 100644 --- a/pypy/doc/you-want-to-help.rst +++ /dev/null @@ -1,81 +0,0 @@ -You want to help with PyPy, now what? -===================================== - -PyPy is a very large project that has a reputation of being hard to dive into. -Some of this fame is warranted, some of it is purely accidental. There are three -important lessons that everyone willing to contribute should learn: - -* PyPy has layers. There are many pieces of architecture that are very well - separated from each other. More about this below, but often the manifestation - of this is that things are at a different layer than you would expect them - to be. For example if you are looking for the JIT implementation, you will - not find it in the implementation of the Python programming language. - -* Because of the above, we are very serious about Test Driven Development. - It's not only what we believe in, but also that PyPy's architecture is - working very well with TDD in mind and not so well without it. Often - development means progressing in an unrelated corner, one unittest - at a time; and then flipping a giant switch, bringing it all together. - (It generally works out of the box. If it doesn't, then we didn't - write enough unit tests.) It's worth repeating - PyPy's - approach is great if you do TDD, and not so great otherwise. - -* PyPy uses an entirely different set of tools - most of them included - in the PyPy repository. There is no Makefile, nor autoconf. More below. - - -Architecture ------------- - -PyPy has layers. The 100 miles view: - -* :ref:`RPython <rpython:language>` is the language in which we write interpreters. Not the entire - PyPy project is written in RPython, only the parts that are compiled in - the translation process. The interesting point is that RPython has no parser, - it's compiled from the live python objects, which makes it possible to do - all kinds of metaprogramming during import time. In short, Python is a meta - programming language for RPython. - - The RPython standard library is to be found in the ``rlib`` subdirectory. - -* The translation toolchain - this is the part that takes care of translating - RPython to flow graphs and then to C. There is more in the :doc:`architecture <architecture>` - document written about it. - - It lives in the ``rpython`` directory: ``flowspace``, ``annotator`` - and ``rtyper``. - -* Python Interpreter and modules - - This is in the ``pypy`` directory. ``pypy/interpreter`` is a standard - interpreter for Python written in RPython. The fact that it is - RPython is not apparent at first. Built-in modules are written in - ``pypy/module/*``. Some modules that CPython implements in C are - simply written in pure Python; they are in the top-level ``lib_pypy`` - directory. The standard library of Python (with a few changes to - accomodate PyPy) is in ``lib-python``. - -* :ref:`Just-in-Time Compiler (JIT) <rpython:jit>`: we have a tracing JIT that traces the - interpreter written in RPython, rather than the user program that it - interprets. As a result it applies to any interpreter, i.e. any - language. But getting it to work correctly is not trivial: it - requires a small number of precise "hints" and possibly some small - refactorings of the interpreter. The JIT itself also has several - almost-independent parts: the tracer itself in ``rpython/jit/metainterp``, the - optimizer in ``rpython/jit/metainterp/optimizer`` that optimizes a list of - residual operations, and the backend in ``rpython/jit/backend/<machine-name>`` - that turns it into machine code. Writing a new backend is a - traditional way to get into the project. - -* Garbage Collectors (GC): as you may notice if you are used to CPython's - C code, there are no ``Py_INCREF/Py_DECREF`` equivalents in RPython code. - :ref:`rpython:garbage-collection` is inserted - during translation. Moreover, this is not reference counting; it is a real - GC written as more RPython code. The best one we have so far is in - ``rpython/memory/gc/incminimark.py``. - - -Toolset -------- - -xxx _______________________________________________ pypy-commit mailing list [email protected] https://mail.python.org/mailman/listinfo/pypy-commit
