Author: Armin Rigo <ar...@tunes.org>
Branch: extradoc
Changeset: r5591:d5d6783367e1
Date: 2016-01-06 11:18 +0100
http://bitbucket.org/pypy/extradoc/changeset/d5d6783367e1/
Log: in-progress
diff --git a/blog/draft/cffi-embedding.rst b/blog/draft/cffi-embedding.rst
--- a/blog/draft/cffi-embedding.rst
+++ b/blog/draft/cffi-embedding.rst
@@ -6,44 +6,55 @@
Python programs, in a way that is both simple and that works across
CPython 2.x and 3.x and PyPy.
-We are now adding support for *embedding* Python inside non-Python
-programs. This is traditionally done using the CPython C API: from C
-code, you call ``Py_Initialize()`` and then some other functions like
+The major news of CFFI 1.4, released last december, was that you can
+now declare C functions with ``extern "Python"``, in the ``cdef()``.
+These magic keywords make the function callable from C (where it is
+defined automatically), but calling it will call some Python code
+(which you attach with the ``@ffi.def_extern()`` decorator). This is
+useful because it gives a more straightforward, faster and
+libffi-independent way to write callbacks. For more details, see `the
+documentation`_.
+
+You are, in effect, declaring a static family of C functions which
+call Python code. The idea is to take pointers to them, and pass them
+around to other C functions, as callbacks. However, the idea of a set
+of C functions which call Python code opens another path: *embedding*
+Python code inside non-Python programs.
+
+Embedding is traditionally done using the CPython C API: from C code,
+you call ``Py_Initialize()`` and then some other functions like
``PyRun_SimpleString()``. In the simple cases it is, indeed, simple
-enough; but it can become a more complicated story if you throw in
-supporting application-dependent object types, and correctly running
-on multiple threads, and so on.
+enough; but it can become a complicated story if you throw in
+supporting application-dependent object types; and a messy story if
+you add correctly running on multiple threads, for example.
-Moreover, this approach is specific to CPython (2.x or 3.x, which you
-can do in a similar way). It does not work on PyPy, which has its own
-smaller but very different `embedding API`_.
+Moreover, this approach is specific to CPython (2.x or 3.x). It does
+not work at all on PyPy, which has its own very different, minimal
+`embedding API`_.
-The new-and-coming thing about CFFI, meant as replacement of the above
-solutions, is direct embedding support---and it does that with no
-fixed API at all. The idea is to write some Python script with a
-``cdef()`` which declares a number of ``extern "Python"`` functions.
-When running the script, it creates the C source code and compiles it
-to a dynamically-linked library (``.so`` on Linux). This is the same
-as in the regular API-mode usage, and ``extern "Python"`` was
-`introduced in CFFI 1.4`_. What is new is that these ``extern
+The new-and-coming thing about CFFI 1.5, meant as replacement of the
+above solutions, is direct embedding support---with no fixed API at
+all. The idea is to write some Python script with a ``cdef()`` which
+declares a number of ``extern "Python"`` functions. When running the
+script, it creates the C source code and compiles it to a
+dynamically-linked library (``.so`` on Linux). This is the same as in
+the regular API-mode usage. What is new is that these ``extern
"Python"`` can now also be *exported* from the ``.so``, in the C
sense. You also give a bit of initialization-time Python code
-directly in the script, which will be compiled into the ``.so``
-too.
+directly in the script, which will be compiled into the ``.so`` too.
-In other words, this library can now be used directly from any C
-program (and it is still importable in Python). It exposes the C API
-of your choice, which you specified with the ``extern "Python"``
-declarations. You can use it to make whatever custom API makes sense
-in your particular case. You can even directly make a "plug-in" for
-any program that supports them, just by exporting the API expected for
-such plugins.
+This library can now be used directly from any C program (and it is
+still importable in Python). It exposes the C API of your choice,
+which you specified with the ``extern "Python"`` declarations. You
+can use it to make whatever custom API makes sense in your particular
+case. You can even directly make a "plug-in" for any program that
+supports them, just by exporting the API expected for such plugins.
-This is still being finalized, but please try it out. (You can also see
-`embedding.py`_ directly online for a quick glance.) These are the
-instructions on Linux with CPython 2.7 (CPython 3.x and non-Linux
-platforms are still a work in progress right now, but this should be
-quickly fixed):
+This is still being finalized, but please try it out. (You can also
+see `embedding.py`_ directly online for a quick glance.) Here are
+below the instructions on Linux with CPython 2.7 (CPython 3.x and
+non-Linux platforms are still a work in progress right now, but this
+should be quickly fixed):
* get the branch ``static-callback-embedding`` of CFFI::
@@ -59,7 +70,7 @@
cd demo
PYTHONPATH=.. python embedding.py
-* run ``gcc`` to build the C sources---on Linux::
+* this produces ``_embedding_cffi.c``; run ``gcc`` to build it---on Linux::
gcc -shared -fPIC _embedding_cffi.c -o _embedding_cffi.so -lpython2.7
-I/usr/include/python2.7
@@ -75,22 +86,39 @@
Very similar steps can be followed on PyPy, but it requires the
``cffi-static-callback-embedding`` branch of PyPy, which you must
-first translate from sources.
+first translate from sources. The difference is only that you need to
+adapt the first ``gcc`` command line: replace ``-lpython2.7`` with
+``-lpypy-c`` and to fix the ``-I`` path (and possibly add a ``-L``
+path).
Note that CPython/PyPy is automatically initialized (using locks in
case of multi-threading) the first time any of the ``extern "Python"``
-functions is called from the C program. At that time, the custom
-initialization-time Python code you put in
+functions is called from the C program. (This should work even if two
+different threads call the first time a function from two *different*
+embedded CFFI extensions; in other words, explicit initialization is
+never needed). The custom initialization-time Python code you put in
``ffi.embedding_init_code()`` is executed. If this code starts to be
-big, you may consider moving it to independent modules or packages;
-then the initialization-time Python code only needs to import them
-(possibly after hacking around with ``sys.path``).
+big, you can move it to independent modules or packages. Then the
+initialization-time Python code only needs to import them. In that
+case, you have to carefully set up ``sys.path`` if the modules are not
+installed in the usual Python way.
+
+A better alternative would be to use virtualenv. How to do that is
+not fully fleshed out so far. You can certainly run the whole program
+with the environment variables set up by the virtualenv's ``activate``
+script first. There are probably other solutions that involve using
+gcc's ``-Wl,-rpath=\$ORIGIN/`` or ``-Wl,-rpath=/fixed/path/`` options
+to load a specific libpython or libypypy-c library. If you try it out
+and it doesn't work the way you would like, please complain :-)
Another point: right now this does not support CPython's notion of
multiple subinterpreters. The logic creates a single global Python
-interpreter, and runs everything in that context. Idea about how to
-support that cleanly would be welcome ``:-)`` More generally, any
-feedback is appreciated.
+interpreter, and runs everything in that context. Maybe a future
+version would have an explicit API to do that---or maybe it should be
+the job of a 3rd-party extension module to provide a Python interface
+over the notion of subinterpreters...
+
+More generally, any feedback is appreciated.
Have fun,
_______________________________________________
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit
