Author: Armin Rigo <ar...@tunes.org> Branch: py3.5 Changeset: r89090:97c916c45ec7 Date: 2016-12-16 10:44 +0100 http://bitbucket.org/pypy/pypy/changeset/97c916c45ec7/
Log: hg merge default diff --git a/pypy/doc/contributor.rst b/pypy/doc/contributor.rst --- a/pypy/doc/contributor.rst +++ b/pypy/doc/contributor.rst @@ -1,3 +1,9 @@ +#encoding utf-8 + +Contributors +------------ +:: + Armin Rigo Maciej Fijalkowski Carl Friedrich Bolz @@ -307,7 +313,7 @@ Mads Kiilerich Antony Lee Jason Madden - Daniel Neuh�user + Daniel Neuhäuser reub...@gmail.com Yaroslav Fedevych Jim Hunziker diff --git a/pypy/doc/cppyy.rst b/pypy/doc/cppyy.rst --- a/pypy/doc/cppyy.rst +++ b/pypy/doc/cppyy.rst @@ -3,16 +3,17 @@ The cppyy module delivers dynamic Python-C++ bindings. It is designed for automation, high performance, scale, interactivity, and -handling all of modern C++. +handling all of modern C++ (11, 14, etc.). It is based on `Cling`_ which, through `LLVM`_/`clang`_, provides C++ reflection and interactivity. Reflection information is extracted from C++ header files. Cppyy itself is built into PyPy (an alternative exists for CPython), but -it requires a backend, installable through pip, to interface with Cling. +it requires a `backend`_, installable through pip, to interface with Cling. .. _Cling: https://root.cern.ch/cling .. _LLVM: http://llvm.org/ .. _clang: http://clang.llvm.org/ +.. _backend: https://pypi.python.org/pypi/PyPy-cppyy-backend Installation @@ -22,25 +23,39 @@ module, which is no longer supported. Both the tooling and user-facing Python codes are very backwards compatible, however. -Further dependencies are cmake (for general build) and Python2.7 (for LLVM). +Further dependencies are cmake (for general build), Python2.7 (for LLVM), and +a modern C++ compiler (one that supports at least C++11). Assuming you have a recent enough version of PyPy installed, use pip to complete the installation of cppyy:: - $ pypy-c -m pip install PyPy-cppyy-backend + $ MAKE_NPROCS=4 pypy-c -m pip install --verbose PyPy-cppyy-backend +Set the number of parallel builds ('4' in this example, through the MAKE_NPROCS +environment variable) to a number appropriate for your machine. The building process may take quite some time as it includes a customized -version of LLVM as part of Cling. +version of LLVM as part of Cling, which is why --verbose is recommended so that +you can see the build progress. + +The default installation will be under +$PYTHONHOME/site-packages/cppyy_backend/lib, +which needs to be added to your dynamic loader path (LD_LIBRARY_PATH). +If you need the dictionary and class map generation tools (used in the examples +below), you need to add $PYTHONHOME/site-packages/cppyy_backend/bin to your +executable path (PATH). Basic bindings example ---------------------- -Now test with a trivial example whether all packages are properly installed -and functional. -First, create a C++ header file with some class in it (note that all functions -are made inline for convenience; a real-world example would of course have a -corresponding source file):: +These examples assume that cppyy_backend is pointed to by the environment +variable CPPYYHOME, and that CPPYYHOME/lib is added to LD_LIBRARY_PATH and +CPPYYHOME/bin to PATH. + +Let's first test with a trivial example whether all packages are properly +installed and functional. +Create a C++ header file with some class in it (all functions are made inline +for convenience; if you have out-of-line code, link with it as appropriate):: $ cat MyClass.h class MyClass { @@ -54,11 +69,11 @@ int m_myint; }; -Then, generate the bindings using ``genreflex`` (part of ROOT), and compile the -code:: +Then, generate the bindings using ``genreflex`` (installed under +cppyy_backend/bin in site_packages), and compile the code:: $ genreflex MyClass.h - $ g++ -fPIC -rdynamic -O2 -shared -I$REFLEXHOME/include MyClass_rflx.cpp -o libMyClassDict.so -L$REFLEXHOME/lib -lReflex + $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyClass_rflx.cpp -o libMyClassDict.so -L$CPPYYHOME/lib -lCling Next, make sure that the library can be found through the dynamic lookup path (the ``LD_LIBRARY_PATH`` environment variable on Linux, ``PATH`` on Windows), @@ -110,7 +125,7 @@ For example:: $ genreflex MyClass.h --rootmap=libMyClassDict.rootmap --rootmap-lib=libMyClassDict.so - $ g++ -fPIC -rdynamic -O2 -shared -I$REFLEXHOME/include MyClass_rflx.cpp -o libMyClassDict.so -L$REFLEXHOME/lib -lReflex + $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyClass_rflx.cpp -o libMyClassDict.so -L$CPPYYHOME/lib -lCling where the first option (``--rootmap``) specifies the output file name, and the second option (``--rootmap-lib``) the name of the reflection library where @@ -212,7 +227,7 @@ Now the reflection info can be generated and compiled:: $ genreflex MyAdvanced.h --selection=MyAdvanced.xml - $ g++ -fPIC -rdynamic -O2 -shared -I$REFLEXHOME/include MyAdvanced_rflx.cpp -o libAdvExDict.so -L$REFLEXHOME/lib -lReflex + $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyAdvanced_rflx.cpp -o libAdvExDict.so -L$CPPYYHOME/lib -lCling and subsequently be used from PyPy:: @@ -271,7 +286,7 @@ bound using:: $ genreflex example.h --deep --rootmap=libexampleDict.rootmap --rootmap-lib=libexampleDict.so - $ g++ -fPIC -rdynamic -O2 -shared -I$REFLEXHOME/include example_rflx.cpp -o libexampleDict.so -L$REFLEXHOME/lib -lReflex + $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include example_rflx.cpp -o libexampleDict.so -L$CPPYYHOME/lib -lCling * **abstract classes**: Are represented as python classes, since they are needed to complete the inheritance hierarchies, but will raise an exception @@ -567,13 +582,10 @@ Templates --------- -A bit of special care needs to be taken for the use of templates. -For a templated class to be completely available, it must be guaranteed that -said class is fully instantiated, and hence all executable C++ code is -generated and compiled in. -The easiest way to fulfill that guarantee is by explicit instantiation in the -header file that is handed to ``genreflex``. -The following example should make that clear:: +Templates can be automatically instantiated, assuming the appropriate header +files have been loaded or are accessible to the class loader. +This is the case for example for all of STL. +For example:: $ cat MyTemplate.h #include <vector> @@ -587,68 +599,10 @@ int m_i; }; - #ifdef __GCCXML__ - template class std::vector<MyClass>; // explicit instantiation - #endif - -If you know for certain that all symbols will be linked in from other sources, -you can also declare the explicit template instantiation ``extern``. -An alternative is to add an object to an unnamed namespace:: - - namespace { - std::vector<MyClass> vmc; - } // unnamed namespace - -Unfortunately, this is not always enough for gcc. -The iterators of vectors, if they are going to be used, need to be -instantiated as well, as do the comparison operators on those iterators, as -these live in an internal namespace, rather than in the iterator classes. -Note that you do NOT need this iterators to iterator over a vector. -You only need them if you plan to explicitly call e.g. ``begin`` and ``end`` -methods, and do comparisons of iterators. -One way to handle this, is to deal with this once in a macro, then reuse that -macro for all ``vector`` classes. -Thus, the header above needs this (again protected with -``#ifdef __GCCXML__``), instead of just the explicit instantiation of the -``vector<MyClass>``:: - - #define STLTYPES_EXPLICIT_INSTANTIATION_DECL(STLTYPE, TTYPE) \ - template class std::STLTYPE< TTYPE >; \ - template class __gnu_cxx::__normal_iterator<TTYPE*, std::STLTYPE< TTYPE > >; \ - template class __gnu_cxx::__normal_iterator<const TTYPE*, std::STLTYPE< TTYPE > >;\ - namespace __gnu_cxx { \ - template bool operator==(const std::STLTYPE< TTYPE >::iterator&, \ - const std::STLTYPE< TTYPE >::iterator&); \ - template bool operator!=(const std::STLTYPE< TTYPE >::iterator&, \ - const std::STLTYPE< TTYPE >::iterator&); \ - } - - STLTYPES_EXPLICIT_INSTANTIATION_DECL(vector, MyClass) - -Then, still for gcc, the selection file needs to contain the full hierarchy as -well as the global overloads for comparisons for the iterators:: - - $ cat MyTemplate.xml - <lcgdict> - <class pattern="std::vector<*>" /> - <class pattern="std::vector<*>::iterator" /> - <function name="__gnu_cxx::operator=="/> - <function name="__gnu_cxx::operator!="/> - - <class name="MyClass" /> - </lcgdict> - Run the normal ``genreflex`` and compilation steps:: $ genreflex MyTemplate.h --selection=MyTemplate.xml - $ g++ -fPIC -rdynamic -O2 -shared -I$REFLEXHOME/include MyTemplate_rflx.cpp -o libTemplateDict.so -L$REFLEXHOME/lib -lReflex - -Note: this is a dirty corner that clearly could do with some automation, -even if the macro already helps. -Such automation is planned. -In fact, in the Cling world, the backend can perform the template -instantations and generate the reflection info on the fly, and none of the -above will any longer be necessary. + $ g++ -std=c++11 -fPIC -rdynamic -O2 -shared -I$CPPYYHOME/include MyTemplate_rflx.cpp -o libTemplateDict.so -L$CPPYYHOME/lib -lCling Subsequent use should be as expected. Note the meta-class style of "instantiating" the template:: @@ -665,8 +619,6 @@ 1 2 3 >>>> -Other templates work similarly, but are typically simpler, as there are no -similar issues with iterators for e.g. ``std::list``. The arguments to the template instantiation can either be a string with the full list of arguments, or the explicit classes. The latter makes for easier code writing if the classes passed to the @@ -676,95 +628,40 @@ The fast lane ------------- -The following is an experimental feature of cppyy. -It mostly works, but there are some known issues (e.g. with return-by-value). -Soon it should be the default mode, however. +By default, cppyy will use direct function pointers through `CFFI`_ whenever +possible. If this causes problems for you, you can disable it by setting the +CPPYY_DISABLE_FASTPATH environment variable. -With a slight modification of Reflex, it can provide function pointers for -C++ methods, and hence allow PyPy to call those pointers directly, rather than -calling C++ through a Reflex stub. +.. _CFFI: https://cffi.readthedocs.io/en/latest/ -The standalone version of Reflex `provided`_ has been patched, but if you get -Reflex from another source (most likely with a ROOT distribution), locate the -file `genreflex-methptrgetter.patch`_ in pypy/module/cppyy and apply it to -the genreflex python scripts found in ``$ROOTSYS/lib``:: - - $ cd $ROOTSYS/lib - $ patch -p2 < genreflex-methptrgetter.patch - -With this patch, ``genreflex`` will have grown the ``--with-methptrgetter`` -option. -Use this option when running ``genreflex``, and add the -``-Wno-pmf-conversions`` option to ``g++`` when compiling. -The rest works the same way: the fast path will be used transparently (which -also means that you can't actually find out whether it is in use, other than -by running a micro-benchmark or a JIT test). - -.. _provided: http://cern.ch/wlav/reflex-2014-10-20.tar.bz2 -.. _genreflex-methptrgetter.patch: https://bitbucket.org/pypy/pypy/src/default/pypy/module/cppyy/genreflex-methptrgetter.patch CPython ------- -Most of the ideas in cppyy come originally from the `PyROOT`_ project. -Although PyROOT does not support Reflex directly, it has an alter ego called -"PyCintex" that, in a somewhat roundabout way, does. -If you installed ROOT, rather than just Reflex, PyCintex should be available -immediately if you add ``$ROOTSYS/lib`` to the ``PYTHONPATH`` environment -variable. +Most of the ideas in cppyy come originally from the `PyROOT`_ project, which +contains a CPython-based cppyy.py module (with similar dependencies as the +one that comes with PyPy). +A standalone pip-installable version is planned, but for now you can install +ROOT through your favorite distribution installer (available in the science +section). .. _PyROOT: https://root.cern.ch/pyroot -There are a couple of minor differences between PyCintex and cppyy, most to do -with naming. -The one that you will run into directly, is that PyCintex uses a function -called ``loadDictionary`` rather than ``load_reflection_info`` (it has the -same rootmap-based class loader functionality, though, making this point -somewhat moot). -The reason for this is that Reflex calls the shared libraries that contain -reflection info "dictionaries." -However, in python, the name `dictionary` already has a well-defined meaning, -so a more descriptive name was chosen for cppyy. -In addition, PyCintex requires that the names of shared libraries so loaded -start with "lib" in their name. -The basic example above, rewritten for PyCintex thus goes like this:: - - $ python - >>> import PyCintex - >>> PyCintex.loadDictionary("libMyClassDict.so") - >>> myinst = PyCintex.gbl.MyClass(42) - >>> print myinst.GetMyInt() - 42 - >>> myinst.SetMyInt(33) - >>> print myinst.m_myint - 33 - >>> myinst.m_myint = 77 - >>> print myinst.GetMyInt() - 77 - >>> help(PyCintex.gbl.MyClass) # shows that normal python introspection works - -Other naming differences are such things as taking an address of an object. -In PyCintex, this is done with ``AddressOf`` whereas in cppyy the choice was -made to follow the naming as in ``ctypes`` and hence use ``addressof`` -(PyROOT/PyCintex predate ``ctypes`` by several years, and the ROOT project -follows camel-case, hence the differences). - -Of course, this is python, so if any of the naming is not to your liking, all -you have to do is provide a wrapper script that you import instead of -importing the ``cppyy`` or ``PyCintex`` modules directly. -In that wrapper script you can rename methods exactly the way you need it. - -In the cling world, all these differences will be resolved. +There are a couple of minor differences between the two versions of cppyy +(the CPython version has a few more features). +Work is on-going to integrate the nightly tests of both to make sure their +feature sets are equalized. Python3 ------- -To change versions of CPython (to Python3, another version of Python, or later -to the `Py3k`_ version of PyPy), the only part that requires recompilation is -the bindings module, be it ``cppyy`` or ``libPyROOT.so`` (in PyCintex). -Although ``genreflex`` is indeed a Python tool, the generated reflection -information is completely independent of Python. +The CPython version of cppyy supports Python3, assuming your packager has +build the backend for it. +The cppyy module has not been tested with the `Py3k`_ version of PyPy. +Note that the generated reflection information (from ``genreflex``) is fully +independent of Python, and does not need to be rebuild when switching versions +or interpreters. .. _Py3k: https://bitbucket.org/pypy/pypy/src/py3k @@ -772,5 +669,4 @@ .. toctree:: :hidden: - cppyy_backend cppyy_example diff --git a/pypy/doc/cppyy_backend.rst b/pypy/doc/cppyy_backend.rst deleted file mode 100644 --- a/pypy/doc/cppyy_backend.rst +++ /dev/null @@ -1,45 +0,0 @@ -Backends for cppyy -================== - -The cppyy module needs a backend to provide the C++ reflection information on -which the Python bindings are build. -The backend is called through a C-API, which can be found in the PyPy sources -in: :source:`pypy/module/cppyy/include/capi.h`. -There are two kinds of API calls: querying about reflection information, which -are used during the creation of Python-side constructs, and making the actual -calls into C++. -The objects passed around are all opaque: cppyy does not make any assumptions -about them, other than that the opaque handles can be copied. -Their definition, however, appears in two places: in the C code (in capi.h), -and on the RPython side (in :source:`capi_types.py <pypy/module/cppyy/capi/capi_types.py>`), so if they are changed, they -need to be changed on both sides. - -There are two places where selections in the RPython code affect the choice -(and use) of the backend. -The first is in :source:`pypy/module/cppyy/capi/__init__.py`:: - - # choose C-API access method: - from pypy.module.cppyy.capi.loadable_capi import * - #from pypy.module.cppyy.capi.builtin_capi import * - -The default is the loadable C-API. -Comment it and uncomment the builtin C-API line, to use the builtin version. - -Next, if the builtin C-API is chosen, the specific backend needs to be set as -well (default is Reflex). -This second choice is in :source:`pypy/module/cppyy/capi/builtin_capi.py`:: - - import reflex_capi as backend - #import cint_capi as backend - -After those choices have been made, built pypy-c as usual. - -When building pypy-c from source, keep the following in mind. -If the loadable_capi is chosen, no further prerequisites are needed. -However, for the build of the builtin_capi to succeed, the ``ROOTSYS`` -environment variable must point to the location of your ROOT (or standalone -Reflex in the case of the Reflex backend) installation, or the ``root-config`` -utility must be accessible through ``$PATH`` (e.g. by adding ``$ROOTSYS/bin`` -to ``PATH``). -In case of the former, include files are expected under ``$ROOTSYS/include`` -and libraries under ``$ROOTSYS/lib``. diff --git a/pypy/doc/extending.rst b/pypy/doc/extending.rst --- a/pypy/doc/extending.rst +++ b/pypy/doc/extending.rst @@ -12,7 +12,7 @@ * Write them in pure Python and use ctypes_. -* Write them in C++ and bind them through Reflex_. +* Write them in C++ and bind them through :doc:`cppyy <cppyy>` using Cling. * Write them in as `RPython mixed modules`_. @@ -61,11 +61,11 @@ .. _libffi: http://sourceware.org/libffi/ -Reflex ------- +Cling and cppyy +--------------- The builtin :doc:`cppyy <cppyy>` module uses reflection information, provided by -`Reflex`_ (which needs to be `installed separately`_), of C/C++ code to +`Cling`_ (which needs to be `installed separately`_), of C/C++ code to automatically generate bindings at runtime. In Python, classes and functions are always runtime structures, so when they are generated matters not for performance. @@ -76,11 +76,14 @@ The :doc:`cppyy <cppyy>` module is written in RPython, thus PyPy's JIT is able to remove most cross-language call overhead. -:doc:`Full details <cppyy>` are `available here <cppyy>`. +:doc:Full details are `available here <cppyy>`. -.. _installed separately: http://cern.ch/wlav/reflex-2013-08-14.tar.bz2 -.. _Reflex: https://root.cern.ch/how/how-use-reflex +.. _installed separately: https://pypi.python.org/pypi/PyPy-cppyy-backend +.. _Cling: https://root.cern.ch/cling +.. toctree:: + + cppyy RPython Mixed Modules --------------------- @@ -94,7 +97,3 @@ This is how the numpy module is being developed. -.. toctree:: - :hidden: - - cppyy diff --git a/pypy/doc/index-of-release-notes.rst b/pypy/doc/index-of-release-notes.rst --- a/pypy/doc/index-of-release-notes.rst +++ b/pypy/doc/index-of-release-notes.rst @@ -59,6 +59,7 @@ .. toctree:: + release-pypy3.3-v5.5.0.rst release-pypy3.3-v5.2-alpha1.rst CPython 3.2 compatible versions diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst --- a/pypy/doc/whatsnew-head.rst +++ b/pypy/doc/whatsnew-head.rst @@ -54,3 +54,14 @@ Assign ``tp_doc`` to the new TypeObject's type dictionary ``__doc__`` key so it will be picked up by app-level objects of that type + +.. branch: cling-support + +Module cppyy now uses cling as its backend (Reflex has been removed). The +user-facing interface and main developer tools (genreflex, selection files, +class loader, etc.) remain the same. A libcppyy_backend.so library is still +needed but is now available through PyPI with pip: PyPy-cppyy-backend. + +The Cling-backend brings support for modern C++ (11, 14, etc.), dynamic +template instantations, and improved integration with CFFI for better +performance. It also provides interactive C++ (and bindings to that). diff --git a/pypy/module/cppyy/test/conftest.py b/pypy/module/cppyy/test/conftest.py --- a/pypy/module/cppyy/test/conftest.py +++ b/pypy/module/cppyy/test/conftest.py @@ -1,7 +1,13 @@ -import py +import py, sys @py.test.mark.tryfirst def pytest_runtest_setup(item): + if 'linux' in sys.platform: + # tests require minimally std=c++11 + cc_info = py.process.cmdexec('gcc -v --help') + if not '-std=c++11' in cc_info: + py.test.skip('skipping tests because gcc does not support C++11') + if py.path.local.sysfind('genreflex') is None: import pypy.module.cppyy.capi.loadable_capi as lcapi if 'dummy' in lcapi.reflection_library: diff --git a/pypy/module/cpyext/stubs.py b/pypy/module/cpyext/stubs.py --- a/pypy/module/cpyext/stubs.py +++ b/pypy/module/cpyext/stubs.py @@ -2158,18 +2158,3 @@ it causes an exception to immediately be thrown; this is used for the throw() methods of generator objects.""" raise NotImplementedError - -@cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL) -def PyWeakref_Check(space, ob): - """Return true if ob is either a reference or proxy object.""" - raise NotImplementedError - -@cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL) -def PyWeakref_CheckRef(space, ob): - """Return true if ob is a reference object.""" - raise NotImplementedError - -@cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL) -def PyWeakref_CheckProxy(space, ob): - """Return true if ob is a proxy object.""" - raise NotImplementedError diff --git a/pypy/module/cpyext/test/test_weakref.py b/pypy/module/cpyext/test/test_weakref.py --- a/pypy/module/cpyext/test/test_weakref.py +++ b/pypy/module/cpyext/test/test_weakref.py @@ -56,3 +56,30 @@ ) ]) module.test_macro_cast() + + def test_weakref_check(self): + module = self.import_extension('foo', [ + ("test_weakref_cast", "METH_O", + """ + return Py_BuildValue("iiii", + (int)PyWeakref_Check(args), + (int)PyWeakref_CheckRef(args), + (int)PyWeakref_CheckRefExact(args), + (int)PyWeakref_CheckProxy(args)); + """ + ) + ]) + import weakref + def foo(): pass + class Bar(object): + pass + bar = Bar() + assert module.test_weakref_cast([]) == (0, 0, 0, 0) + assert module.test_weakref_cast(weakref.ref(foo)) == (1, 1, 1, 0) + assert module.test_weakref_cast(weakref.ref(bar)) == (1, 1, 1, 0) + assert module.test_weakref_cast(weakref.proxy(foo)) == (1, 0, 0, 1) + assert module.test_weakref_cast(weakref.proxy(bar)) == (1, 0, 0, 1) + class X(weakref.ref): + pass + assert module.test_weakref_cast(X(foo)) == (1, 1, 0, 0) + assert module.test_weakref_cast(X(bar)) == (1, 1, 0, 0) diff --git a/pypy/module/cpyext/weakrefobject.py b/pypy/module/cpyext/weakrefobject.py --- a/pypy/module/cpyext/weakrefobject.py +++ b/pypy/module/cpyext/weakrefobject.py @@ -1,6 +1,7 @@ from pypy.module.cpyext.api import cpython_api -from pypy.module.cpyext.pyobject import PyObject +from pypy.module.cpyext.pyobject import PyObject, CANNOT_FAIL from pypy.module._weakref.interp__weakref import W_Weakref, proxy +from pypy.module._weakref.interp__weakref import W_Proxy, W_CallableProxy from rpython.rtyper.lltypesystem import rffi @cpython_api([PyObject, PyObject], PyObject) @@ -54,3 +55,34 @@ PyWeakref_GetObject() and Py_INCREF().) """ return space.call_function(w_ref) + +@cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL) +def PyWeakref_CheckRef(space, w_obj): + """Return true if ob is a reference object. + """ + w_obj_type = space.type(w_obj) + w_type = space.gettypeobject(W_Weakref.typedef) + return (space.is_w(w_obj_type, w_type) or + space.issubtype_w(w_obj_type, w_type)) + +@cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL) +def PyWeakref_CheckRefExact(space, w_obj): + w_obj_type = space.type(w_obj) + w_type = space.gettypeobject(W_Weakref.typedef) + return space.is_w(w_obj_type, w_type) + +@cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL) +def PyWeakref_CheckProxy(space, w_obj): + """Return true if ob is a proxy object. + """ + w_obj_type = space.type(w_obj) + w_type1 = space.gettypeobject(W_Proxy.typedef) + w_type2 = space.gettypeobject(W_CallableProxy.typedef) + return space.is_w(w_obj_type, w_type1) or space.is_w(w_obj_type, w_type2) + +@cpython_api([PyObject], rffi.INT_real, error=CANNOT_FAIL) +def PyWeakref_Check(space, w_obj): + """Return true if ob is either a reference or proxy object. + """ + return (PyWeakref_CheckRef(space, w_obj) or + PyWeakref_CheckProxy(space, w_obj)) diff --git a/pypy/module/posix/interp_posix.py b/pypy/module/posix/interp_posix.py --- a/pypy/module/posix/interp_posix.py +++ b/pypy/module/posix/interp_posix.py @@ -2099,8 +2099,9 @@ Return a string of 'size' random bytes suitable for cryptographic use. """ context = get(space).random_context + signal_checker = space.getexecutioncontext().checksignals try: - return space.newbytes(rurandom.urandom(context, size)) + return space.newbytes(rurandom.urandom(context, n, signal_checker)) except OSError as e: # 'rurandom' should catch and retry internally if it gets EINTR # (at least in os.read(), which is probably enough in practice) diff --git a/rpython/rlib/rurandom.py b/rpython/rlib/rurandom.py --- a/rpython/rlib/rurandom.py +++ b/rpython/rlib/rurandom.py @@ -7,12 +7,12 @@ from rpython.rtyper.lltypesystem import lltype, rffi from rpython.rlib.objectmodel import not_rpython +from rpython.translator.tool.cbuild import ExternalCompilationInfo +from rpython.rtyper.tool import rffi_platform if sys.platform == 'win32': from rpython.rlib import rwin32 - from rpython.translator.tool.cbuild import ExternalCompilationInfo - from rpython.rtyper.tool import rffi_platform eci = ExternalCompilationInfo( includes = ['windows.h', 'wincrypt.h'], @@ -56,7 +56,7 @@ return lltype.malloc(rffi.CArray(HCRYPTPROV), 1, immortal=True, zero=True) - def urandom(context, n): + def urandom(context, n, signal_checker=None): provider = context[0] if not provider: # This handle is never explicitly released. The operating @@ -80,11 +80,71 @@ def init_urandom(): return None - def urandom(context, n): + SYS_getrandom = None + + if sys.platform.startswith('linux'): + eci = ExternalCompilationInfo(includes=['sys/syscall.h']) + class CConfig: + _compilation_info_ = eci + SYS_getrandom = rffi_platform.DefinedConstantInteger( + 'SYS_getrandom') + globals().update(rffi_platform.configure(CConfig)) + + if SYS_getrandom is not None: + from rpython.rlib.rposix import get_saved_errno, handle_posix_error + import errno + + eci = eci.merge(ExternalCompilationInfo(includes=['linux/random.h'])) + class CConfig: + _compilation_info_ = eci + GRND_NONBLOCK = rffi_platform.ConstantInteger('GRND_NONBLOCK') + globals().update(rffi_platform.configure(CConfig)) + + # On Linux, use the syscall() function because the GNU libc doesn't + # expose the Linux getrandom() syscall yet. + syscall = rffi.llexternal( + 'syscall', + [lltype.Signed, rffi.CCHARP, rffi.LONG, rffi.INT], + lltype.Signed, + compilation_info=eci, + save_err=rffi.RFFI_SAVE_ERRNO) + + class Works: + status = True + getrandom_works = Works() + + def _getrandom(n, result, signal_checker): + if not getrandom_works.status: + return n + while n > 0: + with rffi.scoped_alloc_buffer(n) as buf: + got = syscall(SYS_getrandom, buf.raw, n, GRND_NONBLOCK) + if got >= 0: + s = buf.str(got) + result.append(s) + n -= len(s) + continue + err = get_saved_errno() + if (err == errno.ENOSYS or err == errno.EPERM or + err == errno.EAGAIN): # see CPython 3.5 + getrandom_works.status = False + return n + if err == errno.EINTR: + if signal_checker is not None: + signal_checker() + continue + handle_posix_error("getrandom", got) + raise AssertionError("unreachable") + return n + + def urandom(context, n, signal_checker=None): "Read n bytes from /dev/urandom." - result = '' - if n == 0: - return result + result = [] + if SYS_getrandom is not None: + n = _getrandom(n, result, signal_checker) + if n <= 0: + return ''.join(result) + # XXX should somehow cache the file descriptor. It's a mess. # CPython has a 99% solution and hopes for the remaining 1% # not to occur. For now, we just don't cache the file @@ -98,8 +158,8 @@ if e.errno != errno.EINTR: raise data = '' - result += data + result.append(data) n -= len(data) finally: os.close(fd) - return result + return ''.join(result) diff --git a/rpython/rlib/test/test_rurandom.py b/rpython/rlib/test/test_rurandom.py new file mode 100644 --- /dev/null +++ b/rpython/rlib/test/test_rurandom.py @@ -0,0 +1,12 @@ +from rpython.rlib import rurandom + +def test_rurandom(): + context = rurandom.init_urandom() + s = rurandom.urandom(context, 5000) + assert type(s) is str and len(s) == 5000 + for x in [1, 11, 111, 222]: + assert s.count(chr(x)) >= 1 + +def test_rurandom_no_syscall(monkeypatch): + monkeypatch.setattr(rurandom, 'SYS_getrandom', None) + test_rurandom() _______________________________________________ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit