Author: Alex Gaynor <alex.gay...@gmail.com> Branch: Changeset: r71058:1f55d73ad8a0 Date: 2014-04-28 19:58 -0700 http://bitbucket.org/pypy/pypy/changeset/1f55d73ad8a0/
Log: merged upstream diff --git a/pypy/doc/getting-started.rst b/pypy/doc/getting-started.rst --- a/pypy/doc/getting-started.rst +++ b/pypy/doc/getting-started.rst @@ -76,7 +76,7 @@ .. code-block:: console $ curl -O http://python-distribute.org/distribute_setup.py - $ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py + $ curl -O https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py $ ./pypy-2.1/bin/pypy distribute_setup.py $ ./pypy-2.1/bin/pypy get-pip.py $ ./pypy-2.1/bin/pip install pygments # for example diff --git a/pypy/doc/release-2.3.0.rst b/pypy/doc/release-2.3.0.rst --- a/pypy/doc/release-2.3.0.rst +++ b/pypy/doc/release-2.3.0.rst @@ -1,11 +1,17 @@ ======================================= -PyPy 2.3 - XXXX TODO +PyPy 2.3 - Easier Than Ever ======================================= We're pleased to announce PyPy 2.3, which targets version 2.7.6 of the Python language. This release updates the stdlib from 2.7.3, jumping directly to 2.7.6. -This release also contains several bugfixes and performance improvements. +This release also contains several bugfixes and performance improvements, +many generated by real users finding corner cases our `TDD`_ methods missed. +`CFFI`_ has made it easier than ever to use existing C code with both cpython +and PyPy, easing the transition for packages like `cryptography`_, `Pillow`_ +(Python Imaging Library [Fork]), a basic port of `pygame-cffi`_, and others. + +PyPy can now be embedded in a hosting application, for instance inside `uWSGI`_ You can download the PyPy 2.3 release here: @@ -17,72 +23,112 @@ Please consider donating more so we can finish those projects! The three projects are: -* Py3k (supporting Python 3.x): the release PyPy3 2.2 is imminent. +* `Py3k`_ (supporting Python 3.x): the release PyPy3 2.2 is imminent. -* STM (software transactional memory): a preview will be released very soon, - as soon as we fix a few bugs +* `STM`_ (software transactional memory): a preview will be released very soon, + once we fix a few bugs -* NumPy: the work done is included in the PyPy 2.2 release. More details below. +* `NumPy`_ the work done is included in the PyPy 2.2 release. More details below. -.. _`Raspberry Pi Foundation`: http://www.raspberrypi.org +.. _`Py3k`: http://pypy.org/py3donate.html +.. _`STM`: http://pypy.org/tmdonate2.html +.. _ `Numpy`: http://pypy.org/numpydonate.html +.. _`TDD`: http://doc.pypy.org/en/latest/how-to-contribute.html +.. _`CFFI`: http://cffi.readthedocs.org +.. _`cryptography`: https://cryptography.io +.. _`Pillow`: https://pypi.python.org/pypi/Pillow/2.4.0 +.. _`pygame-cffi`: https://github.com/CTPUG/pygame_cffi +.. _`uWSGI`: http://uwsgi-docs.readthedocs.org/en/latest/PyPy.html What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for -CPython 2.7. It's fast (`pypy 2.2 and cpython 2.7.2`_ performance comparison) +CPython 2.7. It's fast (`pypy 2.2 and cpython 2.7.2`_ performance comparison; +note that the latest cpython is not faster than cpython 2.7.2) due to its integrated tracing JIT compiler. -This release supports x86 machines running Linux 32/64, Mac OS X 64, Windows -32, or ARM (ARMv6 or ARMv7, with VFPv3). +This release supports x86 machines running Linux 32/64, Mac OS X 64, Windows, +and OpenBSD, +as well as newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux. -Work on the native Windows 64 is still stalling, we would welcome a volunteer -to handle that. +While we support 32 bit python on Windows, work on the native Windows 64 +bit python is still stalling, we would welcome a volunteer +to `handle that`_. .. _`pypy 2.2 and cpython 2.7.2`: http://speed.pypy.org +.. _`handle that`: http://doc.pypy.org/en/latest/windows.html#what-is-missing-for-a-full-64-bit-translation Highlights ========== -* Our Garbage Collector is now "incremental". It should avoid almost - all pauses due to a major collection taking place. Previously, it - would pause the program (rarely) to walk all live objects, which - could take arbitrarily long if your process is using a whole lot of - RAM. Now the same work is done in steps. This should make PyPy - more responsive, e.g. in games. There are still other pauses, from - the GC and the JIT, but they should be on the order of 5 - milliseconds each. +Bugfixes +-------- -* The JIT counters for hot code were never reset, which meant that a - process running for long enough would eventually JIT-compile more - and more rarely executed code. Not only is it useless to compile - such code, but as more compiled code means more memory used, this - gives the impression of a memory leak. This has been tentatively - fixed by decreasing the counters from time to time. +Many issues were cleaned up after being reported by users to https://bugs.pypy.org (ignore the bad SSL certificate) or on IRC at #pypy. Note that we consider +performance slowdowns as bugs. -* NumPy has been split: now PyPy only contains the core module, called - ``_numpypy``. The ``numpy`` module itself has been moved to - ``https://bitbucket.org/pypy/numpy`` and ``numpypy`` disappeared. - You need to install NumPy separately with a virtualenv: +* The ARM port no longer crashes on unaligned memory access to floats and doubles, + and singlefloats are supported in the JIT. + +* Generators are faster since they now skip unecessary cleanup + +* A first time contributor simplified JIT traces by adding integer bound + propagation in indexing and logical operations. + +* Optimize consecutive dictionary lookups of the same key in a chain + +* Our extensive pre-translation test suite now runs nightly on more platforms + +* Fix issues with reimporting builtin modules + +* Fix a rpython bug with loop-unrolling that appeared in the `HippyVM`_ PHP port + +.. _`HippyVM`: http://www.hippyvm.com + +New Platforms and Features +-------------------------- + +* Support for OpenBSD + +* Code cleanup: we continue to prune out old and unused code, and to refactor + large parts of the codebase. We have sepearated rpython from the PyPy python + interpreter, and rpython is seeing use in other dynamic language projects. + +* Support for precompiled headers in the build process for MSVC + +* Support for objects with __int__ and __float__ methods + +* Tweak support of errno in cpyext (the PyPy implemenation of the capi) + + +Numpy +----- +Numpy support has been split into a builtin ``_numpy`` module and a +fork of the numpy code base adapted to pypy at + ``https://bitbucket.org/pypy/numpy``. +You need to install NumPy separately with a virtualenv: ``pip install git+https://bitbucket.org/pypy/numpy.git``; or directly: ``git clone https://bitbucket.org/pypy/numpy.git``; ``cd numpy``; ``pypy setup.py install``. -* non-inlined calls have less overhead +* NumPy support has been improved, many failures in indexing, dtypes, + and scalars were corrected. We are slowly approaching our goal of passing + the numpy test suite. We still do not support object or unicode ndarrays. -* Things that use ``sys.set_trace`` are now JITted (like coverage) +* speed of iteration in dot() is now within 1.5x of the numpy c + implementation (without BLAS acceleration). Since the same array + iterator is used throughout the ``_numpy`` module, speed increases should + be apparent in all Numpy functionality. -* JSON decoding is now very fast (JSON encoding was already very fast) +* Most of the core functionality of nditer has been implemented. -* various buffer copying methods experience speedups (like list-of-ints to - ``int[]`` buffer from cffi) +* A cffi-based ``numpy.random`` module is available as a branch in the numpy + repository, it will be merged soon after this release. -* We finally wrote (hopefully) all the missing ``os.xxx()`` functions, - including ``os.startfile()`` on Windows and a handful of rare ones - on Posix. +* enhancements to the PyPy JIT were made to support virtualizing the raw_store/raw_load memory operations used in numpy arrays. Further work remains here in virtualizing the alloc_raw_storage when possible. This will allow scalars to have storages but still be virtualized when possible in loops. -* numpy has a rudimentary C API that cooperates with ``cpyext`` +Cheers +The PyPy Team -Cheers, -Armin Rigo and Maciej Fijalkowski diff --git a/pypy/module/sys/version.py b/pypy/module/sys/version.py --- a/pypy/module/sys/version.py +++ b/pypy/module/sys/version.py @@ -14,7 +14,8 @@ if platform.name == 'msvc': COMPILER_INFO = 'MSC v.%d 32 bit' % (platform.version * 10 + 600) -elif platform.cc is not None and platform.cc.startswith(('gcc', 'clang')): +elif platform.cc is not None and \ + os.path.basename(platform.cc).startswith(('gcc', 'clang')): from rpython.rtyper.tool import rffi_platform COMPILER_INFO = 'GCC ' + rffi_platform.getdefinedstring('__VERSION__', '') else: diff --git a/pypy/module/zipimport/test/test_undocumented.py b/pypy/module/zipimport/test/test_undocumented.py --- a/pypy/module/zipimport/test/test_undocumented.py +++ b/pypy/module/zipimport/test/test_undocumented.py @@ -135,8 +135,7 @@ importer = zipimport.zipimporter(os.path.join(zip_path, '_pkg')) assert zip_path in zipimport._zip_directory_cache file_set = set(zipimport._zip_directory_cache[zip_path].iterkeys()) - compare_set = set(path.replace(os.path.sep, '/') + '.py' - for path in self.created_paths) + compare_set = set(path + '.py' for path in self.created_paths) assert file_set == compare_set finally: self.cleanup_zipfile(self.created_paths) diff --git a/pypy/tool/release/force-builds.py b/pypy/tool/release/force-builds.py --- a/pypy/tool/release/force-builds.py +++ b/pypy/tool/release/force-builds.py @@ -9,7 +9,7 @@ modified by PyPy team """ -import os, sys, pwd, urllib +import os, sys, urllib from twisted.internet import reactor, defer from twisted.python import log @@ -34,6 +34,13 @@ 'build-pypy-c-jit-linux-armel', ] +def get_user(): + if sys.platform == 'win32': + return os.environ['USERNAME'] + else: + import pwd + return pwd.getpwuid(os.getuid())[0] + def main(): #XXX: handle release tags #XXX: handle validity checks @@ -49,7 +56,7 @@ print 'Forcing', builder, '...' url = "http://buildbot.pypy.org/builders/" + builder + "/force" args = [ - ('username', pwd.getpwuid(os.getuid())[0]), + ('username', get_user()), ('revision', ''), ('submit', 'Force Build'), ('branch', branch), diff --git a/rpython/rlib/parsing/lexer.py b/rpython/rlib/parsing/lexer.py --- a/rpython/rlib/parsing/lexer.py +++ b/rpython/rlib/parsing/lexer.py @@ -107,7 +107,7 @@ self.matcher = matcher self.lineno = 0 self.columnno = 0 - + def find_next_token(self): while 1: self.state = 0 @@ -126,8 +126,8 @@ i = ~i stop = self.last_matched_index + 1 assert stop >= 0 - if start == stop: - source_pos = SourcePos(i - 1, self.lineno, self.columnno) + if start == stop: + source_pos = self.token_position_class(i - 1, self.lineno, self.columnno) raise deterministic.LexerError(self.text, self.state, source_pos) source = self.text[start:stop] @@ -147,7 +147,7 @@ else: raise StopIteration return result - source_pos = SourcePos(i - 1, self.lineno, self.columnno) + source_pos = self.token_position_class(i - 1, self.lineno, self.columnno) raise deterministic.LexerError(self.text, self.state, source_pos) def adjust_position(self, token): @@ -158,7 +158,7 @@ self.columnno += len(token) else: self.columnno = token.rfind("\n") - + # def inner_loop(self, i): # while i < len(self.text): # char = self.text[i] @@ -186,10 +186,15 @@ class LexingDFARunner(AbstractLexingDFARunner): def __init__(self, matcher, automaton, text, ignore, eof=False, token_class=None): - if token_class is None: + + if not token_class: self.token_class = Token + self.token_position_class = SourcePos + else: self.token_class = token_class + self.token_position_class = token_class.source_position_class + AbstractLexingDFARunner.__init__(self, matcher, automaton, text, eof) self.ignore = ignore @@ -198,8 +203,10 @@ def make_token(self, index, state, text, eof=False): assert (eof and state == -1) or 0 <= state < len(self.automaton.names) - source_pos = SourcePos(index, self.lineno, self.columnno) + + source_pos = self.token_position_class(index, self.lineno, self.columnno) if eof: return self.token_class("EOF", "EOF", source_pos) + return self.token_class(self.automaton.names[self.last_matched_state], text, source_pos) diff --git a/rpython/rlib/rsocket.py b/rpython/rlib/rsocket.py --- a/rpython/rlib/rsocket.py +++ b/rpython/rlib/rsocket.py @@ -1146,9 +1146,9 @@ address_to_fill=None): # port_or_service is a string, not an int (but try str(port_number)). assert port_or_service is None or isinstance(port_or_service, str) - if _c._MACOSX: - if port_or_service is None or port_or_service == '0': - port_or_service = '00' + if _c._MACOSX and flags & AI_NUMERICSERV and \ + (port_or_service is None or port_or_service == '0'): + port_or_service = '00' hints = lltype.malloc(_c.addrinfo, flavor='raw', zero=True) rffi.setintfield(hints, 'c_ai_family', family) rffi.setintfield(hints, 'c_ai_socktype', socktype) diff --git a/rpython/rlib/test/test_streamio.py b/rpython/rlib/test/test_streamio.py --- a/rpython/rlib/test/test_streamio.py +++ b/rpython/rlib/test/test_streamio.py @@ -1104,6 +1104,21 @@ finally: signal(SIGALRM, SIG_DFL) + def test_append_mode(self): + tfn = str(udir.join('streamio-append-mode')) + fo = streamio.open_file_as_stream # shorthand + x = fo(tfn, 'w') + x.write('abc123') + x.close() + + x = fo(tfn, 'a') + x.seek(0, 0) + x.write('456') + x.close() + x = fo(tfn, 'r') + assert x.read() == 'abc123456' + x.close() + # Speed test diff --git a/rpython/rtyper/module/ll_os_path.py b/rpython/rtyper/module/ll_os_path.py --- a/rpython/rtyper/module/ll_os_path.py +++ b/rpython/rtyper/module/ll_os_path.py @@ -6,26 +6,24 @@ import stat import os -from rpython.tool.staticmethods import ClassMethods # Does a path exist? # This is false for dangling symbolic links. -class BaseOsPath: - __metaclass__ = ClassMethods - +class BaseOsPath(object): + @classmethod def ll_os_path_exists(cls, path): """Test whether a path exists""" try: - st = os.stat(cls.from_rstr_nonnull(path)) + os.stat(cls.from_rstr_nonnull(path)) except OSError: return False return True + @classmethod def ll_os_path_isdir(cls, path): try: st = os.stat(cls.from_rstr_nonnull(path)) except OSError: return False return stat.S_ISDIR(st[0]) - diff --git a/rpython/tool/identity_dict.py b/rpython/tool/identity_dict.py --- a/rpython/tool/identity_dict.py +++ b/rpython/tool/identity_dict.py @@ -3,15 +3,15 @@ except ImportError: idict = None -from UserDict import DictMixin +from collections import MutableMapping -class IdentityDictPurePython(object, DictMixin): +class IdentityDictPurePython(MutableMapping): __slots__ = "_dict _keys".split() def __init__(self): self._dict = {} - self._keys = {} # id(obj) -> obj + self._keys = {} # id(obj) -> obj def __getitem__(self, arg): return self._dict[id(arg)] @@ -24,8 +24,11 @@ del self._keys[id(arg)] del self._dict[id(arg)] - def keys(self): - return self._keys.values() + def __iter__(self): + return self._keys.itervalues() + + def __len__(self): + return len(self._keys) def __contains__(self, arg): return id(arg) in self._dict @@ -37,8 +40,7 @@ return d -class IdentityDictPyPy(object, DictMixin): - __slots__ = ["_dict"] +class IdentityDictPyPy(MutableMapping): def __init__(self): self._dict = idict() @@ -52,8 +54,11 @@ def __delitem__(self, arg): del self._dict[arg] - def keys(self): - return self._dict.keys() + def __iter__(self): + return iter(self._dict.keys()) + + def __len__(self): + return len(self._dict) def __contains__(self, arg): return arg in self._dict @@ -64,8 +69,10 @@ assert len(d) == len(self) return d + def __nonzero__(self): + return bool(self._dict) + if idict is None: identity_dict = IdentityDictPurePython else: identity_dict = IdentityDictPyPy - diff --git a/rpython/tool/staticmethods.py b/rpython/tool/staticmethods.py --- a/rpython/tool/staticmethods.py +++ b/rpython/tool/staticmethods.py @@ -10,11 +10,5 @@ class StaticMethods(AbstractMethods): """ Metaclass that turns plain methods into staticmethods. - """ + """ decorator = staticmethod - -class ClassMethods(AbstractMethods): - """ - Metaclass that turns plain methods into classmethods. - """ - decorator = classmethod _______________________________________________ pypy-commit mailing list pypy-commit@python.org https://mail.python.org/mailman/listinfo/pypy-commit