[pypy-commit] pypy default: remove old unused dict

[cfbolz]

Author: Carl Friedrich Bolz-Tereick 
Branch: 
Changeset: r93404:60d1d4bb6379
Date: 2017-12-13 11:17 +0100
http://bitbucket.org/pypy/pypy/changeset/60d1d4bb6379/

Log:remove old unused dict

diff --git a/rpython/jit/metainterp/optimizeopt/optimizer.py 
b/rpython/jit/metainterp/optimizeopt/optimizer.py
--- a/rpython/jit/metainterp/optimizeopt/optimizer.py
+++ b/rpython/jit/metainterp/optimizeopt/optimizer.py
@@ -273,7 +273,6 @@
 self.jitdriver_sd = jitdriver_sd
 self.cpu = metainterp_sd.cpu
 self.interned_refs = self.cpu.ts.new_ref_dict()
-self.interned_ints = {}
 self.resumedata_memo = resume.ResumeDataLoopMemo(metainterp_sd)
 self.pendingfields = None # set temporarily to a list, normally by
   # heap.py, as we're about to generate a guard
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy default: fix typo

[cfbolz]

Author: Carl Friedrich Bolz-Tereick 
Branch: 
Changeset: r93405:36428ed768d3
Date: 2017-12-13 11:57 +0100
http://bitbucket.org/pypy/pypy/changeset/36428ed768d3/

Log:fix typo

diff --git a/pypy/doc/whatsnew-head.rst b/pypy/doc/whatsnew-head.rst
--- a/pypy/doc/whatsnew-head.rst
+++ b/pypy/doc/whatsnew-head.rst
@@ -36,6 +36,6 @@
 
 .. branch: win32-vcvars
 
-.. branch rdict-fast-hash
+.. branch: rdict-fast-hash
 
 Make it possible to declare that the hash function of an r_dict is fast in 
RPython.
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] buildbot default: fix for py3.5 (death by a thousand paper cuts)

[mattip]

Author: Matti Picus 
Branch: 
Changeset: r1050:527820e0350d
Date: 2017-12-13 18:41 +0200
http://bitbucket.org/pypy/buildbot/changeset/527820e0350d/

Log:fix for py3.5 (death by a thousand paper cuts)

diff --git a/bot2/pypybuildbot/builds.py b/bot2/pypybuildbot/builds.py
--- a/bot2/pypybuildbot/builds.py
+++ b/bot2/pypybuildbot/builds.py
@@ -735,7 +735,7 @@
 haltOnFailure=True,
 workdir='.'))
 # copy libpypy-c.so to the expected location within the pypy source 
checkout, if available
-command = 'if [ -e pypy-c/bin/libpypy-c.so ]; then cp -v 
pypy-c/bin/libpypy-c.so build/pypy/goal; fi;'
+command = 'cp -v pypy-c/bin/libpypy*-c.so build/pypy/goal/ || true'
 self.addStep(ShellCmd(
 description="copy libpypy-c.so",
 command=command,
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy regalloc-playground: some more calls supported

[cfbolz]

Author: Carl Friedrich Bolz-Tereick 
Branch: regalloc-playground
Changeset: r93406:5d41d2ca0275
Date: 2017-12-13 22:37 +0100
http://bitbucket.org/pypy/pypy/changeset/5d41d2ca0275/

Log:some more calls supported

diff --git a/rpython/jit/backend/x86/reghint.py 
b/rpython/jit/backend/x86/reghint.py
--- a/rpython/jit/backend/x86/reghint.py
+++ b/rpython/jit/backend/x86/reghint.py
@@ -148,6 +148,20 @@
 consider_call_f = _consider_real_call
 consider_call_n = _consider_real_call
 
+def _consider_call_may_force(self, op, position):
+self._consider_call(op, position, guard_not_forced=True)
+consider_call_may_force_i = _consider_call_may_force
+consider_call_may_force_r = _consider_call_may_force
+consider_call_may_force_f = _consider_call_may_force
+consider_call_may_force_n = _consider_call_may_force
+
+def _consider_call_release_gil(self, op, position):
+# [Const(save_err), func_addr, args...]
+self._consider_call(op, position, guard_not_forced=True, 
first_arg_index=2)
+consider_call_release_gil_i = _consider_call_release_gil
+consider_call_release_gil_f = _consider_call_release_gil
+consider_call_release_gil_n = _consider_call_release_gil
+
 oplist = [X86RegisterHints.not_implemented_op] * rop._LAST
 
 for name, value in X86RegisterHints.__dict__.iteritems():
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy py3.5: Port extra_tests/test_json.py to py3

[rlamy]

Author: Ronan Lamy 
Branch: py3.5
Changeset: r93407:3c274e0f1720
Date: 2017-12-13 21:51 +
http://bitbucket.org/pypy/pypy/changeset/3c274e0f1720/

Log:Port extra_tests/test_json.py to py3

diff --git a/extra_tests/test_json.py b/extra_tests/test_json.py
--- a/extra_tests/test_json.py
+++ b/extra_tests/test_json.py
@@ -7,15 +7,11 @@
 
 def test_no_ensure_ascii():
 assert is_(json.dumps(u"\u1234", ensure_ascii=False), u'"\u1234"')
-assert is_(json.dumps("\xc0", ensure_ascii=False), '"\xc0"')
-with pytest.raises(UnicodeDecodeError) as excinfo:
-json.dumps((u"\u1234", "\xc0"), ensure_ascii=False)
-assert str(excinfo.value).startswith(
-"'ascii' codec can't decode byte 0xc0 ")
-with pytest.raises(UnicodeDecodeError) as excinfo:
-json.dumps(("\xc0", u"\u1234"), ensure_ascii=False)
-assert str(excinfo.value).startswith(
-"'ascii' codec can't decode byte 0xc0 ")
+assert is_(json.dumps(u"\xc0", ensure_ascii=False), u'"\xc0"')
+with pytest.raises(TypeError):
+json.dumps((u"\u1234", b"x"), ensure_ascii=False)
+with pytest.raises(TypeError):
+json.dumps((b"x", u"\u1234"), ensure_ascii=False)
 
 def test_issue2191():
 assert is_(json.dumps(u"xxx", ensure_ascii=False), u'"xxx"')
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy py3.6: The py3k version of the utf32 decoder should not allow lone surrogates.

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: py3.6
Changeset: r93409:8f8a8d212853
Date: 2017-12-13 10:04 +0100
http://bitbucket.org/pypy/pypy/changeset/8f8a8d212853/

Log:The py3k version of the utf32 decoder should not allow lone
surrogates.

diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py
--- a/rpython/rlib/runicode.py
+++ b/rpython/rlib/runicode.py
@@ -489,21 +489,21 @@
 return result, length
 
 def py3k_str_decode_utf_16(s, size, errors, final=True,
-  errorhandler=None):
+   errorhandler=None):
 result, length, byteorder = str_decode_utf_16_helper(s, size, errors, 
final,
  errorhandler, 
"native",
  'utf-16-' + 
BYTEORDER2)
 return result, length
 
 def py3k_str_decode_utf_16_be(s, size, errors, final=True,
- errorhandler=None):
+  errorhandler=None):
 result, length, byteorder = str_decode_utf_16_helper(s, size, errors, 
final,
  errorhandler, "big",
  'utf-16-be')
 return result, length
 
 def py3k_str_decode_utf_16_le(s, size, errors, final=True,
- errorhandler=None):
+  errorhandler=None):
 result, length, byteorder = str_decode_utf_16_helper(s, size, errors, 
final,
  errorhandler, 
"little",
  'utf-16-le')
@@ -714,41 +714,41 @@
 
 def str_decode_utf_32(s, size, errors, final=True,
   errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"native")
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "native")
 return result, length
 
 def str_decode_utf_32_be(s, size, errors, final=True,
  errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, "big")
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "big")
 return result, length
 
 def str_decode_utf_32_le(s, size, errors, final=True,
  errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"little")
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "little")
 return result, length
 
 def py3k_str_decode_utf_32(s, size, errors, final=True,
errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"native",
- 'utf-32-' + 
BYTEORDER2)
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "native",
+'utf-32-' + BYTEORDER2, allow_surrogates=False)
 return result, length
 
 def py3k_str_decode_utf_32_be(s, size, errors, final=True,
   errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, "big",
- 'utf-32-be')
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "big",
+'utf-32-be', allow_surrogates=False)
 return result, length
 
 def py3k_str_decode_utf_32_le(s, size, errors, final=True,
   errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"little",
- 'utf-32-le')
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "little",
+'utf-32-le', allow_surrogates=False)
 return result, length
 
 BOM32_DIRECT  = intmask(0xFEFF)
@@ -757,7 +757,8 @@
 def str_decode_utf_32_helper(s, size, errors, final=True,
  errorhandler=None,
  byteorder="native",
- public_encoding_name='utf32'):
+ public_encoding_name='utf32',
+ allow_surrogates=True):
 if errorhandler is None:
 errorhandler = default_unicode

[pypy-commit] pypy py3.6: Applevel test for the utf32 surrogates.

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: py3.6
Changeset: r93410:c99a2002f6fc
Date: 2017-12-13 10:05 +0100
http://bitbucket.org/pypy/pypy/changeset/c99a2002f6fc/

Log:Applevel test for the utf32 surrogates.

diff --git a/pypy/module/_codecs/test/test_codecs.py 
b/pypy/module/_codecs/test/test_codecs.py
--- a/pypy/module/_codecs/test/test_codecs.py
+++ b/pypy/module/_codecs/test/test_codecs.py
@@ -772,6 +772,18 @@
 '[]'.encode(encoding))
 assert (u'[\udc80]'.encode(encoding, "replace") ==
 '[?]'.encode(encoding))
+for encoding, ill_surrogate in [('utf-8', b'\xed\xb2\x80'),
+('utf-16-le', b'\x80\xdc'),
+('utf-16-be', b'\xdc\x80'),
+('utf-32-le', b'\x80\xdc\x00\x00'),
+('utf-32-be', b'\x00\x00\xdc\x80')]:
+print(encoding)
+before, after = "[", "]"
+before_sequence = before.encode(encoding)
+after_sequence = after.encode(encoding)
+test_string = before + "\uDC80" + after
+test_sequence = before_sequence + ill_surrogate + after_sequence
+raises(UnicodeDecodeError, test_sequence.decode, encoding)
 
 def test_charmap_encode(self):
 assert 'xxx'.encode('charmap') == b'xxx'
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy py3.6: Fix struct test.

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: py3.6
Changeset: r93408:24d2c052c1dd
Date: 2017-12-12 23:21 +0100
http://bitbucket.org/pypy/pypy/changeset/24d2c052c1dd/

Log:Fix struct test.

diff --git a/pypy/module/struct/test/test_struct.py 
b/pypy/module/struct/test/test_struct.py
--- a/pypy/module/struct/test/test_struct.py
+++ b/pypy/module/struct/test/test_struct.py
@@ -49,7 +49,7 @@
 assert calcsize('=Q') == 8
 assert calcsize('d') == 8
-assert calcsize('https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy default: Add support for half floats in the RPython rstruct module.

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: 
Changeset: r93412:ca5586010ac3
Date: 2017-12-11 20:29 +0100
http://bitbucket.org/pypy/pypy/changeset/ca5586010ac3/

Log:Add support for half floats in the RPython rstruct module.

diff --git a/rpython/rlib/rstruct/standardfmttable.py 
b/rpython/rlib/rstruct/standardfmttable.py
--- a/rpython/rlib/rstruct/standardfmttable.py
+++ b/rpython/rlib/rstruct/standardfmttable.py
@@ -105,6 +105,18 @@
 _pack_string(fmtiter, string, count-1)
 
 
+def pack_halffloat(fmtiter):
+size = 2
+fl = fmtiter.accept_float_arg()
+try:
+result = ieee.pack_float(fmtiter.wbuf, fmtiter.pos,
+ fl, size, fmtiter.bigendian)
+except OverflowError:
+raise StructOverflowError("float too large for format 'e'")
+else:
+fmtiter.advance(size)
+return result
+
 def make_float_packer(TYPE):
 size = rffi.sizeof(TYPE)
 def packer(fmtiter):
@@ -247,6 +259,11 @@
 end = count
 fmtiter.appendobj(data[1:end])
 
+@specialize.argtype(0)
+def unpack_halffloat(fmtiter):
+data = fmtiter.read(2)
+fmtiter.appendobj(ieee.unpack_float(data, fmtiter.bigendian))
+
 def make_ieee_unpacker(TYPE):
 @specialize.argtype(0)
 def unpack_ieee(fmtiter):
@@ -374,6 +391,8 @@
   'needcount' : True },
 'p':{ 'size' : 1, 'pack' : pack_pascal, 'unpack' : unpack_pascal,
   'needcount' : True },
+'e':{ 'size' : 2, 'pack' : pack_halffloat,
+'unpack' : unpack_halffloat},
 'f':{ 'size' : 4, 'pack' : make_float_packer(rffi.FLOAT),
 'unpack' : unpack_float},
 'd':{ 'size' : 8, 'pack' : make_float_packer(rffi.DOUBLE),
diff --git a/rpython/rlib/rstruct/test/test_pack.py 
b/rpython/rlib/rstruct/test/test_pack.py
--- a/rpython/rlib/rstruct/test/test_pack.py
+++ b/rpython/rlib/rstruct/test/test_pack.py
@@ -138,6 +138,19 @@
 self.check('f', 123.456)
 self.check('d', 123.456789)
 
+def test_pack_halffloat(self):
+if self.fmttable is nativefmttable.native_fmttable:
+# Host Python cannot handle half floats.
+return
+size = 2
+wbuf = MutableStringBuffer(size)
+self.mypack_into('e', wbuf, 6.5e+04)
+got = wbuf.finish()
+if self.bigendian:
+assert got == b'\x7b\xef'
+else:
+assert got == b'\xef\x7b'
+
 def test_float_overflow(self):
 if self.fmt_prefix == '@':
 # native packing, no overflow
diff --git a/rpython/rlib/rstruct/test/test_runpack.py 
b/rpython/rlib/rstruct/test/test_runpack.py
--- a/rpython/rlib/rstruct/test/test_runpack.py
+++ b/rpython/rlib/rstruct/test/test_runpack.py
@@ -78,6 +78,10 @@
 assert f != 12.34 # precision lost
 assert abs(f - 12.34) < 1E-6
 
+def test_unpack_halffloat(self):
+assert runpack(">e", b"\x7b\xef") == 64992.0
+assert runpack("https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy default: The py3k version of the utf32 decoder should not allow lone surrogates.

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: 
Changeset: r93411:33178f62171f
Date: 2017-12-13 10:04 +0100
http://bitbucket.org/pypy/pypy/changeset/33178f62171f/

Log:The py3k version of the utf32 decoder should not allow lone
surrogates.

diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py
--- a/rpython/rlib/runicode.py
+++ b/rpython/rlib/runicode.py
@@ -489,21 +489,21 @@
 return result, length
 
 def py3k_str_decode_utf_16(s, size, errors, final=True,
-  errorhandler=None):
+   errorhandler=None):
 result, length, byteorder = str_decode_utf_16_helper(s, size, errors, 
final,
  errorhandler, 
"native",
  'utf-16-' + 
BYTEORDER2)
 return result, length
 
 def py3k_str_decode_utf_16_be(s, size, errors, final=True,
- errorhandler=None):
+  errorhandler=None):
 result, length, byteorder = str_decode_utf_16_helper(s, size, errors, 
final,
  errorhandler, "big",
  'utf-16-be')
 return result, length
 
 def py3k_str_decode_utf_16_le(s, size, errors, final=True,
- errorhandler=None):
+  errorhandler=None):
 result, length, byteorder = str_decode_utf_16_helper(s, size, errors, 
final,
  errorhandler, 
"little",
  'utf-16-le')
@@ -714,41 +714,41 @@
 
 def str_decode_utf_32(s, size, errors, final=True,
   errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"native")
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "native")
 return result, length
 
 def str_decode_utf_32_be(s, size, errors, final=True,
  errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, "big")
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "big")
 return result, length
 
 def str_decode_utf_32_le(s, size, errors, final=True,
  errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"little")
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "little")
 return result, length
 
 def py3k_str_decode_utf_32(s, size, errors, final=True,
errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"native",
- 'utf-32-' + 
BYTEORDER2)
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "native",
+'utf-32-' + BYTEORDER2, allow_surrogates=False)
 return result, length
 
 def py3k_str_decode_utf_32_be(s, size, errors, final=True,
   errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, "big",
- 'utf-32-be')
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "big",
+'utf-32-be', allow_surrogates=False)
 return result, length
 
 def py3k_str_decode_utf_32_le(s, size, errors, final=True,
   errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"little",
- 'utf-32-le')
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "little",
+'utf-32-le', allow_surrogates=False)
 return result, length
 
 BOM32_DIRECT  = intmask(0xFEFF)
@@ -757,7 +757,8 @@
 def str_decode_utf_32_helper(s, size, errors, final=True,
  errorhandler=None,
  byteorder="native",
- public_encoding_name='utf32'):
+ public_encoding_name='utf32',
+ allow_surrogates=True):
 if errorhandler is None:
 errorhandler = default_unicode_erro

[pypy-commit] pypy py3.5: Applevel test for the utf32 surrogates.

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: py3.5
Changeset: r93414:c9a148ecf262
Date: 2017-12-13 10:05 +0100
http://bitbucket.org/pypy/pypy/changeset/c9a148ecf262/

Log:Applevel test for the utf32 surrogates.

diff --git a/pypy/module/_codecs/test/test_codecs.py 
b/pypy/module/_codecs/test/test_codecs.py
--- a/pypy/module/_codecs/test/test_codecs.py
+++ b/pypy/module/_codecs/test/test_codecs.py
@@ -778,6 +778,18 @@
 '[]'.encode(encoding))
 assert (u'[\udc80]'.encode(encoding, "replace") ==
 '[?]'.encode(encoding))
+for encoding, ill_surrogate in [('utf-8', b'\xed\xb2\x80'),
+('utf-16-le', b'\x80\xdc'),
+('utf-16-be', b'\xdc\x80'),
+('utf-32-le', b'\x80\xdc\x00\x00'),
+('utf-32-be', b'\x00\x00\xdc\x80')]:
+print(encoding)
+before, after = "[", "]"
+before_sequence = before.encode(encoding)
+after_sequence = after.encode(encoding)
+test_string = before + "\uDC80" + after
+test_sequence = before_sequence + ill_surrogate + after_sequence
+raises(UnicodeDecodeError, test_sequence.decode, encoding)
 
 def test_charmap_encode(self):
 assert 'xxx'.encode('charmap') == b'xxx'
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy py3.5: The py3k version of the utf32 decoder should not allow lone surrogates.

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: py3.5
Changeset: r93413:b0267eee69d8
Date: 2017-12-13 10:04 +0100
http://bitbucket.org/pypy/pypy/changeset/b0267eee69d8/

Log:The py3k version of the utf32 decoder should not allow lone
surrogates.

diff --git a/rpython/rlib/runicode.py b/rpython/rlib/runicode.py
--- a/rpython/rlib/runicode.py
+++ b/rpython/rlib/runicode.py
@@ -489,21 +489,21 @@
 return result, length
 
 def py3k_str_decode_utf_16(s, size, errors, final=True,
-  errorhandler=None):
+   errorhandler=None):
 result, length, byteorder = str_decode_utf_16_helper(s, size, errors, 
final,
  errorhandler, 
"native",
  'utf-16-' + 
BYTEORDER2)
 return result, length
 
 def py3k_str_decode_utf_16_be(s, size, errors, final=True,
- errorhandler=None):
+  errorhandler=None):
 result, length, byteorder = str_decode_utf_16_helper(s, size, errors, 
final,
  errorhandler, "big",
  'utf-16-be')
 return result, length
 
 def py3k_str_decode_utf_16_le(s, size, errors, final=True,
- errorhandler=None):
+  errorhandler=None):
 result, length, byteorder = str_decode_utf_16_helper(s, size, errors, 
final,
  errorhandler, 
"little",
  'utf-16-le')
@@ -714,41 +714,41 @@
 
 def str_decode_utf_32(s, size, errors, final=True,
   errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"native")
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "native")
 return result, length
 
 def str_decode_utf_32_be(s, size, errors, final=True,
  errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, "big")
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "big")
 return result, length
 
 def str_decode_utf_32_le(s, size, errors, final=True,
  errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"little")
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "little")
 return result, length
 
 def py3k_str_decode_utf_32(s, size, errors, final=True,
errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"native",
- 'utf-32-' + 
BYTEORDER2)
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "native",
+'utf-32-' + BYTEORDER2, allow_surrogates=False)
 return result, length
 
 def py3k_str_decode_utf_32_be(s, size, errors, final=True,
   errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, "big",
- 'utf-32-be')
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "big",
+'utf-32-be', allow_surrogates=False)
 return result, length
 
 def py3k_str_decode_utf_32_le(s, size, errors, final=True,
   errorhandler=None):
-result, length, byteorder = str_decode_utf_32_helper(s, size, errors, 
final,
- errorhandler, 
"little",
- 'utf-32-le')
+result, length, byteorder = str_decode_utf_32_helper(
+s, size, errors, final, errorhandler, "little",
+'utf-32-le', allow_surrogates=False)
 return result, length
 
 BOM32_DIRECT  = intmask(0xFEFF)
@@ -757,7 +757,8 @@
 def str_decode_utf_32_helper(s, size, errors, final=True,
  errorhandler=None,
  byteorder="native",
- public_encoding_name='utf32'):
+ public_encoding_name='utf32',
+ allow_surrogates=True):
 if errorhandler is None:
 errorhandler = default_unicode

[pypy-commit] pypy default: mmap.write() return the number of bytes written: RPython part

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: 
Changeset: r93417:5ef9bb870cd2
Date: 2017-12-13 23:50 +0100
http://bitbucket.org/pypy/pypy/changeset/5ef9bb870cd2/

Log:mmap.write() return the number of bytes written: RPython part

diff --git a/rpython/rlib/rmmap.py b/rpython/rlib/rmmap.py
--- a/rpython/rlib/rmmap.py
+++ b/rpython/rlib/rmmap.py
@@ -492,6 +492,7 @@
 
 self.setslice(start, data)
 self.pos = start + data_len
+return data_len
 
 def write_byte(self, byte):
 if len(byte) != 1:
diff --git a/rpython/rlib/test/test_rmmap.py b/rpython/rlib/test/test_rmmap.py
--- a/rpython/rlib/test/test_rmmap.py
+++ b/rpython/rlib/test/test_rmmap.py
@@ -258,7 +258,7 @@
 f.flush()
 def func(no):
 m = mmap.mmap(no, 6, access=mmap.ACCESS_WRITE)
-m.write("ciao\n")
+assert m.write("ciao\n") == 5
 m.seek(0)
 assert m.read(6) == "ciao\nr"
 m.close()
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy py3.6: mmap.write() return the number of bytes written: AppLevel part

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: py3.6
Changeset: r93416:98fb1b0c5570
Date: 2017-12-13 23:52 +0100
http://bitbucket.org/pypy/pypy/changeset/98fb1b0c5570/

Log:mmap.write() return the number of bytes written: AppLevel part

diff --git a/pypy/module/mmap/interp_mmap.py b/pypy/module/mmap/interp_mmap.py
--- a/pypy/module/mmap/interp_mmap.py
+++ b/pypy/module/mmap/interp_mmap.py
@@ -103,7 +103,7 @@
 data = self.space.charbuf_w(w_data)
 self.check_writeable()
 try:
-self.mmap.write(data)
+return self.space.newint(self.mmap.write(data))
 except RValueError as v:
 raise mmap_error(self.space, v)
 
diff --git a/pypy/module/mmap/test/test_mmap.py 
b/pypy/module/mmap/test/test_mmap.py
--- a/pypy/module/mmap/test/test_mmap.py
+++ b/pypy/module/mmap/test/test_mmap.py
@@ -268,7 +268,7 @@
 m = mmap.mmap(f.fileno(), 6, access=mmap.ACCESS_WRITE)
 raises(TypeError, m.write, 123)
 raises(ValueError, m.write, b"c"*10)
-m.write(b"ciao\n")
+assert m.write(b"ciao\n") == 5
 m.seek(0)
 assert m.read(6) == b"ciao\nr"
 m.close()
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy py3.6: mmap.write() return the number of bytes written: RPython part

[amauryfa]

Author: Amaury Forgeot d'Arc 
Branch: py3.6
Changeset: r93415:09b70b8c9aba
Date: 2017-12-13 23:50 +0100
http://bitbucket.org/pypy/pypy/changeset/09b70b8c9aba/

Log:mmap.write() return the number of bytes written: RPython part

diff --git a/rpython/rlib/rmmap.py b/rpython/rlib/rmmap.py
--- a/rpython/rlib/rmmap.py
+++ b/rpython/rlib/rmmap.py
@@ -492,6 +492,7 @@
 
 self.setslice(start, data)
 self.pos = start + data_len
+return data_len
 
 def write_byte(self, byte):
 if len(byte) != 1:
diff --git a/rpython/rlib/test/test_rmmap.py b/rpython/rlib/test/test_rmmap.py
--- a/rpython/rlib/test/test_rmmap.py
+++ b/rpython/rlib/test/test_rmmap.py
@@ -258,7 +258,7 @@
 f.flush()
 def func(no):
 m = mmap.mmap(no, 6, access=mmap.ACCESS_WRITE)
-m.write("ciao\n")
+assert m.write("ciao\n") == 5
 m.seek(0)
 assert m.read(6) == "ciao\nr"
 m.close()
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy py3.5: Port b0267eee69d8 to unicodehelper and fix it

[rlamy]

Author: Ronan Lamy 
Branch: py3.5
Changeset: r93418:badb71ed332d
Date: 2017-12-14 02:22 +
http://bitbucket.org/pypy/pypy/changeset/badb71ed332d/

Log:Port b0267eee69d8 to unicodehelper and fix it

diff --git a/pypy/interpreter/test/test_unicodehelper.py 
b/pypy/interpreter/test/test_unicodehelper.py
--- a/pypy/interpreter/test/test_unicodehelper.py
+++ b/pypy/interpreter/test/test_unicodehelper.py
@@ -2,7 +2,7 @@
 import pytest
 import struct
 from pypy.interpreter.unicodehelper import (
-encode_utf8, decode_utf8, unicode_encode_utf_32_be)
+encode_utf8, decode_utf8, unicode_encode_utf_32_be, str_decode_utf_32_be)
 from pypy.interpreter.unicodehelper import encode_utf8sp, decode_utf8sp
 
 
@@ -90,3 +90,6 @@
 assert replace_with(u'rep', None) == u''.encode('utf-32-be')
 assert (replace_with(None, '\xca\xfe\xca\xfe') ==
 '\x00\x00\x00<\xca\xfe\xca\xfe\x00\x00\x00>')
+
+with pytest.raises(UnicodeDecodeError):
+str_decode_utf_32_be(b"\x00\x00\xdc\x80", 4, None)
diff --git a/pypy/interpreter/unicodehelper.py 
b/pypy/interpreter/unicodehelper.py
--- a/pypy/interpreter/unicodehelper.py
+++ b/pypy/interpreter/unicodehelper.py
@@ -172,19 +172,22 @@
 def str_decode_utf_32(s, size, errors, final=True,
errorhandler=None):
 result, length, byteorder = str_decode_utf_32_helper(
-s, size, errors, final, errorhandler, "native", 'utf-32-' + BYTEORDER2)
+s, size, errors, final, errorhandler, "native", 'utf-32-' + BYTEORDER2,
+allow_surrogates=False)
 return result, length
 
 def str_decode_utf_32_be(s, size, errors, final=True,
   errorhandler=None):
 result, length, byteorder = str_decode_utf_32_helper(
-s, size, errors, final, errorhandler, "big", 'utf-32-be')
+s, size, errors, final, errorhandler, "big", 'utf-32-be',
+allow_surrogates=False)
 return result, length
 
 def str_decode_utf_32_le(s, size, errors, final=True,
   errorhandler=None):
 result, length, byteorder = str_decode_utf_32_helper(
-s, size, errors, final, errorhandler, "little", 'utf-32-le')
+s, size, errors, final, errorhandler, "little", 'utf-32-le',
+allow_surrogates=False)
 return result, length
 
 BOM32_DIRECT = intmask(0xFEFF)
@@ -193,7 +196,8 @@
 def str_decode_utf_32_helper(s, size, errors, final=True,
  errorhandler=None,
  byteorder="native",
- public_encoding_name='utf32'):
+ public_encoding_name='utf32',
+ allow_surrogates=True):
 if errorhandler is None:
 errorhandler = default_unicode_error_decode
 bo = 0
@@ -256,10 +260,17 @@
 continue
 ch = ((ord(s[pos + iorder[3]]) << 24) | (ord(s[pos + iorder[2]]) << 
16) |
 (ord(s[pos + iorder[1]]) << 8) | ord(s[pos + iorder[0]]))
-if ch >= 0x11:
+if not allow_surrogates and 0xD800 <= ch <= 0xDFFF:
+r, pos = errorhandler(errors, public_encoding_name,
+  "code point in surrogate code point "
+  "range(0xd800, 0xe000)",
+  s, pos, pos + 4)
+result.append(r)
+continue
+elif ch >= 0x11:
 r, pos = errorhandler(errors, public_encoding_name,
   "codepoint not in range(0x11)",
-  s, pos, len(s))
+  s, pos, pos + 4)
 result.append(r)
 continue
 
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy py3.5: Adapt test_particularly_evil_undecodable to pypy

[rlamy]

Author: Ronan Lamy 
Branch: py3.5
Changeset: r93419:7b810b0bf663
Date: 2017-12-14 02:47 +
http://bitbucket.org/pypy/pypy/changeset/7b810b0bf663/

Log:Adapt test_particularly_evil_undecodable to pypy

diff --git a/lib-python/3/test/test_compile.py 
b/lib-python/3/test/test_compile.py
--- a/lib-python/3/test/test_compile.py
+++ b/lib-python/3/test/test_compile.py
@@ -524,7 +524,8 @@
 with open(fn, "wb") as fp:
 fp.write(src)
 res = script_helper.run_python_until_end(fn)[0]
-self.assertIn(b"Non-UTF-8", res.err)
+# PyPy change: we have a different error here
+self.assertIn(b"SyntaxError", res.err)
 
 def test_yet_more_evil_still_undecodable(self):
 # Issue #25388
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit

[pypy-commit] pypy py3.5: Add failing test that explains the failure in CPython's test_unencodable_filename()

[rlamy]

Author: Ronan Lamy 
Branch: py3.5
Changeset: r93420:e170f5f30e32
Date: 2017-12-14 04:54 +
http://bitbucket.org/pypy/pypy/changeset/e170f5f30e32/

Log:Add failing test that explains the failure in CPython's
test_unencodable_filename()

diff --git a/pypy/objspace/std/test/test_celldict.py 
b/pypy/objspace/std/test/test_celldict.py
--- a/pypy/objspace/std/test/test_celldict.py
+++ b/pypy/objspace/std/test/test_celldict.py
@@ -76,6 +76,8 @@
 def test_key_not_there(self):
 d = type(__builtins__)("abc").__dict__
 raises(KeyError, "d['def']")
+assert 42 not in d
+assert u"\udc00" not in d
 
 def test_fallback_evil_key(self):
 class F(object):
___
pypy-commit mailing list
pypy-commit@python.org
https://mail.python.org/mailman/listinfo/pypy-commit