[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Changes by Ezio Melotti : -- resolution: -> fixed stage: commit review -> committed/rejected status: open -> closed ___ Python tracker ___ ___

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Roundup Robot added the comment: New changeset 9481e062fe26 by Ezio Melotti in branch '2.7': #5057: the peepholer no longer optimizes subscription on unicode literals (e.g. u"foo"[0]) in order to produce compatible pyc files between narrow and wide builds. http://hg.python.org/cpython/rev/9481e

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Serhiy Storchaka added the comment: issue5057-3.diff LGTM. I added debug output in peepholer, ran tests and found that this optimization happened for unicode strings only in test_multibytecodec (where it used deliberately) and test_peepholer. Seems as this is very rare case. -- nosy:

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

STINNER Victor added the comment: I prefer option (1), remove the buggy optimization. Python 3.3 does solve correctly this issue. -- ___ Python tracker ___ __

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Armin Rigo added the comment: Did anyone ever show that this particular detail, which looks like a completely obscure case to me, has any measurable effect on any code whatsoever? Just coming up with numbers, but I'm sure it gives you 5% on the most specially tuned micro-benchmark, and nothi

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Antoine Pitrou added the comment: Option 2) would have my preference. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsu

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Ezio Melotti added the comment: Attached a patch that implements option 1). -- stage: needs patch -> commit review Added file: http://bugs.python.org/file25575/issue5057-3.diff ___ Python tracker __

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Ezio Melotti added the comment: This whole issue doesn't affect 3.3. For 2.7/3.2 there are three possible options: 1) remove constant folding altogether on unicode (this is the solution adopted by PyPy); 2) scan the string up to the index looking for non-BMP chars and disable the constant f

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Armin Rigo added the comment: Sorry to re-open this issue. The following example shows that it was not fully resolved: def f(): return u'\U00023456abcdef'[3] import dis; dis.dis(f) print f() On a wide build it should print 'c' and on a narrow build it should print 'b'.

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Changes by Ezio Melotti : -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Roundup Robot added the comment: New changeset 3cffa2009a92 by Ezio Melotti in branch '2.7': Issue #5057: fix a bug in the peepholer that led to non-portable pyc files between narrow and wide builds while optimizing BINARY_SUBSCR on non-BMP chars (e.g. u"\U00012345"[0]). http://hg.python.org/c

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Ezio Melotti added the comment: Yes. The original report was for 2.6. I will apply the patch on all the 4 branches then. -- ___ Python tracker ___ ___

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Marc-Andre Lemburg added the comment: Ezio Melotti wrote: > > Ezio Melotti added the comment: > > Do you think this should go in 3.1 too? If the problem triggers there as well: Yes. Is the problem also visible on Python 2.7 ? -- title: Unicode-width dependent optimizationleads

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Ezio Melotti added the comment: Do you think this should go in 3.1 too? -- versions: +Python 3.2, Python 3.3 ___ Python tracker ___ __

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Marc-Andre Lemburg added the comment: Ezio Melotti wrote: > > Ezio Melotti added the comment: > > PEP 3147 says[0]: > """ > For backward compatibility, Python will still support pyc-only distributions, > however it will only do so when the pyc file lives in the directory where the > py file

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Ezio Melotti added the comment: PEP 3147 says[0]: """ For backward compatibility, Python will still support pyc-only distributions, however it will only do so when the pyc file lives in the directory where the py file would have been, i.e. not in the __pycache__ directory. pyc file outside of

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Marc-Andre Lemburg added the comment: Ezio Melotti wrote: > > Ezio Melotti added the comment: > > Here's a new patch that checks that newconst is not NULL and that it's a > unicode object. > I added a test for the case where it's NULL. I don't think it's possible to > test the case when ne

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

STINNER Victor added the comment: > ... which does not give the same result in UCS-2 and UCS-4 builds. > As a result, the pyc file is not portable across those builds. Since Python 3.2, the pyc filename contains a tag (u) to indicate wide build (sys.maxunicode==0x10), instead of narrow (sy

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Ezio Melotti added the comment: Here's a new patch that checks that newconst is not NULL and that it's a unicode object. I added a test for the case where it's NULL. I don't think it's possible to test the case when newconst is not unicode though, because unicode subclasses are not literals

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Marc-Andre Lemburg added the comment: Ezio Melotti wrote: > > Ezio Melotti added the comment: > > Are there any cases where v[w] -- where v is a unicode object -- returns a > non-unicode object? There could be: either from subclasses or from buggy code. In any case, macros should only be us

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Changes by Ezio Melotti : -- versions: +Python 3.3 -Python 2.6, Python 3.1 ___ Python tracker ___ ___ Python-bugs-list mailing list Uns

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Ezio Melotti added the comment: Are there any cases where v[w] -- where v is a unicode object -- returns a non-unicode object? -- ___ Python tracker ___

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Marc-Andre Lemburg added the comment: Ezio Melotti wrote: > > Ezio Melotti added the comment: > > The attached patch skips the peepholer optimizations for BINARY_SUBSCR if the > resulting char is a surrogate on narrow builds or a non-bmp char in wide > builds. > Note that this affects the o

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Ezio Melotti added the comment: The attached patch skips the peepholer optimizations for BINARY_SUBSCR if the resulting char is a surrogate on narrow builds or a non-bmp char in wide builds. Note that this affects the optimization of lone surrogates on narrow builds too, but I think it's not w

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Ezio Melotti added the comment: I have the same failure on trunk (narrow build). -- priority: -> normal stage: -> needs patch versions: +Python 3.2 -Python 3.0 ___ Python tracker _

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Antoine Pitrou added the comment: I don't think so. Issue 3297 seems related to the way unicode objects are marshalled/unmarshalled, even if the build settings don't change. ___ Python tracker __

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Amaury Forgeot d'Arc added the comment: Is this related to issue3297 ? -- nosy: +amaury.forgeotdarc ___ Python tracker ___ ___ Python-

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Changes by Ezio Melotti : -- nosy: +ezio.melotti ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyt

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

Changes by Antoine Pitrou : -- versions: +Python 2.6, Python 2.7, Python 3.0 ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue5057] Unicode-width dependent optimization leads to non-portable pyc file

New submission from Antoine Pitrou : The peephole optimizer can optimize indexed access to an unicode constant which does not give the same result in UCS-2 and UCS-4 builds. As a result, the pyc file is not portable across those builds. This is something I witnessed when recompiling in UCS-2 rat