On 03.11.2021 01:21, Chris Angelico wrote: > On Wed, Nov 3, 2021 at 11:09 AM Steven D'Aprano <st...@pearwood.info> wrote: >> >> On Wed, Nov 03, 2021 at 03:03:54AM +1100, Chris Angelico wrote: >>> On Wed, Nov 3, 2021 at 1:06 AM Petr Viktorin <encu...@gmail.com> wrote: >>>> Let me know if it's clear in the newest version, with this note: >>>> >>>>> Here, ``encoding: unicode_escape`` in the initial comment is an encoding >>>>> declaration. The ``unicode_escape`` encoding instructs Python to treat >>>>> ``\u0027`` as a single quote (which can start/end a string), ``\u002c`` as >>>>> a comma (punctuator), etc. >>>> >>> >>> Huh. Is that level of generality actually still needed? Can Python >>> deprecate all but a small handful of encodings? >> >> To be clear, are you proposing to deprecate the encodings *completely* >> or just as the source code encoding? > > Only source code encodings. Obviously we still need to be able to cope > with all manner of *data*, but Python source code shouldn't need to be > in bizarre, weird encodings. > > (Honestly, I'd love to just require that Python source code be UTF-8, > but that would probably cause problems, so mandating that it be one of > a small set of encodings would be a safer option.)
Most Python code will be written in UTF-8 going forward, but there's still a lot of code out there in other encodings. Limiting this to some reduced set doesn't really make sense, since it's not clear where to draw the line. Coming back to the thread topic, many of the Unicode security considerations don't apply to non-Unicode encodings, since those usually don't support e.g. changing the bidi direction within a stream of text or other interesting features you have in Unicode such as combining code points, invisible (space) code points, font rendering hint code points, etc. So in a sense, those non-Unicode encodings are safer than using UTF-8 :-) Please also note that most character lookalikes are not encoding issues, but instead font issues, which then result in the characters looking similar. There are fonts which are designed to avoid this and it's no surprise that source code fonts typically do make e.g. 0 and O, as well as 1 and l look sufficiently different to be able to notice the difference. Things get a lot harder when dealing with combining characters, since it's not always easy to spot the added diacritics, e.g. try this: >>> print ('a\u0348bc') # strong articulation a͈bc >>> print ('a\u034Fbc') # combining grapheme joiner a͏bc The latter is only "visible" in the unicode_escape encoding: >>> print ('a\u034Fbc'.encode('unicode_escape')) b'a\\u034fbc' Projects wanting to limit code encoding settings, disallow using bidi markers and other special code points in source code, can easily do this via e.g. pre-commit hooks, special editor settings, code linters or security scanners. I don't think limiting the source code encoding is the right approach to making code more secure. Instead, tooling has to be used to detect potentially malicious code points in code. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 03 2021) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MBWBY47ILPL3E6733W4XAZXF2M6RKFH6/ Code of Conduct: http://python.org/psf/codeofconduct/