Nick Coghlan added the comment: Based on the latest round of bytes handling discussions on python-dev, I came up with this updated proposal:
# Constant in the string module (akin to string.ascii_letters et al) escaped_surrogates = bytes(range(128, 256)).decode('ascii', errors='surrogateescape') # Helper to ensure a string contains no escaped surrogates # This allows it to be safely encoded without surrogateescape _match_surrogates = re.compile('[{}]'.format(escaped_surrogates)) def clean(s, repl='\ufffd'): return _match_surrogates.sub(repl, s) # Helper to redecode a string that was decoded incorrectly # For example, WSGI strings are passed from the server to the # framework as latin-1 by default and may need to be redecoded def redecode(s, encoding, errors='strict', old_encoding='latin-1', old_errors='strict'): return s.encode(old_encoding, old_errors).decode(encoding, errors) In addition to the concrete use cases David describes, I think these will also serve a useful documentation purpose, in highlighting the two main mechanisms for "smuggling" raw binary data through text APIs (i.e. surrogate escapes and latin-1 decoding). ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue18814> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com