[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Roundup Robot added the comment: New changeset cf6e782a7f94 by Serhiy Storchaka in branch '2.7': Issue #19543: Emit deprecation warning for known non-text encodings. https://hg.python.org/cpython/rev/cf6e782a7f94 -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Georg Brandl added the comment: Both this backport and #20404 will make it into 3.3.5rc2. -- ___ Python tracker ___ ___ Python-bugs-li

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Georg Brandl added the comment: This is fixed now, right? -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ P

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Roundup Robot added the comment: New changeset 9975f827eefd by Serhiy Storchaka in branch '3.3': Fix typo (issue #19619). http://hg.python.org/cpython/rev/9975f827eefd -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: I think that issue20404 should be backported too. It is common to wrap socket stream with io.TextIOWrapper, and socket.makefile() returns text stream by default. -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Roundup Robot added the comment: New changeset 95386bbf9471 by Serhiy Storchaka in branch '3.3': Issue #19619: Blacklist non-text codecs in method API http://hg.python.org/cpython/rev/95386bbf9471 -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Backporting just the encode/decode changes sounds reasonable to me, since open() isn't likely to be a DoS vector in this particular case. -- ___ Python tracker _

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Georg Brandl added the comment: That looks ok to me. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:/

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: Sorry, I missed the patch. -- Added file: http://bugs.python.org/file34213/issue19619_blacklist_transforms_py33.patch ___ Python tracker ___

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Changes by Serhiy Storchaka : -- nosy: +georg.brandl priority: high -> release blocker stage: needs patch -> patch review versions: -Python 3.4 ___ Python tracker ___ __

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: I don't think that adding underscored parameter to public API is best solution, but we need the fix for 3.3. So here is a patch for backporting d68df99d7a57 to 3.3. -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: Is a bug which allows easily make DDOS attacks important enough? Every Python HTTP server, mail or news client is affected. -- ___ Python tracker ___

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: > It isn't fixed in 3.3 yet. I'm not sure that the issue should be fixed in 3.3 because the patch is quite large. Do you consider that the bug is important enough? -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: As Serhiy noted, 3.3 is still affected. On the other hand, the approach I used for 3.4 is a pretty invasive fix, so I'm not sure it's a good idea to implement something like that in a maintenance release. -- ___ Python

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: It isn't fixed in 3.3 yet. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: Is it still something to do? The initial issue looks to be fixed. You may open new issue if you see more things to do? -- nosy: +haypo ___ Python tracker _

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Roundup Robot added the comment: New changeset f3ec00d2b75e by Nick Coghlan in branch 'default': Close #20404: blacklist non-text encodings in io.TextIOWrapper http://hg.python.org/cpython/rev/f3ec00d2b75e -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Changes by Martin Panter : -- nosy: +vadmium ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Changes by Jakub Wilk : -- nosy: +jwilk ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Roundup Robot added the comment: New changeset 0e10367c88ce by Zachary Ware in branch 'default': Issue19619: skip zlib error test when zlib not available http://hg.python.org/cpython/rev/0e10367c88ce -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: If people don't pay attention on explicit warning not to use certain parameters, is the lack of documentation will stop them? -- ___ Python tracker _

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: I was planning to fix pydoc to not show private keyword only arguments. -- ___ Python tracker ___ ___

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Changes by Serhiy Storchaka : -- versions: +Python 3.4 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: What about my comments in msg203841? See http://comments.gmane.org/gmane.comp.python.devel/143943 for example how hard to get rid of private arguments in public functions. -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Unassigning this one - I don't think the solution we used for 3.4 is appropriate in a maintenance release, but I'm not sure how else to resolve it. -- assignee: ncoghlan -> priority: critical -> high ___ Python tracke

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Changes by STINNER Victor : -- nosy: -haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: The examples from the 3.4 What's New may make it clearer why the exception wrapping in the codec machinery is much broader in scope that just detecting output type errors (even though handling that case was the original motivation for the idea): >>> import

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Just noting the exact list of codecs that currently bypass the full codec machinery and go direct to the C implementation by normalising the codec name (which includes forcing to lowercase) and then using strcmp to check against a specific set of known encodings

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: > Regarding UTF-8 et al, the existing shortcuts in unicodeobject.c already > bypass the full codec machinery, and that includes the exception wrapping > on failures. There are still platforms using locale encodings different than ascii, latin1 or utf8. For examp

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Yay, thanks Victor! Regarding UTF-8 et al, the existing shortcuts in unicodeobject.c already bypass the full codec machinery, and that includes the exception wrapping on failures. -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: With blacklisted transform codecs, I'm fine with the idea of restoring codecs aliases for transform codecs in Python 3.4. Go ahead Nick. -- For Python 3.5, a better solution should be found to declare transform codecs. And I had like to also add transform()/u

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: > Victor, please accept the fact that the codec sub-system in Python > doesn't only have the Unicode implementation as target. It's > true that most codecs were added for the Unicode implementation, > but I deliberately designed the system to be open for other >

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: > * I think you should add the flag to the constructor of the CodecInfo >tuple subclass and then set this in the resp. codecs. The extra >constructor class method looks too much like a hack and is >not needed. I think that adding new keyword argum

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Marc-Andre Lemburg added the comment: Victor, please accept the fact that the codec sub-system in Python doesn't only have the Unicode implementation as target. It's true that most codecs were added for the Unicode implementation, but I deliberately designed the system to be open for other purpos

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Victor, the general purpose codec infrastructure is more than a decade old, and supported in both Python 2 and Python 3, so you're not going to get it deprecated in the last few days before the 3.4 feature freeze. You've already succeeded in inconveniencing affecte

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: > No, we can't remove wrap_codec_error, as it is still needed in order > to cover direct use of the codec machinery and to handle non-text > codecs registered by third parties (the tests cover both these cases) I searched on the WWW for third party codecs, I onl

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: No, we can't remove wrap_codec_error, as it is still needed in order to cover direct use of the codec machinery and to handle non-text codecs registered by third parties (the tests cover both these cases). -- ___ Python

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: Can we now remove wrap_codec_error()? And maybe more changes which were done to workaround issues with non-Unicode codecs. bytes.decode/str.encode should no more raise a TypeError with codecs of the Python standard library. Related changesets: changeset: 8

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: The initial concern was a denial of service. Nick, can you backport your changeset to Python 3.3? > This issue should avoid the denial of service attack when a compression codec > is used, see: > https://mail.python.org/pipermail/python-dev/2013-November/13018

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Roundup Robot added the comment: New changeset 1e3b8601b098 by Nick Coghlan in branch 'default': Issue #19619: Update What's New for codec blacklist http://hg.python.org/cpython/rev/1e3b8601b098 -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: 3.4 has been dealt with. A similar functional patch should work in 3.3, but the test changes will be different (since those test cases don't exist yet in 3.3, and the wrapping ones aren't relevant at all). -- versions: -Python 3.4 _

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Roundup Robot added the comment: New changeset d68df99d7a57 by Nick Coghlan in branch 'default': Issue #19619: Blacklist non-text codecs in method API http://hg.python.org/cpython/rev/d68df99d7a57 -- nosy: +python-dev ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Well, that patch minus the debugging print that was still in there :) -- ___ Python tracker ___ ___ Py

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: It turns out the codec cache and the refleak hunting mechanism in regrtest *really* don't like each other if you use closures to implement your test codecs :) Anyway, the attached patch tweaks the tests to handle refleak hunting (and running the refleak hunter

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: 2013/11/22 Marc-Andre Lemburg : > Anyway: Thanks to Nick for implementing this, to Serhiy for the black > list idea and Victor for the attribute idea :-) In fact, the attribute idea comes from you :-) http://bugs.python.org/issue7475#msg96374 -- __

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Marc-Andre Lemburg added the comment: On 22.11.2013 12:43, STINNER Victor wrote: > > STINNER Victor added the comment: > > If _is_text_encoding may change in Python 3.5, you should add a comment to > warn users to not use it and explain its purpose, maybe with a reference to > this issue. +1

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: If _is_text_encoding may change in Python 3.5, you should add a comment to warn users to not use it and explain its purpose, maybe with a reference to this issue. -- We have talking about a very few codecs: * base64: bytes => bytes * bz2: bytes => bytes * he

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: - switched the private flag from being set in a class method to using a keyword only parameter to __init__ - updated the codecs.h comment as MAL suggested -- Added file: http://bugs.python.org/file32773/issue19619_blacklist_transforms_py34_keyword_only_p

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Marc-Andre Lemburg added the comment: On 22.11.2013 12:24, Marc-Andre Lemburg wrote: > > Nick: I've had a look at your second patch. A couple of notes: > > * I think you should add the flag to the constructor of the CodecInfo >tuple subclass and then set this in the resp. codecs. The extra

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Marc-Andre Lemburg added the comment: Nick: I've had a look at your second patch. A couple of notes: * I think you should add the flag to the constructor of the CodecInfo tuple subclass and then set this in the resp. codecs. The extra constructor class method looks too much like a hack an

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: - switched to LookupError for the dedicated exception - default value moved to a CodecInfo class attribute - new private APIs guarded by PY_LIMITED_API - used repr formatting where appropriate in the tests - cleaned up the tests a bit by using encodings.normalize_e

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Victor, you can propose whatever you like for 3.5, but I'm not adding new interpreter state two days before feature freeze when we don't have to. Looking up the private CodecInfo attribute is still O(1) anyway. -- ___ P

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: > There is no "codec registry" - there is only the default codec search function, the encodings import namespace, the normalisation algorithm and the alias dictionary. interp->codec_search_cache can be seen as the "registry". If you store codecs in two differen

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Victor pointed out this should now raise LookupError rather than TypeError. However, I'm not going to duplicate the manipulation of the private attribute across seven different codecs when a private alternate constructor solves that problem far more cleanly.

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: Why return codecs.CodecInfo._declare_transform( name='base64', encode=base64_encode, decode=base64_decode, incrementalencoder=IncrementalEncoder, incrementaldecoder=IncrementalDecoder, streamwriter=StreamWrit

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: > return codecs.CodecInfo._declare_transform() I also prefer the private attribute option. -- ___ Python tracker ___ __

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: I used the private class method to minimise the per-codec impact (1 modified/added line per codec rather than 3). Your other suggestions look good, so I'll apply those before committing. -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Changes by Nick Coghlan : -- assignee: -> ncoghlan ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: New patch for 3.4 that uses a private attribute on CodecInfo and a private class method to set it appropriately (as I believe that is a better approach than changing the signature of CodecInfo.__init__ at this point, especially if we end up pursuing the codec ty

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Yes, a private attribute on CodecInfo is probably better - the rest of the patch would stay the same, it would just check for that attribute instead of particular names. -- ___ Python tracker

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: Blacklisting by name is slow and it prevents a user from defining a codec with blacklisted name. What if just add private attribute ("_not_text"?) to unsafe codecs? If a codec has this attribute, than it should not be used it text encoding/decoding. Checkin

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Future proofing is irrelevant at this point - this is just about what can realistically be implemented in 3.4, not what can be implemented with the luxury of several months to rearchitect the codec system (and if we were going to do that, we'd just fix the type map

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

STINNER Victor added the comment: +/* A set would be faster, but when to build it, where to store it? */ +if (_PyUnicode_CompareWithId(codec_name, &PyId_base64) == 0 || +_PyUnicode_CompareWithId(codec_name, &PyId_uu) == 0 || +_PyUnicode_CompareWithId(codec_name, &PyId_quop

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Attached is a proof of concept for the blacklist approach (for 3.4, but without the fixes needed for the transform codec handling tests in test_codecs) This does have the potential to add a reasonable amount of additional overhead to encoding and decoding for sh

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Marc-Andre Lemburg added the comment: On 19.11.2013 12:38, Nick Coghlan wrote: > > Given the time frame, how about we just go with Serhiy's suggestion of a > "known non-Unicode codec" internal blacklist for both 3.3 and 3.4? +1 -- ___ Python tracke

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Given the time frame, how about we just go with Serhiy's suggestion of a "known non-Unicode codec" internal blacklist for both 3.3 and 3.4? I still like the idea of exposing codec type maps for introspection, but designing a decent API for that which also handle

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Marc-Andre Lemburg added the comment: On 16.11.2013 15:52, Nick Coghlan wrote: > > The only reasonable way to accurately represent "anything that exposes a > buffer memoryview can read" as a type check is to write an appropriately > duck-typed ABC. You can't enumerate all the types that the bi

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: The only reasonable way to accurately represent "anything that exposes a buffer memoryview can read" as a type check is to write an appropriately duck-typed ABC. You can't enumerate all the types that the binary codecs accept as input, because that list of types

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Marc-Andre Lemburg added the comment: On 16.11.2013 14:37, Nick Coghlan wrote: > > Such codecs can be represented (for 3.4) by simply not setting the attribute > and leaving the output types unspecified. We don't need that complexity for > the standard library, and the "not specified" escape h

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Such codecs can be represented (for 3.4) by simply not setting the attribute and leaving the output types unspecified. We don't need that complexity for the standard library, and the "not specified" escape hatch means complex codecs will be no worse off than the

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Marc-Andre Lemburg added the comment: On 16.11.2013 13:44, Nick Coghlan wrote: > > Nick Coghlan added the comment: > > Now that I understand Victor's proposal better, I actually agree with it, I > just think the attribute names need to be "encodes_to" and "decodes_to". > > With Victor's propo

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Now that I understand Victor's proposal better, I actually agree with it, I just think the attribute names need to be "encodes_to" and "decodes_to". With Victor's proposal, *input* validity checks (including type checks) would remain the responsibility of the co

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: Note that users can completely blacklist any codec that hasn't been imported yet by preventing imports of that codec definition: >>> import sys, encodings >>> blocked_codecs = "bz2_codec", "zlib_codec" >>> for name in blocked_codecs: ... sys.modules["encoding

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Marc-Andre Lemburg added the comment: On 16.11.2013 10:16, Nick Coghlan wrote: > > Nick Coghlan added the comment: > > The full input/output type specifications can't be implemented sensibly > without also defining at least a ByteSequence ABC. While I think it's a good > idea in the long run,

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Serhiy Storchaka added the comment: I think internal blacklist for all standard non-unicode codecs will be enough to prevent the denial of service attack in maintenance releases. -- components: +Unicode nosy: +doerwalter, ezio.melotti priority: normal -> critical stage: -> needs patch

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

Nick Coghlan added the comment: The full input/output type specifications can't be implemented sensibly without also defining at least a ByteSequence ABC. While I think it's a good idea in the long run, there's no feasible way to design such a system in the time remaining before the Python 3.4

[issue19619] Blacklist base64, hex, ... codecs from bytes.decode() and str.encode()

New submission from STINNER Victor: I propose to add new input_type and output_type to CodecInfo. These attributes would only be defined for base64, hex, ... codecs which are not the classic encode: str=>bytes, decode: bytes=>str codecs. I also propose to modify str.encode() and bytes.encode()