[issue18850] xml.etree.ElementTree accepts control chars.
Serhiy Storchaka added the comment: Isn't this a duplicate of issue5166? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Eli Bendersky added the comment: As Serhiy points out, this is a duplicate of #5166 -- superseder: - ElementTree and minidom don't prevent creation of not well-formed XML ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Changes by Eli Bendersky eli...@gmail.com: -- resolution: - duplicate stage: needs patch - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Eli Bendersky added the comment: Can this be transformed into a new issue that succinctly summarizes what the new requested feature is, and why it's useful? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment:
Is that you actual use case? That you *want* to store binary data in XML,
instead of getting it properly rejected as non well-formed content?
No, Stefan.
What I was saying in my last message was just you're right, the user shall
always use repr() when printing an xml tree (msg196313) because xml does
*not* guarantee to have only printable chars by itself (msg196368, msg196379).
As an advice I hope you do not take as insult, saying
in section {section} the spec says {argument}
is much more constructive than
read the spec on that, {extremely_obvious_link},
at least to people not familiar with the spec and asking for the source of your
arguments (msg196360). Can shorten threads, too.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18850
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment:
As an advice I hope you do not take as insult, saying
in section {section} the spec says {argument}
is much more constructive than
read the spec on that, {extremely_obvious_link},
at least to people not familiar with the spec and asking for the source of
your arguments (msg196360). Can shorten threads, too.
No harm done. The reason why I just posted the spec URL is that it's actually
the entire spec that backs the argument. XML is (essentially) specified as a
mapping from a sequence of bytes to a hierarchical structure (and back again).
That's why there is an XML declaration header that names the encoding, for
example. It wouldn't be needed if XML was defined as Unicode data.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18850
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: The parser *is* rejecting control characters. It's an XML parser. See the example in the link you posted. Ehrm, my apologies. That's not an XML specific issue. You are printing a byte string here, so repr() would be the right thing to use (and is actually being used automatically in Py3), instead of plain printing. The fact that you are wrapping the content in XML doesn't matter. [citation needed] After a quick scan in the documentation I did not see anything mentioning this. Instead, I see many cases in which escape chars and binary-to-text encodings are mentioned. What I meant was: at what step of the process from creating an XML tree in memory to serialisation is it a problem that the tree contains control characters? Because once the data is serialised, it will just be rejected on input by any XML parser, and handling bytes data is a thing on its own (e.g. you could serialise to UTF16 and the result would contain null bytes - too bad). m, I think the problem lies in the expectation of having fromstring(tostring(tree)) = tree Unless there is a more dangerous way to exploit this that is actually due to XML being used, I'd suggest changing the type from security back to behaviour. Or maybe even to enhancement. The behaviour that it writes out what you give it isn't exactly wrong, it's just inconvenient that you have to take care yourself that you pass it well-formed XML content. I think the point here is clarifying whether xml expect text or just a byte string. In case that's a stream of byte, I agree with you, is more a behaviour problem. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment: I think the point here is clarifying whether xml expect text or just a byte string. In case that's a stream of byte, I agree with you, is more a behaviour problem. XML is *defined* as a stream of bytes. Regarding the API side in ElementTree, Py2 accepts byte strings and Py3 requires Unicode strings. Py2 will not change in that regard, and I can't see this being a serious enough issue to change the ET-API there, so IMHO we can ignore Py2.x completely for this issue. (changing ticket accordingly) However, Py3 will happily write out control characters if they appear in the Unicode text string, so the issue is the same there. A fix for Py3 would be to add an input validation step, preferably at serialisation time. -- type: security - enhancement versions: -Python 2.6, Python 2.7, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: Incidentally I read today http://blastedbio.blogspot.co.uk/2012/05/blast-tabular-missing-descriptions.html mentioning ^A being used. Maybe that would stop working? I don't see any problem in any xml output. Indeed: You can't put a nasty non-printing ASCII character in XML, so they were switched to something else. That is my working theory for this anyway. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: XML is *defined* as a stream of bytes. Can you *paste* the *source* proving what you are arguing, please? Regarding the API side in ElementTree, Py2 accepts byte strings and Py3 requires Unicode strings. accepts? python3 works with ElementTree(bytes(unicode)) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment: XML is *defined* as a stream of bytes. Can you *paste* the *source* proving what you are arguing, please? http://www.w3.org/TR/REC-xml/ python3 works with ElementTree(bytes(unicode)) What does this sentence mean? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: XML is *defined* as a stream of bytes. Can you *paste* the *source* proving what you are arguing, please? http://www.w3.org/TR/REC-xml/ The first two suggestions are directly derived from the rules given for identifiers in Standard Annex #31 (UAX #31) of the Unicode Standard, version 5.0 [Unicode], and exclude all control characters, enclosing nonspacing marks, non-decimal numbers, private-use characters, punctuation characters (with the noted exceptions), symbol characters, unassigned codepoints, and white space characters. The other suggestions are mostly derived from Appendix B in previous editions of this specification. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: Document authors are encouraged to avoid compatibility characters, as defined in section 2.3 of [Unicode]. The characters defined in the following ranges are also discouraged. They are either control characters or permanently undefined Unicode characters: -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: Does not seem to me just a byte string where you can put binary data. Hence, I expect the xml tree to escape/reject those. Hence, Is not an enhancement, but a bug. Unless we just want to document this. (not going to change the metadata, otherwise we'll end up changing on each message). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment: We are talking about two different things here. I said that (serialised) XML is defined as a sequence of bytes. Read the spec on that. What you are talking about is the Infoset, or the parsed/generated in-memory XML tree. That's obviously not bytes, it's defined based on Unicode. Parsing and serialising does the mapping here. The attack that you presented is based on serialised XML, thus on a sequence of bytes. What I am saying is that this attack can be done by any kind of binary data, so it's not XML specific, thus not a problem with ElementTree. The fact that ElementTree allows you to generate non well-formed 'XML' containing control characters when you tell it to do so is unfortunate, but it's neither a security risk (you already had the non well-formed content in your hands *before* you passed it into ElementTree), nor clearly a bug, because the user specifically requested the serialisation of an in-memory tree that contained these control characters. But, once again, it would be nice if ElementTree rejected this input in one way or another, and that's a feature request. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: I said that (serialised) XML is defined as a sequence of bytes. Read the spec on that. And I'm saying that's inexact. I have expectations that control chars are escaped in the serialized xml, because the spec I'm reading says so, and because the documentation does not warn me about this. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: Just pointed by a friend - http://www.w3.org/TR/REC-xml/#sec-cdata-sect I suppose this is insanely used to put binary blobs inside xml until only the CDEnd string is recognized as markup. That's what I needed. Amen. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +christian.heimes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment: Is that you actual use case? That you *want* to store binary data in XML, instead of getting it properly rejected as non well-formed content? Then I suggest going the canonical route of passing it through base64 first, or any of the other binary-to-characters encoding. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
New submission from Michele Orrù: Got from irc; JerryKwan python bug in xml.etree.ElementTree, from version 2.7 to 3.2 http://www.reddit.com/r/Python/comments/1l6cta/python_bug_in_xmletreeelementtree/ I think we should keep consistency with lxml and forbid control chars in advance. https://en.wikipedia.org/wiki/Valid_characters_in_XML#XML_1.1 Attaching the test file proving the issue, slightly modified to work on 2.7 and 3.x -- files: test.py messages: 196271 nosy: maker priority: normal severity: normal status: open title: xml.etree.ElementTree accepts control chars. Added file: http://bugs.python.org/file31482/test.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Changes by Michele Orrù maker...@gmail.com: -- components: +Library (Lib), XML type: - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +eli.bendersky, scoder, serhiy.storchaka versions: +Python 2.7, Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment: This is a bit tricky in ET because it generally allows you to stick anything into the Element properties (and that's a feature). So catching this at tree building time (as lxml.etree does) isn't really possible. However, at least catching it in the serialiser should be possible and would help. Regexes for well-formed tag names and text could do the job. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: you mind if I try by myself to provide patch and unittest in the next few days? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment: Go for it. That's usually the fastest way to get things done. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
R. David Murray added the comment: Unless it is a security issue, this seems like the kind of fix that shouldn't be applied to maintenance releases. -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment: I suppose it is, David, if in 2 minutes flat I can change your terminal name. -- Added file: http://bugs.python.org/file31484/inject.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
R. David Murray added the comment: In that case, the fix needs to be applied to 3.2 and 2.6 as well. Or at least considered for application. It could be that this will break working (though dangerous) programs. I'll leave it to folks more knowledgeable in this particular area than I to decide where the severity/backward compatibility line lies in this case. -- stage: - needs patch type: behavior - security versions: +Python 2.6, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Serhiy Storchaka added the comment: See also issue7727. Almost any other XML generation code (xml.sax.sautils.XMLGenerator, xml.dom.minidom.Element.writexml(), etc, but not plistlib.PlistWriter) has the same problem. The problem with filtering control characters is that it will significantly slowdown XML generation. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment: Michele, could you elaborate how you would exploit this issue as a security risk? I mean, I can easily create a (non-)XML-document with control characters manually, and the parser would reject it. What part of the create-to-serialise process exactly is a problem here? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Michele Orrù added the comment:
Michele, could you elaborate how you would exploit this issue as a security
risk?
Sure. What I meant in my message is: assume you have a script that simply
stores each message it receives (from stdin, from a tcp stream, whatever)
inside an xml tree like
'text{message1}/texttext{message2}text',
and prints the tree on SIGINT.
What I would expect is the xml document not to allow control chars, as
restricted and discouraged, and consistent with lxml.
What instead happens is that the control chars are not handled, and thus
anybody can send control chars in my terminal. Changing the terminal title is a
trivial example of those.
For sure an echo server may have the same issue, but the premises are
different, because I expect to print just a byte stream.
Mentioning this fact in the documentation may be a possible solution, but I
believe more that keeping consistency with lxml is the right way.
I mean, I can easily create a (non-)XML-document with control characters
manually, and the parser would reject it.
False? The parser is *not* rejecting control chars.
What part of the create-to-serialise process exactly is a problem here?
ElementTree.tostring().
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18850
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment:
The parser is *not* rejecting control chars.
The parser *is* rejecting control characters. It's an XML parser. See the
example in the link you posted.
assume you have a script that simply stores each message it receives (from
stdin, from a tcp stream, whatever) inside an xml tree like
'text{message1}/texttext{message2}text',
and prints the tree on SIGINT.
That's not an XML specific issue. You are printing a byte string here, so
repr() would be the right thing to use (and is actually being used
automatically in Py3), instead of plain printing. The fact that you are
wrapping the content in XML doesn't matter.
What part of the create-to-serialise process exactly is a problem here?
ElementTree.tostring().
What I meant was: at what step of the process from creating an XML tree in
memory to serialisation is it a problem that the tree contains control
characters? Because once the data is serialised, it will just be rejected on
input by any XML parser, and handling bytes data is a thing on its own (e.g.
you could serialise to UTF16 and the result would contain null bytes - too bad).
It may just be a bad example that you chose here, but I really can't see this
being a security problem. You are mishandling arbitrary untrusted binary data,
that's all. Control characters are most likely not the only problem that you
should guard against.
Unless there is a more dangerous way to exploit this that is actually due to
XML being used, I'd suggest changing the type from security back to
behaviour.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18850
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Stefan Behnel added the comment: Or maybe even to enhancement. The behaviour that it writes out what you give it isn't exactly wrong, it's just inconvenient that you have to take care yourself that you pass it well-formed XML content. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18850] xml.etree.ElementTree accepts control chars.
Martin Mokrejs added the comment: Incidentally I read today http://blastedbio.blogspot.co.uk/2012/05/blast-tabular-missing-descriptions.html mentioning ^A being used. Maybe that would stop working? -- nosy: +mmokrejs ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18850 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
