[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Irit Katriel]

Irit Katriel  added the comment:

2.7 is no longer relevant, and it looks like these examples are working now:

>>> xmlrpc.client.dumps(('\u20ac',), encoding='iso-8859-1')
'\n\n€\n\n\n'
>>> xmlrpc.client.dumps((u'\u20ac',), encoding='iso-8859-1')
'\n\n€\n\n\n'

There is possibly still a documentation enhancement to make regarding non-ascii 
strings. This is what I get now with Serhiy's examples:

>>> xmlrpc.client.loads(xmlrpc.client.dumps(('\xe2\x82\xac',)))
(('â\x82¬',), None)
>>> xmlrpc.client.loads(xmlrpc.client.dumps(('\r',)))
(('\n',), None)
>>> xmlrpc.client.loads(xmlrpc.client.dumps(('\r\n',)))
(('\n',), None)

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Fredrik Larsen]

Change by Fredrik Larsen :


--
nosy: +fredrikhl

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Alex Corcoles]

Changes by Alex Corcoles :


--
versions: +Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Alex Corcoles]

Changes by Alex Corcoles :


--
nosy: +Alex Corcoles

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
nosy: +effbot

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Updating tests I found some related errors.

XML-RPC doesn't work in general case for non UTF-8 encoding:

 import xmlrpclib
 xmlrpclib.dumps(('\u20ac',), encoding='iso-8859-1')
'params\nparam\nvaluestring\\u20ac/string/value\n/param\n/params\n'
 xmlrpclib.dumps((u'\u20ac',), encoding='iso-8859-1')
Traceback (most recent call last):
  File stdin, line 1, in module
  File /usr/lib/python2.7/xmlrpclib.py, line 1085, in dumps
data = m.dumps(params)
  File /usr/lib/python2.7/xmlrpclib.py, line 632, in dumps
dump(v, write)
  File /usr/lib/python2.7/xmlrpclib.py, line 654, in __dump
f(self, value, write)
  File /usr/lib/python2.7/xmlrpclib.py, line 700, in dump_unicode
value = value.encode(self.encoding)
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u20ac' in 
position 0: ordinal not in range(256)

We should use 'xmlcharrefreplace' error handler.

Non-ASCII strings is passed as Unicode strings (this should be documented).

 xmlrpclib.loads(xmlrpclib.dumps(('\xe2\x82\xac',)))
((u'\u20ac',), None)

'\r' and '\r\n' are deserialized as '\n'.

 xmlrpclib.loads(xmlrpclib.dumps(('\r',)))
(('\n',), None)
 xmlrpclib.loads(xmlrpclib.dumps(('\r\n',)))
(('\n',), None)

--
Added file: 
http://bugs.python.org/file30360/xmlrpc_dump_invalid_string-2.7_2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


Removed file: http://bugs.python.org/file30339/xmlrpc_dump_invalid_string.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


Removed file: 
http://bugs.python.org/file30343/xmlrpc_dump_invalid_string-2.7.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Martin v . Löwis]

Martin v. Löwis added the comment:

I'm still skeptical that a new exception should be introduced in 2.7.x, or 3.3 
(might this break existing setups?). I suggest to ask the release manager for a 
decision.

But if this is done, then I propose to add the following text to ServerProxy:

versionchanged (2.7.6): Sending strings with characters that are ill-formed in 
XML (e.g. \x00) now raises ValueError.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Antoine Pitrou]

Antoine Pitrou added the comment:

I don't really understand the issue. If you want to pass binary data (rather 
than unicode text), you should use a Binary object as explained in the docs:
http://docs.python.org/2/library/xmlrpclib.html#binary-objects

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Martin v . Löwis]

Martin v. Löwis added the comment:

The original report really includes two parts:
a) when a string containing \0 is marshalled, ill-formed XML is produced
b) the expected behavior is that base64 is used

IMO: While a) is correct, b) is not. Antoine is correct that xmlrpclib.Binary 
should be used if you want to transmit binary data. Consequently, an Error 
should be reported if an attempt is made to produce ill-formed XML.

OTOH, ill-formed XML can also be produced when sending a byte string that does 
not match the encoding declaration. Because of that, I propose to close this by 
documentating the limitations, rather than changing the code.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The limitations is already documented:

However, it’s the caller’s responsibility to ensure that the string is free 
of characters that aren’t allowed in XML, such as the control characters with 
ASCII values between 0 and 31 (except, of course, tab, newline and carriage 
return); failing to do this will result in an XML-RPC request that isn’t 
well-formed XML. If you have to pass arbitrary bytes via XML-RPC, use the bytes 
class or the class:Binary wrapper class described below.

Here is a patch which forbids creating ill-formed XML.

--
nosy: +serhiy.storchaka
stage: test needed - patch review
versions: +Python 2.7, Python 3.3, Python 3.4 -Python 2.6
Added file: http://bugs.python.org/file30339/xmlrpc_dump_invalid_string.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Martin v . Löwis]

Martin v. Löwis added the comment:

Serhiy: The patch fixes the OP's concern, but not the extended concern about 
producing ill-formed XML (at least not for 2.7). If the string contains 
non-UTF-8 data, yet the XML declaration says UTF-8, it's still ill-formed, and 
not caught by your patch.

I wonder whether xmlrpclib.Error would be a better exception than ValueError 
(although ValueError is also plausible); either way, the case should be 
documented.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Indeed, 2.7 needs more work. Here is a patch for 2.7.

UnicodeError (which subclasses ValueError) can be raised implicitly here, that 
is why I think ValueError is a good exception.

I'll be very grateful to you for your help with a documentation.

--
Added file: 
http://bugs.python.org/file30343/xmlrpc_dump_invalid_string-2.7.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Mark Lawrence]

Mark Lawrence added the comment:

Even if the original patch is valid it will need reworking as xmlrpclib isn't 
in Python 3, the code is now in xmlrpc/client.  It also looks as if dump_string 
has been renamed dump_unicode.

--
nosy: +BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

Marshaller.dump_string() encodes a byte string in string.../string using 
the escape() function. A byte string can be encoded in base64 using 
base64.../base64. It's described in the XML-RPC specification, but I don't 
know if all XML-RPC implementations do understand this type.
http://www.xmlrpc.com/spec

Should we change the default type to base64, or only fallback to base64 if the 
byte string cannot be encoded in XML. Test if a byte string can be encoded in 
XML can be slow, and set default type to base64 may cause compatibility issues 
:-/

--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

Here is an example of patch using the following test:

   all(32 = ord(byte) = 127 for byte in value)

I don't know how much slower is the patch, but at least it doesn't raise an 
ExpatError: not well-formed (invalid token): 

--
keywords: +patch
Added file: http://bugs.python.org/file15961/xmlrpc_byte_string.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Steven Hartland]

Steven Hartland steven.hartl...@multiplay.co.uk added the comment:

One thing that springs to mind is how valid is that when applied to utf8 data?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Steven Hartland]

New submission from Steven Hartland steven.hartl...@multiplay.co.uk:

When using SimpleXMLRPCServer that is used to return data that includes strings 
that have a \x00 in them this data is returned, which is invalid.

The expected result is that the data should be treated as binary and base64 
encoded.

The bug appears to be in the core xmlrpc library which relies on type( value ) 
to determine the data type. This returns str for a string even if it includes 
the null char.

--
components: XML
messages: 97972
nosy: Steven.Hartland
severity: normal
status: open
title: xmlrpc library returns string which contain null ( \x00 )
type: behavior
versions: Python 2.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7727] xmlrpc library returns string which contain null ( \x00 )

[Brian Curtin]

Changes by Brian Curtin cur...@acm.org:


--
nosy: +loewis
priority:  - normal
stage:  - test needed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7727
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com