Nir Soffer has uploaded a new change for review. Change subject: compat: Use simplejson instead of json ......................................................................
compat: Use simplejson instead of json We use to require simplejson when using Python 2.6 since the built-in json library was missing the C extension, causing performance issue. When we moved to Python 2.7, we drop this requirement, since Python 2.7 includes the C extension. However json and simplejson are not equivalent. The built-in json library likes to convert plain strings to unicode: >>> import json >>> json.loads(json.dumps({"key": unicode("\xd7\x90", 'utf8')})) {u'key': u'\u05d0'} While simplejson keeps plain strings as strings: >>> json.loads(json.dumps({"key": unicode("\xd7\x90", 'utf8')})) {'key': u'\u05d0'} While "key" and u"key" are equal, they have different behavior when combining them with non-ascii strings: >>> "ascii" + "\xd7\x90" 'ascii\xd7\x90' >>> u"ascii" + "\xd7\x90" Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128) Since u"ascii" is unicode string, Python try to convert "\xd7\x90" assuming the default encoding (ascii), and fails. Another example is logging: >>> import logging >> logging.error("ascii=%s utf8=%s", u"ascii", "\xd7\x90") Traceback (most recent call last): File "/usr/lib64/python2.7/logging/__init__.py", line 859, in emit msg = self.format(record) File "/usr/lib64/python2.7/logging/__init__.py", line 732, in format return fmt.format(record) File "/usr/lib64/python2.7/logging/__init__.py", line 471, in format record.message = record.getMessage() File "/usr/lib64/python2.7/logging/__init__.py", line 335, in getMessage msg = msg % self.args UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128) Again, works fine with using string object: >>> logging.error("ascii=%s utf8=%s", "ascii", "\xd7\x90") ERROR:root:ascii=ascii utf8=א We require now the simplejson library to avoid this issue. Change-Id: I2f0fb2ac45da86a344be556e6474667650bca966 Bug-Url: https://bugzilla.redhat.com/1281940 Signed-off-by: Nir Soffer <nsof...@redhat.com> --- M lib/vdsm/compat.py M vdsm.spec.in 2 files changed, 7 insertions(+), 12 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/65/48565/1 diff --git a/lib/vdsm/compat.py b/lib/vdsm/compat.py index 8ac5201..519943e 100644 --- a/lib/vdsm/compat.py +++ b/lib/vdsm/compat.py @@ -26,15 +26,8 @@ import pickle pickle # yep, this is needed twice. -try: - # on RHEL/Centos 6.x, the JSON module in the python standard - # library does not include significant speedups: - # stdlib is based on simplejson 1.9, speedups were added on 2.0.9. - # In general, speedups are first found on the - # simplejson package. - import simplejson as json - json # make pyflakes happy -except ImportError: - # no big deal, fallback to standard libary - import json - json # yep, this is needed twice. +# We want to work only with simplejson, since the builtin json library +# likes to convert plain ascii strings to unicode, causing failures when +# mixing non-ascii string and "unicode" values. +# See https://bugzilla.redhat.com/1281940 +import simplejson as json diff --git a/vdsm.spec.in b/vdsm.spec.in index 4a8860d..16dc1e9 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -75,6 +75,7 @@ BuildRequires: python-netaddr BuildRequires: python-nose BuildRequires: python-six +BuildRequires: python-simplejson BuildRequires: rpm-build # BuildRequires needed by the tests during the build @@ -118,6 +119,7 @@ Requires: xz Requires: ntp Requires: iproute >= 3.10.0 +Requires: python-simplejson Requires: python-netaddr Requires: python-inotify Requires: python-argparse -- To view, visit https://gerrit.ovirt.org/48565 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2f0fb2ac45da86a344be556e6474667650bca966 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsof...@redhat.com> _______________________________________________ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches