Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On Sun, Nov 23, 2014 at 03:08:59PM -0500, Saggi Mizrahi wrote:
> It's well known.
I'd drop the "well" - it was not known to me, and to others.
> The problem is that there is no simplejson in EL6\7
> (unless the situation changed since I last checked).
Francesco verified that RHEL6 does have simplejson. RHEL7 does not have
it (and does not need it as anxiously).
>
> I already did extensive profiling on the jsonrpc framework.
> The patches [1] move us on par with XML-RPC for sequential requests and
> *way* faster for concurrent commands.
Thanks!
>
> There are thoughts about adding msgpack [2] as an optional encoding for
> responses as it's much smaller and less cpu intensive to encode\decode
> but we have bigger fish to fry ATM (patches welcome).
>
> [1]
> http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:stomp_performance,n,z
> [2] http://msgpack.org/
>
> - Original Message -
> > From: "Francesco Romani"
> > To: devel@ovirt.org
> > Sent: Monday, November 10, 2014 10:02:44 AM
> > Subject: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on
> > RHEL/Centos 6
> >
> > Hi everyone,
> >
> > I was doing JSON-RPC investigation recently, running VDSM on RHEL6.{5,6},
> > and while some initial profile work, I discovered a (little) performance
> > degradation
> > about pure JSONRPC coding/encoding, *only* on the above platforms. Here's
> > why:
> >
> > - the JSON codec module in the python stdlib is based on simplejson
> > - (thus) simplejson API is identical to stdlib's json module API
> > - python 2.6 JSON codec is (based on) simplejson 1.9
> > - python 2.7 JSON codec is (based on) simplejson 2.0.9, with significant
> > speedups. See
> > https://docs.python.org/2/whatsnew/2.7.html#new-and-improved-modules and
> > https://bugs.python.org/issue4136
> > - RHEL6.x/Centos 6.x includes python 2.6, so here (and only here) JSON codec
> > is unnecessarily slower.
> > - RHEL6.x (surely) and CentOS 6.x (need to check, don't see why not)
> > includes
> > anyway simplejson 2.0.9 as external package
> >
> > So, it seems to me that we can get python 2.7 speed even on stable platform
> > with
> > a five line patch:
> >
> > === cut here ===
> > diff --git a/lib/yajsonrpc/__init__.py b/lib/yajsonrpc/__init__.py
> > index b3fd590..6682fc3 100644
> > --- a/lib/yajsonrpc/__init__.py
> > +++ b/lib/yajsonrpc/__init__.py
> > @@ -12,7 +12,10 @@
> > # You should have received a copy of the GNU General Public
> > # License along with this program; if not, write to the Free Software
> > # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> > -import json
> > +try:
> > +import simplejson as json
> > +except ImportError
> > +import json
> > import logging
> > from functools import partial
> > from Queue import Queue
> > === cut here ===
> >
> > Here's an excerpt of the effects of this patch, using an initial still rough
> > bencmark
> > - once again 100 VMs boot, is the beginning of every testing routine from
> > mine,
> > so nothing new here.
> >
> > Let's consider two initial profiles. Please note that
> > 1) the amount of calls is not the same (error from mine in profiling, need
> > more precise bench)
> >but still
> > 2) using 2.0.9 module JSON just vanishes from the profile alltogether
> >
> > Vanilla VDSM, stdlib json, top 20 expensive calls
> >
> >ncalls tottime percall cumtime percall filename:lineno(function)
> >19/100 127.2166.6969.9740.100
> >/usr/lib/python2.6/site-packages/mom/GuestMonitor.py:51(GuestMonitor.run)
> > 1555/1635 41.1800.026 58.5660.036
> > /usr/lib64/python2.6/threading.py:481(Thread.run)
> > 11708/1553160 11.9670.001 37.9210.000
> > /usr/lib64/python2.6/copy.py:144(deepcopy)
> > 12686136.1540.0009.2090.000
> > /usr/lib64/python2.6/copy.py:261(_keep_alive)
> > 4890/1300936.1490.001 37.5960.000
> > /usr/lib64/python2.6/copy.py:251(_deepcopy_dict)
> > 497858/20700785.2240.000 11.1020.000
> > /usr/lib64/python2.6/json/encoder.py:284(JSONEncoder._iterencode)
> > 498440/13334744.1870.000 10.3680.000
> > /usr/lib64/python2.6/json/encoder.py:213(JSONEncoder._iterencode_dict)
> >12/1004.1150.3434.1300.041
> >/usr/share/vdsm/virt/sampling.py:424(VmStatsThread.run)
> > 438263.6920.0006.5160.000
> >
> > /usr/lib64/python2.6/xml/dom/expatbuilder.py:743(ExpatBuilderNS.start_element_handler)
> > 73453.6570.000 11.0470.002
> > /usr/share/vdsm/virt/vm.py:2264(Vm._getRunningVmStats)
> > 9666/3205083.5680.0003.5720.000
> > /usr/lib64/python2.6/xml/dom/minidom.py:305(_get_elements_by_tagName_helper)
> > 274994/2750003.3390.0005.5930.000
> > /usr/lib64/python2.6/StringIO.py:208(StringIO.write)
> > 902171.8230.0001.8700.000
> > /usr/lib64/python2.6/xml/dom/minidom.py:349(Attr.__init_
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
It's well known.
The problem is that there is no simplejson in EL6\7
(unless the situation changed since I last checked).
I already did extensive profiling on the jsonrpc framework.
The patches [1] move us on par with XML-RPC for sequential requests and
*way* faster for concurrent commands.
There are thoughts about adding msgpack [2] as an optional encoding for
responses as it's much smaller and less cpu intensive to encode\decode
but we have bigger fish to fry ATM (patches welcome).
[1]
http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:stomp_performance,n,z
[2] http://msgpack.org/
- Original Message -
> From: "Francesco Romani"
> To: devel@ovirt.org
> Sent: Monday, November 10, 2014 10:02:44 AM
> Subject: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on
> RHEL/Centos 6
>
> Hi everyone,
>
> I was doing JSON-RPC investigation recently, running VDSM on RHEL6.{5,6},
> and while some initial profile work, I discovered a (little) performance
> degradation
> about pure JSONRPC coding/encoding, *only* on the above platforms. Here's
> why:
>
> - the JSON codec module in the python stdlib is based on simplejson
> - (thus) simplejson API is identical to stdlib's json module API
> - python 2.6 JSON codec is (based on) simplejson 1.9
> - python 2.7 JSON codec is (based on) simplejson 2.0.9, with significant
> speedups. See
> https://docs.python.org/2/whatsnew/2.7.html#new-and-improved-modules and
> https://bugs.python.org/issue4136
> - RHEL6.x/Centos 6.x includes python 2.6, so here (and only here) JSON codec
> is unnecessarily slower.
> - RHEL6.x (surely) and CentOS 6.x (need to check, don't see why not) includes
> anyway simplejson 2.0.9 as external package
>
> So, it seems to me that we can get python 2.7 speed even on stable platform
> with
> a five line patch:
>
> === cut here ===
> diff --git a/lib/yajsonrpc/__init__.py b/lib/yajsonrpc/__init__.py
> index b3fd590..6682fc3 100644
> --- a/lib/yajsonrpc/__init__.py
> +++ b/lib/yajsonrpc/__init__.py
> @@ -12,7 +12,10 @@
> # You should have received a copy of the GNU General Public
> # License along with this program; if not, write to the Free Software
> # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> -import json
> +try:
> +import simplejson as json
> +except ImportError
> +import json
> import logging
> from functools import partial
> from Queue import Queue
> === cut here ===
>
> Here's an excerpt of the effects of this patch, using an initial still rough
> bencmark
> - once again 100 VMs boot, is the beginning of every testing routine from
> mine,
> so nothing new here.
>
> Let's consider two initial profiles. Please note that
> 1) the amount of calls is not the same (error from mine in profiling, need
> more precise bench)
>but still
> 2) using 2.0.9 module JSON just vanishes from the profile alltogether
>
> Vanilla VDSM, stdlib json, top 20 expensive calls
>
>ncalls tottime percall cumtime percall filename:lineno(function)
>19/100 127.2166.6969.9740.100
>/usr/lib/python2.6/site-packages/mom/GuestMonitor.py:51(GuestMonitor.run)
> 1555/1635 41.1800.026 58.5660.036
> /usr/lib64/python2.6/threading.py:481(Thread.run)
> 11708/1553160 11.9670.001 37.9210.000
> /usr/lib64/python2.6/copy.py:144(deepcopy)
> 12686136.1540.0009.2090.000
> /usr/lib64/python2.6/copy.py:261(_keep_alive)
> 4890/1300936.1490.001 37.5960.000
> /usr/lib64/python2.6/copy.py:251(_deepcopy_dict)
> 497858/20700785.2240.000 11.1020.000
> /usr/lib64/python2.6/json/encoder.py:284(JSONEncoder._iterencode)
> 498440/13334744.1870.000 10.3680.000
> /usr/lib64/python2.6/json/encoder.py:213(JSONEncoder._iterencode_dict)
>12/1004.1150.3434.1300.041
>/usr/share/vdsm/virt/sampling.py:424(VmStatsThread.run)
> 438263.6920.0006.5160.000
>
> /usr/lib64/python2.6/xml/dom/expatbuilder.py:743(ExpatBuilderNS.start_element_handler)
> 73453.6570.000 11.0470.002
> /usr/share/vdsm/virt/vm.py:2264(Vm._getRunningVmStats)
> 9666/3205083.5680.0003.5720.000
> /usr/lib64/python2.6/xml/dom/minidom.py:305(_get_elements_by_tagName_helper)
> 274994/2750003.3390.0005.5930.000
> /usr/lib64/python2.6/StringIO.py:208(StringIO.write)
> 902171.8230.0001.8700.000
> /usr/lib64/python2.6/xml/dom/minidom.py:349(Attr.__init__)
> 44025/440361.6660.0002.4700.000
> /usr/share/vdsm/storage/lvm.py:217(makeLV)
> 144212/1442261.2870.0001.2920.000
> /usr/lib/python2.6/site-packages/vdsm/utils.py:285(convertToStr)
> 2011.2850.0067.2780.036
> /usr/share/vdsm/storage/lvm.py:411(LVMCache._reloadlvs)
>1098801.2350.0001.2830.000
>/usr/lib64/python2.6/xml/dom/minidom.py:281(Document._append_child)
> 578741.16
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On Thu, Nov 20, 2014 at 12:36:20PM +0100, Sven Kieske wrote: > > > On 20/11/14 12:18, Dan Kenigsberg wrote: > > RHEV cannot depend on EPEL. If a package is not in RHEL, RHEV must > > consume it, which is something that's not worth doing in this case > > (where there's not functional benefit). > > As I said: it already does that: > > an excerpt from the latest ovirt-release35-001-1.noarch.rpm: > in /./usr/share/ovirt-release35/ovirt-el6-deps.repo > you find: > > [ovirt-3.5-epel] > name=Extra Packages for Enterprise Linux 6 - $basearch > #baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch > mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch > failovermethod=priority > enabled=1 > includepkgs=epel-release,python-uinput,puppet,python-lockfile,python-cpopen,python-ordereddict,python-pthreading,python-inotify,python-argparse,novnc,python-ply,python-kitchen,python-daemon,python-websockify,livecd-tools,spice-html5,mom,python-IPy,python-ioprocess,ioprocess,rubygem-rgen > gpgcheck=1 > gpgkey=https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6 > > > So why can engine depend on epel and vdsm can not? engine and vdsm both can, and do, depend on epel. However, RHEV cannot. Taking an epel package has a price of needing to ship it in http://ftp.redhat.com/pub/redhat/rhev-m/3.x/SRPMS/ (see python-inotify there, as a vdsm dependency). In case of simplejson for el7, it simply does not worth it. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On 20/11/14 12:18, Dan Kenigsberg wrote: > RHEV cannot depend on EPEL. If a package is not in RHEL, RHEV must > consume it, which is something that's not worth doing in this case > (where there's not functional benefit). As I said: it already does that: an excerpt from the latest ovirt-release35-001-1.noarch.rpm: in /./usr/share/ovirt-release35/ovirt-el6-deps.repo you find: [ovirt-3.5-epel] name=Extra Packages for Enterprise Linux 6 - $basearch #baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch failovermethod=priority enabled=1 includepkgs=epel-release,python-uinput,puppet,python-lockfile,python-cpopen,python-ordereddict,python-pthreading,python-inotify,python-argparse,novnc,python-ply,python-kitchen,python-daemon,python-websockify,livecd-tools,spice-html5,mom,python-IPy,python-ioprocess,ioprocess,rubygem-rgen gpgcheck=1 gpgkey=https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6 So why can engine depend on epel and vdsm can not? -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On Thu, Nov 20, 2014 at 11:12:44AM +0100, Sven Kieske wrote: > On 19/11/14 18:36, Dan Kenigsberg wrote: > > We've learned the hard way that this was not the case.. > > http://gerrit.ovirt.org/#/c/35354 > > Why don't you just require this one package from epel than? > epel is included in EL7, you just need to activate it. > > Furthermore, some engine packages are also just available > via epel, so I see no reason to not require epel for vdsm too? RHEV cannot depend on EPEL. If a package is not in RHEL, RHEV must consume it, which is something that's not worth doing in this case (where there's not functional benefit). Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On 19/11/14 18:36, Dan Kenigsberg wrote: > We've learned the hard way that this was not the case.. > http://gerrit.ovirt.org/#/c/35354 Why don't you just require this one package from epel than? epel is included in EL7, you just need to activate it. Furthermore, some engine packages are also just available via epel, so I see no reason to not require epel for vdsm too? -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On Tue, Nov 11, 2014 at 05:31:37AM -0500, Nir Soffer wrote: > - Original Message - > > From: "Francesco Romani" > > To: devel@ovirt.org > > Cc: "Nir Soffer" > > Sent: Tuesday, November 11, 2014 12:24:43 PM > > Subject: Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance > > on RHEL/Centos 6 > > > > > > > > - Original Message - > > > From: "Nir Soffer" > > > To: "Francesco Romani" > > > Cc: devel@ovirt.org > > > Sent: Monday, November 10, 2014 1:21:53 PM > > > Subject: Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding > > > performance > > > onRHEL/Centos 6 > > > > > > We have vdsm/lib/compat.py for this stuff. > > > > > > Then everyone needing json will do "from vdsm.compat import json" > > > > Do you think is better to have an hard dependency on 'simplejson' > > or a soft one like I showed above hidden on compat.py? > > It depends on the availability of simplejson. If it available > on all platforms, we can require it. We've learned the hard way that this was not the case.. http://gerrit.ovirt.org/#/c/35354 ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
On Tue, Nov 11, 2014 at 05:24:43AM -0500, Francesco Romani wrote: > > > - Original Message - > > From: "Nir Soffer" > > To: "Francesco Romani" > > Cc: devel@ovirt.org > > Sent: Monday, November 10, 2014 1:21:53 PM > > Subject: Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance > > on RHEL/Centos 6 > > > > We have vdsm/lib/compat.py for this stuff. > > > > Then everyone needing json will do "from vdsm.compat import json" > > Do you think is better to have an hard dependency on 'simplejson' > or a soft one like I showed above hidden on compat.py? > > I think we agree that 'explicit is better than implicit' here, so... (see > below). I know ganeti has a hard dependency on simplejson for performance reasons, so you wouldn't be the first virtualization project to do so :) > > Nice catch - using pure python encoder/decoder is indeed much slower, > > strange > > that Python 2.6 does not include the c extension. > > > > I think we would like to use simplejson on all platforms, to make the build > > and support simpler. > > ...agreed, and posted http://gerrit.ovirt.org/#/c/35024/ ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
- Original Message - > From: "Francesco Romani" > To: devel@ovirt.org > Cc: "Nir Soffer" > Sent: Tuesday, November 11, 2014 12:24:43 PM > Subject: Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance > onRHEL/Centos 6 > > > > - Original Message - > > From: "Nir Soffer" > > To: "Francesco Romani" > > Cc: devel@ovirt.org > > Sent: Monday, November 10, 2014 1:21:53 PM > > Subject: Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance > > on RHEL/Centos 6 > > > > We have vdsm/lib/compat.py for this stuff. > > > > Then everyone needing json will do "from vdsm.compat import json" > > Do you think is better to have an hard dependency on 'simplejson' > or a soft one like I showed above hidden on compat.py? It depends on the availability of simplejson. If it available on all platforms, we can require it. > > I think we agree that 'explicit is better than implicit' here, so... (see > below). I prefer to see "json" in the code, even if we use simplejson library. When we stop supporting Python 2.6 the only change we have to do is remove the "soft" import mechanism. Nir ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
- Original Message - > From: "Nir Soffer" > To: "Francesco Romani" > Cc: devel@ovirt.org > Sent: Monday, November 10, 2014 1:21:53 PM > Subject: Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance > onRHEL/Centos 6 > > We have vdsm/lib/compat.py for this stuff. > > Then everyone needing json will do "from vdsm.compat import json" Do you think is better to have an hard dependency on 'simplejson' or a soft one like I showed above hidden on compat.py? I think we agree that 'explicit is better than implicit' here, so... (see below). > Nice catch - using pure python encoder/decoder is indeed much slower, strange > that Python 2.6 does not include the c extension. > > I think we would like to use simplejson on all platforms, to make the build > and support simpler. ...agreed, and posted http://gerrit.ovirt.org/#/c/35024/ Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
- Original Message -
> From: "Francesco Romani"
> To: devel@ovirt.org
> Sent: Monday, November 10, 2014 10:02:44 AM
> Subject: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on
> RHEL/Centos 6
>
> Hi everyone,
>
> I was doing JSON-RPC investigation recently, running VDSM on RHEL6.{5,6},
> and while some initial profile work, I discovered a (little) performance
> degradation
> about pure JSONRPC coding/encoding, *only* on the above platforms. Here's
> why:
>
> - the JSON codec module in the python stdlib is based on simplejson
> - (thus) simplejson API is identical to stdlib's json module API
> - python 2.6 JSON codec is (based on) simplejson 1.9
> - python 2.7 JSON codec is (based on) simplejson 2.0.9, with significant
> speedups. See
> https://docs.python.org/2/whatsnew/2.7.html#new-and-improved-modules and
> https://bugs.python.org/issue4136
> - RHEL6.x/Centos 6.x includes python 2.6, so here (and only here) JSON codec
> is unnecessarily slower.
> - RHEL6.x (surely) and CentOS 6.x (need to check, don't see why not) includes
> anyway simplejson 2.0.9 as external package
>
> So, it seems to me that we can get python 2.7 speed even on stable platform
> with
> a five line patch:
>
> === cut here ===
> diff --git a/lib/yajsonrpc/__init__.py b/lib/yajsonrpc/__init__.py
> index b3fd590..6682fc3 100644
> --- a/lib/yajsonrpc/__init__.py
> +++ b/lib/yajsonrpc/__init__.py
> @@ -12,7 +12,10 @@
> # You should have received a copy of the GNU General Public
> # License along with this program; if not, write to the Free Software
> # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> -import json
> +try:
> +import simplejson as json
> +except ImportError
> +import json
> import logging
> from functools import partial
> from Queue import Queue
> === cut here ===
We have vdsm/lib/compat.py for this stuff.
Then everyone needing json will do "from vdsm.compat import json"
>
> Here's an excerpt of the effects of this patch, using an initial still rough
> bencmark
> - once again 100 VMs boot, is the beginning of every testing routine from
> mine,
> so nothing new here.
>
> Let's consider two initial profiles. Please note that
> 1) the amount of calls is not the same (error from mine in profiling, need
> more precise bench)
>but still
> 2) using 2.0.9 module JSON just vanishes from the profile alltogether
>
> Vanilla VDSM, stdlib json, top 20 expensive calls
>
>ncalls tottime percall cumtime percall filename:lineno(function)
>19/100 127.2166.6969.9740.100
>/usr/lib/python2.6/site-packages/mom/GuestMonitor.py:51(GuestMonitor.run)
> 1555/1635 41.1800.026 58.5660.036
> /usr/lib64/python2.6/threading.py:481(Thread.run)
> 11708/1553160 11.9670.001 37.9210.000
> /usr/lib64/python2.6/copy.py:144(deepcopy)
> 12686136.1540.0009.2090.000
> /usr/lib64/python2.6/copy.py:261(_keep_alive)
> 4890/1300936.1490.001 37.5960.000
> /usr/lib64/python2.6/copy.py:251(_deepcopy_dict)
> 497858/20700785.2240.000 11.1020.000
> /usr/lib64/python2.6/json/encoder.py:284(JSONEncoder._iterencode)
> 498440/13334744.1870.000 10.3680.000
> /usr/lib64/python2.6/json/encoder.py:213(JSONEncoder._iterencode_dict)
>12/1004.1150.3434.1300.041
>/usr/share/vdsm/virt/sampling.py:424(VmStatsThread.run)
> 438263.6920.0006.5160.000
>
> /usr/lib64/python2.6/xml/dom/expatbuilder.py:743(ExpatBuilderNS.start_element_handler)
> 73453.6570.000 11.0470.002
> /usr/share/vdsm/virt/vm.py:2264(Vm._getRunningVmStats)
> 9666/3205083.5680.0003.5720.000
> /usr/lib64/python2.6/xml/dom/minidom.py:305(_get_elements_by_tagName_helper)
> 274994/2750003.3390.0005.5930.000
> /usr/lib64/python2.6/StringIO.py:208(StringIO.write)
> 902171.8230.0001.8700.000
> /usr/lib64/python2.6/xml/dom/minidom.py:349(Attr.__init__)
> 44025/440361.6660.0002.4700.000
> /usr/share/vdsm/storage/lvm.py:217(makeLV)
> 144212/1442261.2870.0001.2920.000
> /usr/lib/python2.6/site-packages/vdsm/utils.py:285(convertToStr)
> 2011.2850.0067.2780.036
> /usr/share/vdsm/storage/lvm.py:411(LVMCache._reloadlvs)
>1098801.2350.0001.2830.000
>/usr/lib64/python2.6/xml/dom/minidom.py:281(Document._append_child)
> 578741.1630.0001.7990.000
>
> /usr/lib64/python2.6/xml/dom/expatbuilder.py:274(ExpatBuilderNS.character_data_handler_cdata)
> 11308591.1600.0001.1600.000
> /usr/lib64/python2.6/copy.py:197(_deepcopy_atomic)
> 3021/30221.1210.0002.5540.001
> /usr/lib64/python2.6/inspect.py:247(getmembers)
>
>
> patch applied, top 20 expensive calls
>
>24/100 117.2224.884 13.3720.134
>/usr/lib/python2.6/site-packages/mom/GuestMonitor.py:51(GuestMonitor
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
- Original Message - > From: "Piotr Kliczewski" > To: "Francesco Romani" > Cc: devel@ovirt.org > Sent: Monday, November 10, 2014 9:40:22 AM > Subject: Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance > on RHEL/Centos 6 > > Great work, > > +1 to merge it to all branches. > > Can you post results of the same test for xmlrpc? > > I wonder what is the slowdown/speedup between protocols. I'm on it, results will follow asap. I just need to figure out a more precise bench, to rule out unneeded noise. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM][JSON] jsonrpc coding/encoding performance on RHEL/Centos 6
Great work,
+1 to merge it to all branches.
Can you post results of the same test for xmlrpc?
I wonder what is the slowdown/speedup between protocols.
On Mon, Nov 10, 2014 at 9:02 AM, Francesco Romani wrote:
> Hi everyone,
>
> I was doing JSON-RPC investigation recently, running VDSM on RHEL6.{5,6},
> and while some initial profile work, I discovered a (little) performance
> degradation
> about pure JSONRPC coding/encoding, *only* on the above platforms. Here's why:
>
> - the JSON codec module in the python stdlib is based on simplejson
> - (thus) simplejson API is identical to stdlib's json module API
> - python 2.6 JSON codec is (based on) simplejson 1.9
> - python 2.7 JSON codec is (based on) simplejson 2.0.9, with significant
> speedups. See
> https://docs.python.org/2/whatsnew/2.7.html#new-and-improved-modules and
> https://bugs.python.org/issue4136
> - RHEL6.x/Centos 6.x includes python 2.6, so here (and only here) JSON codec
> is unnecessarily slower.
> - RHEL6.x (surely) and CentOS 6.x (need to check, don't see why not) includes
> anyway simplejson 2.0.9 as external package
>
> So, it seems to me that we can get python 2.7 speed even on stable platform
> with
> a five line patch:
>
> === cut here ===
> diff --git a/lib/yajsonrpc/__init__.py b/lib/yajsonrpc/__init__.py
> index b3fd590..6682fc3 100644
> --- a/lib/yajsonrpc/__init__.py
> +++ b/lib/yajsonrpc/__init__.py
> @@ -12,7 +12,10 @@
> # You should have received a copy of the GNU General Public
> # License along with this program; if not, write to the Free Software
> # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> -import json
> +try:
> +import simplejson as json
> +except ImportError
> +import json
> import logging
> from functools import partial
> from Queue import Queue
> === cut here ===
>
> Here's an excerpt of the effects of this patch, using an initial still rough
> bencmark
> - once again 100 VMs boot, is the beginning of every testing routine from
> mine,
> so nothing new here.
>
> Let's consider two initial profiles. Please note that
> 1) the amount of calls is not the same (error from mine in profiling, need
> more precise bench)
>but still
> 2) using 2.0.9 module JSON just vanishes from the profile alltogether
>
> Vanilla VDSM, stdlib json, top 20 expensive calls
>
>ncalls tottime percall cumtime percall filename:lineno(function)
>19/100 127.2166.6969.9740.100
> /usr/lib/python2.6/site-packages/mom/GuestMonitor.py:51(GuestMonitor.run)
> 1555/1635 41.1800.026 58.5660.036
> /usr/lib64/python2.6/threading.py:481(Thread.run)
> 11708/1553160 11.9670.001 37.9210.000
> /usr/lib64/python2.6/copy.py:144(deepcopy)
> 12686136.1540.0009.2090.000
> /usr/lib64/python2.6/copy.py:261(_keep_alive)
> 4890/1300936.1490.001 37.5960.000
> /usr/lib64/python2.6/copy.py:251(_deepcopy_dict)
> 497858/20700785.2240.000 11.1020.000
> /usr/lib64/python2.6/json/encoder.py:284(JSONEncoder._iterencode)
> 498440/13334744.1870.000 10.3680.000
> /usr/lib64/python2.6/json/encoder.py:213(JSONEncoder._iterencode_dict)
>12/1004.1150.3434.1300.041
> /usr/share/vdsm/virt/sampling.py:424(VmStatsThread.run)
> 438263.6920.0006.5160.000
> /usr/lib64/python2.6/xml/dom/expatbuilder.py:743(ExpatBuilderNS.start_element_handler)
> 73453.6570.000 11.0470.002
> /usr/share/vdsm/virt/vm.py:2264(Vm._getRunningVmStats)
> 9666/3205083.5680.0003.5720.000
> /usr/lib64/python2.6/xml/dom/minidom.py:305(_get_elements_by_tagName_helper)
> 274994/2750003.3390.0005.5930.000
> /usr/lib64/python2.6/StringIO.py:208(StringIO.write)
> 902171.8230.0001.8700.000
> /usr/lib64/python2.6/xml/dom/minidom.py:349(Attr.__init__)
> 44025/440361.6660.0002.4700.000
> /usr/share/vdsm/storage/lvm.py:217(makeLV)
> 144212/1442261.2870.0001.2920.000
> /usr/lib/python2.6/site-packages/vdsm/utils.py:285(convertToStr)
> 2011.2850.0067.2780.036
> /usr/share/vdsm/storage/lvm.py:411(LVMCache._reloadlvs)
>1098801.2350.0001.2830.000
> /usr/lib64/python2.6/xml/dom/minidom.py:281(Document._append_child)
> 578741.1630.0001.7990.000
> /usr/lib64/python2.6/xml/dom/expatbuilder.py:274(ExpatBuilderNS.character_data_handler_cdata)
> 11308591.1600.0001.1600.000
> /usr/lib64/python2.6/copy.py:197(_deepcopy_atomic)
> 3021/30221.1210.0002.5540.001
> /usr/lib64/python2.6/inspect.py:247(getmembers)
>
>
> patch applied, top 20 expensive calls
>
>24/100 117.2224.884 13.3720.134
> /usr/lib/python2.6/site-packages/mom/GuestMonitor.py:51(GuestMonitor.run)
> 1209/1302 60.8120.050 64.9630.050
> /usr/lib64/python2.6/threading.py:481(Thread.run)
> 9243/1235834 10.2590.001 33.0470.000
