This is by no means an academic benchmark, but I was doing some testing with
RPC methods and decided to come up with a benchmark to get some average
times for the time it takes to make several types of RPC calls, and if one
protocol has a performance advantage over another. I tested JSON-RPC and
XML-RPC. I also tested using sessions, and using SSL to see how those things
affect performance. I will show all the code used, along with a description
of the testing environment later, but right now, here are the results:
#######################################
##### HTTP with Sessions
#######################################
ADD TWO NUMBERS
==================================
JSON-RPC: 47.74 ms
XML-RPC: 46.54 ms
Result: XML won by: 1.20 ms
CONCATENATE TWO STRINGS
==================================
JSON-RPC: 45.09 ms
XML-RPC: 44.66 ms
Result: XML won by: 0.43 ms
HANDLE BOOLEAN VALUE
==================================
JSON-RPC: 45.35 ms
XML-RPC: 45.43 ms
Result: JSON won by: 0.08 ms
HANDLE DATETIME VALUE
==================================
JSON-RPC: 46.13 ms
XML-RPC: 45.52 ms
Result: XML won by: 0.61 ms
HANDLE COMPLEX OBJECTS
==================================
JSON-RPC: 51.63 ms
XML-RPC: 51.19 ms
Result: XML won by: 0.44 ms
FINAL RESULTS
==================================
JSON-RPC Average Time: 47.19 ms
JSON-RPC Total Time Taken: 8731.09 ms
XML-RPC Average Time: 46.67 ms
XML-RPC Total Time Taken: 8682.93 ms
Result: XML won by: 0.52 ms
#######################################
##### HTTP without Sessions
#######################################
This was done by placing "session.forget(response)" in the call() method.
ADD TWO NUMBERS
==================================
JSON-RPC: 49.10 ms
XML-RPC: 50.45 ms
Result: JSON won by: 1.35 ms
CONCATENATE TWO STRINGS
==================================
JSON-RPC: 45.81 ms
XML-RPC: 49.13 ms
Result: JSON won by: 3.32 ms
HANDLE BOOLEAN VALUE
==================================
JSON-RPC: 45.50 ms
XML-RPC: 49.44 ms
Result: JSON won by: 3.94 ms
HANDLE DATETIME VALUE
==================================
JSON-RPC: 45.97 ms
XML-RPC: 49.08 ms
Result: JSON won by: 3.11 ms
HANDLE COMPLEX OBJECTS
==================================
JSON-RPC: 51.25 ms
XML-RPC: 54.90 ms
Result: JSON won by: 3.65 ms
FINAL RESULTS
==================================
JSON-RPC Average Time: 47.53 ms
JSON-RPC Total Time Taken: 8767.23 ms
XML-RPC Average Time: 50.60 ms
XML-RPC Total Time Taken: 9076.67 ms
Result: JSON won by: 3.07 ms
As you can see, there wasn't much of a change. In fact, turning off sessions
added about 400 ms to the testing time. Now let's try with SSL:
#######################################
##### SSL with Sessions
#######################################
I ran this test several times because of the HUGE difference between the
protocols when used over SSL. I think this may be more of a problem with the
JSON client or server rather than the protocol itself. You'll see what I
mean:
ADD TWO NUMBERS
==================================
JSON-RPC: 335.97 ms
XML-RPC: 60.19 ms
Result: XML won by: 275.78 ms
CONCATENATE TWO STRINGS
==================================
JSON-RPC: 335.83 ms
XML-RPC: 43.74 ms
Result: XML won by: 292.09 ms
HANDLE BOOLEAN VALUE
==================================
JSON-RPC: 336.12 ms
XML-RPC: 43.93 ms
Result: XML won by: 292.20 ms
HANDLE DATETIME VALUE
==================================
JSON-RPC: 334.30 ms
XML-RPC: 45.14 ms
Result: XML won by: 289.17 ms
HANDLE COMPLEX OBJECTS
==================================
JSON-RPC: 337.24 ms
XML-RPC: 50.18 ms
Result: XML won by: 287.06 ms
FINAL RESULTS
==================================
JSON-RPC Average Time: 335.90 ms
JSON-RPC Total Time Taken: 37605.90 ms
XML-RPC Average Time: 48.64 ms
XML-RPC Total Time Taken: 8878.73 ms
Result: XML won by: 287.26 ms
#######################################
##### SSL without Sessions
#######################################
Again using "session.forget(response)" here. And again, using this line
seems to slow things down a bit:
ADD TWO NUMBERS
==================================
JSON-RPC: 341.59 ms
XML-RPC: 61.99 ms
Result: XML won by: 279.60 ms
CONCATENATE TWO STRINGS
==================================
JSON-RPC: 338.62 ms
XML-RPC: 45.59 ms
Result: XML won by: 293.03 ms
HANDLE BOOLEAN VALUE
==================================
JSON-RPC: 333.35 ms
XML-RPC: 46.06 ms
Result: XML won by: 287.28 ms
HANDLE DATETIME VALUE
==================================
JSON-RPC: 332.38 ms
XML-RPC: 45.22 ms
Result: XML won by: 287.16 ms
HANDLE COMPLEX OBJECTS
==================================
JSON-RPC: 336.60 ms
XML-RPC: 51.47 ms
Result: XML won by: 285.13 ms
FINAL RESULTS
==================================
JSON-RPC Average Time: 336.51 ms
JSON-RPC Total Time Taken: 37665.40 ms
XML-RPC Average Time: 50.07 ms
XML-RPC Total Time Taken: 9023.97 ms
Result: XML won by: 286.44 ms
As you can see, JSON-RPC over SSL is a real problem. In fact, while I was
playing around with it, I noticed the 300 ms delay and this was the reason I
decided to do the performance testing in the first place. Now, I have web2py
running from a Ubuntu 10.04 virtual machine (using bridged networking) and
running the tests from the host machine (Mac OS X 10.7.1) over the network.
As for the code used to perform this test, I have attached the controller
and the test script, since they are too large to paste.
I just thought this might be of interest as not only a general performance
test of web2py, but also an RPC performance test, which hasn't gotten as
much attention as web testing.
# -*- coding: utf-8 -*-
# this file is released under public domain and you can use without limitations
from gluon import *
import datetime
#########################################################################
## This is a samples controller
## - index is the default action of any application
## - user is required for authentication and authorization
## - download is for downloading files uploaded in the db (does streaming)
## - call exposes all registered services (none by default)
#########################################################################
def index():
"""
example action using the internationalization operator T and flash
rendered by views/default/index.html or views/generic.html
"""
return dict(message=T('Hello World'))
@service.xmlrpc
@service.jsonrpc
def add(num1, num2):
"""
Adds two numbers together.
@param num1: First number
@param num2: Second number
@return: The sum of the two numbers.
>>> add(2, 2)
4
>>> add(2.5, 2.5)
5.0
"""
session.forget(response)
return num1 + num2
@service.xmlrpc
@service.jsonrpc
def concat(str1, str2, str3=''):
"""
Concatenates two or three strings.
@param str1: First string
@param str2: Second string
@param str3: (Optional) Third string
@return: The concatenated string.
>>> concat('hello', ' world')
'hello world'
>>> concat('hello', ' world', ' !!!')
'hello world !!!'
"""
return str1 + str2 + str3
@service.xmlrpc
@service.jsonrpc
def complex(obj):
"""
Accepts a simple or complex object and returns a complex object. This
tests the performance of a larger payload.
@param obj: Any object
@return: A complex object (list of dictionaries)
"""
return [
{'id': 1, 'name': 'Peter Griffin', 'weight': 350, 'birthdate': datetime.datetime(1965, 3, 21), 'children': [
{'id': 3, 'name': 'Chris Griffin', 'weight': 275, 'birthdate': datetime.datetime(1987, 2, 3), 'children': []},
{'id': 4, 'name': 'Meg Griffin', 'weight': 800, 'birthdate': datetime.datetime(1985, 6, 9), 'children': []},
{'id': 5, 'name': 'Stewie Griffin', 'weight': 30, 'birthdate': datetime.datetime(2010, 6, 5), 'children': []},
]},
{'id': 2, 'name': 'Louis Griffin', 'weight': 350, 'birthdate': datetime.datetime(1964, 9, 8), 'children': [
{'id': 3, 'name': 'Chris Griffin', 'weight': 275, 'birthdate': datetime.datetime(1987, 2, 3), 'children': []},
{'id': 4, 'name': 'Meg Griffin', 'weight': 800, 'birthdate': datetime.datetime(1985, 6, 9), 'children': []},
{'id': 5, 'name': 'Stewie Griffin', 'weight': 30, 'birthdate': datetime.datetime(2010, 6, 5), 'children': []},
]},
{'id': 6, 'name': 'Brian Griffin', 'weight': 40, 'birthdate': datetime.datetime(2002, 11, 14), 'children': []},
{'id': 7, 'name': 'Quagmire', 'weight': 40, 'birthdate': datetime.datetime(1971, 12, 4), 'children': []}
]
@service.xmlrpc
@service.jsonrpc
def fail():
"""
This method is designed to fail and raise an exception.
@raise: Will raise an exception ZeroDivisionError.
@return: Will always raise an exception instead of returning.
"""
return 1/0
@service.xmlrpc
@service.jsonrpc
def boolean():
"""
This method always return True in order to test the bool data type.
@return: always True
"""
return True
@service.xmlrpc
@service.jsonrpc
def now():
"""
Returns the server's current date and time.
@return: datetime
"""
return datetime.datetime.now()
def call():
"""
exposes services. for example:
http://..../[app]/default/call/jsonrpc
decorate with @services.jsonrpc the functions to expose
supports xml, json, xmlrpc, jsonrpc, amfrpc, rss, csv
"""
session.forget(response)
return service()
import time, datetime
from gluon.contrib.simplejsonrpc import ServerProxy as JS
from xmlrpclib import ServerProxy as XM
j = JS('https://172.16.100.111:8000/rpc_test/default/call/jsonrpc')
x = XM('https://172.16.100.111:8000/rpc_test/default/call/xmlrpc')
def test():
def format_time(title, t):
return '%s: %0.2f ms' % (title, t * 1000.0)
def rpc(client):
start_total = time.time()
add_times = []
concat_times = []
bool_times = []
date_times = []
complex_times = []
for i in xrange(20):
start = time.time()
client.add(i, i)
stop = time.time()
add_times.append(stop - start)
start = time.time()
client.concat('hello ', 'world')
stop = time.time()
concat_times.append(stop - start)
start = time.time()
client.boolean()
stop = time.time()
bool_times.append(stop - start)
start = time.time()
client.now()
stop = time.time()
date_times.append(stop - start)
start = time.time()
client.complex([
{'id': 1, 'name': 'Peter Griffin', 'weight': 350, 'birthdate': str(datetime.datetime(1965, 3, 21)), 'children': [
{'id': 3, 'name': 'Chris Griffin', 'weight': 275, 'birthdate': str(datetime.datetime(1987, 2, 3)), 'children': []},
{'id': 4, 'name': 'Meg Griffin', 'weight': 800, 'birthdate': str(datetime.datetime(1985, 6, 9)), 'children': []},
{'id': 5, 'name': 'Stewie Griffin', 'weight': 30, 'birthdate': str(datetime.datetime(2010, 6, 5)), 'children': []},
]},
{'id': 2, 'name': 'Louis Griffin', 'weight': 350, 'birthdate': str(datetime.datetime(1964, 9, 8)), 'children': [
{'id': 3, 'name': 'Chris Griffin', 'weight': 275, 'birthdate': str(datetime.datetime(1987, 2, 3)), 'children': []},
{'id': 4, 'name': 'Meg Griffin', 'weight': 800, 'birthdate': str(datetime.datetime(1985, 6, 9)), 'children': []},
{'id': 5, 'name': 'Stewie Griffin', 'weight': 30, 'birthdate': str(datetime.datetime(2010, 6, 5)), 'children': []},
]},
{'id': 6, 'name': 'Brian Griffin', 'weight': 40, 'birthdate': str(datetime.datetime(2002, 11, 14)), 'children': []},
{'id': 7, 'name': 'Quagmire', 'weight': 40, 'birthdate': str(datetime.datetime(1971, 12, 4)), 'children': []}
])
stop = time.time()
complex_times.append(stop - start)
time.sleep(0.2)
stop_total = time.time()
time_total = stop_total - start_total
def get_avg(l):
total = 0.0
for t in l:
total += t
return total / len(l)
add_avg = get_avg(add_times)
concat_avg = get_avg(concat_times)
bool_avg = get_avg(bool_times)
date_avg = get_avg(date_times)
complex_avg = get_avg(complex_times)
total_avg = get_avg([add_avg, concat_avg, bool_avg, date_avg, complex_avg])
return (time_total, add_avg, concat_avg, bool_avg, date_avg, complex_avg, total_avg)
json_total, json_add, json_concat, json_bool, json_date, json_complex, json_avg = rpc(j)
xml_total, xml_add, xml_concat, xml_bool, xml_date, xml_complex, xml_avg = rpc(x)
def compare_avg(json, xml):
if xml < json:
return format_time('Result: XML won by', json - xml)
elif json < xml:
return format_time('Result: JSON won by', xml - json)
else:
return 'Result: XML and JSON tied'
print ''
print ''
print 'ADD TWO NUMBERS'
print '=================================='
print format_time('JSON-RPC', json_add)
print format_time('XML-RPC', xml_add)
print compare_avg(json_add, xml_add)
print ''
print ''
print 'CONCATENATE TWO STRINGS'
print '=================================='
print format_time('JSON-RPC', json_concat)
print format_time('XML-RPC', xml_concat)
print compare_avg(json_concat, xml_concat)
print ''
print ''
print 'HANDLE BOOLEAN VALUE'
print '=================================='
print format_time('JSON-RPC', json_bool)
print format_time('XML-RPC', xml_bool)
print compare_avg(json_bool, xml_bool)
print ''
print ''
print 'HANDLE DATETIME VALUE'
print '=================================='
print format_time('JSON-RPC', json_date)
print format_time('XML-RPC', xml_date)
print compare_avg(json_date, xml_date)
print ''
print ''
print 'HANDLE COMPLEX OBJECTS'
print '=================================='
print format_time('JSON-RPC', json_complex)
print format_time('XML-RPC', xml_complex)
print compare_avg(json_complex, xml_complex)
print ''
print ''
print 'FINAL RESULTS'
print '=================================='
print format_time('JSON-RPC Average Time', json_avg)
print format_time('JSON-RPC Total Time Taken', json_total)
print format_time('XML-RPC Average Time', xml_avg)
print format_time('XML-RPC Total Time Taken', xml_total)
print compare_avg(json_avg, xml_avg)
print ''
print ''
test()
