I currently have a patch that adds a C extension module to wrap the
in-tree JSON parser. It works with both Python 2 and 3. The
performance increase is quite large. For example, parsing a 100Mb JSON
file:
Without extension: 273 seconds
With extension: 3.7 seconds
On the object->string side, I've tried to just replace the _Serializer
class and to_{stream, file, string} with using Python's built-in
json.dumps, since we don't have to worry about streaming when going
from object->string. The problem is that there are 8 tests related to
float handling that yield slightly different output. In each case, it
looks like Python's version returns slightly more precise results. The
existing implementation also seems to convert from things like 1e-9999
to 0 whereas Python's converts to 0.0. None of these differences
should really matter.
I'd like to avoid having to wrap the serialization stuff when there is
something else that already works and is better, but if I have to I
will. Just looking for opinions on what I should do. Here's the test
failure output:
## ---------------------- ##
## Detailed failed tests. ##
## ---------------------- ##
# -*- compilation -*-
633. json.at:219: testing large integers that overflow to reals - Python2 ...
./json.at:219: printf %s "[9223372036854775807000,
-92233720368547758080000]" > input
./json.at:219: $PYTHON $srcdir/test-json.py input
stdout:
[9.223372036854776e+21,-9.223372036854776e+22]
./json.at:219: cat stdout
--- - 2016-04-30 13:18:34.042338507 -0500
+++ /home/terry/src/ovs/tests/testsuite.dir/at-groups/633/stdout
2016-04-30 13:18:34.031984775 -0500
@@ -1,2 +1,2 @@
-[9.22337203685478e+21,-9.22337203685478e+22]
+[9.223372036854776e+21,-9.223372036854776e+22]
input:
> [9223372036854775807000, -92233720368547758080000]633. json.at:219: 633.
> large integers that overflow to reals - Python2 (json.at:219): FAILED
> (json.at:219)
# -*- compilation -*-
634. json.at:219: testing large integers that overflow to reals - Python3 ...
./json.at:219: printf %s "[9223372036854775807000,
-92233720368547758080000]" > input
./json.at:219: $PYTHON3 $srcdir/test-json.py input
stdout:
[9.223372036854776e+21,-9.223372036854776e+22]
./json.at:219: cat stdout
--- - 2016-04-30 13:18:34.043961319 -0500
+++ /home/terry/src/ovs/tests/testsuite.dir/at-groups/634/stdout
2016-04-30 13:18:34.041734444 -0500
@@ -1,2 +1,2 @@
-[9.22337203685478e+21,-9.22337203685478e+22]
+[9.223372036854776e+21,-9.223372036854776e+22]
input:
> [9223372036854775807000, -92233720368547758080000]634. json.at:219: 634.
> large integers that overflow to reals - Python3 (json.at:219): FAILED
> (json.at:219)
# -*- compilation -*-
636. json.at:224: testing negative zero - Python2 ...
./json.at:224: printf %s "[-0, -0.0, 1e-9999, -1e-9999]" > input
./json.at:224: $PYTHON $srcdir/test-json.py input
stdout:
[0,0,0.0,0.0]
./json.at:224: cat stdout
--- - 2016-04-30 13:18:34.072286699 -0500
+++ /home/terry/src/ovs/tests/testsuite.dir/at-groups/636/stdout
2016-04-30 13:18:34.068816855 -0500
@@ -1,2 +1,2 @@
-[0,0,0,0]
+[0,0,0.0,0.0]
input:
> [-0, -0.0, 1e-9999, -1e-9999]636. json.at:224: 636. negative zero - Python2
> (json.at:224): FAILED (json.at:224)
# -*- compilation -*-
637. json.at:224: testing negative zero - Python3 ...
./json.at:224: printf %s "[-0, -0.0, 1e-9999, -1e-9999]" > input
./json.at:224: $PYTHON3 $srcdir/test-json.py input
stdout:
[0,0,0.0,0.0]
./json.at:224: cat stdout
--- - 2016-04-30 13:18:34.054673696 -0500
+++ /home/terry/src/ovs/tests/testsuite.dir/at-groups/637/stdout
2016-04-30 13:18:34.050400815 -0500
@@ -1,2 +1,2 @@
-[0,0,0,0]
+[0,0,0.0,0.0]
input:
> [-0, -0.0, 1e-9999, -1e-9999]637. json.at:224: 637. negative zero - Python3
> (json.at:224): FAILED (json.at:224)
# -*- compilation -*-
645. json.at:240: testing +/- DBL_MAX - Python2 ...
./json.at:240: printf %s "[1.7976931348623157e+308,
-1.7976931348623157e+308]" > input
./json.at:240: $PYTHON $srcdir/test-json.py input
stdout:
[1.7976931348623157e+308,-1.7976931348623157e+308]
./json.at:240: cat stdout
--- - 2016-04-30 13:18:34.193220996 -0500
+++ /home/terry/src/ovs/tests/testsuite.dir/at-groups/645/stdout
2016-04-30 13:18:34.191229356 -0500
@@ -1,2 +1,2 @@
-[1.79769313486232e+308,-1.79769313486232e+308]
+[1.7976931348623157e+308,-1.7976931348623157e+308]
input:
> [1.7976931348623157e+308, -1.7976931348623157e+308]645. json.at:240: 645. +/-
> DBL_MAX - Python2 (json.at:240): FAILED (json.at:240)
# -*- compilation -*-
646. json.at:240: testing +/- DBL_MAX - Python3 ...
./json.at:240: printf %s "[1.7976931348623157e+308,
-1.7976931348623157e+308]" > input
./json.at:240: $PYTHON3 $srcdir/test-json.py input
stdout:
[1.7976931348623157e+308,-1.7976931348623157e+308]
./json.at:240: cat stdout
--- - 2016-04-30 13:18:34.182268493 -0500
+++ /home/terry/src/ovs/tests/testsuite.dir/at-groups/646/stdout
2016-04-30 13:18:34.179313094 -0500
@@ -1,2 +1,2 @@
-[1.79769313486232e+308,-1.79769313486232e+308]
+[1.7976931348623157e+308,-1.7976931348623157e+308]
input:
> [1.7976931348623157e+308, -1.7976931348623157e+308]646. json.at:240: 646. +/-
> DBL_MAX - Python3 (json.at:240): FAILED (json.at:240)
# -*- compilation -*-
654. json.at:253: testing 1e-9999 underflows to 0 - Python2 ...
./json.at:253: printf %s "[1e-9999]" > input
./json.at:253: $PYTHON $srcdir/test-json.py input
stdout:
[0.0]
./json.at:253: cat stdout
--- - 2016-04-30 13:18:34.234905408 -0500
+++ /home/terry/src/ovs/tests/testsuite.dir/at-groups/654/stdout
2016-04-30 13:18:34.231311325 -0500
@@ -1,2 +1,2 @@
-[0]
+[0.0]
input:
> [1e-9999]654. json.at:253: 654. 1e-9999 underflows to 0 - Python2
> (json.at:253): FAILED (json.at:253)
# -*- compilation -*-
655. json.at:253: testing 1e-9999 underflows to 0 - Python3 ...
./json.at:253: printf %s "[1e-9999]" > input
./json.at:253: $PYTHON3 $srcdir/test-json.py input
stdout:
[0.0]
./json.at:253: cat stdout
--- - 2016-04-30 13:18:34.319529142 -0500
+++ /home/terry/src/ovs/tests/testsuite.dir/at-groups/655/stdout
2016-04-30 13:18:34.314725153 -0500
@@ -1,2 +1,2 @@
-[0]
+[0.0]
input:
> [1e-9999]655. json.at:253: 655. 1e-9999 underflows to 0 - Python3
> (json.at:253): FAILED (json.at:253)
_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev
