Hi,
I did a tiny bit of profiling on Cython compiling lxml.etree. Here are the
numbers I get:
"""
3127018 function calls (2777951 primitive calls) in 25.128 CPU seconds
Ordered by: internal time, call count
List reduced from 1035 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
119144 4.579 0.000 4.671 0.000
Scanners.py:148(run_machine_inlined)
18316 1.586 0.000 1.586 0.000 codecs.py:371(read)
119144 1.074 0.000 5.746 0.000 Scanners.py:109(scan_a_token)
88362 1.023 0.000 7.976 0.000 Scanners.py:88(read)
29680 0.673 0.000 1.659 0.000 Code.py:103(mark_pos)
77 0.588 0.008 0.588 0.008 posixpath.py:168(exists)
88362 0.538 0.000 8.514 0.000 Scanning.py:397(next)
23318/2985 0.495 0.000 0.517 0.000 Nodes.py:155(end_pos)
70446 0.442 0.000 0.591 0.000 Code.py:62(put)
90657/10562 0.351 0.000 4.549 0.000 Parsing.py:59(p_binop_expr)
[...]
"""
So the major headache here is Scanners.py in Plex. The method at the top-rank
is a huge function. According to the comments, it's the result of inlining a
couple of method calls that originally lead to slow code, and it looks heavily
profiled already.
Assuming that further optimisation attempts were rather futile, I just
compiled the module with Cython. The first (obvious) result is that the
internal calls disappear from the profile log, as they are now internal C
calls. The call that remains is Scanner.next(), which originally took an
accumulated 8.5 seconds. In the compiled version, it's down to just over 5
seconds, that's more than 40 percent faster.
"""
2595627 function calls (2246560 primitive calls) in 18.681 CPU seconds
Ordered by: internal time, call count
List reduced from 1028 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
88362 4.246 0.000 5.041 0.000 Scanning.py:397(next)
29680 0.673 0.000 1.632 0.000 Code.py:103(mark_pos)
70446 0.439 0.000 0.586 0.000 Code.py:62(put)
90657/10562 0.335 0.000 3.228 0.000 Parsing.py:59(p_binop_expr)
23318/2985 0.316 0.000 0.338 0.000 Nodes.py:155(end_pos)
65791 0.295 0.000 0.903 0.000 Code.py:47(putln)
29677 0.289 0.000 0.915 0.000 Code.py:93(file_contents)
4724 0.287 0.000 0.292 0.000 Symtab.py:532(allocate_temp)
88070 0.247 0.000 0.247 0.000 ExprNodes.py:192(subexpr_nodes)
52071 0.232 0.000 0.232 0.000 Nodes.py:82(__init__)
[...]
"""
In total, I get an improvement of 12% in compilation time. That makes me think
that it's actually worth putting the compilation into Cython's own setup.py,
and installing the compiled Scanning module next to the Python one (Python
prefers C extensions on import). Here's a patch, what do you think?
Stefan
# HG changeset patch
# User Stefan Behnel <[EMAIL PROTECTED]>
# Date 1212929816 -7200
# Node ID 5562d59529350243c8d7d1130e0f3541f2c5eb9e
# Parent fb47d44008f43174c14cc2c446d6bfb3bf85ce29
compile Cython.Plex.Scanners module on install to speed up compilation
diff -r fb47d44008f4 -r 5562d5952935 setup.py
--- a/setup.py Sat Jun 07 10:45:33 2008 -0700
+++ b/setup.py Sun Jun 08 14:56:56 2008 +0200
@@ -1,6 +1,6 @@ from distutils.core import setup
-from distutils.core import setup
+from distutils.core import setup, Extension
from distutils.sysconfig import get_python_lib
-import os, sys
+import os, os.path
import sys
from Cython.Compiler.Version import version
@@ -21,6 +21,31 @@ if os.name == "posix":
scripts = ["bin/cython"]
else:
scripts = ["cython.py"]
+
+try:
+ sys.argv.remove("--no-compile")
+except ValueError:
+ try:
+ from Cython.Compiler.Main import compile
+ source_root = os.path.dirname(__file__)
+ compiled_modules = ["Cython.Plex.Scanners"]
+ extensions = []
+ for module in compiled_modules:
+ source_file = os.path.join(source_root, *module.split('.'))
+ print("Compiling module %s ..." % module)
+ result = compile(source_file + ".py")
+ if result.c_file:
+ extensions.append(
+ Extension(module, sources = [result.c_file])
+ )
+ else:
+ print("Compilation failed")
+ if extensions:
+ setup_args['ext_modules'] = extensions
+ except Exception:
+ print("ERROR: %s" % sys.exc_info()[1])
+ print("Extension module compilation failed, using plain Python implementation")
+
setup(
name = 'Cython',
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
