On Thu, Feb 4, 2010 at 12:30 AM, Stefan Behnel <[email protected]> wrote: > > David Cournapeau, 03.02.2010 14:45: >> On Wed, Feb 3, 2010 at 9:11 PM, Stefan Behnel wrote: >>> Dag Sverre Seljebotn, 03.02.2010 12:20: >>>> We should have an option to split Cython-generated code into reusable >>>> include-files at some point too, and perhaps make use of precompiled >>>> headers. >>> Given that the largest part of code in a Cython generated C file is due to >>> real user code, I doubt that there is much Cython can improve by spitting >>> out multiple files here. >> >> I don't think that's always true. Cython quickly generates files which >> are about 1 Mb of content. For relatively simple code (with a couple >> of python-visible functions), generated files are 5000 LOC, whereas a >> hand-made file would be much smaller. > > What I meant, was, that the amount of code and the compiler runtime is not > due to code that Cython could extract into header files. Cython generates a > lot of (type) special casing code that you'd never write manually, e.g. for > loops or function argument extraction. The result is that your hand-written > C-API-using code simply isn't as fast as the code that Cython generates > (except for selected cases where Cython isn't good enough yet). > > I would also guess that the compile time is substantially increased by the > amount of inline functions that Cython uses for type conversions.
That was one of my first guess last time I looked at the problem, but I don't think that's the real culprit : I tried simple things like removing every inline, and it did not make a difference (both for -O0 and -O2). > The C compiler would still have to handle it, > regardless of where it came from. Not if it is pre-compiled. > Parsing is only a tiny part of what the C > compiler does these days. Optimisation takes several times longer. I don't think parsing takes time either. But I have generally observed that for a unit of compilation of N lines taking T time, 2 * N lines takes more, sometimes much more than 2 * T, especially for N >= few thousand lines. > When I pass -O3, however, the compile time > jumps up to some 40 seconds, i.e. more than a factor of 4, which is quite > noticeable in comparison. Yes, one of the main advantage of numscons for numpy (and even more for scipy) is that I can control CFLAGS easily and reliably when working on it. It is almost one order of magnitude faster than with distutils if you care about reliability (which often means removing a lot of built files with distutils). > BTW, I know that this is not directly comparable to other projects that use > several separate modules. I guess there is more overhead if you have a > larger number of small Cython modules. But that should be mitigated by > running partial builds (which you'd certainly do during a compile-test cycle). Yes // builds help a lot for large projects, for example scipy - a fully optimized scipy build takes < 2 minutes on a beefy computer, and ~ 45 seconds for debug builds. At that point, numscons and scons become the bottleneck. cheers, David _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
