On Thu, Sep 18, 2008 at 1:29 AM, Robert Kern <[EMAIL PROTECTED]> wrote: > On Wed, Sep 17, 2008 at 18:09, Ondrej Certik <[EMAIL PROTECTED]> wrote: >> On Wed, Sep 17, 2008 at 3:56 AM, Robert Kern <[EMAIL PROTECTED]> wrote: >>> On Mon, Sep 15, 2008 at 11:13, Arnar Flatberg <[EMAIL PROTECTED]> wrote: >>>> That would make me an extremely happy user, I've been looking for this for >>>> years! >>>> I can't imagine I'm the only one who profiles some hundred lines of code >>>> and >>>> ends up with 90% of total time in the dot-function >>> >>> For the time being, you can grab it here: >>> >>> http://www.enthought.com/~rkern/cgi-bin/hgwebdir.cgi/line_profiler/ >>> >>> It requires Cython and a C compiler to build. I'm still debating >>> myself about the desired workflow for using it, but for now, it only >>> profiles functions which you have registered with it. I have made the >>> profiler work as a decorator to make this easy. E.g., >>> >>> from line_profiler import LineProfiler >>> >>> profile = LineProfiler() >>> >>> @profile >>> def my_slow_func(): >>> ... >>> >>> profile.dump_stats('my_slow_func.lprof') >>> >>> This is kind of inconvenient, so I have a little script that handles >>> everything except for the @profile itself. It started as a script to >>> run cProfile nicely, so it actually does that by default. >>> >>> I took pystone.py from the Python source and added a couple of >>> @profile decorators to demonstrate: >>> >>> >>> [line_profiler]$ ./kernprof.py --help >>> Usage: ./kernprof.py [-s setupfile] [-o output_file_path] scriptfile [arg] >>> ... >>> >>> Options: >>> -h, --help show this help message and exit >>> -l, --line-by-line Use the line-by-line profiler from the line_profiler >>> module instead of cProfile. Implies --builtin. >>> -b, --builtin Put 'profile' in the builtins. Use 'profile.enable()' >>> and 'profile.disable()' in your code to turn it on >>> and >>> off, or '@profile' to decorate a single function, or >>> 'with profile:' to profile a single section of code. >>> -o OUTFILE, --outfile=OUTFILE >>> Save stats to <outfile> >>> -s SETUP, --setup=SETUP >>> Code to execute before the code to profile >>> >>> [line_profiler]$ ./kernprof.py -l pystone.py >>> Pystone(1.1) time for 50000 passes = 11.34 >>> This machine benchmarks at 4409.17 pystones/second >>> Wrote profile results to pystone.py.lprof >>> >>> [line_profiler]$ ./view_line_prof.py pystone.py.lprof >>> Timer unit: 1e-06 s >>> >>> File: pystone.py >>> Function: Proc0 at line 79 >>> Total time: 8.46516 s >> [...] >> >> This is what I am getting: >> >> $ ./kernprof.py -l pystone.py >> Wrote profile results to pystone.py.lprof >> $ ./view_line_prof.py pystone.py.lprof >> Timer unit: 1e-06 s >> >> $ >> >> So I think you meant: >> >> $ ./kernprof.py -l mystone.py >> 20628 >> Wrote profile results to mystone.py.lprof >> $ ./view_line_prof.py mystone.py.lprof >> Timer unit: 1e-06 s >> >> File: pystone.py >> Function: Proc0 at line 79 >> Total time: 13.0803 s >> [...] >> >> Now it works. > > No, I meant pystone.py. My script-finding code may have (incorrectly) > found a different, uninstrumented pystone.py file somewhere else, > though. Try with "./pystone.py". > >> This is an excellent piece of software! Nice job. Just today I needed >> such a thing! >> >> >> How do you easily install it? > > "python setup.py install" should have installed the module. I haven't > done anything with the scripts, yet. > >> I usually do "python setup.py install >> --home=~/lib" and I have the PYTHONPATH + PATH setup in my .bashrc, >> but I then need to manually remove the stuff from my ~/lib if I want >> to uninstall, which sucks. So this time I just did "python setup.py >> build" and moved the .so file manually to the current dir. But there >> must be a better way. What is your workflow? > > For things I am developing on, I build them in-place, using the > setuptools "develop" command to add the appropriate path to the > easy-install.pth file. To remove, I would just edit that file. > > For thing I'm not developing on, I usually build and install an egg if > at all possible. > > But then, I'm typically on a single-user box where I'm root, so I > sometimes do nasty and unsanitary things like chown -R rkern:rkern > /usr/local/. > >> Anyway, so I used it on my code and here is what I got: >> >> File: hermes1d/mesh.py >> Function: integrate_function at line 119 >> Total time: 0.647412 s >> >> Line # Hits Time % Time Line Contents >> ===================================================== >> 119 @profile >> 120 def integrate_function(self, f): >> 121 """ >> 122 Integrate the function >> "f" on the element. >> 123 """ >> 124 96 1091 0.2 from numpy import array >> 125 96 461070 71.2 from scipy.integrate >> import quadrature >> 126 96 496 0.1 a, b = >> self.nodes[0].x, self.nodes[1].x >> 127 96 418 0.1 def func(x): >> 128 #print x >> 129 #print >> array([f.f(y) for y in x]) >> 130 return >> array([f.f(y) for y in x]) >> 131 96 183903 28.4 val, err = >> quadrature(func, a, b) >> 132 #print val, a, b >> 133 #stop >> 134 96 434 0.1 return val >> >> >> >> This is interesting, because 70% of time is spent on "from >> scipy.integrate import quadrature". So this is not good. >> >> So I moved it out and voila! My code is 3.4x faster. I really didn't >> know that importing (scipy.integrate) is *so* slow. > > It should only be slow the first time. If it's not, then there's a > problem somewhere (and I wouldn't put it past being in the profiler; > it might screw up something in the import mechanism). Can you do the > import both outside the function *and* inside to see if that changes > the result?
Sure. "b.py": from scipy.integrate import quadrature --------------- def a(): b = 1 for x in range(1000000): a() --------------- "a.py": --------------- def a(): from scipy.integrate import quadrature b = 1 for x in range(1000000): a() --------------- and: [EMAIL PROTECTED]:~/repos/hermes1d$ time python a.py real 0m2.655s user 0m2.612s sys 0m0.044s [EMAIL PROTECTED]:~/repos/hermes1d$ time python b.py real 0m0.620s user 0m0.540s sys 0m0.080s So I guess that speaks for itself. Don't have time to dig more, but I think it's this line in scipy/integrate/__init__: __all__ = filter(lambda s:not s.startswith('_'),dir()) But maybe also something else, I don't know. Ondrej Ondrej _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion