Re: [matplotlib-devel] Experimental, traited config module available in svn
Hi Eric, On Monday 30 July 2007 10:14:22 pm Eric Firing wrote: > I have not made any matplotlib.conf other than the one that is now being > installed correctly in mpl-data, and it contains > > # display grid on regular or polar axes > grid = False > polargrid = True > > grep indicates the True value is coming from mplconfig: > > config/mplconfig.py:grid = T.Trait(True, mplT.BoolHandler()) > > config/rcsetup.py:'axes.grid' : [False, validate_bool] > > so it looks like somehow this mplconfig initialization is overriding the > value in matplotlib.conf. Ok, sorry for the confusion. There were two problems: -The .conf file in mpl-data was not being read. It is fixed in svn 3645. -The .conf file is not auto-generated by setup.py. This is a more difficult problem, so I am not going to work on it unless we decide to adopt the new module. In the mean time, the file can be auto-generated by running mplconfig. Darren - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Experimental, traited config module available in svn
On Monday 30 July 2007 07:10:01 pm Eric Firing wrote:
> Darren Dale wrote:
> > I just committed changes in svn that will allow matplotlib to use the
> > experimental traited mplconfig module. The traited config object itself
> > is called mplConfig, but I wrapped it in an object called rcParams to
> > make it compatible with matplotlib, so we can kick the tires without
> > extensive changes.
>
> Darren,
>
> There is *major* speed problem. Running "backend_driver.py Template":
>
> 0.95 minutes with NEWCONFIG = True
> 0.51 minutes False
I used the following script in the matplotlib/config directory to profile the
mplconfig module:
def main():
import mplconfig
if __name__=='__main__':
import profile
import pstats
profile.run('main()', 'config.prof')
p = pstats.Stats('config.prof')
p.sort_stats('cumulative').print_stats(55)
I am attaching the results for two different tests. One is loading the default
mpl-data/matplotlib.conf, the other is loading an empty
~/.matplotlib/matplotlib.conf. I haven't done much work with profiling, maybe
others can make some comments about the results.
Tue Jul 31 10:50:12 2007config.prof
192414 function calls (182927 primitive calls) in 2.032 CPU seconds
Ordered by: internal time
List reduced from 925 to 55 due to restriction <55>
ncalls tottime percall cumtime percall filename:lineno(function)
2440.1480.0010.3200.001
/usr/lib64/python2.5/compiler/transformer.py:105(__init__)
32329/319520.1160.0000.1160.000 :0(len)
120440.1080.0000.1080.000 :0(append)
76770.1080.0000.1440.000
/usr/lib64/python2.5/sre_parse.py:188(__next)
182360.0960.0000.1040.000 :0(getattr)
216440.0920.0000.0960.000 :0(hasattr)
358/400.0840.0000.3880.010
/usr/lib64/python2.5/sre_parse.py:385(_parse)
0.0720.0000.0720.000 :0(isinstance)
480.0520.0010.1320.003
/usr/lib64/python2.5/site-packages/enthought.traits-2.0b2.dev_r13165-py2.5-linux-x86_64.egg/enthought/traits/has_traits.py:558(__init__)
1904/2440.0480.0000.1600.001
/usr/lib64/python2.5/compiler/transformer.py:1060(com_binary)
723/400.0440.0000.1600.004
/usr/lib64/python2.5/sre_compile.py:38(_compile)
1027/3470.0400.0000.0440.000
/usr/lib64/python2.5/sre_parse.py:146(getwidth)
2440.0360.0000.0640.000
/usr/lib64/python2.5/compiler/transformer.py:984(com_assign)
69370.0320.0000.1680.000
/usr/lib64/python2.5/sre_parse.py:207(get)
1760.0320.0000.1200.001
/usr/lib64/python2.5/site-packages/pkg_resources.py:1645(find_on_path)
246/2450.0280.0000.1400.001
/usr/lib64/python2.5/site-packages/enthought.traits-2.0b2.dev_r13165-py2.5-linux-x86_64.egg/enthought/traits/traits.py:1577(define)
248/400.0240.0000.3880.010
/usr/lib64/python2.5/sre_parse.py:307(_parse_sub)
48150.0240.0000.0240.000 :0(endswith)
40.0240.0060.7720.193
/usr/local/src/matplotlib/matplotlib/lib/matplotlib/config/configobj.py:1410(_parse)
141/1390.0240.0000.0720.001
/usr/lib64/python2.5/site-packages/enthought.traits-2.0b2.dev_r13165-py2.5-linux-x86_64.egg/enthought/traits/traits.py:1710(do_list)
8320.0200.0000.0280.000
/usr/lib64/python2.5/posixpath.py:56(join)
1220/2440.0200.0000.0200.000
/usr/lib64/python2.5/compiler/transformer.py:1349(get_docstring)
2412/2440.0200.0000.1400.001
/usr/lib64/python2.5/compiler/transformer.py:787(com_node)
20040.0200.0000.0280.000
/usr/lib64/python2.5/sre_parse.py:201(match)
1490.0200.0000.0800.001
/usr/lib64/python2.5/posixpath.py:410(realpath)
2760.0200.0000.0200.000
/usr/lib64/python2.5/site-packages/enthought.traits-2.0b2.dev_r13165-py2.5-linux-x86_64.egg/enthought/traits/traits.py:292(_default_value_type)
30170.0200.0000.0200.000 :0(get)
27500.0200.0000.0200.000
/usr/lib64/python2.5/sre_parse.py:136(__getitem__)
332/2440.0200.0000.1520.001
/usr/lib64/python2.5/compiler/transformer.py:579(test)
8910.0160.0000.0320.000
/usr/lib64/python2.5/site-packages/enthought.traits-2.0b2.dev_r13165-py2.5-linux-x86_64.egg/enthought/traits/has_traits.py:128(_get_def)
6910.0160.0000.0160.000 :0(match)
20270.0160.0000.0160.000 :0(range)
332/2440.0160.0000.1160.000
/usr/lib64/python2.5/compiler/transformer.py:611(comparison)
32870.0160.0000.0160.000 :0(lower)
2310.0160.0000.0240.000
/usr/lib64/python2.5/sre_compile.py:213(_optimize_charset)
2440.0160.0000.300
Re: [matplotlib-devel] Experimental, traited config module available in svn
On Tuesday 31 July 2007 11:08:42 am Darren Dale wrote:
> On Monday 30 July 2007 07:10:01 pm Eric Firing wrote:
> > Darren Dale wrote:
> > > I just committed changes in svn that will allow matplotlib to use the
> > > experimental traited mplconfig module. The traited config object itself
> > > is called mplConfig, but I wrapped it in an object called rcParams to
> > > make it compatible with matplotlib, so we can kick the tires without
> > > extensive changes.
> >
> > Darren,
> >
> > There is *major* speed problem. Running "backend_driver.py Template":
> >
> > 0.95 minutes with NEWCONFIG = True
> > 0.51 minutes False
>
> I used the following script in the matplotlib/config directory to profile
> the mplconfig module:
> def main():
> import mplconfig
>
> if __name__=='__main__':
> import profile
> import pstats
> profile.run('main()', 'config.prof')
>
> p = pstats.Stats('config.prof')
> p.sort_stats('cumulative').print_stats(55)
sorry, that should say 'time' instead of 'cumulative'
> I am attaching the results for two different tests. One is loading the
> default mpl-data/matplotlib.conf, the other is loading an empty
> ~/.matplotlib/matplotlib.conf. I haven't done much work with profiling,
> maybe others can make some comments about the results.
-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Experimental, traited config module available in svn
On 7/31/07, Darren Dale <[EMAIL PROTECTED]> wrote: > I am attaching the results for two different tests. One is loading the default > mpl-data/matplotlib.conf, the other is loading an empty > ~/.matplotlib/matplotlib.conf. I haven't done much work with profiling, maybe > others can make some comments about the results. I'm afraid I can't help with this right now, but you may want to look at http://amath.colorado.edu/faculty/fperez/python/profiling/ the little code I put in there makes using kcachegrind very easy for python profiling, and it's really a useful tool for analyzing profiling info. Cheers, f - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Experimental, traited config module available in svn
Darren,
The two cases you ran have almost identical timing, so the problem is
not in reading matplotlib.conf. Instead, it seems to be in the
initialization of all the machinery, and I suspect that something may
be getting run many times in a loop instead of just once. I used the
profile.py script method to profile the whole simple_plot.py demo
modified to use Agg, with NEWCONFIG True versus False. Attached are the
top 100 lines of the output, sorted by cumulative time, with "new"
meaning NEWCONFIG=True. The execution time of the new version is 3.2
seconds versus 1.8 for the old. (Both are pretty slow for making a
minimal plot from a script.)
Looking at the profile outputs, a couple of things stand out in the new
version. First, various Traits things are prominent near the top.
Second, look at the lines for posixpath functions, which are taking
quite a bit of time. Here are two examples:
56510.1080.0000.1880.000 posixpath.py:56(join)
[...]
2730.0120.0000.1640.001 posixpath.py:410(realpath)
If I understand correctly, path.join is being called 5651 times! And
realpath is being called 273 times! Now, these calls are contributing
only a little to the total time difference between the mpl versions, but
they may be providing clues as to what is wrong. My guess is that there
is a glitch in the program logic that is causing some operations that
should be done once or a few times to be done hundreds or thousands of
times, and this will turn out to be the biggest factor in the slowdown.
If so, then we will still have to see whether a significant slowdown
remains after the glitch is corrected.
I don't have a lot of experience with profiling, but it seems to me that
ordering results by cumulative time is good for finding the major
operations that are slow, whereas ordering by time is good for finding
the lowest-level functions that are holding things up. Ordering by time
shows these as the top two for the new system:
32581/316160.1520.0000.1680.000 :0(len)
980.1320.0010.3720.004 has_traits.py:558(__init__)
This doesn't mean that the len builtin is too slow, but that it is
getting called a lot of times, and we have to look elsewhere to figure
out how to reduce that, if possible. The second line, however, suggests
to me that even after the glitch problem is solved, we will be stuck
with a distressing startup overhead from using traits. This line is
saying that we are initializing 98 trait instances, and this
initialization alone is taking 0.37 seconds, of which 0.13 seconds is in
the init code, not in functions that it is calling. If you look at the
code for that __init__, you will see that it is huge. It may be that
the traits overhead is small once a program is up and running, but it
certainly looks to me like the startup overhead is a killer if more than
a few traits are to be used.
I would not like to see mpl become a library that is painfully sluggish
to start up; it is already too slow for my liking, with font setup being
a major contributor.
Eric
Darren Dale wrote:
On Monday 30 July 2007 07:10:01 pm Eric Firing wrote:
Darren Dale wrote:
I just committed changes in svn that will allow matplotlib to use the
experimental traited mplconfig module. The traited config object itself
is called mplConfig, but I wrapped it in an object called rcParams to
make it compatible with matplotlib, so we can kick the tires without
extensive changes.
Darren,
There is *major* speed problem. Running "backend_driver.py Template":
0.95 minutes with NEWCONFIG = True
0.51 minutes False
I used the following script in the matplotlib/config directory to profile the
mplconfig module:
def main():
import mplconfig
if __name__=='__main__':
import profile
import pstats
profile.run('main()', 'config.prof')
p = pstats.Stats('config.prof')
p.sort_stats('cumulative').print_stats(55)
I am attaching the results for two different tests. One is loading the default
mpl-data/matplotlib.conf, the other is loading an empty
~/.matplotlib/matplotlib.conf. I haven't done much work with profiling, maybe
others can make some comments about the results.
/home/efiring/programs/py/mpl/tests/matplotlibrc converted to
/home/efiring/programs/py/mpl/tests/matplotlib.conf
276205 function calls (272272 primitive calls) in 3.248 CPU seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
10.0080.0083.2483.248 simple_plot.py:5()
10.0000.0003.2483.248 :0(execfile)
137/10.0000.0003.2483.248 :1()
10.0000.0003.2483.248
profile:0(execfile('simple_plot.py'))
10.0040.0041.6481.648 __init__.py:55()
10.0000.0001.5921.592 __init__.py:3()
10.0040.004
Re: [matplotlib-devel] Experimental, traited config module available in svn
Eric Firing wrote: > I don't have a lot of experience with profiling... Slightly OT, but I do have a recommendation for anyone doing Python profiling. I don't usually get very excited about GUI tools, but KCachegrind (Linux only) is very slick and works great with Python. It's really handy to be able to "browse" the calltree interactively to explore what's going on. The trick (which is very non-obvious from the documentation), is that you use Python's 'hotspot' profiler, then convert that to 'calltree' format using hotspot2calltree (included with KCachegrind), and then load that with KCachegrind. Just my two cents for something you may want to try, Mike - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Experimental, traited config module available in svn
On 7/31/07, Michael Droettboom <[EMAIL PROTECTED]> wrote: > Eric Firing wrote: > > I don't have a lot of experience with profiling... > Slightly OT, but I do have a recommendation for anyone doing Python > profiling. I don't usually get very excited about GUI tools, but > KCachegrind (Linux only) is very slick and works great with Python. > It's really handy to be able to "browse" the calltree interactively to > explore what's going on. > > The trick (which is very non-obvious from the documentation), is that > you use Python's 'hotspot' profiler, then convert that to 'calltree' > format using hotspot2calltree (included with KCachegrind), and then load > that with KCachegrind. > > Just my two cents for something you may want to try, Yup. The link I posted earlier is basically a little wrapper that does all that. I wrote it so you wouldn't have to think :) I keep it in my path and can profile any python script with kcachegrind just by running one command. And yes, kcachegrind rocks. I'm also very much a non-gui person, yet I've found it invaluable for doing really complex profiling of very large python codes. Cheers, f - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Experimental, traited config module available in svn
On Tuesday 31 July 2007 02:12:06 pm you wrote: > Looking at the profile outputs, a couple of things stand out in the new > version. First, various Traits things are prominent near the top. pycachegrind is telling me that traits_db is responsible for about 20%, configobj is responsible for about 30%, and the config package another 30%. > Second, look at the lines for posixpath functions, which are taking > quite a bit of time. Here are two examples: > > 5651 0.108 0.000 0.188 0.000 posixpath.py:56(join) > [...] > 273 0.012 0.000 0.164 0.001 posixpath.py:410(realpath) > > If I understand correctly, path.join is being called 5651 times! And > realpath is being called 273 times! I think the posix_path stuff is being called most often by python while it attempts to locate modules. [...] > I would not like to see mpl become a library that is painfully sluggish > to start up; it is already too slow for my liking, with font setup being > a major contributor. I agree. Darren - This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
