Just for everyone's FYI while we are talking about profilers, Floris Bruynooghe (who I am cc'ing on this so he can contribute to the conversation), for Google's Summer of Code, wrote a replacement for 'profile' that uses Hotshot directly. Thanks to his direct use of Hotshot and rewrite of pstats it loads Hotshot data 30% faster and also alleviates keeping 'profile' around and its slightly questionable license.
You can find his project at http://savannah.nongnu.org/projects/pyprof/ . I believe he also tweaked Hotshot to accept custom timing functions. I have not had a chance to go over his code to clean it up for putting it up on SF, but I thought people should be aware of it. -Brett On 11/19/05, Armin Rigo <[EMAIL PROTECTED]> wrote: > Hi! > > The current Python profilers situation is a mess. > > 'profile.Profile' is the ages-old pure Python profiler. At the end of a > run, it builds a dict that is inspected by 'pstats.Stats'. It has some > recent support for profiling C calls, which however make it crash in > some cases [1]. And of course it's slow (makes a run take about 10x > longer). > > 'hotshot', new from 2.2, is quite faster (reportedly, only 30% added > overhead). The log file is then loaded and turned into an instance of > the same 'pstats.Stats'. This loading takes ages. The reason is that > the log file only records events, and loading is done by instantiating a > 'profile.Profile' and sending it all the events. In other words, it > takes exactly as long as the time it spared in the first place! > Moreover, for some reasons, the results given by hotshot seem sometimes > quite wrong. (I don't understand why, but I've seen it myself, and it's > been reported by various people, e.g. [2].) 'hotshot' doesn't know > about C calls, but it can log line events, although this information is > lost(!) in the final conversion to a 'pstats.Stats'. > > 'lsprof' is a third profiler by Brett Rosen and Ted Czotter, posted on > SF in June [2]. Michael Hudson and me did some minor clean-ups and > improvements on it, and it seems to be quite useful. It is, for > example, the only of the three profilers that managed to give sensible > information about the PyPy translation process without crashing, > allowing us to accelerate it from over 30 to under 20 minutes. The SF > patch contains a more detailed account on the reasons for writing > 'lsprof'. The current version [3] does not support C calls nor line > events. It has its own simple interface, which is not compatible with > any of the other two profilers. However, unlike the other two > profilers, it can record detailed stats about children, which I found > quite useful (e.g. how much take is spent in a function when it is > called by another specific function). > > Therefore, I think it would be a great idea to add 'lsprof' to the > standard library. Unless there are objections, it seems that the best > plan is to keep 'profile.py' as a pure Python implementation and replace > 'hotshot' with 'lsprof'. Indeed, I don't see any obvious advantage that > 'hotshot' has over 'lsprof', and I certainly see more than one downside. > Maybe someone has a use for (and undocumented ways to fish for) line > events generated by hotshot. Well, there is a script [4] to convert > hotshot log files to some format that a KDE tool [5] can display. (It > even looks like hotshot files were designed with this in mind.) Given > that the people doing that can still compile 'hotshot' as a separate > extension module, it doesn't strike me as a particularly good reason to > keep Yet Another Profiler in the standard library. > > So here is my plan: > > Unify a bit more the interfaces of the pure Python and the C profilers. > This also means that 'lsprof' should be made to use a pstats-compatible > log format. The 'pstats' documentation specifically says that the file > format can change: that would give 'lsprof' a place to store its > detailed children stats. > > Then we can provide a dummy 'hotshot.py' for compatibility, remove its > documentation, and provide documentation for 'lsprof'. > > If anyone feels like this is a bad idea, please speak up. > > > A bientot, > > Armin > > > [1] > https://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=1117670 > > [2] > http://sourceforge.net/tracker/?group_id=5470&atid=305470&func=detail&aid=1212837 > > [3] http://codespeak.net/svn/user/arigo/hack/misc/lsprof (Subversion) > > [4] http://mail.python.org/pipermail/python-list/2003-September/183887.html > > [5] http://kcachegrind.sourceforge.net/cgi-bin/show.cgi > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com