On Apr 7, 2011, at 9:19 AM, Andrew Bennetts wrote:
> Itamar Turner-Trauring wrote:
> […]
>> So, the design has to *not* rely on caching working.
>
> FWIW: this is an achievable goal. I have 32 different bzr plugins
> currently installed, and here's the difference they make:
>
> $ time bzr --no-plugins rocks
> It sure does!
>
> real 0m0.075s
>
> $ time bzr rocks
> It sure does!
>
> real 0m0.119s
>
> So that's about 1.5ms per plugin, on average. With a hot disk cache, at
> least…
Is your cache as hot for Twisted as for bzr? Have you replicated these results
in a randomized, double-blind clinical trial? ;-)
I'm not surprised that bzr has faster startup though; twistd has not been (and
doubtful will ever be) nearly so ruthlessly optimized. Maybe it's time to put
a startup benchmark on <http://speed.twistedmatrix.com/>, at least that way we
could keep track.
> For comparison, 'twistd --version' takes 116ms, with a dropin.cache and
> (I think, although how can I tell?) no plugins installed.
Twisted itself installs 22 dropins (python files which each define at least one
plugin), which comprise 48 plugins of various types, so there are always some.
You should be able to tell, though. It's pathetic that we don't have a
command-line tool to inspect the available plugins and what they're doing.
Independent of the other issues under discussion here:
<http://twistedmatrix.com/trac/ticket/5039>.
But this is all moot. 'twistd --version' doesn't scan for plugins, so that's
all just the normal startup time; apparently we import too much in the first
place. The thing to compare with is 'twistd --help' or even just 'twistd
[some-plugin]' (since invoking one plugin actually loads all of them). Plus -
this is really the genesis for this thread - the dropin.cache isn't really
saving us much work at all right now, because all the plugins get loaded anyway
for all practical uses of plugin scanning.
> In part, we achieve this via the bzrlib.lazy_import hack, which plugins
> can and often do use, and by encouraging plugin authors to put as little
> code into their __init__.py files as possible. A typical plugin's
> __init__ might do just:
>
> # This is example_plugin/__init__.py
> # The actual command implementation is in
> # example_plugin/example_commands.py
> from bzrlib import commands
> commands.plugin_cmds.register_lazy('cmd_class_name', [],
> 'bzrlib.plugins.example_plugin.example_commands')
This looks very similar to ServiceMaker.
> Glyph's expressed scepticism that plugin authors and maintainers will
> know to keep their __init__.py files cheap to import. Bazaar's
> experience is different. Partly that's probably because the Bazaar
> community has paid a fair bit of attention to start up time and I
> suppose Twisted doesn't have that.
Yeah, bzr's audience makes this easier. For one thing, the audience is much
bigger :), but more importantly, bzr is a user-facing tool which users are
running _constantly_ at the command line. The only visible consequence of a
rogue twistd plugin is that your server which runs for days at a time takes
0.2s longer to start; the real problem sets in later, where your 25
subprocesses are suddenly consuming an additional 50meg each because of the
extra plugin they loaded. You do find this eventually, it's just rare to find
it while you're writing the plugin.
> But I think also it's partly because
> we've provided tools to help people diagnose what/who to blame for bzr
> being slow to start, like 'bzr --profile-imports', and even the crude
> 'time bzr rocks'.
Yes. These are a great idea, and there's no excuse that Twisted's plugin
system is so difficult to inspect and debug. A couple of good tools would
address a wide range of plugin issues, many of them much more interesting than
performance, like the ever-popular "why isn't my plugin getting loaded".
Thanks for the impetus to file the ticket above. (I kinda hope it's a dup, but
I couldn't find one.)_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
