On Dec 16, 2010, at 5:00 AM, Brice Figureau wrote:
> Hi,
>
> I noticed lately a couple of threads on the puppet-users list about the
> puppetmaster processes taking 100% CPU for an unknown reasons.
> There might be a bug there, which we have to find, but the real issue I
> think is that we don't really know what a given process is doing at a
> given time (and that's also true for apparently stuck puppetd).
>
> I was thinking about abusing the process name to display what the
> process is currently doing ala:
>
> puppet master: compiling node.masterzen.fr since 0.2s
>
> or
>
> puppet master: serving metadata for /modules/mysql/mysql.cnf
>
> or
>
> puppet agent: exec /usr/bin/apt-get update
>
> This is quite easy to add in an (almost) unobtrusely(?) way to the
> puppet core. My idea would be that we could trigger this functionality
> by firing USR1 (or whatever signal) to the process (the feature being
> inactive by default).
>
> In terms of use in the code, I was thinking about something like
> (pseudo-ruby-code):
>
> class ...
>
> include ProcessNameIntrumentation
>
> ...
>
> def compile(node)
> instrument("compiling #{node}") do
> # normal process compilation
> end
> end
> end
>
> It's still yet unknown how I can display the "since 0.2s" part, which
> would require a thread (but with MRI green thread implementations this
> part might not change).
>
> There's also the question about what to do for multiple inner
> instrumentation like:
>
> instrument("compiling #{node}") do
> instrument("parsing file #{file}") do
> ..
> end
> instrument("evaluating #{klass}") do
> end
> end
>
> I originally planed to concatenate all the strings, but something
> ultra-über-cool would be to have a kind of ticker things that "scrolls"
> the long string (so that in top or by using watch you can see the thing
> really scrolling).
>
> My plan is to work on this during the next (rainy) week-end, but I
> thought I should first ask if the idea sounds as cool/useful as I think
> and/or if the usage is as unobstrusive as I think.
>
> In short: is this a good idea? is there a better solution?
I guess I want to separate the question a bit, between is it a good idea to
have instrumentation, and is the proposed solution a good one?
To the first, I say easily yes - some form of instrumentation in Puppet beyond
the basic benchmarking of very limited things we have would be fantastic.
To the second, I guess there are a few questions I'd like to think about before
being able to answer it. Feel free to ignore any of these if you think they're
irrelevant; they're just what popped into my head as a means of breaking it
down.
* Are there other examples of this problem being solved in the ruby community
that we can crib off of? It'd be great not to have to reinvent the wheel,
especially in terms of design. I really wish we could just add dtrace probes
for it, but that only works on solaris and os x.
* Is there a difference between instrumentation to discover the problem behind
stuck processes, and instrumentation to generally know what's going on in
Puppet? I.e., people probably would like some more understanding of what's
slow without having to open a debugger, but this probably doesn't require the
extra step of threads reporting separately.
* Do you have an idea of how you might implement the methods that do the
instrumentation?
* How do you decide where to put the instrumentation? And how worried are you
about doing so at the subsystem level (e.g., networking, parser, indirector)
vs. individual classes?
I know this is a lot of background, but it's a top-level cross-cutting design,
and I think it's hard enough that I've thought about it for a long time and
never delivered on anything.
--
I don't want the world, I just want your half.
---------------------------------------------------------------------
Luke Kanies -|- http://puppetlabs.com -|- +1(615)594-8199
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en.
