On Sep 17, 2010, at 4:41 AM, David Schmitt wrote:
> On 9/16/2010 1:26 AM, Luke Kanies wrote:
>> Hi all,
>>
>> I've just stuck my proposal for a Catalog Service (which we've been
>> bandying about internally for a while, and which I've been thinking
>> about even longer) on the wiki:
>>
>> http://projects.puppetlabs.com/projects/puppet/wiki/CatalogServiceArchitecture
>
> Interesting read :-) here're a few notes:
I'll respond to the notes as necessary and update the document (probably on
this next flight) as appropriate, but separately.
I misread your notes at first - this first section is really about being very
clear as to the steps necessary to create this, right? I.e., it's an explicit
description of the work necessary to implement the document's goals?
> * document needs list of proposed functional changes, afaict:
> * insert RESTful API between puppetmaster and Catalog storage,
> thereby exposing a proper interface
> * decouple compilation and catalog serving completely
> * btw, using futures, one could compile a "template" catalog
> and only insert the changing fact values quickly?
This could be done, but I doubt that futures (which are a function in the
parser) would be the mechanism. I certainly wouldn't want to tie this to
futures, though.
> * enrichen search API to cover all resources and complex queries
> * implement additional backends
> * simple, no external dependencies
> * massively scalable, using some nosql solution
>
> * I'm wondering how the flat file based backend will perform in the face of
> 100 systems. My intuition says that traditional SQL storage will remain a
> viable (perfomance vs. configuration) solution in this space.
I expect a file back end to perform poorly with 100 systems - I think 30 is a
reasonable amount. I agree that SQL will continue to be viable, and quite
possibly a better long-term direction, at least for the next few years.
> * re directly exposing back-end interface: only an artifact of a badly
> designed API. If this really becomes a problem, perhaps building a more
> complex query, e.g. looking for multiple resource types, might be viable to
> avoid strong coupling to the backend
>
> * I'm reminded of a trick i used in the early days to emulate a Catalog query
> in the main scope:
>
> case $hostname {
> 'monitoring': {
> # apply monitoring stuff
> }
> 'webserver': {
> # install webserver
> }
> }
>
> Today it looks like an awful hack, but the underlying principle might prove
> interesting, even if only to strengthen the case of Catalog storage by
> discarding it.
Yep, I did something very similar with Cfengine in 2003, and that work is in
large part what drove me to write exported resources into Puppet. It works
just as well in Puppet as it did in Cfengine, though, and in some ways it's
superior. Note that I would tend to branch this by class membership rather
than hostname.
In particular, it gives you the option of having an application-stack view;
i.e., you can effectively say that a host is both a member of a given
application stack and and also performs the database function, and from there
Puppet can use conditionals to figure out all of the details. That's not
always as visible using exported resources, although of course there are other
benefits.
> To contrast this with a modern implementation:
>
> class monitoring {
> Monitoring::Service<<||>>
> }
>
> define monitoring::service::http() {
> @@monitoring::service {
> "http_${fqdn}_${name}":
> command => "check_http",
> target => $fqdn,
> args => $port;
> }
> }
>
> class webserver {
> monitoring::service::http { $port: }
> }
>
> The main difference between the two solutions is the dataflow. In the first
> solution, different resources are created from the same configuration,
> depending on the environment. In the latter version, compiling one manifest
> alters the environment for the other nodes.
>
> Suddenly that sounds so wrong :) If all facts/nodes are available on the
> server, shouldn't the puppetmaster be able to compile all Catalogs in one
> step? Is the next manifest legal? Discuss!
Yes, it should. Well, one step might be a stretch, but yeah, it should. I
envision a catalog service dishing catalogs to clients, and a pool of compiler
processes that pull compile requests off of a queue and compile as necessary.
The compile requests can be created by the client -- which would be a normal
model -- or by the dashboard, or as part of a commit hook in git, or whatever
you want.
> node A { y { 1: } }
> node B { x { 1: } }
>
> define y() {
> $next = $name+1
> @@x { $next: }
> Y<<||>>
> }
>
> define x() {
> $next = $name+1
> @@y { $next: }
> X<<||>>
> }
>
> If I'm not completely off, this will create lots and lots of resources as A
> and B are evaluated alternatively.
This might quite possible destroy the universe if resource collection didn't
ignore resources exported by the compiling host. Given that they do, though,
you'd likely just get flapping and some very pissed coworkers.
> The last part might be a little bit off-topic, but I think it does pertain to
> the whole "all-nodes-are-part-of-the-system" thinking that is the motivation
> for Catalog storage/queries.
Yeah, that's a good point - one of the big goals here is to lose the 'nodes sit
alone' perspective and really give them membership of part of a larger whole.
--
Men never do evil so completely and cheerfully as when they do it from a
religious conviction. --Blaise Pascal
---------------------------------------------------------------------
Luke Kanies -|- http://puppetlabs.com -|- +1(615)594-8199
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en.
