On Sep 17, 2010, at 6:08 AM, Brice Figureau wrote: > On Wed, 2010-09-15 at 16:26 -0700, Luke Kanies wrote: >> Hi all, >> >> I've just stuck my proposal for a Catalog Service (which we've been >> bandying about internally for a while, and which I've been thinking >> about even longer) on the wiki: >> >> http://projects.puppetlabs.com/projects/puppet/wiki/CatalogServiceArchitecture >> >> Comments appreciated, and ideas for how you would use it appreciated >> even more. > > I need to read it again, but the envisioned change looks really > interesting. I like this move :) > > There's something interesting about node dependencies. I understand how > all the nodes catalogs form a graph, with one node catalog being a > subgraph. However, is there a plan to extend the puppet/ruby DSL to > express/enforce such dependencies?
Not yet, but I've been thinking about it. Really, how they're used is more of a problem than the expression, since they should generally be created automatically from resource collection. However, my 'external resource' prototype from April or so of this year is a decent example of how this might work, if you want to start taking advantage of them. One of the things this will force us to begin to resolve is what these dependencies should change in terms of behaviour. Obviously you've got things like a required host's catalog being updated resulting in the requiring host's catalog getting updated, but should the requiring host wait on a service to come up when running the transaction? All unclear, at this point. > I'm also wondering what would happen when serving a catalog of node A > which requires some resources from node B which hasn't been yet > compiled. Compiling node B might not be an option because we need those > node facts (BTW, how do the facts fit in this system?). Theoretically, this can't happen - the dependencies are all automatic, resulting from a host pulling another host's resources into its catalog, which means that you don't actually know about a dependency until both hosts have compiled their catalog. This could easily be seen as either a bug or a feature. As to facts, this service definitely (and explicitly, I believe) requires the Inventory Service, which is basically just a service that stores and dishes facts, and from the agent's perspective everything's exactly the same - upload facts, download catalog. I do expect this to change at some point, with the agent having a separate 'register' operation that sends facts, and then the existing operation that pulls and runs the catalog. Or really, breaking it into three operations: puppet agent register puppet agent retrieve/update/whatever puppet agent run > It would be great if your document could more clearly explain the > various possible architectures (ie combined master/catalog service, > multiple master to one catalog service, decentralized catalog services > if possible...). Also it seems you imply it is needed to run both a > master and a catalog service as separate processes (even on the same > host), which I thought wasn't necessary (ie indirect directly to the > terminus instead of :rest). I'll make this more clear. You should definitely be able to run it all in one process. > Regarding queries: > I'm not sure I like the second proposed query system where you separate > tags and parameters. I prefer the current system where parameters are in > the "top namespace" like tags. I agree, but I wanted to make it clear both were at least feasible. I'll more obviously express a preference in the document. > I remember we also support OR which are not in your proposal. I > personally don't use any ORs, and I can understand how it can make > everything complex... I don't believe we do in the Rails integration - in the build_active_record_query method, we just do ANDing. I thought we'd added it, but the last time I said that, I think you corrected me. :) > Regarding back-ends: > I have lots of doubts about the performance of the text-file solution, > though. I completely agree. Can you think of another solution that doesn't add dependencies? > One of the nice things of using a RDBMS is that you can use one or > another quite independently. Once we'll chose a specific NoSQL or a > graph database, I'm afraid we'll have to live a long time with it. > I'm not talking about implementation, but actual use of the datastore, > which might prove to be painful, if for some reason we find it doesn't > work as advertised and we need to switch to the new kid on the block. > > If possible, I think your document should better enforce the fact > back-ends will be plugins (ie abstracted), so that we could let the user > chose among RDBMS or NoSQL (provided those plugins exists of course). Ok. > And finally, you're talking about migration tool. I don't really see how > it will be possible to migrate from the current storeconfigs back-end to > a newer system since you told us that one of the issue of storeconfigs > was it was lacking some necessary information. The only way I would see > a migration is: wait all your nodes ask for a catalog, then switch off > storeconfigs :) Yeah, probably true. > About storing all the revisions of a given catalog, one of the issue > would be that you need to implement in the service an option to retrieve > one specific version or the list of all versions. This can be easy to do > with couchdb because it's how it works, but other NoSQL solutions might > not be able to provide an easy way to do this. Of course this is an > interesting and powerful feature for auditing changes (especially if > there is a corresponding audit tool). I think this is going to be important for the future, but I didn't want to make it absolutely required for the first version. I agree that the api would need support for that, but it can't be that hard - get a specific version or date of a catalog for a specific host, right? > Oh and I learnt a new word: paucity. Not sure I'll be able to use it in > conversations, but thanks anyway :) :) -- I have never met a man so ignorant that I couldn't learn something from him. --Galileo Galilei --------------------------------------------------------------------- Luke Kanies -|- http://puppetlabs.com -|- +1(615)594-8199 -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
