On Mon, 2010-09-27 at 18:17 -0700, Luke Kanies wrote: > Hi all, > > This is (hopefully) just a short write-up of some prototyping Volcane > and I have done today. > > Here's what I did: > > * Added a CatalogRequest class to indicate that a compile is needed > > * Modified the agent to submit a CatalogRequest into a Stomp queue and > block waiting for a catalog to appear on a different (per-host) queue > > * Created a new 'compiler' daemon (run as 'puppet compiler') to pull > CatalogRequests off of the queue, compile the catalogs, and store the > results into a different queue
It looks like it is completely decorrelated from the usual indirector system? > Thus, we have completely asynchronous compiles - the client currently > blocks waiting for a catalog, but it doesn't really have to. You could > start one compiler process or 1000, because they're share-nothing and > they have no state other than the individual catalog being compiled. This is awesome! How does it fit in your "Catalog Service" model you were proposing a couple of weeks ago? > I've pushed the code into a 'feature/master/asynchrony' branch in my > repo on github. > > Downsides of my implementation, from my commit: > > * It's a big conglomeration of hack, just to prove the idea > > * It adds some unrelated stuff > > * It completely breaks our existing Queue class > > * There aren't any tests for any of the new work > > Additional downsides: > > * The Queueing subsystem has very clearly become separate from the > Indirector subsystem, with duplicated and confusing effort That's not a goog news. You (and the rest of the team) had spent lots of time polishing the indirector, up to the point it is a central and clean point in the puppet code. > * I'd hoped to have an option that had the client still speaking > http to the server and the server doing all of the async bits, so you > wouldn't need to upgrade your client to get this functionality, but I > did not succeed. I think I could do it now, though. > > Questions I have from this: > > * Is this useful enough to make sure it's in 2.7 (which should be > out in December)? I think it would be useful to some of the biggest infrastructure that already use a queueing system. It looks like most big players are still using 0.25. I think most people would use this system if it has a proven better scalability than the current synchronous model. > * Do you have any comments or concerns on the basic idea? I really like the idea. I've always be a proponent user of queueing systems. I'm more reluctant about having the end clients connecting to the queueing system directly. REST/HTTP is imho preferrable over Stomp over SSL (ie I see a potential problem of client trust, or at least setup complexity regarding SSL). Also it might be easier to firewall HTTP than Stomp (excuse my complete ignorance about stomp here). > * Should we attempt to reconcile the model of the queueing and > indirector systems, or should we just have a sync and async subsystem > that are nearly entirely separate? *shudder* That'd be great to be able to transform the indirector in an async system, which would be still usable in a synchronous mode. Best of both worlds... I'm simply not sure how it can be done (even though the asynchronous system could just be a layer above the indirector). More generally, I think I'd prefer puppetd to still connect using REST/HTTP/SSL to the local/remote master which in turn could use the queued compiler (through Stomp). The client can still be using poor-man's asynchronoucity by actively polling his master. I think there will be some interesting stuff to discuss at Puppetcamp :) -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
