Hi Deepak, great to hear from you. Didn't expect you to join the conversation, but as I offended your baby... sorry for this ;)
Am 01.04.2016 um 20:42 schrieb Deepak Giridharagopal: > ... Maybe the thing to do here is to try some experiments and post back > some numbers that could hopefully ground the discussion with some data? Absolutely, I guess this is one of the things I was indirectly asking for. No objection against letting C++ do the lexing work. But please let's get some numbers before introducing the next caching mechanism. > The vibe I'm getting from this line of feedback is that we should > perhaps better articulate the longer-term plan around the native > compiler in general, instead of focusing on increments (like .xpp) that, > absent the larger context, may seem unhelpful in their own right? Yes, please - that would be awesome! > "Databases are slow" > > We had active-records hammering our databases. The conclusion wasn't > that someone with SQL knowledge should design a good schema. The > publicly stated reasoning was "well, databases are slow, so we need more > cores to hammer the database, Ruby has not threading, Clojure is cool". > It still was slow by the end, so we added a message queue, a dead letter > office and more to the mix. > > > With respect, I think this is a pretty unfair retelling of history. Even > in the Puppetconf talk where I introduced PDB, this was not the story. > I'm comfortable letting all the public video footage of us discussing > the rationale rebut this. Sorry Deepak, it wasn't meant like this. But yes, I was sitting there in 2012, listening to your talk. You're absolutely right, these where not your words, you said nothing like this. But given the context this was the message one could have read between the lines. At least I did. The introduction of PuppetDB was that single piece that was "a little too much" for me at that time. Remember, with this we started to have to run two different RDBMS in parallel (old Dashboard), two distinct message queues (MCO was also there), Ruby, Java, Clojure... for many people just to configure NTP and Apache "the right way". It wasn't for PuppetDB, that's a cool product. And it's getting better with every release. It was for the Puppet-picture as a whole. I was sitting there and listened, I "hated" you in that moment. I had the chance to personally meet you twice. You are a very intelligent, handsome and friendly person. And I never thought I would state this in public. But yes, I was sitting there and I really hated you. Please believe me, it was nothing personal. I hope you will forgive me one far day ;-) > Queueing had nothing to do with speed of persistence... That's true. There is nothing wrong with queuing. What I wanted to state was that at that time for most if not all "storeconfig users" a better schema and well-thought queries would have solved all of their issues. Even if blocking and run by plain Ruby. > The dead-letter-office is unrelated to performance ... so we can debug > it more thoroughly Sure, absolutely makes sense. But how many people do you think know that it even exists? It's a perfectly valid feature, a wonderful addition to solve a specific problem. I would never drop it. In the context of shiny new XPP this should just have been an example for "a far simpler solution would eventually also work out". > If writes take 0.5 seconds, then you'd start failing agent runs on any > site that was > 3600 nodes using a 30 minute runinterval. It's 1400 new, unseen resources in 0.5 seconds. So, 3000 inserts a second, single connection, index rebuilding, full ACID, no cheating, no SSD. No problem to run lot's of those transactions in parallel I guess. Single transaction, so no excessive syncing/index update involved. MUCH faster on the next catalog for the same node, as 99% of the resources will already be there. One single query figures out which resources are to be stored. Nothing complex, but not that bad either. I mentioned those numbers with the intent to say "Hey, if we need seconds to build a catalog plus time to serialize and ship that catalog, a fraction of a second to persist it should be fine." PuppetDB is a great product. It also was a big investment. > In any case, if you've got your own well-tuned system for persisting > catalogs, you can use that in place of puppetdb if you like. You could > reuse the puppetdb terminus and swap your backend in (the wire formats > are documented > here > https://docs.puppetlabs.com/puppetdb/4.0/api/wire_format/catalog_format_v6.html, > and the spec for queries are documented in the same spot). Is your code > in a place where you can open source it? No objection against open sourcing it. I also have no problem with completely dropping it while trying to bring some of it's ideas to PuppetDB. In case I'm going to make it public, I'd however need to sort out some ideas. Just to give you an idea of what else is in the mix: * first, I absolutely didn't want to write a new PuppetDB. It was meant to be a quick & dirty tool to diff lot's of catalogs in preparation for a migration to Puppet 4. Got other task to do and it was then left untouched for a long time. It doesn't even have a Git repo. * of course with some cleanup it could perfectly be used as a serious alternative puppetdb terminus. But hey, there is PuppetDB, "just another database" makes not much sense to me * it's fact and catalog diffing capabilities are fantastic, even in it's early stage. Before making it public I'd however love to figure out whether and how it could scale out * I have some very simple but nice ideas for a better Puppet module lifecycle management in combination with this db. Wouldn't it be great to be able to restore the exact module combination used to built a specific historic catalog? * I first played with the puppetdb terminus but then really fast opted for an "inversion of control". To gain more flexibility I decided to feed the compiler by myself. Remember, it was about running multiple Puppet and Facter versions and comparing their catalogs. So I ended up with flags allowing me to switch Puppet and Facter version for every single run. * At that time I also played a lot with Puppet over SSH. I recently mentioned this in an Ignite talk: http://cfgmgmtcamp.eu/schedule/ignites/Thomas.pdf It wasn't meant as a serious proposal, it was about encouraging people to think around the corner. But hey, it works. And as pluginsync and file-shipping is amazingly fast that way, it easily outperforms "legacy" Master/Agent communication. I did different attempts to let all those prototypes work together in a meaningful way. You can imagine what it looks like right now ;) Sure, I'd love to make this little baby become a shiny new tool loved by many Puppet users. However, I'm currently bound in too many parallel projects, and sadly most of them not directly related to Puppet at all. It would require 2-3 dedicated weeks to get ready for a first useful release with some related documentation. > The point about it being harder for folks to quickly fix facts behaving > weird on their systems is one worth talking more about. Would you mind > starting a separate thread about the debugging experience so we could > talk through that independent of the xpp discussion? You're right, this is the wrong thread for this. However, I wouldn't even know how to start such a new one. I have no easy solution to propose for this issue. A fact that behaves strange in Ruby facter can be fixed. Even on an outdated Linux or a Unix system like AIX, Solaris, HP-UX, whatever. cFacter IS faster. But it's an all-or-nothing choice. Like AIO ;) > I do think it's worth keeping in mind that there are more puppet users > now than ever; it's a very big tent. In my humble opinion, > generalizations about what "most average admins" can do are increasingly > fraught with peril the bigger and more diverse our user base has gotten. Full ack. Thanks you Deepak for your comments, and sorry again for my little rant against PuppetDB. Cheers, Thomas -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/ndn7bq%24g9e%241%40ger.gmane.org. For more options, visit https://groups.google.com/d/optout.
