Hey all, TL;DR: We're adding support to environments to PuppetDB but have a small migration hassle we wanted some community opinion on. If you're interested in PuppetDB and environments read on.
So we're looking at adding first class support to PuppetDB for environments. In the past we would happily accept facts/reports & catalogs from multiple different environments, but that information was never stored with the data. As a consequence people trying to use the same PuppetDB instance for environments would find they are collecting globally across all environments thus creating a high chance of resource collision. For many people this just wouldn't work, but for some they probably found horrible ways of working around this (if you are one of these people, I'd probably want to hear from you). Not to mention we also have no way to represent this data in the PE console either, basically PuppetDB just has no environment awareness. So we're fixing this now (at least the PDB part). Our current Epic: https://tickets.puppetlabs.com/browse/PDB-47 and relevant child tickets covers this work if you want to follow along at home. The issue we've hit however is how to cleanly migrate from a world of no environments, to one where environments are going to be constraining collections. Let me break it down: * Currently all submissions to PDB have no environment knowledge, and we can't store it anyway. * During an upgrade to this new feature we have to set the environment of all items in our database to 'something' internally and externally, however without prior knowledge everything will be labelled the same. * Once environment awareness is added to the terminus we only want to collect on environments the agent transaction was run with, this complies with the semantics of collections going back to whenever storeconfigs & environments were added to Puppet. One strategy for upgrades is to set the environment to 'production' for existing data. This will stay like this until another catalog is submitted with the true environment for that node. Now the problem with that solution is related to people who might be using exported resources (and somehow avoiding collection collisions): * In a single environment setup, where the name of the environment is not 'production'. * In a multi-environment setup where environment names are quite different (but only 1 is production). For both of these cases, if we just default the environment to 'production' there will be a short time where collections will return nothing for environments not called 'production' - until the next catalog submission occurs. This could be detrimental, and we'd like to understand how many users this might impact. Please let me know if this is you. An alternate solution is to migrate existing data to have the environment set to something completely different (like 'nil' or something else that can't possibly collide with a real name). With this solution, we can put in a collection query that not only collects for the current environment, but also for 'nil' to pick up the older global resources ... until such time as all catalogs have submitted thus putting the catalogs in the correct environment. This concept of 'nil' is almost transient (although we be longer lived for old reports obviously) its a temporary marker to say 'we don't know the environment'. So in most cases, once all catalogs have been submitted by all nodes the concept disappears. No new data should be added to this internal special environment. Anyway - I'm looking for some feedback for these two alternate solutions (or a third more whacky solution - whatever :-). The first one is obviously easiest and avoids leaving behind a transient solution, the second we think solves this concern to some extent (at least we think so). Any feedback or opinion on this would be much appreciated. ken. -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAE4bNT%3DwHZtttzvAuL7qYX_6Ao%2BJuPggY%2BosgwP1%2BB5%2B7QhOJA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
