Re: [Puppet Users] satellite sites management
Michael DeHaan wrote: On Thu, Feb 11, 2010 at 7:20 PM, Nat qwerty@gmail.com wrote: Hi, We have got puppet set up and running at our main office with no issues. We are using an external node classifier instead of directly creating node definition files. We would like to manage our remote offices using puppet also. A little about our set up. From our main site we have VPN links out to a remote site. each site is generally identical with the same number of servers and roughly the same services running on each server. Essentially the only differences at each remote site the subnet and related IP addresses. Since we are using an external node classifier we do not explicitly have node definition so we can not inherit a class and override a default value. Is there a way to do this using node classifiers? An example will probably show this better Site1: + location UK + subnet 192.168.1.0/24 + gateway 192.168.1.254 (acts also as nameserver and local dns etc for all servers at site 1, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.1.1 gateway of 192.168.1.254 + sever2 ip - 192.168.1.2 gateway of 192.168.1.254 Site 2: + location US + subnet 192.168.2.0/24 + gateway 192.168.2.254 (acts also as nameserver and local dns etc for all servers at site 2, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.2.1 gateway of 192.168.2.254 + sever2 ip - 192.168.2.2 gateway of 192.168.2.254 As you can see most details are identical between sites except for a few network and geographical differences. Has there been any consensus within the community on the best way to manage situations like this? I was talking with Eric yesterday about his external nodes regex classifier: http://github.com/reductivelabs/puppet/tree/master/ext/regexp_nodes/ This might be a start to some sort of evolved smart node idea (that we could stick in Dashboard and also build a CLI tool to) that could support the concept of variable inheritance. So not just define what machines are webservers (rather than what webservers are what machine) but use similar regexen (or another system of groups) to classify what machines live in what areas -- and blend the two groups together. Dan Bode mentions he sees several logical groups here -- there's what type of a machine you have, whether it's a stage/prod machine, and what location (datacenter) it is in (i.e. what is the machine's geographic location). Some variables may come from one or more of those sources, and they can have some basic defaults. (This is somewhat similar to Cobbler's blender inheritance for groups of things... allowing extension of arrays and adding keys to hashes, or overriding of scalars, as we evaluate the group orders.The location groups and the classification groups would not need to be chained (i..e one a parent of another) but we'd want to support the idea of inherited subgroups (acme-datacenter is a subset of us-datacenters is a subset of datacenters).Apologies if I'm being confusing :) There's obviously a lot to do here, but I can see the need for a intelligent external nodes classifier that understands those kinds of ideas that can really model a multi-site environment as a first class concept. --Michael I don't want to sound a fanboy or one of Volcane's minions, but I think his idea is spot on for this kind of thing. If it had LDAP/other DB support it would be even more ridiculously awesome. At the base of this problem, your base solution is you just need a function that can look up arbitrary data. Hostname can come into play, but with extlookup you can choose an order of precedence (IE overriding: hostname.csv trumps domain.csv or vice versa and it's all configurable). I use CSV files cause I am lazy, it is easy, and my environment is small. Someone at $FORTUNE500 could use LDAP if the tool was extended slightly. As I recall however, extlookup.rb is not included in mainline puppet because Luke disagrees with the implementation I think. Could be wrong here. Best to ask him. I deploy it as part of my deploy:cold cap task for new masters. My uses for extlookup: * combined with ERB to create iptables firewalls * in puppetdoc, $location shows you where it lives since you can set it in the csv files (huge, huge win) * override DNS stupidity, which a lot people have. Or to simple force hostnames in case a master is having performance issues and you need to re route some clients. You just change the value in the csv. * Current list of admins to realize even can be done in the csv. The point is, a tool like this should be in puppet, extlookup.rb
Re: [Puppet Users] satellite sites management
On Fri, Feb 12, 2010 at 9:41 PM, Michael DeHaan mich...@reductivelabs.comwrote: This might be a start to some sort of evolved smart node idea (that we could stick in Dashboard and also build a CLI tool to) that could support the concept of variable inheritance. So not just define what machines are webservers (rather than what webservers are what machine) but use similar regexen (or another system of groups) to classify what machines live in what areas -- and blend the two groups together. I hate to bring it up again, but this kind of functionality is already in Foreman since 0.1. cheers, Ohad -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] satellite sites management
On Sun, Feb 14, 2010 at 10:57 PM, Julian Simpson simpsonjul...@gmail.comwrote: Foreman already has integration on its side for extlookup (it's in trunk or something) as soon as Julian and I finish making the backend pluggable it will be able to integrate easily into foreman. It should be very simple to add foreman support now. There's an ExtLookup class that has a backend. There's already the CSV backend that R.I. originally wrote, and it's pretty straightforward to see how you could pass it another backend. Thats great, please let me know when, and I'll merge the code into the main branch and the official release. cheers, Ohad -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] satellite sites management
On Thu, Feb 11, 2010 at 7:20 PM, Nat qwerty@gmail.com wrote: Hi, We have got puppet set up and running at our main office with no issues. We are using an external node classifier instead of directly creating node definition files. We would like to manage our remote offices using puppet also. A little about our set up. From our main site we have VPN links out to a remote site. each site is generally identical with the same number of servers and roughly the same services running on each server. Essentially the only differences at each remote site the subnet and related IP addresses. Since we are using an external node classifier we do not explicitly have node definition so we can not inherit a class and override a default value. Is there a way to do this using node classifiers? An example will probably show this better Site1: + location UK + subnet 192.168.1.0/24 + gateway 192.168.1.254 (acts also as nameserver and local dns etc for all servers at site 1, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.1.1 gateway of 192.168.1.254 + sever2 ip - 192.168.1.2 gateway of 192.168.1.254 Site 2: + location US + subnet 192.168.2.0/24 + gateway 192.168.2.254 (acts also as nameserver and local dns etc for all servers at site 2, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.2.1 gateway of 192.168.2.254 + sever2 ip - 192.168.2.2 gateway of 192.168.2.254 As you can see most details are identical between sites except for a few network and geographical differences. Has there been any consensus within the community on the best way to manage situations like this? I was talking with Eric yesterday about his external nodes regex classifier: http://github.com/reductivelabs/puppet/tree/master/ext/regexp_nodes/ This might be a start to some sort of evolved smart node idea (that we could stick in Dashboard and also build a CLI tool to) that could support the concept of variable inheritance. So not just define what machines are webservers (rather than what webservers are what machine) but use similar regexen (or another system of groups) to classify what machines live in what areas -- and blend the two groups together. Dan Bode mentions he sees several logical groups here -- there's what type of a machine you have, whether it's a stage/prod machine, and what location (datacenter) it is in (i.e. what is the machine's geographic location). Some variables may come from one or more of those sources, and they can have some basic defaults. (This is somewhat similar to Cobbler's blender inheritance for groups of things... allowing extension of arrays and adding keys to hashes, or overriding of scalars, as we evaluate the group orders.The location groups and the classification groups would not need to be chained (i..e one a parent of another) but we'd want to support the idea of inherited subgroups (acme-datacenter is a subset of us-datacenters is a subset of datacenters).Apologies if I'm being confusing :) There's obviously a lot to do here, but I can see the need for a intelligent external nodes classifier that understands those kinds of ideas that can really model a multi-site environment as a first class concept. --Michael -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] satellite sites management
On Fri, Feb 12, 2010 at 11:41 AM, Michael DeHaan mich...@reductivelabs.com wrote: On Thu, Feb 11, 2010 at 7:20 PM, Nat qwerty@gmail.com wrote: Hi, We have got puppet set up and running at our main office with no issues. We are using an external node classifier instead of directly creating node definition files. We would like to manage our remote offices using puppet also. A little about our set up. From our main site we have VPN links out to a remote site. each site is generally identical with the same number of servers and roughly the same services running on each server. Essentially the only differences at each remote site the subnet and related IP addresses. Since we are using an external node classifier we do not explicitly have node definition so we can not inherit a class and override a default value. Is there a way to do this using node classifiers? An example will probably show this better Site1: + location UK + subnet 192.168.1.0/24 + gateway 192.168.1.254 (acts also as nameserver and local dns etc for all servers at site 1, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.1.1 gateway of 192.168.1.254 + sever2 ip - 192.168.1.2 gateway of 192.168.1.254 Site 2: + location US + subnet 192.168.2.0/24 + gateway 192.168.2.254 (acts also as nameserver and local dns etc for all servers at site 2, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.2.1 gateway of 192.168.2.254 + sever2 ip - 192.168.2.2 gateway of 192.168.2.254 As you can see most details are identical between sites except for a few network and geographical differences. Has there been any consensus within the community on the best way to manage situations like this? I was talking with Eric yesterday about his external nodes regex classifier: http://github.com/reductivelabs/puppet/tree/master/ext/regexp_nodes/ I see this classifier uses hostname to refer to what strictly speaking is the certname... This might be a start to some sort of evolved smart node idea (that we could stick in Dashboard and also build a CLI tool to) that could support the concept of variable inheritance. So not just define what machines are webservers (rather than what webservers are what machine) but use similar regexen (or another system of groups) to classify what machines live in what areas -- and blend the two groups together. Aren't we going to need more info than just the certname for external nodes to be able to really be able to functionally classify them? I realize this is a bit of a hobby horse for me as we don't use hostnames for the certname... :) but even if you're using hostnames as certnames do you really want to have to encode all this info into the hostnames? Dan Bode mentions he sees several logical groups here -- there's what type of a machine you have, whether it's a stage/prod machine, and what location (datacenter) it is in (i.e. what is the machine's geographic location). Some variables may come from one or more of those sources, and they can have some basic defaults. (This is somewhat similar to Cobbler's blender inheritance for groups of things... allowing extension of arrays and adding keys to hashes, or overriding of scalars, as we evaluate the group orders. The location groups and the classification groups would not need to be chained (i..e one a parent of another) but we'd want to support the idea of inherited subgroups (acme-datacenter is a subset of us-datacenters is a subset of datacenters). Apologies if I'm being confusing :) There's obviously a lot to do here, but I can see the need for a intelligent external nodes classifier that understands those kinds of ideas that can really model a multi-site environment as a first class concept. --Michael -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. -- nigel -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] satellite sites management
Hi, We have got puppet set up and running at our main office with no issues. We are using an external node classifier instead of directly creating node definition files. We would like to manage our remote offices using puppet also. A little about our set up. From our main site we have VPN links out to a remote site. each site is generally identical with the same number of servers and roughly the same services running on each server. Essentially the only differences at each remote site the subnet and related IP addresses. Since we are using an external node classifier we do not explicitly have node definition so we can not inherit a class and override a default value. Is there a way to do this using node classifiers? An example will probably show this better Site1: + location UK + subnet 192.168.1.0/24 + gateway 192.168.1.254 (acts also as nameserver and local dns etc for all servers at site 1, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.1.1 gateway of 192.168.1.254 + sever2 ip - 192.168.1.2 gateway of 192.168.1.254 Site 2: + location US + subnet 192.168.2.0/24 + gateway 192.168.2.254 (acts also as nameserver and local dns etc for all servers at site 2, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.2.1 gateway of 192.168.2.254 + sever2 ip - 192.168.2.2 gateway of 192.168.2.254 As you can see most details are identical between sites except for a few network and geographical differences. Has there been any consensus within the community on the best way to manage situations like this? -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] satellite sites management
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nat wrote: Hi, We have got puppet set up and running at our main office with no issues. We are using an external node classifier instead of directly creating node definition files. We would like to manage our remote offices using puppet also. A little about our set up. From our main site we have VPN links out to a remote site. each site is generally identical with the same number of servers and roughly the same services running on each server. Essentially the only differences at each remote site the subnet and related IP addresses. Since we are using an external node classifier we do not explicitly have node definition so we can not inherit a class and override a default value. Is there a way to do this using node classifiers? An example will probably show this better Site1: + location UK + subnet 192.168.1.0/24 + gateway 192.168.1.254 (acts also as nameserver and local dns etc for all servers at site 1, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.1.1 gateway of 192.168.1.254 + sever2 ip - 192.168.1.2 gateway of 192.168.1.254 Site 2: + location US + subnet 192.168.2.0/24 + gateway 192.168.2.254 (acts also as nameserver and local dns etc for all servers at site 2, for example ntp will use the closest time source geographically) + sever1 ip - 192.168.2.1 gateway of 192.168.2.254 + sever2 ip - 192.168.2.2 gateway of 192.168.2.254 As you can see most details are identical between sites except for a few network and geographical differences. Has there been any consensus within the community on the best way to manage situations like this? Well, I think a large portion of users have discovered that RI Pienaar's (Volcane on IRC) extlookup.rb add on is good for this sort of stuff. It's basically a function that looks up data in a csv file. So say your locations are domains like us.yourcompany.com. You can have a file called us.yourcompany.com.csv in a directory under /etc/puppet, or you set a location some other way you can have a us.csv file in the directory. I call my external lookup directory 'extdata', and it looks up in the order of something like fqdn, domain, location, common (if it fails to find one csv, it moves on to the next). Then inside your manifest you set variables like: $something = extlookup(something) It's really got so many uses I encourage you to use it. It was the only way I could solve a lot of puppet problems I ran into. Here is the page: http://www.devco.net/?s=extlookup - -- Joe McDonagh AIM: YoosingYoonickz IRC: joe-mac on freenode L'ennui est contre-révolutionnaire -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkt05q4ACgkQRkBieEaRmuZWDgCfdk89Fk1eyC9ichJIbjN9Jldj z1wAn0lUYhjxeqAinqjSZuS7MqkG+hnp =SZh3 -END PGP SIGNATURE- -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
