Re: [gentoo-user] Managing multiple Gentoo systems
On Thursday 07 July 2011 14:01:55 kashani did opine thusly: > On 7/7/2011 1:37 PM, Alan McKinnon wrote: > > On Thursday 07 July 2011 11:23:15 kashani did opine thusly: > >> On 7/2/2011 3:14 PM, Grant wrote: > >>> After a frustrating experience with a Linksys WRT54GL, I've > >>> decided to stick with Gentoo routers. This increases the > >>> number of Gentoo systems I'm responsible for and they're > >>> nearing double-digits. What can be done to make the > >>> management > >>> of multiple Gentoo systems easier? I think identical > >>> hardware > >>> in each system would help a lot but I'm not sure that's > >>> practical. I need to put together a bunch of new > >>> workstations > >>> and I'm thinking some sort of server/client arrangement with > >>> the only Gentoo install being on the server could be > >>> appropriate. > >>> > >>> - Grant > >>> > >>You may want to look at something like a config management > > > > system. > > > >> I'm using Puppet these days, but Gentoo support isn't > >> spectacular. It would be a bit complex to have Puppet install > >> the packages with the correct USE flags. However you could > >> use Puppet to manage all the text files and then manage the > >> packages somewhat manually. > > > > Give chef a try. > > > > It overcomes a lot of the issue puppet ran into, and of course > > makes new ones all of it's won, but by and large chef is more > > flexible. > > Too late. I've already put a year in with Puppet and have too much > working code to switch. Also I'm not much of a programmer so I get a > bit more out of the DSL though my templates are getting fairly > fancy these days. For anyone else interested in what we're talking > about, here's a fairly balanced and up to date link talking about > some of the differences. > > http://redbluemagenta.com/2011/05/21/puppet-vs-chef/ At least with puppet you can still work around shortcomings as you find them (no black box tricks in puttet) But regardless of it's quality, it's still 1,000,000's of times better than doing it all manually! -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] Managing multiple Gentoo systems
On 7/7/2011 1:37 PM, Alan McKinnon wrote: On Thursday 07 July 2011 11:23:15 kashani did opine thusly: On 7/2/2011 3:14 PM, Grant wrote: After a frustrating experience with a Linksys WRT54GL, I've decided to stick with Gentoo routers. This increases the number of Gentoo systems I'm responsible for and they're nearing double-digits. What can be done to make the management of multiple Gentoo systems easier? I think identical hardware in each system would help a lot but I'm not sure that's practical. I need to put together a bunch of new workstations and I'm thinking some sort of server/client arrangement with the only Gentoo install being on the server could be appropriate. - Grant You may want to look at something like a config management system. I'm using Puppet these days, but Gentoo support isn't spectacular. It would be a bit complex to have Puppet install the packages with the correct USE flags. However you could use Puppet to manage all the text files and then manage the packages somewhat manually. Give chef a try. It overcomes a lot of the issue puppet ran into, and of course makes new ones all of it's won, but by and large chef is more flexible. Too late. I've already put a year in with Puppet and have too much working code to switch. Also I'm not much of a programmer so I get a bit more out of the DSL though my templates are getting fairly fancy these days. For anyone else interested in what we're talking about, here's a fairly balanced and up to date link talking about some of the differences. http://redbluemagenta.com/2011/05/21/puppet-vs-chef/ kashani
Re: [gentoo-user] Managing multiple Gentoo systems
On Thursday 07 July 2011 11:23:15 kashani did opine thusly:
> On 7/2/2011 3:14 PM, Grant wrote:
> > After a frustrating experience with a Linksys WRT54GL, I've
> > decided to stick with Gentoo routers. This increases the
> > number of Gentoo systems I'm responsible for and they're
> > nearing double-digits. What can be done to make the management
> > of multiple Gentoo systems easier? I think identical hardware
> > in each system would help a lot but I'm not sure that's
> > practical. I need to put together a bunch of new workstations
> > and I'm thinking some sort of server/client arrangement with
> > the only Gentoo install being on the server could be
> > appropriate.
> >
> > - Grant
>
> You may want to look at something like a config management
system.
> I'm using Puppet these days, but Gentoo support isn't spectacular.
> It would be a bit complex to have Puppet install the packages with
> the correct USE flags. However you could use Puppet to manage all
> the text files and then manage the packages somewhat manually.
Give chef a try.
It overcomes a lot of the issue puppet ran into, and of course makes
new ones all of it's won, but by and large chef is more flexible.
>
> Here's a snippet of a template for nrpe.cfg
>
> <% if processorcount.to_i >= 12 then -%>
> command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
> %>/check_load -w 35,25,25 -c 35,25,25
> <% elsif fqdn =~ /(.*)stage|demo(.*)/ then -%>
> command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
> %>/check_load -w 10,10,10 -c 10,10,10
> <% else -%>
> command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
> %>/check_load -w 10,7,5 -c 10,7,5
> <% end -%>
>
> If you were managing a make.conf you could set -j<%=
> processorcount*2 %> or whatever as well as pass in your own
> settings etc. Once you have things working it's pretty good at
> keeping your servers in sync and doing minor customization per
> server based on OS, hardware, IP, hostname, etc.
>
> kashani
--
alan dot mckinnon at gmail dot com
Re: [gentoo-user] Managing multiple Gentoo systems
On 7/2/2011 3:14 PM, Grant wrote:
After a frustrating experience with a Linksys WRT54GL, I've decided to
stick with Gentoo routers. This increases the number of Gentoo
systems I'm responsible for and they're nearing double-digits. What
can be done to make the management of multiple Gentoo systems easier?
I think identical hardware in each system would help a lot but I'm not
sure that's practical. I need to put together a bunch of new
workstations and I'm thinking some sort of server/client arrangement
with the only Gentoo install being on the server could be appropriate.
- Grant
You may want to look at something like a config management system. I'm
using Puppet these days, but Gentoo support isn't spectacular. It would
be a bit complex to have Puppet install the packages with the correct
USE flags. However you could use Puppet to manage all the text files and
then manage the packages somewhat manually.
Here's a snippet of a template for nrpe.cfg
<% if processorcount.to_i >= 12 then -%>
command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
%>/check_load -w 35,25,25 -c 35,25,25
<% elsif fqdn =~ /(.*)stage|demo(.*)/ then -%>
command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
%>/check_load -w 10,10,10 -c 10,10,10
<% else -%>
command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
%>/check_load -w 10,7,5 -c 10,7,5
<% end -%>
If you were managing a make.conf you could set -j<%= processorcount*2 %>
or whatever as well as pass in your own settings etc. Once you have
things working it's pretty good at keeping your servers in sync and
doing minor customization per server based on OS, hardware, IP,
hostname, etc.
kashani
Re: [gentoo-user] Managing multiple Gentoo systems
>> After a frustrating experience with a Linksys WRT54GL, I've decided to >> stick with Gentoo routers. > > Out of curiosity, could you tell us more about this experience? Sure, I was using the stock firmware and I didn't like that you couldn't specify a source IP address when punching a hole in the firewall for a particular port, and I also couldn't coax "Remote Access" into working no matter what I tried. > The WRT54G(L) is quite dated, and the OpenWRT devs recommend against trying > to do anything fancy on it. I chose the WRT54GL because it has the best ratings on newegg.com. I looked into OpenWRT once but decided against it after I decided installation and possibly management was not nearly as trivial as I had imagined. > In another post you mentioned that you have a TP-Link TL-WR1043ND, which is a > bunch newer, I think, and should run OpenWRT quite well. I got rid of the TL-WR1043ND a while back because I couldn't get packet shaping to work with the stock firmware no matter how I tried. At this point I've sworn off mystery boxes. I even had a Dlink router die on me recently. If I'm not using mystery boxes for greater hardware reliability, why am I using them? Power consumption would be a good reason but it's not worth it IMO. - Grant
Re: [gentoo-user] Managing multiple Gentoo systems
On Sun, 3 Jul 2011 16:46:30 -0400, Simon wrote: > I have 2 world file "styles", one has "everything" in it and the other > is just the minimal stuff. I used to maintain the world files directly > but I've opted for creating 2 ebuilds in my own portage overlay which > are basically called sys-apps/everything and another called > sys-apps/minimalist. Maintenance of world files moved to the > maintenance of these ebuilds. You can do much the same, but in a simpler way, by using sets. -- Neil Bothwick Master of all I survey (at the moment, empty pizza boxes) signature.asc Description: PGP signature
Re: [gentoo-user] Managing multiple Gentoo systems
On Sat, Jul 2, 2011 at 6:14 PM, Grant wrote: > After a frustrating experience with a Linksys WRT54GL, I've decided to > stick with Gentoo routers. This increases the number of Gentoo > systems I'm responsible for and they're nearing double-digits. What > can be done to make the management of multiple Gentoo systems easier? > I think identical hardware in each system would help a lot but I'm not > sure that's practical. I need to put together a bunch of new > workstations and I'm thinking some sort of server/client arrangement > with the only Gentoo install being on the server could be appropriate. > > - Grant > > I've been managing 3 very very different gentoo systems for more than 12 months now. I used to have 2 more machines but I decomissioned them for they were old and noisy and it's summer and it's hot enough like that! haha I do not have a lazy-proof way of administrating this whole stuff, but I've come up with some kind of methodology that helps... I found it really helps to have the same profile on all machines, same /etc/portage everything. The /etc/make.conf can vary, but I would make it vary in the most minimal way. For example, all my kernels and /etc/make.conf are compiled for an i686, global use flags are exactly the same... in fact the only difference is my host has extra make option "-j5" and their distfiles mirrors will point to my own computers. Basically, downgrade all computers to the lowest common denominator of all your computers. I don't use binpkgs but I'm working on that... and also for the world file, I have 2 world file "styles", one has "everything" in it and the other is just the minimal stuff. I used to maintain the world files directly but I've opted for creating 2 ebuilds in my own portage overlay which are basically called sys-apps/everything and another called sys-apps/minimalist. Maintenance of world files moved to the maintenance of these ebuilds. Also, it helps to have your faster computer install all packages that would be installed on the others. If there's any trouble doing the emerge on that one, you'll have the same problems on the others. If the solution was to add some keywords in /etc/portage or something like that, then you can copy the change over to your other PCs and their emerge should be fine. To reduce network load, I sync my master server (my server online) against official sync server, then I sync my main computer at home against my server, then I sync any other computers at home against the main computer at home. But the activity of making your systems up-to-date will vary in intensity depending on how well it goes. When things are fine, it used to take 30 minutes to get all my systems updated, with the slowest ones taking 2-3 times that but requiring no intervention until the end. I'm currently in the process of writing a script that fully automates the update process. It basically tries things like "emerge -uDN --keep-going world" and checks for errors, if there are, it then fires a few safe "responses" which have helped in the past. For example, if the above emerge fails, it then tries a revdep-rebuild, an lafilefixer, python-updater and a few more, and then it restart from scratch trying the emerge again, over and over. Rarely, this will not work and will require that I mask something, add some keyword or update my ebuild to get rid of some package. The script is clearly not intelligent and is wasting a lot of time, but it eventually gets it done. Upgrades now take at least 1 hour, but are fully unattended. Also, I think Wonko's suggestion is pretty good in combination with my suggestions, I haven't tried but it does seem very attractive. Simon
Re: [gentoo-user] Managing multiple Gentoo systems
Grant (Sat, 2 Jul 2011 15:14:38 -0700):
> After a frustrating experience with a Linksys WRT54GL, I've decided to
> stick with Gentoo routers. This increases the number of Gentoo
> systems I'm responsible for and they're nearing double-digits. What
> can be done to make the management of multiple Gentoo systems easier?
> I think identical hardware in each system would help a lot but I'm not
> sure that's practical. I need to put together a bunch of new
> workstations and I'm thinking some sort of server/client arrangement
> with the only Gentoo install being on the server could be appropriate.
I used to do it the rsync way. Dozens of machines with varying
hardware, although not profoundly. All machines had all the software any
single machine could need. CFLAGS, kernel config and such were filled
with the greatest common divisor of all the boxes. There was one
'reference box' which did all the compiling and from which all the
other machines would rsync /, minus selected variables, such
as /dev, /etc/mtab, /mnt, /proc, /var/log, etc.
Another set of things excluded from rsync were things that do not
handle themselves locally, but are different accross computers. I had
groups of computers that I wanted to, for example, run different set of
boot-up services. That means that /etc/runlevels was excluded from
rsync'ing, but what was being rsynced were the dirs
/etc/runlevels-group{1,2,3,...} and every machine had a local
symlink /etc/runlevels -> one of the runlevels-groupX. Administrative
tasks are still limited to working with the single reference box and
its single filesystem (plus a few more runlevels-group's and similar
exceptions).
That way you can have specialized runlevel layouts, specialized fstabs
and other configs, etc. Even specialized kernels for every group of
machines, in theory.
There was also a script being distributed that was called by local crons
and allowed for batch rebooting at midnight, or whatever you may need
to run locally.
You may want to check out lsyncd for keeping dirs in sync in a smart
way on-the-fly, but rsyncing from a cronjob at a safe moment may be the
recommended option, depending on your environment. By 'safe' I mean
when there are no users logged in, for example.
The most tricky part was fine-tuning the set of stuff to exclude
from rsyncing. But if sure can be done in reasonable environments.
HTH,
-rz
Re: [gentoo-user] Managing multiple Gentoo systems
On 2 July 2011, at 23:14, Grant wrote: > After a frustrating experience with a Linksys WRT54GL, I've decided to > stick with Gentoo routers. Out of curiosity, could you tell us more about this experience? The WRT54G(L) is quite dated, and the OpenWRT devs recommend against trying to do anything fancy on it. In another post you mentioned that you have a TP-Link TL-WR1043ND, which is a bunch newer, I think, and should run OpenWRT quite well. Even if you were to use an Atom-based Gentoo system, I'm sure it would use more power than an ARM- or MIPS-based commodity router - perhaps twice as much. Stroller.
Re: [gentoo-user] Managing multiple Gentoo systems
Grant writes: > After a frustrating experience with a Linksys WRT54GL, I've decided to > stick with Gentoo routers. This increases the number of Gentoo > systems I'm responsible for and they're nearing double-digits. What > can be done to make the management of multiple Gentoo systems easier? In the case of editing identical files on different machines (Gentoo or not), I like clusterssh. It opens an xterm for every destination host and an additional command window. What you type here happens in all windows. Wonko
Re: [gentoo-user] Managing multiple Gentoo systems
Grant wrote: After a frustrating experience with a Linksys WRT54GL, I've decided to stick with Gentoo routers. This increases the number of Gentoo systems I'm responsible for and they're nearing double-digits. What can be done to make the management of multiple Gentoo systems easier? I think identical hardware in each system would help a lot but I'm not sure that's practical. I need to put together a bunch of new workstations and I'm thinking some sort of server/client arrangement with the only Gentoo install being on the server could be appropriate. - Grant If I had a dozen or so rigs, I would want them to be identical at least as far as mobos and such. Drives and such could vary depending on other things tho. If you use the same USE flags, you could update one and then copy /etc/portage/* to the other rigs then the updates could just be binary installs from the first rig updated. The only thing that might bite you is if you have different uses for each rig. If you have different packages installed, then you may have to compile for that. Most likely tho, you could use binaries for at least the system packages and save some time. I think Alan has some rigs and may be able to share more ideas. Maybe some others could comment that have dealt with the same thing. Dale :-) :-)
