Re: [Puppet Users] Puppet "autoconfigured" clients on AWS - classes from EC2 tags/userdata

[Martin Langhoff]

On Fri, May 27, 2016 at 6:51 AM, Gareth Rushgrove 
wrote:

>
> There are lots of hints and tips in this white paper.
>
>
> https://puppet.com/blog/making-life-puppet-and-aws-or-other-cloud-services-easier
>
> In particular it covers using the policy based autosigning and trusted
> facts to secure doing what I think you're doing.
>

Thank you. It mostly covers other topics around these practices, but not
the specific point I am trying to figure out. At least not that I can see
(maybe it's hiding somewhere?).

To recap what I am looking for: I want to build VMs (on AWS or something
else) that Puppet has no name for, using EC2 tags and userdata (and similar
facilities on RHEV-M, VMWare etc) _to list a number of puppet modules and
puppet variables_.

What I have seen proposed/used is to read in data from EC2 tags and
userdata via facter, and use conditionals in puppet code. While workable,
this means writing a lot of pointless conditionals in puppet code.

What I would prefer is a bit of magic "include all classes listed by name
in this variable", like it's done with hiera classes.

cheers,



m
-- 
 martin.langh...@gmail.com
 - ask interesting questions  ~  http://linkedin.com/in/martinlanghoff
 - don't be distracted~  http://github.com/martin-langhoff
   by shiny stuff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCLDCYN5znaOaM74ren-ZzNhrk3ytf3PgzPoxUNYv4YpkQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Puppet "autoconfigured" clients on AWS - classes from EC2 tags/userdata

[Martin Langhoff]

Hi Puppeteers,

folks are mapping "role" from EC2 tags or userdata into a `case`
statement in site.pp to choose a pre-built configuration.

I wonder whether there is a way to bring in a listing of classes, as
can be done with hiera (`hiera_include("classes")`).

thoughts?



m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCJK%2BAAVgpdo7LBi_H1AGfKyuF9HJbPaD21FMn_dcQzFDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Script to track orphaned resources

[Martin Langhoff]

For context... Our puppet setup is complex, with many behaviors controlled
by facter facts, in part controlled by a .INI file that support personnel
can edit. We manage thousands of VMs.

So unit tests are interesting but offer very limited coverage. Tracking
orphans on live nodes is much more comprehensive. In particular it catches
things folks haven't thought about.

For example, a node had a class webserver applied. For whatever (bad)
reasons that class is no longer applied... hey this node has apache
installed and running, but now unmanaged! We sure as heck want an alert
over that.

Any long-term puppet infra has high risk of orphaned resources. There's so
many ways this can happen.

cheers,

m
On Aug 21, 2014 6:08 PM, "Garrett Honeycutt" 
wrote:

> On 8/21/14 5:45 PM, Manuel Quiñones wrote:
> > Hello,
> >
> > I'm working on a utility script to track orphaned resources.  With
> > orphans I mean: resources that were previously managed by Puppet, but
> > they no longer are.  I want to track those while I do a refactor in my
> > manifests.
> >
> > Here is the script I wrote:
> >
> > https://gist.github.com/manuq/eec269ce7ba00974f46e
> >
> > It is based on some assumptions, and here is my question: are these
> > assumptions correct?
> >
> > - Puppet generates the following files on each run, even when called
> > with --noop:
> > - last_run_report.yaml: contains the resources currently managed, in
> > full detail (serialized Puppet objects)
> > - state.yaml: contains the resources Puppet ever managed since the file
> > was created, only their name and some timestamps "checked" and "synced"
> > - last_run_summary.yaml: among other things, contain the timestamp of
> > the run, and the total time it took
> >
> > Based on that, I have two methods that output the orphans:
> >
> > Method 1: use state.yaml and read the "checked" timestamp. If it was not
> > checked in the last run, then it is an orphan.
> > Method 2: orphans are the subset of resources that are contained in
> > state.yaml and are not contained in last_run_report.yaml.
> >
> > Critics and suggestions welcome.  Also I hope this can be useful to
> others.
> > Cheers,
> >
> > PS Note that this topic was discussed earlier in May.  I took it as
> > initial reference:
> >
> https://groups.google.com/forum/#!searchin/puppet-users/orphan/puppet-users/ghKfRBkPD5A/m7KTeymd2XwJ
>
> Hi Manuel,
>
> Your plan is quite clever though if your goal is to refactor your puppet
> modules and not leave anything out, spec tests are the way to go.
>
> http://rspec-puppet.com/tutorial/
>
> Best regards,
> -g
>
> --
> Garrett Honeycutt
> @learnpuppet
> Puppet Training with LearnPuppet.com
> Mobile: +1.206.414.8658
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/53F66DE9.4020705%40garretthoneycutt.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCJsMK0oOMOZH6WcL27VDk3At3Dz8Pp1Q2o27zPv%2BsmvaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Tracking orphaned resources

[Martin Langhoff]

As the complexity of our $workplace puppet configuration grows, I am
increasingly worried that puppet gives us very limited visibility over
resources it no longer manages.

In practical terms: if I mess up my class include/require/inherit structure
so that a node A no longer indirectly includes module Foo, resources
managed by Foo are present in A but "orphaned".

This is a lurking gotcha; and it can lead to subtle problems.

Is there any tool that helps here, for example keeping a manifest of all
resources ever managed by this puppet install? If not, I will probably try
build that into ppg.

Is there a way to ask puppet for a fuller, more explicit report of all
resources tracked during a run?

Thanks,

m

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCLhd34r%3DNjC91kFOq97VGbUpbdWmYC1Z75CgirmO%2Btsug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Assign group membership in a module or conditionally?

[Martin Langhoff]

On Mon, Mar 31, 2014 at 10:09 AM, Martin Langhoff
 wrote:
> This is really awkward for what I see as a "natural" operation. Am I doing
> something wrong in my setup?
>
> And also... this is funny, but I discovered a change including this syntax
> had already been rolled out into production, so I thought I would find all
> the virtual user accounts created. In fact I did not.

Ok, I found both the explanation of why this is not creating users in
our env, and I guess a good workaround to this situation.

In our env, virtual users are not realized directly, they are realized
through a "define ssh_user" that manages authorized_keys and assigns
them to wheel.

That layer of indirection _is_ in Puppet's state, but it means that
the virtual user does not have the membership set in the "virtual"
definition.

At some point of the discussion I also lost sight of the
conditionality of the materialization (will only materialize
/matching/ resources) and I was temporarily a bit more anxious than I
should have been.

This is a relief. I still consider resource collectors dangerous, but
usable in clearly defined situations.



m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCJNXs6dnpKYri89MaHKGpwgecKG-iVwFwqYfbWZ%3DBwcFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Assign group membership in a module or conditionally?

[Martin Langhoff]

On Friday, March 28, 2014 3:48:47 PM UTC-4, jcbollinger wrote:

> Puppet DSL provides no mechanism, however, for selecting resources via a 
> search expression without realizing all virtual resources among those 
> selected.
>

This is really awkward for what I see as a "natural" operation. Am I doing 
something wrong in my setup?

And also... this is funny, but I discovered a change including this syntax 
had already been rolled out into production, so I thought I would find all 
the virtual user accounts created. In fact I did not.

Could it be that there are additional conditions that control whether the 
accounts are realized 
 

Perhaps, however, you could do something clever at the point where you 
> declare the users in the first place.  If class our_users has some kind of 
> visibility of whether the target node is (supposed to be) a web server, 
> then it could initially declare users with the correct groups, so that you 
> don't have to perform any fixup later.  "Some kind of visibility" could be 
> achieved via hiera or with the help of the roles & profiles pattern; there 
> are probably other alternatives as well.
>

We already use roles/profiles pattern, but without Hiera. So we have 
our_users module which only declares virtual users, then 
our_users::some_group subclasses that realize them.

We use site.pp (split per data center) to define what classes apply to a 
given node. Most of our nodes are identical, so we have "wildcard" regex 
node definitions. Nodes with specialized roles have a dedicated node stanza.

Not all user accts are everywhere: some have special security req's and can 
only allow a very restricted set of sysadmins.

It is in this kind of pattern that I'd like to say, in a "profile" module 
(our_webservers) that all users in wheel also have to be in 'apache'.

cheers,



m

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/ac5bd6ba-963d-4b86-a2d5-3e3f71e3bc70%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Assign group membership in a module or conditionally?

[Martin Langhoff]

On Friday, March 28, 2014 3:08:00 PM UTC-4, Jose Luis Ledesma wrote:

> Realize some of them
>
Right. That does not address my concern, which is the implicit realize.

The rule for which users we realize on our infra are orthogonal to the 
modules that define functionality; and need to remain so...




m

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/97c6916d-61b6-427b-8327-a0e63a380c8a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Assign group membership in a module or conditionally?

[Martin Langhoff]

On Friday, March 28, 2014 2:30:31 PM UTC-4, Jose Luis Ledesma wrote:

> I would really like to know what other people may suggest, the only thing 
> that comes to my mind is to make use of tags for your users.
>
How would you do this with tags _without_ realizing virtual users?




m

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/e5d904f8-ce23-4ec3-8488-017e9c6c2317%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Assign group membership in a module or conditionally?

[Martin Langhoff]

Scenario - we have modules:
 - our_users: defines all the (virtual) users (sysadmins, support,
developers) as members of wheel
 - our_base: gets applied to all nodes
 - our_webserver: defines the apache group

What I want to achieve is: on nodes that use our_webserver (where the
apache group exists), any member of wheel should also be in apache.

We tried

   User <| groups == 'wheel' |> {
  groups = ['wheel', 'apache']
   }

but this realizes all virtual users (which breaks other stuff we do), and
would not very well if another module was to do this too.

Is there a good way to do this? Either in "our_webserver" -- which I would
prefer -- or in our_users (if apache group exists, be a member).




m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCLtFeMR0RpZ6Aha%3DnZe9Fuu_stAzigOhsvnGHVDewz-xg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] filebucket --local should have default bucket matching "apply"

[Martin Langhoff]

Background I am using puppet via direct invokation of puppet apply
path/to/manifest.pp -- no puppet server involved. Puppet is 3.1.1 on
RHEL6.x and CentOS6.x boxes.

I needed to study a file change, coming from a concat. Looking at the
report, I had the hash. This did not work:

# puppet filebucket get --local d49537fd4cd4a5e588b2775f25f511e6
Error: Could not run: File not found

instead, this does work (but how would one know the right bucket?):

# puppet filebucket  get -b /var/lib/puppet/clientbucket  --local
d49537fd4cd4a5e588b2775f25f511e6

OTOH, this Just Works:

# puppet filebucket backup --local  /etc/motd
/etc/motd: d41d8cd98f00b204e9800998ecf8427e

# puppet filebucket get --local d41d8cd98f00b204e9800998ecf8427e

However, looking into /var/lib/puppet is is clear that they use
different buckets. Using puppet apply will use
/var/lib/puppet/clientbucket , while using puppet filebucket --local
will use /var/lib/puppet/bucket .

In my case I used strace and find to figure out where things were
going, which is hardly recommendable.

cheers,


m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCKqiE8FU%3DLmAXkFWpR%3DGpjSQXdWCaxSxYamsqSzZxbdiA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] DR hot/warm spare node patterns / anti-patterns?

[Martin Langhoff]

With our without puppet. We are looking at setting up DR nodes with
some draft ideas. We use puppet against a git repository (with ppg)
instead of against a puppet server. Nodes are actually VMs.

Configuration ideas:

 - a filesystem-level flag that indicates that the node is running as
standby spare, to use as conditional in bash/python scripts as well as
in puppet

 - some services enabled/disabled based on the standby flag

 - some cronjobs check for standby flag

Areas of exploration

 - keep standby nodes down, bring up periodically so that puppet gets
a chance to apply changes vs keep standby nodes up (perhaps with a
smaller/limited VM configuration).

 - handling 'environment' differences
   - mountpoints for app data point to different nfs servers
   - different syslog, email servers
   - ...

ideas? suggestions?



m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCJmy90zF7mgiyDG-3vOvVvoHPF7E6j%3DtnNDarGJEqY5fA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Actual diffs in puppetdb?

[Martin Langhoff]

On Tue, Nov 19, 2013 at 12:25 PM, Bruce  wrote:
> Maybe.  But usually I don't know I want this information, until I need it.
> So having to turn on some debugging ahead of time doesn't help.

I am using etckeeper in conjunction with Puppet for exactly this use
case. Additionally, my puppet configuration itself is under git.

With those two things, it is trivial to walk back to any change. As
etckeeper is not part of puppet, it tracks _any_ change on /etc, and
this adds very valuable coverage.

For example: if you install an rpm, and one of its dependencies
installs a new cronjob (with some unexpected side-effect), Puppet will
not give you visibility on that unexpected cronjob; etckeeper will.

etckeeper is a keeper ;-)

Aside: If puppet would replace its "file bucket" functionality with a
git-based store I would be the happiest man on earth. The contents of
files changed would be available by path as well as hash. git tags
would also allow for a clean migration (preserving md5-based lookups
for preexisting files).

I am familiar with git internals -- I've authored parts of early git,
and various importers still in use -- so I'd be glad to help flesh
this out, if anyone is ever interested.

Current file bucket is pretty useless to me in practice :-/




m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFC%2BEAQ2sDAvQAu1h%2BWUQUE_36vt%3DDiS76CE%2BWBQMDfcQCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: File() or Exec() temporary override?

[Martin Langhoff]

On Wed, Nov 6, 2013 at 9:55 AM, jcbollinger  wrote:
> .  If that's not viable, then something close to the idea you proposed
> should be possible:

By "should be possible, do you mean that you know or think that Puppet
supports it?

> node 'fqdn' { # I work for RL :-)
>   include rl_users
>   include rl_base
>   include rl_webserver
>   File<| title == '/etc/httpd/conf.d/foo.conf' |> {
> ensure => 'present',
> content => undef
> # override other properties to undef (no quotes)
> # as needed.
>   }
> }

Does this syntax work, and does it elegantly override any other File
stanzas that point to that file path? Does content=>undef override
'source' or do I need to also say source->undef ?

If that actually works, what other types support this trick?

thanks,




m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCKrnLdT9XpQ6uXa6%2BVqBPXWTM8UohdTuhLRR0y6sn1ShQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Dashboard running in masterless puppet

[Martin Langhoff]

Hi Dan,

there's quite a bit. Google for 'masterless puppet howto' to find a
couple tutorials and discussions that are popular. Add 'git' to the
search string for more quality options...

The main benefits are

  - scalability
  - pull model -- see http://www.infrastructures.org/bootstrap/pushpull.shtml

The main drawbacks are

 - loss of some dynamic configuration features -- like the
configuration database, what'sitsname?
 - all clients see the whole configuration -- however I would be
personally unwilling to rely too much on this feature of the puppet
server


Myself, I "seeded" my architecture with those tutorials, and wrote the
ppg wrapper/scaffolding/infra I discussed above, which adds some good
bits that Puppet lacks:

 - scheduled rollouts (i.e.: make this effective at 3am)
 - force a specific rollout to happen _now_ (using an ssh loop with an
unprivileged account to touch an inotify  trigger)

Here's a reasonably good thread
https://groups.google.com/forum/#!topic/puppet-users/7ZpAMrMb2NQ

cheers,



m

On Tue, Nov 5, 2013 at 3:11 PM, Dan Ng  wrote:
> I just came across the dynamic of running Puppet in a masterless mode.  Was
> there a tutorial that you had followed in order to configure it that way?
>
> Thanks!
>
>
> On Monday, November 4, 2013 12:21:14 AM UTC-5, Gonzalo wrote:
>>
>> Hi All,
>>
>> I'm running Puppet in masterless mode and trying to make Puppet dashboard
>> play nice with it in this non-standard setup.
>>
>> I'd love to hear how other people are doing this. The process I have in
>> mind is:
>>
>> 1) Run "puppet apply" from cron on each node
>> 2) Rsync (using --remove-sent-files) the reports from each node's
>> /var/lib/puppet/reports dir back to the puppet dashboard server
>> 3) Run rake:import
>>
>> The issue I'm having is that rake:import will create a new failed task for
>> every report that already exists in the database. It does skip them, but I
>> don't want to be notified it's a failed task when it skips them. I was
>> thinking of deleting old reports, but given that they get imported by the
>> delayed task workers, it is hard to know which reports have been processed
>> to avoid deleting a report that hasn't been imported yet.
>>
>> If I can find a way to stop it reporting skipped reports as failed tasks,
>> then I should be OK.
>>
>> Any ideas? Anyone else running Puppet dashboard with masterless Puppet?
>>
>> - GS
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/1ba01e3a-f206-4aca-b7ab-79e924d05d81%40googlegroups.com.
>
> For more options, visit https://groups.google.com/groups/opt_out.



-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCLC9jSgiHwJhjae6CttpPpEVGPEXZ3i_-X4-fcajtPmqA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Dashboard running in masterless puppet

[Martin Langhoff]

Hi Gonzalo,

> I'm running Puppet in masterless mode and trying to make Puppet dashboard
> play nice with it in this non-standard setup.

some of the tricks that are useful in this kind of scenario are
encapsulated in puppet-git / ppg, which I've written, and use at
Remote Learner, where it is gradually taking over an infra with >2000
VMs.

   http://repo.or.cz/w/puppet-git.git

> I'd love to hear how other people are doing this. The process I have in mind
> is:
>
> 1) Run "puppet apply" from cron on each node

yep. You can use --detailed-exitcodes and do something smart about
failures. At the risk of sounding like a broken record... ppg has
example code...

> 2) Rsync (using --remove-sent-files) the reports from each node's
> /var/lib/puppet/reports dir back to the puppet dashboard server

In my case, the chosen transport mechanism is git protocol. ppg pushes
it back to a different git repo (pulls from puppet.git, pushes reports
to reports.git ) . There's code in ppg to "trim" git history.

> 3) Run rake:import

Instead of that, my code is using curl:

http://repo.or.cz/w/puppet-git.git/blob/670a12233f563d47e32e47f09214590e26451a5a:/ppg-reports-to-dashboard#l38

this is barely tested, and we haven't put it in prod yet. It mimics
what the puppet clients do.

> The issue I'm having is that rake:import will create a new failed task

I haven't got the foggiest idea about using rake:import. Have not dug
into Puppet/Dashboard/Ruby internals too deep yet.

cheers,


m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFC%2BBgrLz9P7DYh48B8_H_Zr4FqReTdVg77YijZ38fQTp-A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] File() or Exec() temporary override?

[Martin Langhoff]

Hi Puppeteers,

When you have complex/rich classes, and large numbers of machines/VMs,
sometimes there is a machine that needs a temporary override on a
file.

Is there a way to say something like... ?

  node 'fqdn' { # I work for RL :-)
  include rl_users
  include rl_base
  include rl_webserver # defines /etc/httpd/conf.d/foo.conf
  ignore File['/etc/httpd/conf.d/foo.conf']
  }

or something conceptually equivalent?

I know of a couple inelegant options: I can disable puppet on that VM
for the duration, or comment out the relevant class.

A more precise override/ignore is of course better, as other
components of the configuration are still applied correctly.



m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACPiFCKXXujd0BxbV9PtOT6xe4fn7uXCaTH3UJkVGY5PPq0T%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Best practices for infrastructure

[Martin Langhoff]

On Thu, Aug 29, 2013 at 9:02 AM, jcbollinger  wrote:
> The master will always choose the node block to use based on the client's
> SSL certname (spelled $::clientcert in Puppet DSL).

Oh, that is considerably safer than what I feared. Thanks for the clarification.

My comments earlier in this thread were under the mistaken
understanding that the Puppet master in its default behaviour would
allow match nodename based on $::hostname.

thank you!




m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Best practices for infrastructure

[Martin Langhoff]

On Wed, Aug 28, 2013 at 9:31 AM, jcbollinger  wrote:
> If the objective is to render it into a small number of words,

Just to double-check my understanding is right. If the client-reported
$::hostname does not match the certname, _and_ I am only using 'node
"fqdn"' entries in my Puppet manifests, puppet will use... certname or
client-reported $::hostname?

cheers,



m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Best practices for infrastructure

[Martin Langhoff]

On Tue, Aug 27, 2013 at 2:41 PM, jcbollinger  wrote:
> The client can provide a $::hostname fact that is different from the
> certname it presents, but that is perfectly valid and expected under some
> circumstances.  It is possible that a client doing so is thereby able to
> exploit weaknesses in (user-provided) manifest files required anyway for its
> catalog, thereby extracting information to which it is not intended to have
> access, but that is possible to some degree or another with any fact.  It
> does not constitute a flaw in Puppet itself, but rather in the manifests in
> question.

That's roughly what I recall.

So, in less words: from a security perspective, do not count on Puppet
only serving the right config for the host. It is a very flimsy
security boundary.

cheers,



m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Best practices for infrastructure

[Martin Langhoff]

On Sat, Aug 24, 2013 at 5:18 PM, Jakov Sosic  wrote:
> Only if you use autosign option. After the certificate is signed, agents
> report certname and not hostname.

Well-behaved clients report certname. A malicious client could use one
cert, but report a different name. AIUI the puppet master checks the
certificate to allow connection, but uses the client-reported name to
pick the configuration served.

cheers,


m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Best practices for infrastructure

[Martin Langhoff]

On Sat, Aug 24, 2013 at 6:33 AM, Félix Barbeira  wrote:

> Speaking in security terms, could be masterless puppet configuration less
> secure? I mean, the puppet code is in *all* the clients. On the other hand,
> the puppet code is only in the master, which I think is more secure (you
> can isolate it on a restricted VLAN, private network, etc). If the security
> of one client is vulnerated the hacker gets nothing, otherwise he would be
> able to read the whole puppet code.
>

The difference is minimal. The master will happily serve any config to any
host. The puppet server relies on the self-reported hostname, so a
compromised host can go "fishing" for configurations.

In my git-as-a-master configurations I use ssh to connect to the master.

Yes, all hosts using the same master see the "full" set of configs.

If I ever have a clearly separate security domain of sorts, plan would be
to set up a separate git master. I think that makes sense too with a puppet
master.

cheers,



m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Best practices for infrastructure

[Martin Langhoff]

On Fri, Aug 23, 2013 at 12:03 PM, Paul Archer  wrote:

> I'm thinking about setting up a master in the colo with a slaved master at
> each site,
>

I would strongly recommend using "master-less" recipes, which are actually
"a git repository as a master, and cronjobs running puppet apply as client".

On that track, I have recently implemented just that, rolling out to 4
sites totalling a couple thousand client nodes. I have posted on this list
about my glue / tools, which I am publishing at
http://repo.or.cz/w/puppet-git.git . They allow you to feed the reports to
a puppet dashboard (something that you usually lose in "master-less"
setups.

An addition to puppet-git being triggered via cron, I have a configuration
that sets up an incrond rule, so if we need an immediate rollout, touching
a file in a magic directory triggers the clients to update their config
right now.

My puppet-git is good, I recommend it :-) -- but YMMV on that. But using
git as a master is, IMHO, best practice at scale.

cheers,



m
-- 
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] any elegant way to iterate/map over data types?

[Martin Langhoff]

On Tue, Jun 11, 2013 at 7:36 AM, Matthias Saou  wrote:
> When it comes to iterating with puppet, the usual way to get where you
> want is to apply a definition to an array. From there, you need to avoid
> the (also usual) duplicate declarations, by extending and abusing the
> $title if needed in order to make sure it's unique.

This graf gave me what I needed, thanks! I used md5 to give each key a
unique name and avoid the key appearing twice in the file, which makes
the file unreadable.

class rl_users {
## "ssh_user" pulls together the handling of
##  - usergroup
##  - user
##  - ssh key
## which normally Puppet tracks independently
define ssh_user($uid, $gid, $password, $akeys, $ensure=present) {
user{ $name :
ensure   => $ensure, managehome => true,
uid  => $uid,gid=> $gid,
password => $password,
groups   => ['wheel'],
require  => Group[$name],
}
group { $name :
ensure => $ensure,
gid=> $gid,
}
multi_ssh_authorized_key { $akeys:
ensure   => $ensure,
username => $name,
}
}
define multi_ssh_authorized_key ($ensure, $username) {
ssh_authorized_key { $name:
name=> md5($title), # a shorter name
ensure  => $ensure,
key => $title,
type=> 'ssh-rsa',
user=> $username,
require => User[$username],
   }
}

}

so now a user definition can look like

@ssh_user { 'martin.langhoff':
uid=> 2000 , gid => 2000,
password => '$6$gaga.',
akeys => [ 'onekey' , 'anotherkey' ]
}

and it all works.

thank you!



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] any elegant way to iterate/map over data types?

[Martin Langhoff]

Sysadmins have the (reasonable?) expectation of installing more than
one ssh key.

Relevant bits from my current config follows:

class rl_users {
define ssh_user($uid, $gid, $password, $akey, $ensure=present) {
user{ $name :
ensure   => $ensure, managehome => true,
uid  => $uid,gid=> $gid,
password => $password,
groups   => ['wheel'],
require  => Group[$name],
}
group { $name :
ensure => $ensure,
gid=> $gid,
}
ssh_authorized_key { "${name}-akey":
ensure  => $ensure,
key => $akey,
type=> 'ssh-rsa',
user=> $name,
require => User[$name],
}
}

@ssh_user { 'foo':
uid=> 2004 , gid => 2004,
password => '$6$foo',
akey => 'B3xyz/VFwxhtYhw==',
}

# how can we support user bar?
@ssh_user { 'bar':
uid=> 2005 , gid => 2005,
password => '$6$bar',
akey => [ 'B3xyz/VFwxhtYhw==',
   ''Bz==' ]
}

Right now I have a fugly kludge in place to support a second "akey0" slot.



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] puppet apply -- override node name, module path

[Martin Langhoff]

Hi folks,

testing puppet configs, I have

  /home/martin/mytestingpuppetconfigs/{manifests,modules}

and while working in there, I would like to be able to say something
along the lines of:

 puppet apply --noop --nodename=foo01 --modulepath=./modules manifests/site.pp

which should tell me what puppet wants to do to my system.

Is there any practical way to achieve this? I am not planning on
having one test VM per "victim" system :-p

cheers,


m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Very slow puppet agent runs on empty/noop manifests (v3.2.1)

[Martin Langhoff]

On Thu, May 23, 2013 at 12:25 PM, Keith Burdis  wrote:
> Are you running into the Socket.gethostbyname(Socket.gethostname) issue
> pointed out by Wil Cooley a few days ago?
>
>   https://groups.google.com/forum/?fromgroups#!topic/puppet-dev/z09Nkk18tRE
>
> If so there is potential /etc/hosts change in that thread that might sort
> out your issue.

And indeed it is. Funny thing, the hostname resolves fine and quick

# time host rl01m-puppet
rl01m-puppet.remote-learner.net has address 204.13.103.68

real 0m0.025s
user 0m0.009s
sys 0m0.009s

I am guessing the timeout is on a reverse resolution.

thank you!



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Very slow puppet agent runs on empty/noop manifests (v3.2.1)

[Martin Langhoff]

This is a VM with 1GB RAM and almost nothing running on it. It takes
10s to read a manifest that defines one node, one class, and checks
whether puppet package is installed...

Where is all the time going? Is something wrong?

Details -

  # cat /etc/redhat-release
  CentOS release 6.4 (Final)

  # vmstat
  procs ---memory-- ---swap-- -io --system--
-cpu-
   r  b   swpd   free   buff  cache   si   sobibo   in   cs us
sy id wa st
   0  0  0 705996  32180 20747600 2 2   164  0
 0 100  0  0

  # time ruby -e 'puts "1"'
  1
  real 0m0.012s
  user 0m0.003s
  sys 0m0.007s

Just shy of 6s to display help!

  # time puppet help | head -n1
  Usage: puppet  [options]  [options]

  real 0m5.990s
  user 0m0.764s
  sys 0m0.212s

And 10s to do nothing!
  # time puppet apply --noop --verbose manifests/site.pp
  Info: Applying configuration version '1369325470'
  Notice: Finished catalog run in 0.29 seconds

  real 0m10.226s
  user 0m3.065s
  sys 0m1.981s

[root@rl01m-puppet puppet]# rpm -qi puppet
Name: puppet   Relocations: (not relocatable)
Version : 3.2.1 Vendor: Puppet Labs
Release : 1.el6 Build Date: Wed 22 May
2013 12:29:05 PM CDT
Install Date: Wed 22 May 2013 04:09:54 PM CDT  Build Host:
verne-builder-1.delivery.puppetlabs.net
Group   : System Environment/Base   Source RPM:
puppet-3.2.1-1.el6.src.rpm
Size: 3239292  License: ASL 2.0
Signature   : RSA/SHA1, Wed 22 May 2013 02:14:33 PM CDT, Key ID 1054b7a24bd6ec30
URL : http://puppetlabs.com

[root@rl01m-puppet puppet]# rpm -qi ruby
Name: ruby Relocations: (not relocatable)
Version : 1.8.7.352 Vendor: CentOS
Release : 10.el6_4  Build Date: Fri 08 Mar
2013 08:27:43 AM CST
Install Date: Wed 22 May 2013 04:09:51 PM CDT  Build Host:
c6b10.bsys.dev.centos.org
Group   : Development/Languages Source RPM:
ruby-1.8.7.352-10.el6_4.src.rpm
Size: 1897682  License: Ruby or GPLv2
Signature   : RSA/SHA1, Fri 08 Mar 2013 10:50:23 AM CST, Key ID 0946fca2c105b9de
Packager: CentOS BuildSystem 
URL : http://www.ruby-lang.org/


cheers,



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Puppet, git & security

[Martin Langhoff]

On Wed, May 15, 2013 at 2:44 AM, Stephen Gran
 wrote:
> Your push server can run git update and then rsync to the masters.

Why rsync if you have git?

You have

 - the machine(s) where you edit and make commits on git, you then
_git push_ to what I'll call a "gold" git server

 - the gold git server, normally it answers requests, but in Alex's
company it would run a cron to push to essentially local mirrors in
their various networks.

 - "local puppet masters" -- given that you are using push from the
gold server to the "local puppet masters", you can use a post-receive
hook to auto-checkout the update as part of the push. so as soon as
the push completes, the puppet server code is serving the new files...

 - puppet clients that pull from their local puppet masters.

 --

if you use rsync, you have a potentially large window of time during
which the tree of files is inconsistent on the local puppet masters --
larger if the network connection is slow or unreliable. rsync updates
each file atomically, but not the tree atomically.

Using git shrinks that window considerably -- but not completely.

I am currently working on a similar infra, but local puppet masters
are allowed to "git fetch" from the gold server. The tooling I've
developed around it is here http://repo.or.cz/w/puppet-git.git/ with
some discussion of usage and design here
https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/OilxMytnD_k

cheers,



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: High Availability of Puppet server for separate geographical location

[Martin Langhoff]

On Tue, May 14, 2013 at 7:35 AM, Erik Dalén  wrote:
> We are using SRV records for running multiple puppetmasters and selecting a
> site local but allowing fallback to others in case it is down.
> We have 6 puppetmasters for the production environment running in this way
> currently. Each normally handling 500-1000 nodes. The git repository is push
> replicated to each one of them.

Interesting - do you use dashboard or anything similar to track the
state of the nodes?

cheers,




m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: High Availability of Puppet server for separate geographical location

[Martin Langhoff]

On Fri, May 10, 2013 at 1:52 PM, Ramin K  wrote:
> reasonably resilient or at least able to localize failure. Certainly some
> designs and technology are better than others, but implementation always
> matters.

Of course. I think we're saying the same thing, at the end of the day.


> Masterless Puppet with git as a distribution method does have some
> things going for it as a design. You are giving up things like collected
> resources and standard reporting which may or may not matter you.

Thanks for these notes.

I am definitely supporting reporting -- finishing the implementation
today/Monday.

Reading the thread I mentioned before, ISTM that resources can be
trivially added with the same machinery.

> Additionally you're building a distribution system of some sort even if it's
> just git and ssh where you'd need to decide how to deal with the failure of
> the parts.

Absolutely, and that's at the center of my design

 - It is a pull design (clients pull from a "gold" server, or through
"local proxy" servers)

 - If the pull fails, it's a soft error; we soldier on. config changes
that have been fetched earlier still apply. importantly, config
changes that were fetched and have scheduling data have their
scheduled rollout still apply.

   For example, I make a commit with a change at midday today, and
schedule it for midnight. I publish it on the gold server. Clients
fetch it, but don't apply it yet. Gold server goes down at 10pm --
this is a soft problem: clients will apply the changes
published&distributed earlier when their time comes.

 - Communication back to the gold server is store-and-forward, so that
midnight puppet run collects reports (and perhaps resources in the
future), and they are spooled locally until the gold server is ready
to receive them.

> In any case I'd like to see more discussion on highly available
> Puppet regardless of way it's implemented.

+100



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: High Availability of Puppet server for separate geographical location

[Martin Langhoff]

On Thu, May 9, 2013 at 2:31 PM, Ramin K  wrote:
> Hubris, today thy name is Martin. :-)

Fair enough. I am happy about the tool I am writing (almost finished!)
but, as the followup post makes clear, it isn't about the designe of
ppg. It is about the design of git.

> I'd argue that people have stressed about DNS availability for just
> under three decades and that we are currently enjoying the fruits of that
> labor. Personally, I have yet to work at a company where DNS has not caused
> a significant outage.

I am really surprised at your statement. Of course mishaps can happen,
or someone can mess up configuration DNS royally. But setting up a
primary and secondary setup is trivial.

SMTP and LDAP are also examples where resilience was baked into the
design. With those two, the quality of implementation, and
complications in setup make for a lot more breakage.

Compare to HTTP, databases etc where there's a whole industry of tools
to make things somewhat reliable.

Maybe we are talking about different things.

> Your ppg tooling does look interesting, but there is a large trade
> off in functionality

What is the loss of functionality you see? Anything that you use in practice?

(Reading here 
https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/7ZpAMrMb2NQ
I can't spot anything major, but I may be missing something...)

cheers,


m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: High Availability of Puppet server for separate geographical location

[Martin Langhoff]

On Thu, May 9, 2013 at 10:42 AM, Martin Langhoff
 wrote:
> I am writing some tooling for git+puppet (search for ppg in recent
> posts to this list), and it's trivial to add N-tiers of redundant
> servers...

Heh, so trivial in fact that you can use round-robin DNS and it'll just work :-)

I looked into adding the "feature" to ppg... then realized that git
plays very well with round-robin DNS.



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: High Availability of Puppet server for separate geographical location

[Martin Langhoff]

On Thu, May 9, 2013 at 12:06 AM, John Warburton  wrote:
> I suppose all HA solutions are difficult

Nah. A service correctly designed to be resilient can be HA with
trivial investment.

DNS is a good example. It may have blemishes but nobody stresses about
its availability. Setup as many tiers of redundancy as you want,
easily.

Puppet has no need to be centralized -- a git-based puppet setup can
handle it just fine.

I am writing some tooling for git+puppet (search for ppg in recent
posts to this list), and it's trivial to add N-tiers of redundant
servers...

cheers,



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Practices: what _not_ to manage with Puppet?

[Martin Langhoff]

On Mon, May 6, 2013 at 8:49 AM, Bernardo Costa  wrote:
> Well, I'll tell you that for now some kind of configuration is difficult to
> be done with puppet. At least I couldn't find a way to do it. Ex:
> controlling a /etc/passwd file but partially with a libnss compat syntax.
> This means entries of local users are no controlled but entries beginning
> with a '+' are. As I couldn't find a way to do it, for now it is not being
> controlled by puppet.

Interesting -- that sounds like something that Puppet's
user/useradd.rb could be taught about? If /sbin/useradd has support
for it, I guess it should be a reasonable patch. If not, that's a
whole another kettle of fish...

I guess what I mean to say is: if Puppet had support for that case,
you'd use it. There aren't fundamental or practical reasons no to. Is
that correct?

cheers,



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Practices: what _not_ to manage with Puppet?

[Martin Langhoff]

On Sun, May 5, 2013 at 2:11 PM, Larry Fast  wrote:
> What about the larger processes involved in incremental updates?  Eg.
> sequencing your updates so that the service keeps running.   I'm considering
> using Jenkins to orchestrate sequencial activity.

Coming from an ISConf background, I'd do it makefile-style.

Have actions that indicate completion by touching an empty file, and
tell puppet about it with a "creates".

 exec { "/usr/local/puppetactions/frambulant-upgrade-v1-v2":
   ...
   creates => '/var/lib/puppetactions/frambulant-upgrade-v1-v2'}
 }
 exec { "/usr/local/puppetactions/frambulant-upgrade-v2-v3":
   ...
   requires => Exec['/usr/local/puppetactions/frambulant-upgrade-v1-v2']
   creates => '/var/lib/puppetactions/frambulant-upgrade-v2-v3'}
 }

Haven't actually used this yet, but my reading of
https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/ssn6t2B8us0
is that it would work...

TBH, I am not familiar with Jenkins, perhaps it does something else?



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Practices: what _not_ to manage with Puppet?

[Martin Langhoff]

On Fri, May 3, 2013 at 4:43 PM, Schofield  wrote:
> Everything else is managed by puppet.

Do you manage complex network setups (bonding, routing) via puppet?
There is a certain degree of chicken-and-egg in that; how do you
handle managing configuration without breaking the network that
delivers the puppet config to the host?

Do you manage complex disk setups (RAID arrays, DRBD) via Puppet? Any
hints as to how?

Or perhaps you only use Puppet so extensively in VMs, where you don't
have to deal with all these pesky issues?

For some tasks we _don't_ use VMs (high perf HA DB servers, asterisk
servers are two top examples). I find that managing the config of
those boxes is enormously important to retain sanity...

Of course, we use lots of almost-identical VMs for things that are a
good fit for VMs (webservers, etc)...



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Practices: what _not_ to manage with Puppet?

[Martin Langhoff]

On Sat, May 4, 2013 at 7:33 AM, Erik Dalén  wrote:
> I'd definitely configure the servers and
> clients for those protocols using Puppet.

Is that what you do, or what you _would_ do? ;-)

> Or using DHCP to configure networking instead of having puppet setting it
> statically on your hosts.

Do you actually do that? How do you handle complex routing/bonding setups?

cheers,



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: ppg: Scheduled rollouts and dashboard with git in decentralized setup

[Martin Langhoff]

On Tue, Apr 23, 2013 at 5:22 PM, Martin Langhoff
 wrote:
> We will have a wrapper, "ppg" for puppetgit -- and avoiding confusion
> with PostgreSQL tools.

After some delays in getting started...

   http://repo.or.cz/w/puppet-git.git/

Still a work in progress, but if I can get two more productive days
in, it'll be close to complete.

cheers,



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Practices: what _not_ to manage with Puppet?

[Martin Langhoff]

While I prep my scripts and tool up for a large infra, I want to
revisit a question that I ask myself regularly: what do people not
manage with Puppet (or wish they weren't)?

In my situation (a RH-style world), initial base system install, inc
disk layout and initial networking is handled with kickstart

For example: Do you exclude mountpoints? Network/SAN mountpoints?
Advanced network configs?

What are the reasons to exclude a particular item? How do you manage it instead?

cheers,



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] High Availability of Puppet server for separate geographical location

[Martin Langhoff]

On Mon, Apr 29, 2013 at 9:55 AM, shyam sundar Keshari
 wrote:
> I have to configure puppet server in Primary-Secondary mode for 2
> distributed location .

I am in a similar situation. Not liking the options available, I am
building "puppetgit"
https://groups.google.com/forum/?fromgroups#!topic/puppet-users/OilxMytnD_k

cheers,



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] ppg: Scheduled rollouts and dashboard with git in decentralized setup

[Martin Langhoff]

On Mon, Apr 29, 2013 at 6:19 PM, Felix Frank
 wrote:
> Interesting. It seems nicely thought out, but I stumbled here, reading:

thanks for reading!

> On 04/23/2013 11:22 PM, Martin Langhoff wrote:
>> I am less certain of this part, and input will be specially valuable here.
>>
>> ppg pullapply will...
>>
>>  -  apply changes locally, capture stderr/stdout, perhaps more info
>> that can be negotiated with the puppet client ("facts"?).
>>  - write state to file(s) in a "puppet-feedback" git repo, commit that state
>>  - push to a "feedback" rw repo on the gold server (or on the proxy server)
>
> Uhm, what? Why? Why is there a git repository for your transient puppet
> reports?

Well, the assumption of this setup is that the server where you'd run
dashboard isn't necessarily reachable all the time.

For example, during a network outage, or an uplink DoS.

So I need some store-and-forward facility. Using git for this purpose
isn't the absolute best-fit but limits my tool use, my dependencies. I
could use some other tool (sqlite?) but git handles store-and-forward
setup pretty well (with the normal git push semantics).

Hard to justify added deps and complications unless there's a great
fit to the proposed alternative...

> You're reinventing the wheel I think (although your's a bit square-ish ;)

But isn't it cute how it goes thunk! four times per turn?

> Doesn't the dashboard usually consume the report as generated by the
> agent? Therefor, isn't what you want a way to transfer that very report
> from the agents to the dashboard? I vaguely remember an issue with
> masterless not generating reports, but I may misremember this one.

Correct, masterless won't generate reports, and that's part of what I
am trying to address.


m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] ppg: Scheduled rollouts and dashboard with git in decentralized setup

[Martin Langhoff]

For a "server-less" puppet setup using git for config distribution, I
am drafting out some scaffolding...

Some background in the message I just posted:
https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/A5Ywi1V1OrA

Plan is to have two branches in git: master and production. Commits
will be normally be made to master (which actually acts as the
"queue").

We will have a wrapper, "ppg" for puppetgit -- and avoiding confusion
with PostgreSQL tools.

=Commits and scheduling=

Commits can only be made using ppg, enforced through a commit hook.

Commits with ppg can be --immediate, in which case they are committed
to master and prod (in case they are the same).

Alternatively, commits with ppg can be --schedule [timestamp].

On every commit, ppg checks that the "production" branch is a subset
of master, that is, that a `git merge master` will just mean a
fast-forward. If the two branches have diverged, these checks will
force the user to merge back into master to ensure any differences are
resolved and accounted for.

When using --schedule, ppg checks whether an earlier commit is
scheduled for a later time -- and errors out to prevent premature
rollout of changes due to conflicting schedules.

ppg also runs puppet validate over the files being committed.

=Scheduling happens on the Gold server=

To implement the scheduled rollouts, ppg tags the commit with a
specially crafted tag. The gold server runs a periodic cron that scans
unmerged changes on master and merges them if the timestamp in the tag
has been reached and the merge is a fast-forward.

=Client side apply=
On the client side, `ppg pullapply -- [puppet params]` runs a git pull
and only invokes puppet apply if git has brought changes to the local
branch (normally production).

ppg pullaply collects the output from puppet and somehow pushes it all
the way back.

=Store-and-forward feedback channel=

I am less certain of this part, and input will be specially valuable here.

ppg pullapply will...

 -  apply changes locally, capture stderr/stdout, perhaps more info
that can be negotiated with the puppet client ("facts"?).
 - write state to file(s) in a "puppet-feedback" git repo, commit that state
 - push to a "feedback" rw repo on the gold server (or on the proxy server)

ppg on the proxy and gold servers will take care of store-and-forward
until it reaches its destination (a dashboard server). ppg also takes
care of pruning very old data that has already been delivered.

Once the data reaches the dashboard server, it gets fed to the Puppet
Dashboard thingamajig, butterfly-mode is automagically enabled in your
emacs session and you're so so glad you took the blue pill.

thoughts? comments? bikesheds?



m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] git-based workflow with local proxy, dashboard and scheduled/embargoed rollouts

[Martin Langhoff]

Hi Puppeteers,

I am working with a team to manage a large, complex infra covering
several thousand VMs plus specialized hardware boxes in several data
centers. I have some backstory with Puppet (and isconf2, and
infrastructures.org) and I authored several tools in the
git-swiss-army-chainsaw.

Our current approach is to avoid master/server due to scalability
issues and not-always-reliable connectivity across locations.

So our (trivial) plan is

 - setup a "gold" git server, where changes are published
 - each datacenter has a "proxy" (git mirror, or similar) that pulls
from gold server
 - each client pulls from their local proxy

In case of problems with the gold server or connectivity (outage,
DDoS), the "proxy" can be detached and updated independently.

However I am planning to do two things differently (from what I have
seen discussed):

 - Starting to draft some scaffolding ("gp" for gitpuppet) to allow us to say

  gp commit --deploy-at "05-01-2013 12:00:00" -m "Strangle the frog at
the strike of midnight"

Is there any existing tool that will provide scheduled rollouts?
Searching around I haven't found anything...

 - Looking into whether I can feed data from the puppet client runs
back to the dashboard. My current plan is to push that data back the
same way -- store-and-forward style through the proxy to the gold
server, then to a machine running the dashboard.

Is anyone working on a similar track? Hints?

cheers,


m
--
 martin.langh...@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Do people walk the filebucket tree searching by path?

[Martin Langhoff]

On Thu, Feb 17, 2011 at 3:55 PM, Nigel Kersten  wrote:
> ok. So it's unacceptable for you to refer to logs or reports to get
> the checksum for a given replacement and then restore the file that

It's really damn fiddly :-)

As a git guts hacker, I appreciate that puppet stores things in a
content addressable filesystem. But when I need to use the info in git
or in puppet, I refer to it by path :-) ..

Actually git has some rich syntax to say "the previous version", like

   git diff HEAD^ # the prev commit
   git diff HEAD^^ # two commits back
   git diff HEAD^^ # three commits back ;-)

that kind of glue is of enormous value.



m
-- 
 martin.langh...@gmail.com
 mar...@laptop.org -- Software Architect - OLPC
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Re: best practice for removing a module & maintained resources from a system ?

[Martin Langhoff]

On Thu, Oct 21, 2010 at 4:56 AM, Felix Frank
 wrote:
> I guess what you're getting at is this: No, puppet is not exactly good
> at "uninstall this now and from then on, don't care about it anymore".
> This is not what puppet has been conceived for, though.

OK - but putting (config) files into place is most of what we do with it right?

We need to be able to manage them after putting them there, and that
includes removing them.

> If you're on the purge train, you won't want your package manager to
> interfere with your conf.d directories. Instead, puppet will need the
> whole picture of what should be in the conf.d.
>
> Most puppet providers (package, host, mount, cron etc.) strive to do the
> opposite, and amend to a given state. Purge is a way to switch
> paradigms. If you choose to do that, be prepared to deal with the
> consequences.

Can you flesh out what other consequences you see?



m
-- 
 martin.langh...@gmail.com
 mar...@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Re: best practice for removing a module & maintained resources from a system ?

[Martin Langhoff]

On Wed, Oct 20, 2010 at 3:00 PM, Mohit Chawla
 wrote:
>> Except that some definitions may be gone. That's what worries me. Sure
>> I can read the pp files as they are today.
>
> That's probably true for any tool or method. Unless it was in version
> control.

Not true of packages under any modern packaging system, as per the
example given.



m
-- 
 martin.langh...@gmail.com
 mar...@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Re: best practice for removing a module & maintained resources from a system ?

[Martin Langhoff]

On Wed, Oct 20, 2010 at 12:03 PM, Patrick  wrote:
> I wouldn't call it unreasonable.  I would call it "lack of a really cool 
> feature".

Pretty fundamental feature :-)

I am not saying Puppet needs to magically know what I'd like to happen
with the file. However, it should at least give us enough tools/info
to operate.

>> Imagine you are joining a puppet-using company. To "remove a class"
>> from a server you now need to know all the files ever installed or
>> managed by that class, since puppet started to be used there.
>
> Except if you look through the definitions in the class it shouldn't be too 
> manually write rules that do the opposite.

Except that some definitions may be gone. That's what worries me. Sure
I can read the pp files as they are today.

 - John had a webmail server class assigned to server Frodo. Service
apache must be running, an /etc/apache/conf.d/webmail.conf must be in
place.

 - Frodo is no longer a webmail server. John added a rule to disable
apache service.

 - Here I come new job! First task: set Frodo to be a moodle server.
New class has moodle dpkg, /etc/apache/conf.d/moodle.conf -- and
enables apache.

Hell-O!

> Second, you don't need to do this for config files in directories managed 
> with "purge".

How would you tackle the example above with 'purge'?  Note that
/etc/apache/conf.d/ has other files too, provided by the apache dpkg
(and similar in /etc/httpd/conf.d on RH/Fedora machines).

> A much nastier example is File.  I've been managing a config file.  The file 
> existed before and File replaces it.  Later the file is replaced by something 
> other than Puppet.  File dutifully replaces the file again.  Which file 
> should puppet restore?

I think we should at least be able to say "remove it when we no longer
mention it". Something like 'purgewhengone' => YesPlease

Don't know if Puppet as it stands today keeps enough state for this.

Any kind of 'restoreoriginal' would get into the problems you outline,
and then some more. Which "original"?

cheers,


m
-- 
 martin.langh...@gmail.com
 mar...@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Re: best practice for removing a module & maintained resources from a system ?

[Martin Langhoff]

On Mon, Oct 18, 2010 at 9:13 AM, jcbollinger  wrote:
> I'm guessing you mean you have written sub-*classes* to do that job.
> That is indeed the Puppet way to do it, and I don't find it at all
> ridiculous.

As a puppet newcomer, that is a bit surprising, and IMO unreasonable.

Imagine you are joining a puppet-using company. To "remove a class"
from a server you now need to know all the files ever installed or
managed by that class, since puppet started to be used there.

There is no place (in puppet) to query that info! Hopefully some SCM
was used, and used consistently. And you have to review by hand each
revision. Joy!

Compare/contrast this to what package mgmt tools do: they keep a small
DB of what files they are tracking for each package on a given
machine.

A new version of the package needs only declare what files it has. The
pkg manager will helpfully remove any stale files. From the PoV of the
packager... I don't need to know what files any and all releases of
this package installed -- it'd be impossible to know anyway. I don't
need to remove any "possible" file ever installed by this package.

Puppet manages many resource types, so this isn't trivial. For some
resource types "what to do" won't be super clear.

Puppet (currently doesn't, but ) should track what it manages, and
help us remove it when appropriate.

cheers,



m
-- 
 martin.langh...@gmail.com
 mar...@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Re: Using puppet to update itself

[Martin Langhoff]

On Fri, Oct 8, 2010 at 4:56 PM, Forrie  wrote:
> That's more of what we're looking to do.  I think it would be a bad
> idea to have puppet automatically updating clients.  This would need
> to be a one-off, scheduled item you would trigger from the puppet
> master server, under the default {} node, I would presume;

I don't think it's a matter of working on it on the server side of puppet.

> Perhaps having the puppet (and facter) gem reside in a puppet://files/
> URL

rpm, dpkg or gems, it doesn't really change the risk, nor the
complication. You need something that says:

  - stop the service
  - run command X (with no calls to elements of the package itself) --
rollback if it fails
  - start the service

Hard to do right.

cheers,


m
--
 martin.langh...@gmail.com
 mar...@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Using puppet to update itself

[Martin Langhoff]

On Fri, Oct 8, 2010 at 4:18 PM, Disconnect  wrote:
> We just use packages:
>  package { "puppet": ensure => latest }

If the rpm/deb script attempts to restart the service, that will stop
puppet in the middle of the execution of the pkg manager itself.
(Unless there's special handling of this case implemented somewhere in
the toolchain -- see sshd).

Seems safer to download the package file and schedule through
something that decouples the pkg manager from the puppet client
process.



m
-- 
 martin.langh...@gmail.com
 mar...@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Re: Puppet use with OLPC's XS

[Martin Langhoff]

On Fri, Sep 10, 2010 at 9:56 AM, Luke Kanies  wrote:
> It looks like the list has you pretty well handled, so mostly I'm writing 
> back to say I'm glad to see ISconf users still around, and very glad to see 
> it looks like Puppet will work for you.

Indeed, very well treated. Even my oddest questions are being answered! Thanks!

> That being said, I'll throw my responses in here, too, since I've got more of 
> the ISconf context.
>
>> 1 - With a high number of identical nodes, we are very precious about
>> deploying the exact same sw package (rpm in this case) to all nodes.
>> Can I declare an exact rpm version/revision in a packages[] section?
>> Can I later update it with a later version and will it know to upgrade
>> via yum?
>
> ensure => '1.0' or whatever works fine.

Excellent - thanks! I was going to write full package+version/revision
names instead (which wouldn't have worked properly).

>> 2 - Can I say "install this shellscript and run it once"? How?
>
> Not in so many words, but as Patrick said, use the 'creates' argument to 
> provide idempotency and you'll actually be happier.

Cool! I do write my scripts to be (internally) idempotent, but I hate
to see them re-run everytime (as time goes by, the number grows...) so
touching a file and using the 'creates' arg will DTRT.

>> 5 - Is there any "server status" monitoring tool that
>> integrates/piggybacks with Puppet? Our NAT'ted clients make most
>> monitoring tools a pain...
>
> Our own Dashboard, along with Foreman and a few others.

Will look at Dashboard more in depth - thanks!

>> 6 - For disconnected configuration clients -- is there a way to tell
>> the puppet client: the puppet config of the server is available to you
>> on file:///media/usb/puppetconfig ? The XS already has a mechanism
>> that can guarantee the integrity of files coming from the sysadmin
>> team (so only "signed" files are accepted over usb).
>
> Yes, use stand-alone 'puppet' rather than a client/server.  You can still 
> report back in this mode, although that probably won't help you much if 
> you're disconnected.

Overall, I have to say puppet is treating me a bit too well. Thanks!!!

cheers,


m
-- 
 martin.langh...@gmail.com
 mar...@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Re: Puppet use with OLPC's XS

[Martin Langhoff]

On Wed, Sep 8, 2010 at 11:45 AM, alcy  wrote:
>> 2a - Is there an ISConf-like facility that says "run it until it
>> succeeds once"? [ Happy to use Makefiles, but if there's a
>> Puppet-supported elegant way of doing it... ]
>
> If the code of the shell script can be minimized to use as much as of
> Puppet's existing resources, that would be a good way to go about
> things. Of course, you can execute the shell scripts, and define the
> conditions that you mentioned. Have a look at the exec resource type
> in puppet for such things.

Thanks! Will look at exec. I think I'll make a /var/lib/mypuppetstate
directory and touch files in there on successful completion of
scripts.

>> 3 - Can I hook up a trigger so when a new conffile for a service is
>> installed the service is restarted? IE: I provide a new
>> /etc/named.conf, I want to add a trigger that says that everytime it
>> changes, "services named restart" should be invoked...
>
> Yes, you can have the service "subscribe" to the configuration file:
>
> service{"named":

And I assume that it knows that on Fedora the magic incantation is
service  restart...

That'll only work if the file is updated by Puppet, correct? Or is
there an inotify side to Puppet that I didn't know before...?

If the service is sub'd to several files, and all are updated "in one
go" from Puppet, will there be one service restart, or one per file?

>> 4 - Can the clients send a "deployed configuration successfully" msg
>> to the server, so we keep a tally of
>>
>>   - clients seen recently
>>   - success vs failure in deployment of latest config
>>   - what clients are failing or behind in their sync
>
> At the most basic level, you can do this by simply enabling report =
> true in clients' configuration and at the server reports = log. This

Hmmm... from what I've seen, that gets me when the puppet client
_requested_ stuff, but not a "puppet client deployed successfully".

Am I missing something?

> like Dashboard & Foreman.

Will investigate ;-)



m
-- 
 martin.langh...@gmail.com
 mar...@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

[Puppet Users] Puppet use with OLPC's XS

[Martin Langhoff]

Hi Puppet Users, Hi Luke,

Happy to find myself here. I am an old ISConf user -- Luke might
remember my random questions back in the infrastructures / isconf list
-- and have been developing the School Server ("XS") for OLPC for a
while (and lately doing a dozen more things at OLPC which has shrunk
my XS time).

Writing here because my original intention with the XS was to dust-off
some ISConf concepts and have a minimalistic modern implementation of
something isconf-ish that should integrate well with package mgmt
tools (dpkg/apt, rpm/yum). By the time I managed to look at the
situation, I was very happy to find Luke had done something similar
(but much better than I'd be able to) -- Puppet!

[ Did I say THANKS!? Should say it lots, and often. ]

So I've just been working with a deployment team in the field, and
working through the Puppet docs, defined an initial configuration
strategy, and rolled out the puppet client to the initial 10 XSs (soon
to be joined by approx 800 more).

I have a series of questions, coming from an old hand in terms of
config mgmt, but absolutely new to Puppet. Happy to RTFM, or to code
what's missing; pointers welcome. In many cases I might be missing the
right terminology for things in puppet-land.

Your help is appreciated. Just in the deployment above, the "userbase"
for this infra is 50 000 children with XOs. Demanding userbase, let me
tell you... but they sure know how to say thanks, and how to get a lot
from what we put into it :-)

First, I'll confess to some oddities in our usage:

 - All our nodes are identical (for the purposes of Puppet anyway --
differences are handled by code installed on the node).

 - Currently our nodes run Fedora 9, with an old (0.24.x I think)
Puppet. And update is in the works, but bear in mind for my questions
later...

 - Some of our nodes are behind NAT... multiple layers of it sometimes.

 - Some of our nodes have no WAN/Internet -- questions to follow on this. -

=Questions=

1 - With a high number of identical nodes, we are very precious about
deploying the exact same sw package (rpm in this case) to all nodes.
Can I declare an exact rpm version/revision in a packages[] section?
Can I later update it with a later version and will it know to upgrade
via yum?

2 - Can I say "install this shellscript and run it once"? How?

2a - Is there an ISConf-like facility that says "run it until it
succeeds once"? [ Happy to use Makefiles, but if there's a
Puppet-supported elegant way of doing it... ]

3 - Can I hook up a trigger so when a new conffile for a service is
installed the service is restarted? IE: I provide a new
/etc/named.conf, I want to add a trigger that says that everytime it
changes, "services named restart" should be invoked...

4 - Can the clients send a "deployed configuration successfully" msg
to the server, so we keep a tally of

  - clients seen recently
  - success vs failure in deployment of latest config
  - what clients are failing or behind in their sync

5 - Is there any "server status" monitoring tool that
integrates/piggybacks with Puppet? Our NAT'ted clients make most
monitoring tools a pain...

6 - For disconnected configuration clients -- is there a way to tell
the puppet client: the puppet config of the server is available to you
on file:///media/usb/puppetconfig ? The XS already has a mechanism
that can guarantee the integrity of files coming from the sysadmin
team (so only "signed" files are accepted over usb).

That's all -- Thanks in advance for your help.

Are these very odd questions?

cheers,


m
-- 
 martin.langh...@gmail.com
 mar...@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.