[Puppet Users] Re: defined function notparse order dependant

[Mike Power]

I appreciate the reply but I think the thrust of my point is getting missed.

Any other solution is far more cryptic, some reader coming and reading the 
code will not understand the code nearly as well.  

The fact that puppet has a dimension (parse order) that is so convoluted 
that its behavior cannot be understood is not a virtue.  It is a very bad 
thing.  

On Wednesday, April 24, 2013 6:19:13 AM UTC-7, jcbollinger wrote:
>
>
>
> On Tuesday, April 23, 2013 4:52:11 PM UTC-5, Mike Power wrote:
>>
>> I have been backed into a corner, by they way puppet works, but some 
>> third party module.
>>
>> Basically I have two resources defined:
>>
>> a {$somevar:}
>> b::b {$somrvar:}
>>
>> both have code that looks something like this:
>> if (!defined(File[$name])) {
>> file { $name:
>> ...
>> }
>>
>> According to the documentation 'defined' is dependent on parse order.  So 
>> resource definition 'a' should be parsed first and win.
>
>
>
> This is not a characteristic to manipulate, but rather a strong reason to 
> avoid defined() altogether.  I cannot say this more strongly: DO NOT USE IT.
>
>  
>
>>   But it doesn't, no matter how I order 'a' and 'b::b', resource 
>> definition 'b::b' always defines the file.  I even tried puppet version 
>> 3.1.1.  Same problem.  My problem is that resource "b::b" does it horribly 
>> wrong.  It causes the runtime of my manifest to balloon out from 2mins to 
>> over 10 minutes.  It uses the wrong group.  That would be okay if I could 
>> just put 'a' first and have 'b::b' go silent.
>>
>> Does anyone know why this is happening?  How is "b::b" being parsed first 
>> even though "a" is ahead in the file?
>>
>
>
> The *declaration* of resource A[$somevar] is surely parsed before the 
> declaration of B::B[$somevar], but that's not at all the same thing as the 
> body of defined type 'a' being parsed before the body of defined type 
> 'b::b'.  It's pretty much irrelevant why the parse order you see happens -- 
> Puppet guarantees parse order only within each manifest file, so manifests 
> that depend on parse order across multiple files are flawed.  Even if you 
> found a magic formulation that achieved the order you want on a given 
> version of Puppet, there is no guarantee that it would continue to work 
> even in the next maintenance release.
>
> As I say here from time to time, your manifests should not attempt to 
> query Puppet about what has already been declared.  There are a number of 
> techniques by which that is possible, but generally they can be described 
> as using "data-driven" approaches to manifest and manifest-set design.  As 
> an over-simplified example, you could set a boolean node variable 
> $i_will_use_a, and then change the condition in b::b like so
>
> define b::b (...) {
>   if ! $i_will_use_a {
> file { $name:
>   ...
> }
>   }
>   ...
> }
>
> There are a host of other ways to do essentially the same thing.
>
>
> John
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: defined function notparse order dependant

[Mike Power]

The difference between 'a' and 'b::b' was that 'a' didn't have a file 
declaration, it had a resource declaration that had a file declaration.  So 
if I created a new resource that wrapped 'b::b' then I got a straight 
forward parse-order dependency.  

I tend to use defined because it allows you to under gird something without 
that something actually knowing who you are.  In this case the plugin was 
git.  I was using the repo resource to checkout some code.  It was 
declaring the folder that was going to be checked out.  Defined works nice 
because when you read it, you understand what they are trying to do, even 
if puppet has a hard time producing that result.  Alternatives I have read 
involve defining a class and including it.  But that requires git to know 
about said class, also it doesn't work for resources which applies in this 
case.  The other alternative I have read about is something like 
virtualized resources?  You manifest them at each point.  I have not really 
seen any good documentation on what it is, how it works, what it requires.  
The code I have seen does not make it clear to the user what is taking 
pace, it just doesn't read well.

If you declare things something like this psuedo code:
A
   C
  F1
B
   D
  F2

When people read it they are going to more commonly expect the order to go
A
C
F1
B
D
F2

Not
A
B
C
D
F1
F2


In git they set recurse to true.  Compound that with a resource default up 
the declaration stack and you have puppet spending a long time uselessly 
changing the ownership and permission on a ton a files when they get 
checked out.  I find the dynamic scope of resource defaults to be more of a 
surprise and a burden then a useful tool.  

Anyway I am resolved, thanks for the help.  

On Tuesday, April 23, 2013 4:09:02 PM UTC-7, Nick Fagerlund wrote:
>
> If the relevant code were just sitting there naked in your site manifest, 
> I think you'd probably see a fairly simple parse-order dependency -- I 
> think it's the fact that they're in defined types that's shifting things 
> around. Actually, one of the core team surprised me the other week by 
> telling me that defined types are somehow late-binding when creating their 
> resources, in a way that classes aren't; I can't remember why they thought 
> it had been implemented that way, though.
>
> The point is, this is EXACTLY why we say to not use `defined()` like that, 
> is because it can cause havoc for downstream users like you. I'm afraid 
> you're going to have to fork b::b and go in and muck with its 
> implementation if you want any certainty around this behavior. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] defined function notparse order dependant

[Mike Power]

I have been backed into a corner, by they way puppet works, but some third 
party module.

Basically I have two resources defined:

a {$somevar:}
b::b {$somrvar:}

both have code that looks something like this:
if (!defined(File[$name])) {
file { $name:
...
}

According to the documentation 'defined' is dependent on parse order.  So 
resource definition 'a' should be parsed first and win.  But it doesn't, no 
matter how I order 'a' and 'b::b', resource definition 'b::b' always 
defines the file.  I even tried puppet version 3.1.1.  Same problem.  My 
problem is that resource "b::b" does it horribly wrong.  It causes the 
runtime of my manifest to balloon out from 2mins to over 10 minutes.  It 
uses the wrong group.  That would be okay if I could just put 'a' first and 
have 'b::b' go silent.

Does anyone know why this is happening?  How is "b::b" being parsed first 
even though "a" is ahead in the file?

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] puppet inverts dependencies?

[Mike Power]

Have anyone of you every seen puppet invert a dependency?


Say I have a class like so:
class jenkins {
user {"jenkins":
ensure  => present,
}
}

That should form a dependency like so User[jenkins] => Class[jenkins].
That dependency means that the user jenkins will be done before Class 
jenkins.  However every once (twice now) something disturbs pupplets 
delicate sensibilities and it seems to create a new dependency that goes 
backward from what it is supposed to.  This causes a dependency cycle and 
breaks my catalog.  Right now I have a git module checking out the source 
for a jenkins server.  I end up with a dependency cycle like this:
(Exec[git_http://mich...@github.com/somevalidgiturl] => 
Git::Repo[http://mich...@github.com/somevalidgiturl] => Class[Jenkins] => 
User[jenkins] => Exec[git_http://mich...@github.com/somevalidgiturl])

Notice how it now says Class[Jenkins] needs to be done before 
User[jenkins].  But the jenkins user is defined in the class jenkins.  So 
the dependency should be the other way, It seems to have inverted that 
dependency.  I have found that if I tweak with related components of the 
catalog then it will go away.

https://projects.puppetlabs.com/issues/19718

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: replacing mkdir -p

[Mike Power]

I'll see if I can open source the component I wrote and upload it to puppet 
forge.  In this way the open source community can continue the debate about 
what is the best way to do this, while at the same time those who want can 
use some solution other than mkdir -p

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: replacing mkdir -p

[Mike Power]

Actually I found if I created a resource between path and file called 
element, I could give it a unique name.  Then inside the body I could check 
to see if the File is declared, if not I could declare it.  

On Thursday, April 4, 2013 9:23:54 AM UTC-7, Mike Power wrote:
>
> Puppet right now requires every element of a path to have an individual 
> file definition.  This makes it had to take an arbitrary path as a 
> parameter.  You are forced to require your client to make the entire path 
> structure for you or instead you use an exec resource and call mkdir -p.  
> Using an exec resource does not generate an File resources so autorequire 
> does not work.
>
> I didn't like this, I wanted to be able to once specify a path and have 
> puppet do that autorequire as needed.
>
> Something like:
> path {"/blah/blah/blah/and/blah":
> }
>
>
> In order to make this happen I would have to manually define each file:
> file {"/blah/":
> ensure  => directory,
> }
>
> file {"/blah/blah/":
> ensure  => directory,
> }
>
> file {"/blah/blah/blah/":
> ensure  => directory,
> }
>
> file {"/blah/blah/blah/and/":
> ensure  => directory,
> }
>
> file {"/blah/blah/blah/and/blah/":
> ensure  => directory,
> }
>
> Of course there is a short hand for this:
> file {["/blah/", "/blah/blah/", "/blah/blah/blah/", 
> "/blah/blah/blah/and/","/blah/blah/blah/and/blah/"]:
> ensure  => directory,
> }
>
> Then it occurred to me I could parse the path and produce the array of 
> elements needed.  Something like:
> $path = "/blah/blah/blah/and/blah"
> $file_list = split($path, $file_separator)
> $paths = inline_template('<% parent = nil %><%=@file_list.collect{ 
> |file| parent.nil? ? parent = "#{@file_separator}":parent = 
> "#{parent}#{file}#{@file_separator}"}.join(@path_separator) %>')
> $path_list = split($paths, $path_separator)
> file{$path_list:
> ensure  => directory,
> }
>
> This works great once.  Then you get errors like:
> Error: Duplicate declaration: File[/]
>
> If there anyway to trim down the produced array by removing the resources 
> that already exist?
>
>
>
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] replacing mkdir -p

[Mike Power]

Puppet right now requires every element of a path to have an individual 
file definition.  This makes it had to take an arbitrary path as a 
parameter.  You are forced to require your client to make the entire path 
structure for you or instead you use an exec resource and call mkdir -p.  
Using an exec resource does not generate an File resources so autorequire 
does not work.

I didn't like this, I wanted to be able to once specify a path and have 
puppet do that autorequire as needed.

Something like:
path {"/blah/blah/blah/and/blah":
}


In order to make this happen I would have to manually define each file:
file {"/blah/":
ensure  => directory,
}

file {"/blah/blah/":
ensure  => directory,
}

file {"/blah/blah/blah/":
ensure  => directory,
}

file {"/blah/blah/blah/and/":
ensure  => directory,
}

file {"/blah/blah/blah/and/blah/":
ensure  => directory,
}

Of course there is a short hand for this:
file {["/blah/", "/blah/blah/", "/blah/blah/blah/", 
"/blah/blah/blah/and/","/blah/blah/blah/and/blah/"]:
ensure  => directory,
}

Then it occurred to me I could parse the path and produce the array of 
elements needed.  Something like:
$path = "/blah/blah/blah/and/blah"
$file_list = split($path, $file_separator)
$paths = inline_template('<% parent = nil %><%=@file_list.collect{ 
|file| parent.nil? ? parent = "#{@file_separator}":parent = 
"#{parent}#{file}#{@file_separator}"}.join(@path_separator) %>')
$path_list = split($paths, $path_separator)
file{$path_list:
ensure  => directory,
}

This works great once.  Then you get errors like:
Error: Duplicate declaration: File[/]

If there anyway to trim down the produced array by removing the resources 
that already exist?







-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: Installing puppet modules

[Mike Power]

I use vagrant.  So first I apply on the host machine, then I apply on the 
virtual machine.  

Either solution I use whether master slave or apply I still have not solved 
the problem of downloading the right modules.

On Friday, March 15, 2013 8:31:58 PM UTC-7, Ellison Marks wrote:
>
> Unless you're using puppet apply, you don't download modules to the 
> individual nodes, you download them to your master. The master uses the 
> modules to compile a catalog which it then ships off to a node. Or in other 
> words, I'm not sure what your question is. Why do your nodes need to 
> download modules?
>
> On Friday, March 15, 2013 9:24:36 AM UTC-7, Mike Power wrote:
>>
>> I want to setup a node using a three step process
>> 1) download node requirements (git)
>> 2) download modules to fit requirements (puppet)
>> 3) Bring up the node in the required state (puppet)
>>
>> The second step is the one I am having trouble with.  I want a file 
>> (master.pp) to list out all the modules and what versions are needed before 
>> applying the site.pp file.  From what I have been able to find there are 
>> two possible approaches.
>>
>> 1) Use some module to download and unzip tars.  In this master.pp file 
>> use the module to download the other modules from urls and extract them in 
>> the module path.  The problem I have is that I cannot figure out how to get 
>> the modulepath in the master.pp file and parse it such that I can extract 
>> the tars in the right location.  I tried something like:
>> archive {"$file":
>> ensure  => present,
>> url => "$url",
>> checksum=> false,
>> extension   => "tar.gz",
>> target  => split("$::modulepath", ":")[0],
>> }
>>
>> But that doesn't work.  
>>
>>
>> 2) There is another module called puppet_module (
>> https://github.com/rcoleman/puppet_module_provider) which allows you to 
>> download and install modules from puppetforge.  This module works great.  
>> Except that I need to download modules from an intranet location not an 
>> internet location. I do not have control over puppetforge so I do not have 
>> control over how long stuff stays up there, when it changes etc.  I need to 
>> house the modules on a local file store such that as we make the decision 
>> to upgrade we can do those upgrades other then that the modules will always 
>> remain available.  The puppet_module (reading the source code) does have a 
>> source parameter.  I assume then you could point it at something other than 
>> puppetforge.  But it is not documented and I am not sure I can easily 
>> produce a in house puppet forge that would have the needed filesystem 
>> layout for puppet_module to work.
>>
>> Have other people looked into this problem?  What have they come up with?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Installing module of the correct version

[Mike Power]

I want to install a module into the modulepath.  I can provide either a 
http url to a tar that contains the modules (like puppetforge but the path 
is not as structured) or I could provide a git url and path under that url 
where the module is.

I want to first run puppet standalone on the vagrant vm host.  This should 
download the appropriate modules and make them available on the filesystem 
in the modulepath.  Then I'll stand up a vm and have puppet run standalone 
on the vm using the modules to setup the vm.  

I was looking at two different solutions. 

1) using some sort of archive module to download and extract the module 
(like: http://forge.puppetlabs.com/mthibaut/archive).  The problem being 
puppet requires all paths to be absolute.  I would like to be able to hack 
apart the modulepath configuration in the manifest file.  But I have no 
idea how to do that.  I was trying something like:
archive {"$filename":
ensure  => present,
url => "$url",
checksum=> false,
extension   => "tar.gz",
target  => split("$::modulepath", ":")[0] + "/modules/",
}

But obviously I am a bit out of my league getting that to work.

2) Using the puppet module 
provider(https://github.com/rcoleman/puppet_module_provider) to download 
and install a module.  This works great... if the module is on puppet 
forge.  But I need to keep the connections local, for connectivity outages 
and backups in case third party modules disappear from their place on the 
internet.  Also internal modules are  not going to make it out to the 
forge.  I see in the code something about a source.  But it seems that 
would point to some root folder on the web server and expect a certain 
structure underneath that folder.  I am not sure what the folder is and I 
am even less sure that I could produce it in house.


Have people solved this problem?  What has been their approach?

Mike Power

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Installing puppet modules

[Mike Power]

I want to setup a node using a three step process
1) download node requirements (git)
2) download modules to fit requirements (puppet)
3) Bring up the node in the required state (puppet)

The second step is the one I am having trouble with.  I want a file 
(master.pp) to list out all the modules and what versions are needed before 
applying the site.pp file.  From what I have been able to find there are 
two possible approaches.

1) Use some module to download and unzip tars.  In this master.pp file use 
the module to download the other modules from urls and extract them in the 
module path.  The problem I have is that I cannot figure out how to get the 
modulepath in the master.pp file and parse it such that I can extract the 
tars in the right location.  I tried something like:
archive {"$file":
ensure  => present,
url => "$url",
checksum=> false,
extension   => "tar.gz",
target  => split("$::modulepath", ":")[0],
}

But that doesn't work.  


2) There is another module called puppet_module 
(https://github.com/rcoleman/puppet_module_provider) which allows you to 
download and install modules from puppetforge.  This module works great.  
Except that I need to download modules from an intranet location not an 
internet location. I do not have control over puppetforge so I do not have 
control over how long stuff stays up there, when it changes etc.  I need to 
house the modules on a local file store such that as we make the decision 
to upgrade we can do those upgrades other then that the modules will always 
remain available.  The puppet_module (reading the source code) does have a 
source parameter.  I assume then you could point it at something other than 
puppetforge.  But it is not documented and I am not sure I can easily 
produce a in house puppet forge that would have the needed filesystem 
layout for puppet_module to work.

Have other people looked into this problem?  What have they come up with?

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.