Re: [Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
On Wed, Nov 30, 2011 at 8:40 AM, Josh Cooper wrote: > On Wed, Nov 30, 2011 at 6:48 AM, Felix Frank > wrote: >> >> Hi, >> >> On 11/01/2011 02:42 PM, Luke Bigum wrote: >> > It also has to do with Puppet's implementation of File resources: it >> > creates in memory Ruby objects for every file and directory it finds >> > recursively, so combine that with the md5 summing and you'll blow out >> > your CPU and memory usage very quickly. I've done something like this >> > in the past: >> >> seeing as this isn't mentioned in this thread yet: >> >> When recursing through directory trees, you most likely want to specify >> >> checksum => "none" >> >> in your file resource. This didn't help in puppet 0.25, but since 2.6 >> I've used it to great benefit. >> >> Still, large-ish trees with lots of (small) files will still take a very >> long time due to the other effect noted above. > > Also if you are running 2.7.0 to 2.7.6, we recently fixed a performance > issue, which you'd see when recursing large numbers of > files: https://projects.puppetlabs.com/issues/9671 > It's fixed in 2.7.7 Or 2.7.8rc series (coming later this week), since 2.7.7 got nixed. > Josh > -- > Josh Cooper > Developer, Puppet Labs > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
On Wed, Nov 30, 2011 at 6:48 AM, Felix Frank < felix.fr...@alumni.tu-berlin.de> wrote: > Hi, > > On 11/01/2011 02:42 PM, Luke Bigum wrote: > > It also has to do with Puppet's implementation of File resources: it > > creates in memory Ruby objects for every file and directory it finds > > recursively, so combine that with the md5 summing and you'll blow out > > your CPU and memory usage very quickly. I've done something like this > > in the past: > > seeing as this isn't mentioned in this thread yet: > > When recursing through directory trees, you most likely want to specify > > checksum => "none" > > in your file resource. This didn't help in puppet 0.25, but since 2.6 > I've used it to great benefit. > > Still, large-ish trees with lots of (small) files will still take a very > long time due to the other effect noted above. > Also if you are running 2.7.0 to 2.7.6, we recently fixed a performance issue, which you'd see when recursing large numbers of files: https://projects.puppetlabs.com/issues/9671 It's fixed in 2.7.7 Josh -- Josh Cooper Developer, Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
Hi, On 11/01/2011 02:42 PM, Luke Bigum wrote: > It also has to do with Puppet's implementation of File resources: it > creates in memory Ruby objects for every file and directory it finds > recursively, so combine that with the md5 summing and you'll blow out > your CPU and memory usage very quickly. I've done something like this > in the past: seeing as this isn't mentioned in this thread yet: When recursing through directory trees, you most likely want to specify checksum => "none" in your file resource. This didn't help in puppet 0.25, but since 2.6 I've used it to great benefit. Still, large-ish trees with lots of (small) files will still take a very long time due to the other effect noted above. Cheers, Felix -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
On Nov 4, 9:42 pm, Christopher Wood
wrote:
> I think you still want chown -R here:
>
> command => "/bin/chown -R jetty:users ${jetty_install_dir}",
Of course I do, I fat-fingered it. That's why it wasn't working.
Thanks for that (also, thanks to Stefan for the advice about
refreshonly=>true.)
Cheers, Robert.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
On Thu, Nov 03, 2011 at 10:39:29PM -0700, Robert Atkins wrote:
> On Nov 2, 7:33 pm, Christopher Wood
> wrote:
> > It's worth mentioning that find -exec this way forks a separate copy of
> > find for each file. You'll notice how much slower this is on a really large
> > set of files (possibly larger than yours). If you have a recursion-capable
> > chown it's quicker to use chown -R.
>
> Noted, this is what I'm using now:
>
> exec { "enforce ${jetty_install_dir} permissions":
> command => "/bin/chown jetty:users ${jetty_install_dir}",
> onlyif => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} ! -
> user jetty -o ! -group users | wc -l) -gt 0",
> subscribe => [File["/opt/jetty/bin/jetty.sh"], File["/opt/jetty/
> resources/log4j.xml"]],
> refreshonly => true,
> }
>
> I can see it execute ("notice: /Stage[main]//Exec[enforce /opt/
> jetty-6.1.26 permissions]: Triggered 'refresh' from 1 events") but the
> new files are still left with the wrong ownership. When I run the
> "onlyif" command manually and echo $? I get 0, which says it should
> run the chown. Can anyone see what I'm missing?
>
> Cheers, Robert.
>
I don't see the need to specify refreshonly => true because you already
have a propert onlyif statement. Refreshonly means that the command is
ONLY run when the exec resource receives a refresh event. A refresh event
is triggered when the subscribed resource has changed or a changed
resource has set the notify metaparamter.
So in your case the refreshonly renders your onlyif useless; the chown
command is only run, when /opt/jetty/bin/jetty.sh or
/opt/jetty/resources/log4j.xml changes.
-Stefan
pgpw7TQgzyTfF.pgp
Description: PGP signature
Re: [Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
On Thu, Nov 03, 2011 at 10:39:29PM -0700, Robert Atkins wrote:
> On Nov 2, 7:33 pm, Christopher Wood
> wrote:
> > It's worth mentioning that find -exec this way forks a separate copy of
> > find for each file. You'll notice how much slower this is on a really large
> > set of files (possibly larger than yours). If you have a recursion-capable
> > chown it's quicker to use chown -R.
>
> Noted, this is what I'm using now:
>
> exec { "enforce ${jetty_install_dir} permissions":
> command => "/bin/chown jetty:users ${jetty_install_dir}",
I think you still want chown -R here:
command => "/bin/chown -R jetty:users ${jetty_install_dir}",
> onlyif => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} ! -
> user jetty -o ! -group users | wc -l) -gt 0",
> subscribe => [File["/opt/jetty/bin/jetty.sh"], File["/opt/jetty/
> resources/log4j.xml"]],
> refreshonly => true,
> }
>
> I can see it execute ("notice: /Stage[main]//Exec[enforce /opt/
> jetty-6.1.26 permissions]: Triggered 'refresh' from 1 events") but the
> new files are still left with the wrong ownership. When I run the
> "onlyif" command manually and echo $? I get 0, which says it should
> run the chown. Can anyone see what I'm missing?
>
> Cheers, Robert.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
On Nov 2, 7:33 pm, Christopher Wood
wrote:
> It's worth mentioning that find -exec this way forks a separate copy of find
> for each file. You'll notice how much slower this is on a really large set of
> files (possibly larger than yours). If you have a recursion-capable chown
> it's quicker to use chown -R.
Noted, this is what I'm using now:
exec { "enforce ${jetty_install_dir} permissions":
command => "/bin/chown jetty:users ${jetty_install_dir}",
onlyif => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} ! -
user jetty -o ! -group users | wc -l) -gt 0",
subscribe => [File["/opt/jetty/bin/jetty.sh"], File["/opt/jetty/
resources/log4j.xml"]],
refreshonly => true,
}
I can see it execute ("notice: /Stage[main]//Exec[enforce /opt/
jetty-6.1.26 permissions]: Triggered 'refresh' from 1 events") but the
new files are still left with the wrong ownership. When I run the
"onlyif" command manually and echo $? I get 0, which says it should
run the chown. Can anyone see what I'm missing?
Cheers, Robert.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
It's worth mentioning that find -exec this way forks a separate copy of find
for each file. You'll notice how much slower this is on a really large set of
files (possibly larger than yours). If you have a recursion-capable chown it's
quicker to use chown -R.
On Wed, Nov 02, 2011 at 01:30:04AM -0700, Robert Atkins wrote:
> Thanks for your solution Luke, this is much faster. For future
> reference here's the modification I made (there were a couple of typos
> and/or I've got a different dialect of find):
>
> exec { "enforce ${jetty_install_dir} permissions":
> command => "/usr/bin/find ${jetty_install_dir} ! -user jetty -o ! -
> group users -exec chown jetty:users {} \\;",
> onlyif => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} ! -
> user jetty -o ! -group users | wc -l) -gt 0",
> }
>
> Cheers, Robert.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
Thanks for your solution Luke, this is much faster. For future
reference here's the modification I made (there were a couple of typos
and/or I've got a different dialect of find):
exec { "enforce ${jetty_install_dir} permissions":
command => "/usr/bin/find ${jetty_install_dir} ! -user jetty -o ! -
group users -exec chown jetty:users {} \\;",
onlyif => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} ! -
user jetty -o ! -group users | wc -l) -gt 0",
}
Cheers, Robert.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
On Nov 1, 9:42 pm, Luke Bigum wrote: > It also has to do with Puppet's implementation of File resources: it > creates in memory Ruby objects for every file and directory it finds [...] >From some other reading I gathered this was what it's doing. The jetty dir is up to half a dozen levels deep with almost 4500 files. I'll use the solution you mentioned below unless there's a better way of doing it. Cheers, Robert. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
It also has to do with Puppet's implementation of File resources: it
creates in memory Ruby objects for every file and directory it finds
recursively, so combine that with the md5 summing and you'll blow out
your CPU and memory usage very quickly. I've done something like this
in the past:
$path="/opt/jetty-6.1.26"
exec { "enforce ${path} permissions":
command => "/usr/bin/find ${path} ! -uid jetty -exec chown jetty {}
\;",
onlyif => "test $(/usr/bin/find ${path} -uid jetty | wc -l) -gt 0",
}
I wouldn't call it elegant, but much faster.
On Nov 1, 1:34 pm, madAndroid wrote:
> how big is the directory structure?
>
> we've had incredibly painful experiences trying to manage directory
> perms/ownerships on large directory trees...
> so much so that we only set the perms on a few of the top level
> directories and left the rest
>
> it's something to do with needing to do an md5 and stat on every file
> in the tree that slows it down
>
> how important is it that the permissions are forced?
> we decided eventually that the file attr wouldn't necessarily change
> unless someone had access to the directory via ssh..
> and only the sys admins do to the server in question anyway
>
> hopefully there's a better way of doing it ... calling all gurus?
>
> cheers,
> Andrew
>
> On Nov 1, 11:15 am, Robert Atkins wrote:
>
>
>
>
>
>
>
> > I've just tried this (we assume /opt/jetty-6.1.26 already exists):
>
> > file { "/opt/jetty-6.1.26":
> > owner => "jetty",
> > group => "users",
> > recurse => true,
>
> > }
>
> > ... but it's taking an *age*. What's the Right Way?
>
> > Cheers, Robert.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: What's the canonical way to enforce permissions/ownership on a directory subtree?
how big is the directory structure?
we've had incredibly painful experiences trying to manage directory
perms/ownerships on large directory trees...
so much so that we only set the perms on a few of the top level
directories and left the rest
it's something to do with needing to do an md5 and stat on every file
in the tree that slows it down
how important is it that the permissions are forced?
we decided eventually that the file attr wouldn't necessarily change
unless someone had access to the directory via ssh..
and only the sys admins do to the server in question anyway
hopefully there's a better way of doing it ... calling all gurus?
cheers,
Andrew
On Nov 1, 11:15 am, Robert Atkins wrote:
> I've just tried this (we assume /opt/jetty-6.1.26 already exists):
>
> file { "/opt/jetty-6.1.26":
> owner => "jetty",
> group => "users",
> recurse => true,
>
> }
>
> ... but it's taking an *age*. What's the Right Way?
>
> Cheers, Robert.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
