Re: [Puppet Users] Re: workarounds for ruby segfaults on puppet master

2014-11-25 Thread Tim Mooney

In regard to: [Puppet Users] Re: workarounds for ruby segfaults on puppet...:


Since RHEL 6.x has alternate versions of some packages (including ruby)
available via its Software Collections Library (SCL), I'm tempted to
try switching our puppet master to use the ruby193-* packages from
SCL.  A minor downside is that I won't be able to use the Puppet Labs
packages
anymore, at least on the master.



Hi Tim, why is it that you wouldn't be able to use the packages on the
master?

I think you should be able to point your apache 'PassengerRuby' directive
at the SCL ruby and be good to go.


As John Julien already responded, unfortunately that won't work.

When you use an updated package from the Software Collections Library,
it's essentially installed under a separate installation root, outside
of the /usr space.  The Puppet Labs packages install their bits to e.g.

/usr/lib/ruby/site_ruby/1.8/

but with the SCL packages, that directory isn't going to be anything that
the SCL ruby would search.  It's going to instead look for things
installed in various places under its root, which is

/opt/rh/ruby193/root/usr

I don't know if the package root directory is different for CentOS,
Scientific Linux, or other RHEL derivatives.


Another alternative that I'd recommend is to use the new puppetserver
package, which runs the master under JRuby and replaces the whole
Apache+Passenger+MRI ruby part of the stack.


We just got to 3.7.3, after running with 3.4.2 for about 9 months.  I
haven't yet had a chance to enable the future parser with our traditional
stack, so I don't think we're quite ready to bite the bullet and go to
puppetserver.  It's in our future, to be sure, but it has been painful
enough just getting the traditional stack upgraded.

Thanks for your thoughts on this!  Very much appreciated.

Tim
--
Tim Mooney tim.moo...@ndsu.edu
Enterprise Computing  Infrastructure  701-231-1076 (Voice)
Room 242-J6, Quentin Burdick Building  701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164


Re: [Puppet Users] Re: workarounds for ruby segfaults on puppet master

2014-11-25 Thread Tim Mooney

In regard to: [Puppet Users] Re: workarounds for ruby segfaults on puppet...:


FWIW here's what I put in /etc/puppet/rack/config.ru that has resolved it
for me:


Thanks Trey!  It was your original post about the issue that got me
on a track that eventually got me a workaround.  We're now on 3.7.3,
and so far (fingers crossed) John's suggestion of just adding
--profile, without any of the other options, has allowed us to
avoid the problem.


ARGV  --confdir  /etc/puppet
ARGV  --vardir   /var/lib/puppet
*ARGV  --debug*
*ARGV  --trace*
*ARGV  --profile*
ARGV  --logdest  /var/log/puppet/puppetmaster.log

If I remove the lines the segfaults become a problem.  I'm now on Puppet
3.6.2 and this is still an issue that requires the above work around.




Just be sure if you use that work-around to update logrotate to clean out
puppetmaster.log as that file will get very large very quickly.  The
--logdest portion I used to keep the logs out of syslog and so they could
be cleaned up more easily using logrotate.


Yeah, I was worried about log growth when using those options, but just
--profile hasn't had any impact on our logs.  If it turns out that we
start having the segfaults again, I'll pursue your recommendation and
add the remaining options that you're using.

Thanks again!

Tim


On Wednesday, November 19, 2014 11:02:00 AM UTC-6, Tim.Mooney wrote:



All-

For those of you that are using puppet on RHEL 6.x (/CentOS/Oracle
Linux/Scientific Linux/etc.) and have experienced ruby segfaults on
your puppet master(s), what workaround or workarounds have you been
using?

We have been using puppet 3.4.2 (from Puppet Labs repos) for some time,
with a RHEL 6.x puppetmaster under mod_passenger.  RHEL 6.x currently
has ruby 1.8.7 patchlevel 374 as its default ruby version.

In the past couple weeks we've started to see a couple of different
clients that are triggering segfaults in ruby on the master during a
puppet agent run.  Examples include:

/usr/lib/ruby/site_ruby/1.8/puppet/util/profiler.rb:30: [BUG] Segmentation
fault ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]

/usr/lib/ruby/site_ruby/1.8/puppet/parser/type_loader.rb:110: [BUG]
Segmentation fault ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]

Web searches related to this issue turned up a thread from puppet-users
earlier this year started by treydock:

 https://groups.google.com/forum/#!topic/puppet-users/qWN6j-eNiZ0

Unfortunately, I've tried a lot of the workarounds suggested in that
thread, and none of them seem to reliably avoid the problem.

- I tried back-porting the small patch from PUP-1592 to our 3.4.2
   puppet master.  No luck.

- Yesterday, I bit the bullet and upgraded our entire puppet
   infrastructure from 3.4.2 to 3.7.3.  We still see the same
   segfaults on the master, both under mod_passenger and when
   running the master in standalone mode for testing.

Since RHEL 6.x has alternate versions of some packages (including ruby)
available via its Software Collections Library (SCL), I'm tempted to
try switching our puppet master to use the ruby193-* packages from
SCL.  A minor downside is that I won't be able to use the Puppet Labs
packages
anymore, at least on the master.

The big concern I have relates to how advisable it is to use a different
version of ruby on the master vs. all of the clients?  Have other RHEL
users tried this, with any success?

Thanks,

Tim
--
Tim Mooney tim.m...@ndsu.edu
javascript:
Enterprise Computing  Infrastructure  701-231-1076
(Voice)
Room 242-J6, Quentin Burdick Building  701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164






--
Tim Mooney tim.moo...@ndsu.edu
Enterprise Computing  Infrastructure  701-231-1076 (Voice)
Room 242-J6, Quentin Burdick Building  701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164


[Puppet Users] Re: workarounds for ruby segfaults on puppet master

2014-11-21 Thread Eric Sorenson


On Wednesday, November 19, 2014 9:02:00 AM UTC-8, Tim.Mooney wrote:


 Since RHEL 6.x has alternate versions of some packages (including ruby) 
 available via its Software Collections Library (SCL), I'm tempted to 
 try switching our puppet master to use the ruby193-* packages from 
 SCL.  A minor downside is that I won't be able to use the Puppet Labs 
 packages 
 anymore, at least on the master. 


Hi Tim, why is it that you wouldn't be able to use the packages on the 
master?

I think you should be able to point your apache 'PassengerRuby' directive 
at the SCL ruby and be good to go.

Another alternative that I'd recommend is to use the new puppetserver 
package, which runs the master under JRuby and replaces the whole 
Apache+Passenger+MRI ruby part of the stack.
 

 The big concern I have relates to how advisable it is to use a different 
 version of ruby on the master vs. all of the clients?  Have other RHEL 
 users tried this, with any success? 


It's generally fine as the execution paths are quite different for agent 
code vs master.

That's why it's been possible to move to JRuby under Puppet Server and 
leave the agents as they are on MRI.

--eric0

 

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/696d901f-b007-481c-b082-127bfca8e78c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [Puppet Users] Re: workarounds for ruby segfaults on puppet master

2014-11-21 Thread John Julien

On Nov 21, 2014, at 3:04 PM, Eric Sorenson eric.soren...@puppetlabs.com wrote:

 
 
 On Wednesday, November 19, 2014 9:02:00 AM UTC-8, Tim.Mooney wrote:
 
 Since RHEL 6.x has alternate versions of some packages (including ruby) 
 available via its Software Collections Library (SCL), I'm tempted to 
 try switching our puppet master to use the ruby193-* packages from 
 SCL.  A minor downside is that I won't be able to use the Puppet Labs 
 packages 
 anymore, at least on the master. 
 
 
 Hi Tim, why is it that you wouldn't be able to use the packages on the master?
 
 I think you should be able to point your apache 'PassengerRuby' directive at 
 the SCL ruby and be good to go.
 
If you point Passenger at the SCL ruby (1.9.3) it will not be able to find 
Puppet, since Puppet is installed in the system ruby path (1.8.7).

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/1FB60192-0728-4D79-B98D-2649017261BC%40julienfamily.com.
For more options, visit https://groups.google.com/d/optout.


[Puppet Users] Re: workarounds for ruby segfaults on puppet master

2014-11-21 Thread treydock
FWIW here's what I put in /etc/puppet/rack/config.ru that has resolved it 
for me:

ARGV  --confdir  /etc/puppet
ARGV  --vardir   /var/lib/puppet
*ARGV  --debug*
*ARGV  --trace*
*ARGV  --profile*
ARGV  --logdest  /var/log/puppet/puppetmaster.log

If I remove the lines the segfaults become a problem.  I'm now on Puppet 
3.6.2 and this is still an issue that requires the above work around.

I have a second Puppetmaster that's on EL7 and I don't have to add this 
workaround thankfully.  They serve the same clients and use the same 
modules as the masters are load balanced using round-robin DNS, so it's 
definitely an EL6 issue.

Just be sure if you use that work-around to update logrotate to clean out 
puppetmaster.log as that file will get very large very quickly.  The 
--logdest portion I used to keep the logs out of syslog and so they could 
be cleaned up more easily using logrotate.

- Trey

On Wednesday, November 19, 2014 11:02:00 AM UTC-6, Tim.Mooney wrote:


 All- 

 For those of you that are using puppet on RHEL 6.x (/CentOS/Oracle 
 Linux/Scientific Linux/etc.) and have experienced ruby segfaults on 
 your puppet master(s), what workaround or workarounds have you been 
 using? 

 We have been using puppet 3.4.2 (from Puppet Labs repos) for some time, 
 with a RHEL 6.x puppetmaster under mod_passenger.  RHEL 6.x currently 
 has ruby 1.8.7 patchlevel 374 as its default ruby version. 

 In the past couple weeks we've started to see a couple of different 
 clients that are triggering segfaults in ruby on the master during a 
 puppet agent run.  Examples include: 

 /usr/lib/ruby/site_ruby/1.8/puppet/util/profiler.rb:30: [BUG] Segmentation 
 fault ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux] 

 /usr/lib/ruby/site_ruby/1.8/puppet/parser/type_loader.rb:110: [BUG] 
 Segmentation fault ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux] 

 Web searches related to this issue turned up a thread from puppet-users 
 earlier this year started by treydock: 

  https://groups.google.com/forum/#!topic/puppet-users/qWN6j-eNiZ0 

 Unfortunately, I've tried a lot of the workarounds suggested in that 
 thread, and none of them seem to reliably avoid the problem. 

 - I tried back-porting the small patch from PUP-1592 to our 3.4.2 
puppet master.  No luck. 

 - Yesterday, I bit the bullet and upgraded our entire puppet 
infrastructure from 3.4.2 to 3.7.3.  We still see the same 
segfaults on the master, both under mod_passenger and when 
running the master in standalone mode for testing. 

 Since RHEL 6.x has alternate versions of some packages (including ruby) 
 available via its Software Collections Library (SCL), I'm tempted to 
 try switching our puppet master to use the ruby193-* packages from 
 SCL.  A minor downside is that I won't be able to use the Puppet Labs 
 packages 
 anymore, at least on the master. 

 The big concern I have relates to how advisable it is to use a different 
 version of ruby on the master vs. all of the clients?  Have other RHEL 
 users tried this, with any success? 

 Thanks, 

 Tim 
 -- 
 Tim Mooney tim.m...@ndsu.edu 
 javascript: 
 Enterprise Computing  Infrastructure  701-231-1076 
 (Voice) 
 Room 242-J6, Quentin Burdick Building  701-231-8541 (Fax) 
 North Dakota State University, Fargo, ND 58105-5164 


-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/c7c531ce-3598-471f-befa-51867c3abf89%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[Puppet Users] Re: workarounds for ruby segfaults on puppet master

2014-11-20 Thread John Julien


On Wednesday, November 19, 2014 11:02:00 AM UTC-6, Tim.Mooney wrote:


 All- 

 For those of you that are using puppet on RHEL 6.x (/CentOS/Oracle 
 Linux/Scientific Linux/etc.) and have experienced ruby segfaults on 
 your puppet master(s), what workaround or workarounds have you been 
 using? 


I found that adding the following to config.ru stops the problem.
ARGV  --profile
I am presuming this is because the --profile option is incrementing the 
reference count to all the objects in memory for a single catalog compile 
and prevents the ruby GC bug from thinking something is no longer needed.

I saw this problem only on our large catalogs, I presume that was because 
the large memory requirement of the catalog triggered the ruby GC.

Another option that will be crazy memory intensive is to disable GC in 
config.ru
GC.disable

This worked fine on our test machines, but the only way to make it 
sustainable in production is to set a really low PassengerMaxRequests.  
This wasn't a good option for us in production either, as it removed the 
performance gains of hiera file caching.  If you don't have a lot of 
parameterized classes or your Puppet Masters have a lot of capacity, 
disabling GC and lowering PassengerMaxRequests might be an option, but I 
still think --profile is the better way to go.

I found a bugzilla on the GC bug, don't have the number handy, but it 
basically said that GC in ruby 1.8.7 has a flawed design and this can't be 
fixed.  So the only real way to avoid all the headaches this bug can cause 
is repackage to ruby193 or upgrade to Red Hat 7.

Hope this helps,
John


-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/dd475be7-9845-4d2a-8dcd-c0b049c7dee5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [Puppet Users] Re: workarounds for ruby segfaults on puppet master

2014-11-20 Thread Tim Mooney

In regard to: [Puppet Users] Re: workarounds for ruby segfaults on puppet...:


For those of you that are using puppet on RHEL 6.x (/CentOS/Oracle
Linux/Scientific Linux/etc.) and have experienced ruby segfaults on
your puppet master(s), what workaround or workarounds have you been
using?



I found that adding the following to config.ru stops the problem.
ARGV  --profile
I am presuming this is because the --profile option is incrementing the
reference count to all the objects in memory for a single catalog compile
and prevents the ruby GC bug from thinking something is no longer needed.


Thanks John!  I had seen that + other options, but I was pretty concerned
about running our production master with the amount of logging it would
generate.  I'll give just --profile a try and see if the situation
improves.


Another option that will be crazy memory intensive is to disable GC in
config.ru
GC.disable

This worked fine on our test machines, but the only way to make it
sustainable in production is to set a really low PassengerMaxRequests.
This wasn't a good option for us in production either, as it removed the
performance gains of hiera file caching.  If you don't have a lot of
parameterized classes or your Puppet Masters have a lot of capacity,
disabling GC and lowering PassengerMaxRequests might be an option, but I
still think --profile is the better way to go.


We don't have a lot of parameterized classes, but their use is definitely
growing, especially as we start making more use of the best forge modules.
GC.disable is a good fall-back idea for us, if the --profile option
doesn't completely fix it.


I found a bugzilla on the GC bug, don't have the number handy, but it
basically said that GC in ruby 1.8.7 has a flawed design and this can't be
fixed.  So the only real way to avoid all the headaches this bug can cause
is repackage to ruby193 or upgrade to Red Hat 7.


Yeah, I had seen that one too:

https://bugzilla.redhat.com/show_bug.cgi?id=843199

It was the it's so broken it can't be fixed that caused me to ask here.

You have been extremely helpful!  Thanks so much for the information
you've provided!  It very likely has saved me a lot of work.

Tim
--
Tim Mooney tim.moo...@ndsu.edu
Enterprise Computing  Infrastructure  701-231-1076 (Voice)
Room 242-J6, Quentin Burdick Building  701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164