Re: [Puppet Users] Re: workarounds for ruby segfaults on puppet master
In regard to: [Puppet Users] Re: workarounds for ruby segfaults on puppet...: Since RHEL 6.x has alternate versions of some packages (including ruby) available via its Software Collections Library (SCL), I'm tempted to try switching our puppet master to use the ruby193-* packages from SCL. A minor downside is that I won't be able to use the Puppet Labs packages anymore, at least on the master. Hi Tim, why is it that you wouldn't be able to use the packages on the master? I think you should be able to point your apache 'PassengerRuby' directive at the SCL ruby and be good to go. As John Julien already responded, unfortunately that won't work. When you use an updated package from the Software Collections Library, it's essentially installed under a separate installation root, outside of the /usr space. The Puppet Labs packages install their bits to e.g. /usr/lib/ruby/site_ruby/1.8/ but with the SCL packages, that directory isn't going to be anything that the SCL ruby would search. It's going to instead look for things installed in various places under its root, which is /opt/rh/ruby193/root/usr I don't know if the package root directory is different for CentOS, Scientific Linux, or other RHEL derivatives. Another alternative that I'd recommend is to use the new puppetserver package, which runs the master under JRuby and replaces the whole Apache+Passenger+MRI ruby part of the stack. We just got to 3.7.3, after running with 3.4.2 for about 9 months. I haven't yet had a chance to enable the future parser with our traditional stack, so I don't think we're quite ready to bite the bullet and go to puppetserver. It's in our future, to be sure, but it has been painful enough just getting the traditional stack upgraded. Thanks for your thoughts on this! Very much appreciated. Tim -- Tim Mooney tim.moo...@ndsu.edu Enterprise Computing Infrastructure 701-231-1076 (Voice) Room 242-J6, Quentin Burdick Building 701-231-8541 (Fax) North Dakota State University, Fargo, ND 58105-5164
Re: [Puppet Users] Re: workarounds for ruby segfaults on puppet master
In regard to: [Puppet Users] Re: workarounds for ruby segfaults on puppet...: FWIW here's what I put in /etc/puppet/rack/config.ru that has resolved it for me: Thanks Trey! It was your original post about the issue that got me on a track that eventually got me a workaround. We're now on 3.7.3, and so far (fingers crossed) John's suggestion of just adding --profile, without any of the other options, has allowed us to avoid the problem. ARGV --confdir /etc/puppet ARGV --vardir /var/lib/puppet *ARGV --debug* *ARGV --trace* *ARGV --profile* ARGV --logdest /var/log/puppet/puppetmaster.log If I remove the lines the segfaults become a problem. I'm now on Puppet 3.6.2 and this is still an issue that requires the above work around. Just be sure if you use that work-around to update logrotate to clean out puppetmaster.log as that file will get very large very quickly. The --logdest portion I used to keep the logs out of syslog and so they could be cleaned up more easily using logrotate. Yeah, I was worried about log growth when using those options, but just --profile hasn't had any impact on our logs. If it turns out that we start having the segfaults again, I'll pursue your recommendation and add the remaining options that you're using. Thanks again! Tim On Wednesday, November 19, 2014 11:02:00 AM UTC-6, Tim.Mooney wrote: All- For those of you that are using puppet on RHEL 6.x (/CentOS/Oracle Linux/Scientific Linux/etc.) and have experienced ruby segfaults on your puppet master(s), what workaround or workarounds have you been using? We have been using puppet 3.4.2 (from Puppet Labs repos) for some time, with a RHEL 6.x puppetmaster under mod_passenger. RHEL 6.x currently has ruby 1.8.7 patchlevel 374 as its default ruby version. In the past couple weeks we've started to see a couple of different clients that are triggering segfaults in ruby on the master during a puppet agent run. Examples include: /usr/lib/ruby/site_ruby/1.8/puppet/util/profiler.rb:30: [BUG] Segmentation fault ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux] /usr/lib/ruby/site_ruby/1.8/puppet/parser/type_loader.rb:110: [BUG] Segmentation fault ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux] Web searches related to this issue turned up a thread from puppet-users earlier this year started by treydock: https://groups.google.com/forum/#!topic/puppet-users/qWN6j-eNiZ0 Unfortunately, I've tried a lot of the workarounds suggested in that thread, and none of them seem to reliably avoid the problem. - I tried back-porting the small patch from PUP-1592 to our 3.4.2 puppet master. No luck. - Yesterday, I bit the bullet and upgraded our entire puppet infrastructure from 3.4.2 to 3.7.3. We still see the same segfaults on the master, both under mod_passenger and when running the master in standalone mode for testing. Since RHEL 6.x has alternate versions of some packages (including ruby) available via its Software Collections Library (SCL), I'm tempted to try switching our puppet master to use the ruby193-* packages from SCL. A minor downside is that I won't be able to use the Puppet Labs packages anymore, at least on the master. The big concern I have relates to how advisable it is to use a different version of ruby on the master vs. all of the clients? Have other RHEL users tried this, with any success? Thanks, Tim -- Tim Mooney tim.m...@ndsu.edu javascript: Enterprise Computing Infrastructure 701-231-1076 (Voice) Room 242-J6, Quentin Burdick Building 701-231-8541 (Fax) North Dakota State University, Fargo, ND 58105-5164 -- Tim Mooney tim.moo...@ndsu.edu Enterprise Computing Infrastructure 701-231-1076 (Voice) Room 242-J6, Quentin Burdick Building 701-231-8541 (Fax) North Dakota State University, Fargo, ND 58105-5164
[Puppet Users] Re: workarounds for ruby segfaults on puppet master
On Wednesday, November 19, 2014 9:02:00 AM UTC-8, Tim.Mooney wrote: Since RHEL 6.x has alternate versions of some packages (including ruby) available via its Software Collections Library (SCL), I'm tempted to try switching our puppet master to use the ruby193-* packages from SCL. A minor downside is that I won't be able to use the Puppet Labs packages anymore, at least on the master. Hi Tim, why is it that you wouldn't be able to use the packages on the master? I think you should be able to point your apache 'PassengerRuby' directive at the SCL ruby and be good to go. Another alternative that I'd recommend is to use the new puppetserver package, which runs the master under JRuby and replaces the whole Apache+Passenger+MRI ruby part of the stack. The big concern I have relates to how advisable it is to use a different version of ruby on the master vs. all of the clients? Have other RHEL users tried this, with any success? It's generally fine as the execution paths are quite different for agent code vs master. That's why it's been possible to move to JRuby under Puppet Server and leave the agents as they are on MRI. --eric0 -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/696d901f-b007-481c-b082-127bfca8e78c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: workarounds for ruby segfaults on puppet master
On Nov 21, 2014, at 3:04 PM, Eric Sorenson eric.soren...@puppetlabs.com wrote: On Wednesday, November 19, 2014 9:02:00 AM UTC-8, Tim.Mooney wrote: Since RHEL 6.x has alternate versions of some packages (including ruby) available via its Software Collections Library (SCL), I'm tempted to try switching our puppet master to use the ruby193-* packages from SCL. A minor downside is that I won't be able to use the Puppet Labs packages anymore, at least on the master. Hi Tim, why is it that you wouldn't be able to use the packages on the master? I think you should be able to point your apache 'PassengerRuby' directive at the SCL ruby and be good to go. If you point Passenger at the SCL ruby (1.9.3) it will not be able to find Puppet, since Puppet is installed in the system ruby path (1.8.7). -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1FB60192-0728-4D79-B98D-2649017261BC%40julienfamily.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: workarounds for ruby segfaults on puppet master
FWIW here's what I put in /etc/puppet/rack/config.ru that has resolved it for me: ARGV --confdir /etc/puppet ARGV --vardir /var/lib/puppet *ARGV --debug* *ARGV --trace* *ARGV --profile* ARGV --logdest /var/log/puppet/puppetmaster.log If I remove the lines the segfaults become a problem. I'm now on Puppet 3.6.2 and this is still an issue that requires the above work around. I have a second Puppetmaster that's on EL7 and I don't have to add this workaround thankfully. They serve the same clients and use the same modules as the masters are load balanced using round-robin DNS, so it's definitely an EL6 issue. Just be sure if you use that work-around to update logrotate to clean out puppetmaster.log as that file will get very large very quickly. The --logdest portion I used to keep the logs out of syslog and so they could be cleaned up more easily using logrotate. - Trey On Wednesday, November 19, 2014 11:02:00 AM UTC-6, Tim.Mooney wrote: All- For those of you that are using puppet on RHEL 6.x (/CentOS/Oracle Linux/Scientific Linux/etc.) and have experienced ruby segfaults on your puppet master(s), what workaround or workarounds have you been using? We have been using puppet 3.4.2 (from Puppet Labs repos) for some time, with a RHEL 6.x puppetmaster under mod_passenger. RHEL 6.x currently has ruby 1.8.7 patchlevel 374 as its default ruby version. In the past couple weeks we've started to see a couple of different clients that are triggering segfaults in ruby on the master during a puppet agent run. Examples include: /usr/lib/ruby/site_ruby/1.8/puppet/util/profiler.rb:30: [BUG] Segmentation fault ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux] /usr/lib/ruby/site_ruby/1.8/puppet/parser/type_loader.rb:110: [BUG] Segmentation fault ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux] Web searches related to this issue turned up a thread from puppet-users earlier this year started by treydock: https://groups.google.com/forum/#!topic/puppet-users/qWN6j-eNiZ0 Unfortunately, I've tried a lot of the workarounds suggested in that thread, and none of them seem to reliably avoid the problem. - I tried back-porting the small patch from PUP-1592 to our 3.4.2 puppet master. No luck. - Yesterday, I bit the bullet and upgraded our entire puppet infrastructure from 3.4.2 to 3.7.3. We still see the same segfaults on the master, both under mod_passenger and when running the master in standalone mode for testing. Since RHEL 6.x has alternate versions of some packages (including ruby) available via its Software Collections Library (SCL), I'm tempted to try switching our puppet master to use the ruby193-* packages from SCL. A minor downside is that I won't be able to use the Puppet Labs packages anymore, at least on the master. The big concern I have relates to how advisable it is to use a different version of ruby on the master vs. all of the clients? Have other RHEL users tried this, with any success? Thanks, Tim -- Tim Mooney tim.m...@ndsu.edu javascript: Enterprise Computing Infrastructure 701-231-1076 (Voice) Room 242-J6, Quentin Burdick Building 701-231-8541 (Fax) North Dakota State University, Fargo, ND 58105-5164 -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c7c531ce-3598-471f-befa-51867c3abf89%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: workarounds for ruby segfaults on puppet master
On Wednesday, November 19, 2014 11:02:00 AM UTC-6, Tim.Mooney wrote: All- For those of you that are using puppet on RHEL 6.x (/CentOS/Oracle Linux/Scientific Linux/etc.) and have experienced ruby segfaults on your puppet master(s), what workaround or workarounds have you been using? I found that adding the following to config.ru stops the problem. ARGV --profile I am presuming this is because the --profile option is incrementing the reference count to all the objects in memory for a single catalog compile and prevents the ruby GC bug from thinking something is no longer needed. I saw this problem only on our large catalogs, I presume that was because the large memory requirement of the catalog triggered the ruby GC. Another option that will be crazy memory intensive is to disable GC in config.ru GC.disable This worked fine on our test machines, but the only way to make it sustainable in production is to set a really low PassengerMaxRequests. This wasn't a good option for us in production either, as it removed the performance gains of hiera file caching. If you don't have a lot of parameterized classes or your Puppet Masters have a lot of capacity, disabling GC and lowering PassengerMaxRequests might be an option, but I still think --profile is the better way to go. I found a bugzilla on the GC bug, don't have the number handy, but it basically said that GC in ruby 1.8.7 has a flawed design and this can't be fixed. So the only real way to avoid all the headaches this bug can cause is repackage to ruby193 or upgrade to Red Hat 7. Hope this helps, John -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/dd475be7-9845-4d2a-8dcd-c0b049c7dee5%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Re: workarounds for ruby segfaults on puppet master
In regard to: [Puppet Users] Re: workarounds for ruby segfaults on puppet...: For those of you that are using puppet on RHEL 6.x (/CentOS/Oracle Linux/Scientific Linux/etc.) and have experienced ruby segfaults on your puppet master(s), what workaround or workarounds have you been using? I found that adding the following to config.ru stops the problem. ARGV --profile I am presuming this is because the --profile option is incrementing the reference count to all the objects in memory for a single catalog compile and prevents the ruby GC bug from thinking something is no longer needed. Thanks John! I had seen that + other options, but I was pretty concerned about running our production master with the amount of logging it would generate. I'll give just --profile a try and see if the situation improves. Another option that will be crazy memory intensive is to disable GC in config.ru GC.disable This worked fine on our test machines, but the only way to make it sustainable in production is to set a really low PassengerMaxRequests. This wasn't a good option for us in production either, as it removed the performance gains of hiera file caching. If you don't have a lot of parameterized classes or your Puppet Masters have a lot of capacity, disabling GC and lowering PassengerMaxRequests might be an option, but I still think --profile is the better way to go. We don't have a lot of parameterized classes, but their use is definitely growing, especially as we start making more use of the best forge modules. GC.disable is a good fall-back idea for us, if the --profile option doesn't completely fix it. I found a bugzilla on the GC bug, don't have the number handy, but it basically said that GC in ruby 1.8.7 has a flawed design and this can't be fixed. So the only real way to avoid all the headaches this bug can cause is repackage to ruby193 or upgrade to Red Hat 7. Yeah, I had seen that one too: https://bugzilla.redhat.com/show_bug.cgi?id=843199 It was the it's so broken it can't be fixed that caused me to ask here. You have been extremely helpful! Thanks so much for the information you've provided! It very likely has saved me a lot of work. Tim -- Tim Mooney tim.moo...@ndsu.edu Enterprise Computing Infrastructure 701-231-1076 (Voice) Room 242-J6, Quentin Burdick Building 701-231-8541 (Fax) North Dakota State University, Fargo, ND 58105-5164
