[Puppet Users] Puppet performance on Ubuntu Lucid with mod_passanger

2010-09-24 Thread jtournier 
Hi,

i'm actually using puppet+mongrel for 630 servers. Performances are
correct, but we need to reboot the puppetmaster every 3 hours because
of memory problems :-(

I decided to migrate to a configuration puppet+apache2+passanger on a
Ubuntu 10.04 system.

During my tests, everything was working correctly but when i moved in
production with the 630 servers, performances was very bad: most of
the clients could not retreived their configurations (with an error
message err: Configuration retrieval timed out) and the puppetmaster
has a high load (load=~20 with 10 puppetmaster launched, load=4 with 1
puppetmaster launched).

I check everywhere what could be wrong on my system without finding
something :-(
Maybe one here may have a clue ?

Thanks !!


My configuration:
OS: ubuntu 10.04
puppetmaster0.25.4-2ubuntu6
libapache2-mod-passenger2.2.7debian-1

My installation procedure (maybe could help) :
   $ apt-get install puppetmaster
   $ update-rc.d -f puppetmaster remove
   $ perl -i -pe's...@^start=yes@START=no@' /etc/default/puppetmaster
   $ mkdir /etc/puppet/run
   $ chown puppet:root /etc/puppet/run
   $ apt-get install apache2 libapache2-mod-passenger rails librack-
ruby
   $ a2enmod ssl
   $ a2enmod headers
   = configure Apache and passanger
   $ cat  /etc/apache2/mods-available/passenger.conf  EOF
IfModule mod_passenger.c
PassengerRoot /usr
PassengerRuby /usr/bin/ruby
PassengerHighPerformance on
PassengerMaxPoolSize 10
PassengerPoolIdleTime 300
PassengerUseGlobalQueue on
PassengerStatThrottleRate 120
RackAutoDetect Off
RackBaseURI /
RailsAutoDetect Off
/IfModule
EOF
   $ cat  /etc/apache2/sites-available/puppetmasterd  EOF
# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 1
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off

Listen 8140
VirtualHost *:8140
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/
frcrbvesrv0029.ymagis.net.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/
frcrbvesrv0029.ymagis.net.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
# CRL checking should be enabled; if you have problems with
Apache complaining about the CRL,
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars

ErrorLog /var/log/apache2/puppetmaster-error.log
CustomLog /var/log/apache2/puppetmaster-access.log combined
CustomLog /var/log/apache2/puppetmaster-ssl_request.log %t %h
%{SSL_PROTOCOL}x %{SSL_CIPHER}x \%r\ %b

DocumentRoot /etc/puppet/rack/public
RackBaseURI /
Directory /etc/puppet/rack/
Options None
AllowOverride None
Order allow,deny
allow from all
/Directory
/VirtualHost
EOF
   $ a2ensite puppetmasterd
   mkdir -p /etc/puppet/rack/public
   # ex: http://www.puppetlabs.com/downloads/puppet/puppet-0.25.0beta1.tar.gz
   # see the file ext/rack/README
   cat  /etc/puppet/rack/config.ru  EOF
# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
# $:.push('/opt/puppet/lib')

$0 = puppetmasterd
require 'puppet'

# if you want debugging:
# ARGV  --debug

ARGV  --rack
require 'puppet/application/puppetmasterd'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Application[:puppetmasterd].run
EOF
   chown puppet:root /etc/puppet/rack/config.ru

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Puppet performance on Ubuntu Lucid with mod_passanger

2010-09-24 Thread Patrick 

On Sep 24, 2010, at 5:03 AM, jtournier wrote:

 Hi,
 
 i'm actually using puppet+mongrel for 630 servers. Performances are
 correct, but we need to reboot the puppetmaster every 3 hours because
 of memory problems :-(
 
 I decided to migrate to a configuration puppet+apache2+passanger on a
 Ubuntu 10.04 system.
 
 During my tests, everything was working correctly but when i moved in
 production with the 630 servers, performances was very bad: most of
 the clients could not retreived their configurations (with an error
 message err: Configuration retrieval timed out) and the puppetmaster
 has a high load (load=~20 with 10 puppetmaster launched, load=4 with 1
 puppetmaster launched).
 
 I check everywhere what could be wrong on my system without finding
 something :-(
 Maybe one here may have a clue ?

Any chance you ran out of RAM?  This happened to me and the swapping killed my 
performance.  Webrick only runs one instance and you're now running 10 
instances which will use up much more RAM.

Also, are you using storedconfigs with mysql-lite?  That can cause strange 
things too.

 Thanks !!
 
 
 My configuration:
 OS: ubuntu 10.04
 puppetmaster0.25.4-2ubuntu6
 libapache2-mod-passenger2.2.7debian-1
 
 My installation procedure (maybe could help) :
   $ apt-get install puppetmaster
   $ update-rc.d -f puppetmaster remove
   $ perl -i -pe's...@^start=yes@START=no@' /etc/default/puppetmaster
   $ mkdir /etc/puppet/run
   $ chown puppet:root /etc/puppet/run
   $ apt-get install apache2 libapache2-mod-passenger rails librack-
 ruby
   $ a2enmod ssl
   $ a2enmod headers
   = configure Apache and passanger
   $ cat  /etc/apache2/mods-available/passenger.conf  EOF
 IfModule mod_passenger.c
PassengerRoot /usr
PassengerRuby /usr/bin/ruby
PassengerHighPerformance on
PassengerMaxPoolSize 10
PassengerPoolIdleTime 300
PassengerUseGlobalQueue on
PassengerStatThrottleRate 120
RackAutoDetect Off
RackBaseURI /
RailsAutoDetect Off
 /IfModule
 EOF
   $ cat  /etc/apache2/sites-available/puppetmasterd  EOF
 # you probably want to tune these settings
 PassengerHighPerformance on
 PassengerMaxPoolSize 1
 PassengerPoolIdleTime 1500
 # PassengerMaxRequests 1000
 PassengerStatThrottleRate 120
 RackAutoDetect Off
 RailsAutoDetect Off
 
 Listen 8140
 VirtualHost *:8140
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/
 frcrbvesrv0029.ymagis.net.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/
 frcrbvesrv0029.ymagis.net.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
# CRL checking should be enabled; if you have problems with
 Apache complaining about the CRL,
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars
 
ErrorLog /var/log/apache2/puppetmaster-error.log
CustomLog /var/log/apache2/puppetmaster-access.log combined
CustomLog /var/log/apache2/puppetmaster-ssl_request.log %t %h
 %{SSL_PROTOCOL}x %{SSL_CIPHER}x \%r\ %b
 
DocumentRoot /etc/puppet/rack/public
RackBaseURI /
Directory /etc/puppet/rack/
Options None
AllowOverride None
Order allow,deny
allow from all
/Directory
 /VirtualHost
 EOF
   $ a2ensite puppetmasterd
   mkdir -p /etc/puppet/rack/public
   # ex: http://www.puppetlabs.com/downloads/puppet/puppet-0.25.0beta1.tar.gz
   # see the file ext/rack/README
   cat  /etc/puppet/rack/config.ru  EOF
 # a config.ru, for use with every rack-compatible webserver.
 # SSL needs to be handled outside this, though.
 
 # if puppet is not in your RUBYLIB:
 # $:.push('/opt/puppet/lib')
 
 $0 = puppetmasterd
 require 'puppet'
 
 # if you want debugging:
 # ARGV  --debug
 
 ARGV  --rack
 require 'puppet/application/puppetmasterd'
 # we're usually running inside a Rack::Builder.new {} block,
 # therefore we need to call run *here*.
 run Puppet::Application[:puppetmasterd].run
 EOF
   chown puppet:root /etc/puppet/rack/config.ru
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 Puppet Users group.
 To post to this group, send email to puppet-us...@googlegroups.com.
 To unsubscribe from this group, send email to 
 puppet-users+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/puppet-users?hl=en.
 

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at

Re: [Puppet Users] Puppet performance on Ubuntu Lucid with mod_passanger

2010-09-24 Thread Jerome Tournier 
On Fri, Sep 24, 2010 at 6:44 PM, Patrick kc7...@gmail.com wrote:

 Any chance you ran out of RAM?  This happened to me and the swapping killed
 my performance.  Webrick only runs one instance and you're now running 10
 instances which will use up much more RAM.


No, not a RAM related problem:
$ free -m
  total   used   free sharedbuffers
cached
Mem:  2013332   1680  0 67142
-/+ buffers/cache:123   1889
Swap: 1303  0   1303



 Also, are you using storedconfigs with mysql-lite?  That can cause strange
 things too.


I'm not using storedconfigs.

puppetmasterd can compile the catalogs always in a time  0.5s. Everything
is looks ok on this side.
But the puppetd client need between 5 and 20 minutes to finish. Maybe the
problem is related to Apache2, but i can't see where :-(

-- 
Jérôme





  Thanks !!
 
 
  My configuration:
  OS: ubuntu 10.04
  puppetmaster0.25.4-2ubuntu6
  libapache2-mod-passenger2.2.7debian-1
 
  My installation procedure (maybe could help) :
$ apt-get install puppetmaster
$ update-rc.d -f puppetmaster remove
$ perl -i -pe's...@^start=yes@START=no@' /etc/default/puppetmaster
$ mkdir /etc/puppet/run
$ chown puppet:root /etc/puppet/run
$ apt-get install apache2 libapache2-mod-passenger rails librack-
  ruby
$ a2enmod ssl
$ a2enmod headers
= configure Apache and passanger
$ cat  /etc/apache2/mods-available/passenger.conf  EOF
  IfModule mod_passenger.c
 PassengerRoot /usr
 PassengerRuby /usr/bin/ruby
 PassengerHighPerformance on
 PassengerMaxPoolSize 10
 PassengerPoolIdleTime 300
 PassengerUseGlobalQueue on
 PassengerStatThrottleRate 120
 RackAutoDetect Off
 RackBaseURI /
 RailsAutoDetect Off
  /IfModule
  EOF
$ cat  /etc/apache2/sites-available/puppetmasterd  EOF
  # you probably want to tune these settings
  PassengerHighPerformance on
  PassengerMaxPoolSize 1
  PassengerPoolIdleTime 1500
  # PassengerMaxRequests 1000
  PassengerStatThrottleRate 120
  RackAutoDetect Off
  RailsAutoDetect Off
 
  Listen 8140
  VirtualHost *:8140
 SSLEngine on
 SSLProtocol -ALL +SSLv3 +TLSv1
 SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
 SSLCertificateFile /var/lib/puppet/ssl/certs/
  frcrbvesrv0029.ymagis.net.pem
 SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/
  frcrbvesrv0029.ymagis.net.pem
 SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
 SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
 # CRL checking should be enabled; if you have problems with
  Apache complaining about the CRL,
 SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
 SSLVerifyClient optional
 SSLVerifyDepth 1
 SSLOptions +StdEnvVars
 
 ErrorLog /var/log/apache2/puppetmaster-error.log
 CustomLog /var/log/apache2/puppetmaster-access.log combined
 CustomLog /var/log/apache2/puppetmaster-ssl_request.log %t %h
  %{SSL_PROTOCOL}x %{SSL_CIPHER}x \%r\ %b
 
 DocumentRoot /etc/puppet/rack/public
 RackBaseURI /
 Directory /etc/puppet/rack/
 Options None
 AllowOverride None
 Order allow,deny
 allow from all
 /Directory
  /VirtualHost
  EOF
$ a2ensite puppetmasterd
mkdir -p /etc/puppet/rack/public
# ex:
 http://www.puppetlabs.com/downloads/puppet/puppet-0.25.0beta1.tar.gz
# see the file ext/rack/README
cat  /etc/puppet/rack/config.ru  EOF
  # a config.ru, for use with every rack-compatible webserver.
  # SSL needs to be handled outside this, though.
 
  # if puppet is not in your RUBYLIB:
  # $:.push('/opt/puppet/lib')
 
  $0 = puppetmasterd
  require 'puppet'
 
  # if you want debugging:
  # ARGV  --debug
 
  ARGV  --rack
  require 'puppet/application/puppetmasterd'
  # we're usually running inside a Rack::Builder.new {} block,
  # therefore we need to call run *here*.
  run Puppet::Application[:puppetmasterd].run
  EOF
chown puppet:root /etc/puppet/rack/config.ru
 
  --
  You received this message because you are subscribed to the Google Groups
 Puppet Users group.
  To post to this group, send email to puppet-us...@googlegroups.com.
  To unsubscribe from this group, send email to
 puppet-users+unsubscr...@googlegroups.compuppet-users%2bunsubscr...@googlegroups.com
 .
  For more options, visit this group at
 http://groups.google.com/group/puppet-users?hl=en.
 

 --
 You received this message because you are subscribed to the Google Groups
 Puppet Users group.
 To post to this group, send email to puppet-us...@googlegroups.com.
 To unsubscribe from this group, send email to
 puppet-users+unsubscr...@googlegroups.compuppet-users%2bunsubscr...@googlegroups.com
 .
 For more options, visit this group at