Hi Matthaus, My hostname is misleading. I upgraded at the time of posting to Ubuntu 15.10. Now I have upgraded to 16.04.
I ran the command you suggested instead of service puppetserver start and I don't see OutOfMemory errors process. root@jim-Ubuntu1504:~# ps -aef|grep -i puppet|grep -v grep root 1116 1 0 19:34 ? 00:00:02 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize root 1719 1 0 19:38 ? 00:00:08 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived --config=/etc/puppetlabs/mcollective/server.cfg --pidfile=/var/run/puppetlabs/mcollective.pid --daemonize root 2644 1544 0 22:35 pts/0 00:00:00 bash /opt/puppetlabs/server/apps/puppetserver/cli/apps/foreground root 2650 2644 0 22:35 pts/0 00:00:00 runuser puppet -s /bin/bash -c /usr/bin/java -Xms512m -Xmx512m -XX:MaxPermSize=512m -Dlogappender=STDOUT -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/bootstrap.cfg puppet 2651 2650 55 22:35 ? 00:00:42 /usr/bin/java -Xms512m -Xmx512m -XX:MaxPermSize=512m -Dlogappender=STDOUT -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/bootstrap.cfg root@jim-Ubuntu1504:~# Also when I do see that error as like below when I do Ctrl + C and execute service puppetserver start I get the OutOfMemory error like below. root@jim-Ubuntu1504:~# ps -aef|grep -i puppet|grep -v grep root 1116 1 0 19:34 ? 00:00:02 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize root 1719 1 0 19:38 ? 00:00:08 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived --config=/etc/puppetlabs/mcollective/server.cfg --pidfile=/var/run/puppetlabs/mcollective.pid --daemonize puppet 3386 1 66 22:41 ? 00:00:45 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xms512m -Xmx512m -XX:MaxPermSize=512m -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg root@jim-Ubuntu1504:~# I wonder if I am doing start-up incorrectly ? Either even in this state where it sayOutOfMemoryError my puppetserver still serves all of its agents. The reason I say so is I changed motd file on all agents + installed vim package on agents. I am not sure which is the right method to start the server ? Also instead of running in foreground can I run it in background using nohup ? Regards, Jim On Friday, 27 May 2016 02:46:14 UTC+10, Matthaus Litteken wrote: > > Jim, > For debugging failures on ubuntu 15.04, I would recommend trying > `/opt/puppetlabs/bin/puppetserver foreground`, it may show you the errors > that are causing the service to shutdown. Ubuntu 15.04 is a bit difficult > to debug on, because while it has systemd support, the puppetserver > packages for that platform still ship sysv init scripts. In my experience, > the sysv-systemd integration seems to swallow some startup errors and > messages that would usually go to journalctl on a systemd platform. We > started shipping systemd service files for puppetserver starting in Ubuntu > 16.04. > > HTH > > On Thu, May 26, 2016 at 6:56 AM, Jim <[email protected] <javascript:>> > wrote: > >> Hi Trevor, >> >> No luck. >> >> 2016-05-26 23:46:26,745 INFO [Thread-1] [p.t.internal] Shutting down due >> to JVM shutdown hook. >> 2016-05-26 23:46:26,747 INFO [Thread-1] [p.t.internal] Beginning >> shutdown sequence >> 2016-05-26 23:46:26,761 INFO [clojure-agent-send-pool-1] >> [p.s.j.jruby-puppet-agents] Flush request received; creating new JRuby pool. >> 2016-05-26 23:46:26,765 INFO [clojure-agent-send-pool-1] >> [p.s.j.jruby-puppet-agents] Replacing old JRuby pool with new instance. >> 2016-05-26 23:46:26,765 INFO [clojure-agent-send-pool-1] >> [p.s.j.jruby-puppet-agents] Swapped JRuby pools, beginning cleanup of old >> pool. >> 2016-05-26 23:46:26,770 INFO [clojure-agent-send-pool-1] >> [p.s.j.jruby-puppet-internal] Cleaned up old JRuby instance with id 1. >> 2016-05-26 23:46:26,772 INFO [Thread-1] [p.t.s.w.jetty9-service] >> Shutting down web server(s). >> 2016-05-26 23:46:26,781 INFO [Thread-1] [p.t.s.w.jetty9-core] Shutting >> down web server. >> 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.ServerConnector] >> Stopped ServerConnector@79303966{SSL-HTTP/1.1}{0.0.0.0:8140} >> 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler] >> Stopped o.e.j.s.h.ContextHandler@2b87522b{/,null,UNAVAILABLE} >> 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler] >> Stopped >> o.e.j.s.h.ContextHandler@1f2ffb51{/puppet-admin-api,null,UNAVAILABLE} >> 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler] >> Stopped o.e.j.s.h.ContextHandler@609b2b8a{/puppet,null,UNAVAILABLE} >> 2016-05-26 23:46:26,825 INFO [Thread-1] [o.e.j.s.h.ContextHandler] >> Stopped o.e.j.s.h.ContextHandler@5b1cbf6d{/puppet-ca,null,UNAVAILABLE} >> 2016-05-26 23:46:26,827 INFO [Thread-1] [p.t.s.w.jetty9-core] Web server >> shutdown >> 2016-05-26 23:46:26,837 INFO [Thread-1] [p.t.internal] Finished shutdown >> sequence >> 2016-05-26 23:49:33,878 INFO [main] [o.e.j.u.log] Logging initialized >> @14524ms >> 2016-05-26 23:49:41,413 INFO [async-dispatch-2] >> [p.s.v.versioned-code-service] No code-id-command set for >> versioned-code-service. Code-id will be nil. >> 2016-05-26 23:49:41,414 INFO [async-dispatch-2] >> [p.s.v.versioned-code-service] No code-content-command set for >> versioned-code-service. Attempting to fetch code content will fail. >> 2016-05-26 23:49:41,417 INFO [async-dispatch-2] [p.t.s.w.jetty9-service] >> Initializing web server(s). >> 2016-05-26 23:49:41,457 INFO [async-dispatch-2] >> [p.s.j.jruby-puppet-service] Initializing the JRuby service >> 2016-05-26 23:49:41,457 WARN [async-dispatch-2] >> [p.s.j.jruby-puppet-service] The 'jruby-puppet.use-legacy-auth-conf' >> setting is set to 'true'. Support for the legacy Puppet auth.conf file is >> deprecated and will be removed in a future release. Change this setting to >> 'false' and migrate your authorization rule definitions in the >> /etc/puppetlabs/puppet/auth.conf file to the >> /etc/puppetlabs/puppetserver/conf.d/auth.conf file. >> 2016-05-26 23:49:41,500 INFO [clojure-agent-send-pool-0] >> [p.s.j.jruby-puppet-internal] Creating JRuby instance with id 1. >> 2016-05-26 23:49:58,448 INFO [clojure-agent-send-pool-0] [puppet-server] >> Puppet Puppet settings initialized; run mode: master >> 2016-05-26 23:49:59,624 INFO [clojure-agent-send-pool-0] >> [p.s.j.jruby-puppet-agents] Finished creating JRubyPuppet instance 1 of 1 >> 2016-05-26 23:49:59,636 INFO [async-dispatch-2] >> [p.s.c.puppet-server-config-core] Initializing webserver settings from core >> Puppet >> 2016-05-26 23:49:59,672 INFO [async-dispatch-2] >> [p.p.certificate-authority] CA already initialized for SSL >> 2016-05-26 23:49:59,674 INFO [async-dispatch-2] >> [p.s.c.certificate-authority-service] CA Service adding a ring handler >> 2016-05-26 23:49:59,749 INFO [async-dispatch-2] >> [p.p.certificate-authority] Master already initialized for SSL >> 2016-05-26 23:49:59,750 INFO [async-dispatch-2] [p.s.m.master-service] >> Master Service adding ring handlers >> 2016-05-26 23:49:59,813 INFO [async-dispatch-2] >> [p.s.p.puppet-admin-service] Starting Puppet Admin web app >> 2016-05-26 23:49:59,844 WARN [async-dispatch-2] >> [o.e.j.s.h.ContextHandler] Empty contextPath >> 2016-05-26 23:49:59,849 INFO [async-dispatch-2] [p.t.s.w.jetty9-service] >> Starting web server(s). >> 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] >> webserver config overridden for key 'ssl-cert' >> 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] >> webserver config overridden for key 'ssl-key' >> 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] >> webserver config overridden for key 'ssl-ca-cert' >> 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] >> webserver config overridden for key 'ssl-crl-path' >> 2016-05-26 23:50:00,249 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] >> Starting web server. >> 2016-05-26 23:50:00,260 INFO [async-dispatch-2] [o.e.j.s.Server] >> jetty-9.2.z-SNAPSHOT >> 2016-05-26 23:50:00,360 INFO [async-dispatch-2] >> [o.e.j.s.h.ContextHandler] Started >> o.e.j.s.h.ContextHandler@62f22fd{/puppet-ca,null,AVAILABLE} >> 2016-05-26 23:50:00,360 INFO [async-dispatch-2] >> [o.e.j.s.h.ContextHandler] Started >> o.e.j.s.h.ContextHandler@dc925ec{/puppet,null,AVAILABLE} >> 2016-05-26 23:50:00,360 INFO [async-dispatch-2] >> [o.e.j.s.h.ContextHandler] Started >> o.e.j.s.h.ContextHandler@55c827e3{/puppet-admin-api,null,AVAILABLE} >> 2016-05-26 23:50:00,360 INFO [async-dispatch-2] >> [o.e.j.s.h.ContextHandler] Started >> o.e.j.s.h.ContextHandler@3b82279{/,null,AVAILABLE} >> 2016-05-26 23:50:00,472 INFO [async-dispatch-2] >> [o.e.j.s.ServerConnector] Started ServerConnector@2fcb900d{SSL-HTTP/1.1}{ >> 0.0.0.0:8140} >> 2016-05-26 23:50:00,473 INFO [async-dispatch-2] [o.e.j.s.Server] Started >> @41120ms >> 2016-05-26 23:50:00,486 INFO [async-dispatch-2] [p.s.m.master-service] >> Puppet Server has successfully started and is now ready to handle requests >> 2016-05-26 23:50:00,488 INFO [async-dispatch-2] >> [p.s.l.legacy-routes-service] The legacy routing service has successfully >> started and is now ready to handle requests >> 2016-05-26 23:50:00,602 INFO [clojure-agent-send-off-pool-0] >> [p.d.version-check] Newer version 2.4.0 is available! Visit >> https://github.com/puppetlabs/puppetserver/tree/stable for details. >> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >> service puppetserver status >> ◠puppetserver.service - LSB: puppetserver >> Loaded: loaded (/etc/init.d/puppetserver) >> Active: active (running) since Thu 2016-05-26 23:50:01 AEST; 4min 55s >> ago >> Docs: man:systemd-sysv-generator(8) >> Process: 4397 ExecStart=/etc/init.d/puppetserver start (code=exited, >> status=0/SUCCESS) >> CGroup: /system.slice/puppetserver.service >> └─4403 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p >> -Djava.security.egd=/dev/urandom -Xms512m -Xmx512m -XX:MaxPermSize=512m -cp >> /opt/puppetlabs/server/apps/puppetserver/puppet-ser... >> >> May 26 23:49:19 jim-Ubuntu1504 systemd[1]: Starting LSB: puppetserver... >> May 26 23:50:01 jim-Ubuntu1504 systemd[1]: Started LSB: puppetserver. >> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >> cat /etc/default/puppetserver >> ########################################### >> # Init settings for puppetserver >> ########################################### >> >> # Location of your Java binary (version 7 or higher) >> JAVA_BIN="/usr/bin/java" >> >> # Modify this if you'd like to change the memory allocation, enable JMX, >> etc >> JAVA_ARGS="-Xms512m -Xmx512m -XX:MaxPermSize=512m" >> >> # These normally shouldn't need to be edited if using OS packages >> USER="puppet" >> GROUP="puppet" >> INSTALL_DIR="/opt/puppetlabs/server/apps/puppetserver" >> CONFIG="/etc/puppetlabs/puppetserver/conf.d" >> BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetserver/bootstrap.cfg" >> SERVICE_STOP_RETRIES=60 >> >> # START_TIMEOUT can be set here to alter the default startup timeout in >> # seconds. This is used in System-V style init scripts only, and will >> have no >> # effect in systemd. >> # START_TIMEOUT=120 >> >> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >> >> Cheers >> Jim >> >> On Wednesday, 25 May 2016 19:56:43 UTC+10, Trevor Vaughan wrote: >>> >>> Hi Jim, >>> >>> Your logs should be in /var/log/puppetserver/puppetserver.log. >>> >>> Also, I'm guessing that you need to change the Xmx java option in your >>> puppetserver start script. >>> >>> Trevor >>> >>> On Tue, May 24, 2016 at 8:51 AM, Jim <[email protected]> wrote: >>> >>>> Any idea which log file to look at when I run the service puppetserver >>>> start command? >>>> >>>> >>>> On Tuesday, 24 May 2016 22:44:03 UTC+10, Jim wrote: >>>>> >>>>> Hi Johan, >>>>> >>>>> I did try 512m option not just 256m before posting. and it didn't >>>>> work. The link which you have provided talks basically about >>>>> >>>>> 1. Changing parameters for java options >>>>> 2. max active instances(Its commented out by default) >>>>> 3.num-cpus configuration. I did recursive grep but couldn't find it. >>>>> Puppetlabs says that they ship it anyway with low value so I don't think >>>>> that should be cause or concerb. >>>>> >>>>> I would probably get hold of logs and see if I can find any valuable >>>>> information. >>>>> >>>>> Kind Regards, >>>>> Jim >>>>> >>>>> On Tuesday, 24 May 2016 15:39:05 UTC+10, Johan De Wit wrote: >>>>>> >>>>>> Hi Jim, >>>>>> >>>>>> >>>>>> You should find some answers here : >>>>>> https://docs.puppet.com/puppetserver/2.3/tuning_guide.html >>>>>> >>>>>> >>>>>> per jruby instance, we assign 512mb tot the VM, so 256mb is a little >>>>>> too less. >>>>>> >>>>>> >>>>>> hth >>>>>> >>>>>> >>>>>> Johan >>>>>> >>>>>> >>>>>> >>>>>> -----Original message----- >>>>>> *From:* Jim <[email protected]> >>>>>> *Sent:* Tuesday 24th May 2016 4:54 >>>>>> *To:* Puppet Users <[email protected]> >>>>>> *Subject:* [Puppet Users] Outofmemory error for puppetserver process >>>>>> >>>>>> Hey Guys, >>>>>> >>>>>> I am wondering what is the problem with the following puppetserver >>>>>> process and how to have it resolved. >>>>>> >>>>>> Interesting thing is puppetserver is still running ! and is able to >>>>>> communicate it with agents so I am not sure what is the problem ! >>>>>> >>>>>> I still have 2GB of free memory on my VM >>>>>> >>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >>>>>> >>>>>> ps -aef|grep -i puppet|grep -v grep >>>>>> root 742 1 0 09:39 ? 00:00:02 >>>>>> /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent >>>>>> --no-daemonize >>>>>> root 1369 1 0 09:39 ? 00:00:08 >>>>>> /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived >>>>>> --config=/etc/puppetlabs/mcollective/server.cfg >>>>>> --pidfile=/var/run/puppetlabs/mcollective.pid --daemonize >>>>>> puppet 8044 1 0 09:53 ? 00:01:09 /usr/bin/java >>>>>> -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom >>>>>> -Xms256M >>>>>> -Xmx256M -XX:MaxPermSize=256m -cp >>>>>> /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar >>>>>> clojure.main -m puppetlabs.trapperkeeper.main --config >>>>>> /etc/puppetlabs/puppetserver/conf.d -b >>>>>> /etc/puppetlabs/puppetserver/bootstrap.cfg >>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >>>>>> >>>>>> free -g >>>>>> total used free shared buffers >>>>>> cached >>>>>> Mem: 3 1 2 0 0 >>>>>> 0 >>>>>> -/+ buffers/cache: 1 2 >>>>>> Swap: 3 0 3 >>>>>> >>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >>>>>> >>>>>> >>>>>> >>>>>> As you can see I have set puppetserver memory to as low as 256MB. I >>>>>> tried to change it to 512MB as well but still it reports "OutOfMemory" >>>>>> so >>>>>> kept it on 256MB anyway. >>>>>> >>>>>> Here is the output which confirms that puppetserver is running. >>>>>> >>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >>>>>> >>>>>> service puppetserver status >>>>>> ◠puppetserver.service - LSB: puppetserver >>>>>> Loaded: loaded (/etc/init.d/puppetserver) >>>>>> Active: active (running) since Tue 2016-05-24 12:53:06 AEST; 16s >>>>>> ago >>>>>> Docs: man:systemd-sysv-generator(8) >>>>>> Process: 10986 ExecStart=/etc/init.d/puppetserver start >>>>>> (code=exited, status=0/SUCCESS) >>>>>> CGroup: /system.slice/puppetserver.service >>>>>> └─10992 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 >>>>>> %p -Djava.security.egd=/dev/urandom -Xms256M -Xmx256M >>>>>> -XX:MaxPermSize=256m >>>>>> -cp /opt/puppetlabs/server/apps/puppetserver/puppet-se... >>>>>> >>>>>> May 24 12:52:22 jim-Ubuntu1504 systemd[1]: Starting LSB: >>>>>> puppetserver... >>>>>> May 24 12:53:06 jim-Ubuntu1504 systemd[1]: Started LSB: puppetserver. >>>>>> >>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >>>>>> >>>>>> >>>>>> Any advice would be much appreciated. >>>>>> >>>>>> Cheers >>>>>> Jim >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Puppet Users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/puppet-users/60c8e923-10ca-4d01-a09c-7e87a5036fdb%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/puppet-users/60c8e923-10ca-4d01-a09c-7e87a5036fdb%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/puppet-users/375e3af0-4592-4ac8-b87d-473b85107b70%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/puppet-users/375e3af0-4592-4ac8-b87d-473b85107b70%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Trevor Vaughan >>> Vice President, Onyx Point, Inc >>> (410) 541-6699 x788 >>> >>> -- This account not approved for unencrypted proprietary information -- >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/97f14473-d7cc-4cab-be6f-9e726a686dad%40googlegroups.com >> >> <https://groups.google.com/d/msgid/puppet-users/97f14473-d7cc-4cab-be6f-9e726a686dad%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/da1a5527-cd82-4b6d-adfd-5789b96ab32f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
