Jim, For debugging failures on ubuntu 15.04, I would recommend trying `/opt/puppetlabs/bin/puppetserver foreground`, it may show you the errors that are causing the service to shutdown. Ubuntu 15.04 is a bit difficult to debug on, because while it has systemd support, the puppetserver packages for that platform still ship sysv init scripts. In my experience, the sysv-systemd integration seems to swallow some startup errors and messages that would usually go to journalctl on a systemd platform. We started shipping systemd service files for puppetserver starting in Ubuntu 16.04.
HTH On Thu, May 26, 2016 at 6:56 AM, Jim <[email protected]> wrote: > Hi Trevor, > > No luck. > > 2016-05-26 23:46:26,745 INFO [Thread-1] [p.t.internal] Shutting down due > to JVM shutdown hook. > 2016-05-26 23:46:26,747 INFO [Thread-1] [p.t.internal] Beginning shutdown > sequence > 2016-05-26 23:46:26,761 INFO [clojure-agent-send-pool-1] > [p.s.j.jruby-puppet-agents] Flush request received; creating new JRuby pool. > 2016-05-26 23:46:26,765 INFO [clojure-agent-send-pool-1] > [p.s.j.jruby-puppet-agents] Replacing old JRuby pool with new instance. > 2016-05-26 23:46:26,765 INFO [clojure-agent-send-pool-1] > [p.s.j.jruby-puppet-agents] Swapped JRuby pools, beginning cleanup of old > pool. > 2016-05-26 23:46:26,770 INFO [clojure-agent-send-pool-1] > [p.s.j.jruby-puppet-internal] Cleaned up old JRuby instance with id 1. > 2016-05-26 23:46:26,772 INFO [Thread-1] [p.t.s.w.jetty9-service] Shutting > down web server(s). > 2016-05-26 23:46:26,781 INFO [Thread-1] [p.t.s.w.jetty9-core] Shutting > down web server. > 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.ServerConnector] Stopped > ServerConnector@79303966{SSL-HTTP/1.1}{0.0.0.0:8140} > 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler] > Stopped o.e.j.s.h.ContextHandler@2b87522b{/,null,UNAVAILABLE} > 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler] > Stopped o.e.j.s.h.ContextHandler@1f2ffb51 > {/puppet-admin-api,null,UNAVAILABLE} > 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler] > Stopped o.e.j.s.h.ContextHandler@609b2b8a{/puppet,null,UNAVAILABLE} > 2016-05-26 23:46:26,825 INFO [Thread-1] [o.e.j.s.h.ContextHandler] > Stopped o.e.j.s.h.ContextHandler@5b1cbf6d{/puppet-ca,null,UNAVAILABLE} > 2016-05-26 23:46:26,827 INFO [Thread-1] [p.t.s.w.jetty9-core] Web server > shutdown > 2016-05-26 23:46:26,837 INFO [Thread-1] [p.t.internal] Finished shutdown > sequence > 2016-05-26 23:49:33,878 INFO [main] [o.e.j.u.log] Logging initialized > @14524ms > 2016-05-26 23:49:41,413 INFO [async-dispatch-2] > [p.s.v.versioned-code-service] No code-id-command set for > versioned-code-service. Code-id will be nil. > 2016-05-26 23:49:41,414 INFO [async-dispatch-2] > [p.s.v.versioned-code-service] No code-content-command set for > versioned-code-service. Attempting to fetch code content will fail. > 2016-05-26 23:49:41,417 INFO [async-dispatch-2] [p.t.s.w.jetty9-service] > Initializing web server(s). > 2016-05-26 23:49:41,457 INFO [async-dispatch-2] > [p.s.j.jruby-puppet-service] Initializing the JRuby service > 2016-05-26 23:49:41,457 WARN [async-dispatch-2] > [p.s.j.jruby-puppet-service] The 'jruby-puppet.use-legacy-auth-conf' > setting is set to 'true'. Support for the legacy Puppet auth.conf file is > deprecated and will be removed in a future release. Change this setting to > 'false' and migrate your authorization rule definitions in the > /etc/puppetlabs/puppet/auth.conf file to the > /etc/puppetlabs/puppetserver/conf.d/auth.conf file. > 2016-05-26 23:49:41,500 INFO [clojure-agent-send-pool-0] > [p.s.j.jruby-puppet-internal] Creating JRuby instance with id 1. > 2016-05-26 23:49:58,448 INFO [clojure-agent-send-pool-0] [puppet-server] > Puppet Puppet settings initialized; run mode: master > 2016-05-26 23:49:59,624 INFO [clojure-agent-send-pool-0] > [p.s.j.jruby-puppet-agents] Finished creating JRubyPuppet instance 1 of 1 > 2016-05-26 23:49:59,636 INFO [async-dispatch-2] > [p.s.c.puppet-server-config-core] Initializing webserver settings from core > Puppet > 2016-05-26 23:49:59,672 INFO [async-dispatch-2] > [p.p.certificate-authority] CA already initialized for SSL > 2016-05-26 23:49:59,674 INFO [async-dispatch-2] > [p.s.c.certificate-authority-service] CA Service adding a ring handler > 2016-05-26 23:49:59,749 INFO [async-dispatch-2] > [p.p.certificate-authority] Master already initialized for SSL > 2016-05-26 23:49:59,750 INFO [async-dispatch-2] [p.s.m.master-service] > Master Service adding ring handlers > 2016-05-26 23:49:59,813 INFO [async-dispatch-2] > [p.s.p.puppet-admin-service] Starting Puppet Admin web app > 2016-05-26 23:49:59,844 WARN [async-dispatch-2] > [o.e.j.s.h.ContextHandler] Empty contextPath > 2016-05-26 23:49:59,849 INFO [async-dispatch-2] [p.t.s.w.jetty9-service] > Starting web server(s). > 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] > webserver config overridden for key 'ssl-cert' > 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] > webserver config overridden for key 'ssl-key' > 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] > webserver config overridden for key 'ssl-ca-cert' > 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] > webserver config overridden for key 'ssl-crl-path' > 2016-05-26 23:50:00,249 INFO [async-dispatch-2] [p.t.s.w.jetty9-core] > Starting web server. > 2016-05-26 23:50:00,260 INFO [async-dispatch-2] [o.e.j.s.Server] > jetty-9.2.z-SNAPSHOT > 2016-05-26 23:50:00,360 INFO [async-dispatch-2] > [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@62f22fd > {/puppet-ca,null,AVAILABLE} > 2016-05-26 23:50:00,360 INFO [async-dispatch-2] > [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@dc925ec > {/puppet,null,AVAILABLE} > 2016-05-26 23:50:00,360 INFO [async-dispatch-2] > [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@55c827e3 > {/puppet-admin-api,null,AVAILABLE} > 2016-05-26 23:50:00,360 INFO [async-dispatch-2] > [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@3b82279 > {/,null,AVAILABLE} > 2016-05-26 23:50:00,472 INFO [async-dispatch-2] [o.e.j.s.ServerConnector] > Started ServerConnector@2fcb900d{SSL-HTTP/1.1}{0.0.0.0:8140} > 2016-05-26 23:50:00,473 INFO [async-dispatch-2] [o.e.j.s.Server] Started > @41120ms > 2016-05-26 23:50:00,486 INFO [async-dispatch-2] [p.s.m.master-service] > Puppet Server has successfully started and is now ready to handle requests > 2016-05-26 23:50:00,488 INFO [async-dispatch-2] > [p.s.l.legacy-routes-service] The legacy routing service has successfully > started and is now ready to handle requests > 2016-05-26 23:50:00,602 INFO [clojure-agent-send-off-pool-0] > [p.d.version-check] Newer version 2.4.0 is available! Visit > https://github.com/puppetlabs/puppetserver/tree/stable for details. > root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# > service puppetserver status > ◠puppetserver.service - LSB: puppetserver > Loaded: loaded (/etc/init.d/puppetserver) > Active: active (running) since Thu 2016-05-26 23:50:01 AEST; 4min 55s > ago > Docs: man:systemd-sysv-generator(8) > Process: 4397 ExecStart=/etc/init.d/puppetserver start (code=exited, > status=0/SUCCESS) > CGroup: /system.slice/puppetserver.service > └─4403 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p > -Djava.security.egd=/dev/urandom -Xms512m -Xmx512m -XX:MaxPermSize=512m -cp > /opt/puppetlabs/server/apps/puppetserver/puppet-ser... > > May 26 23:49:19 jim-Ubuntu1504 systemd[1]: Starting LSB: puppetserver... > May 26 23:50:01 jim-Ubuntu1504 systemd[1]: Started LSB: puppetserver. > root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# > cat /etc/default/puppetserver > ########################################### > # Init settings for puppetserver > ########################################### > > # Location of your Java binary (version 7 or higher) > JAVA_BIN="/usr/bin/java" > > # Modify this if you'd like to change the memory allocation, enable JMX, > etc > JAVA_ARGS="-Xms512m -Xmx512m -XX:MaxPermSize=512m" > > # These normally shouldn't need to be edited if using OS packages > USER="puppet" > GROUP="puppet" > INSTALL_DIR="/opt/puppetlabs/server/apps/puppetserver" > CONFIG="/etc/puppetlabs/puppetserver/conf.d" > BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetserver/bootstrap.cfg" > SERVICE_STOP_RETRIES=60 > > # START_TIMEOUT can be set here to alter the default startup timeout in > # seconds. This is used in System-V style init scripts only, and will > have no > # effect in systemd. > # START_TIMEOUT=120 > root@jim-Ubuntu1504 > :/etc/puppetlabs/code/environments/production/manifests# > > Cheers > Jim > > On Wednesday, 25 May 2016 19:56:43 UTC+10, Trevor Vaughan wrote: >> >> Hi Jim, >> >> Your logs should be in /var/log/puppetserver/puppetserver.log. >> >> Also, I'm guessing that you need to change the Xmx java option in your >> puppetserver start script. >> >> Trevor >> >> On Tue, May 24, 2016 at 8:51 AM, Jim <[email protected]> wrote: >> >>> Any idea which log file to look at when I run the service puppetserver >>> start command? >>> >>> >>> On Tuesday, 24 May 2016 22:44:03 UTC+10, Jim wrote: >>>> >>>> Hi Johan, >>>> >>>> I did try 512m option not just 256m before posting. and it didn't work. >>>> The link which you have provided talks basically about >>>> >>>> 1. Changing parameters for java options >>>> 2. max active instances(Its commented out by default) >>>> 3.num-cpus configuration. I did recursive grep but couldn't find it. >>>> Puppetlabs says that they ship it anyway with low value so I don't think >>>> that should be cause or concerb. >>>> >>>> I would probably get hold of logs and see if I can find any valuable >>>> information. >>>> >>>> Kind Regards, >>>> Jim >>>> >>>> On Tuesday, 24 May 2016 15:39:05 UTC+10, Johan De Wit wrote: >>>>> >>>>> Hi Jim, >>>>> >>>>> >>>>> You should find some answers here : >>>>> https://docs.puppet.com/puppetserver/2.3/tuning_guide.html >>>>> >>>>> >>>>> per jruby instance, we assign 512mb tot the VM, so 256mb is a little >>>>> too less. >>>>> >>>>> >>>>> hth >>>>> >>>>> >>>>> Johan >>>>> >>>>> >>>>> >>>>> -----Original message----- >>>>> *From:* Jim <[email protected]> >>>>> *Sent:* Tuesday 24th May 2016 4:54 >>>>> *To:* Puppet Users <[email protected]> >>>>> *Subject:* [Puppet Users] Outofmemory error for puppetserver process >>>>> >>>>> Hey Guys, >>>>> >>>>> I am wondering what is the problem with the following puppetserver >>>>> process and how to have it resolved. >>>>> >>>>> Interesting thing is puppetserver is still running ! and is able to >>>>> communicate it with agents so I am not sure what is the problem ! >>>>> >>>>> I still have 2GB of free memory on my VM >>>>> >>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >>>>> ps -aef|grep -i puppet|grep -v grep >>>>> root 742 1 0 09:39 ? 00:00:02 >>>>> /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent >>>>> --no-daemonize >>>>> root 1369 1 0 09:39 ? 00:00:08 >>>>> /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived >>>>> --config=/etc/puppetlabs/mcollective/server.cfg >>>>> --pidfile=/var/run/puppetlabs/mcollective.pid --daemonize >>>>> puppet 8044 1 0 09:53 ? 00:01:09 /usr/bin/java >>>>> -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom >>>>> -Xms256M >>>>> -Xmx256M -XX:MaxPermSize=256m -cp >>>>> /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar >>>>> clojure.main -m puppetlabs.trapperkeeper.main --config >>>>> /etc/puppetlabs/puppetserver/conf.d -b >>>>> /etc/puppetlabs/puppetserver/bootstrap.cfg >>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >>>>> free -g >>>>> total used free shared buffers >>>>> cached >>>>> Mem: 3 1 2 0 0 >>>>> 0 >>>>> -/+ buffers/cache: 1 2 >>>>> Swap: 3 0 3 >>>>> root@jim-Ubuntu1504 >>>>> :/etc/puppetlabs/code/environments/production/manifests# >>>>> >>>>> >>>>> As you can see I have set puppetserver memory to as low as 256MB. I >>>>> tried to change it to 512MB as well but still it reports "OutOfMemory" so >>>>> kept it on 256MB anyway. >>>>> >>>>> Here is the output which confirms that puppetserver is running. >>>>> >>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests# >>>>> service puppetserver status >>>>> ◠puppetserver.service - LSB: puppetserver >>>>> Loaded: loaded (/etc/init.d/puppetserver) >>>>> Active: active (running) since Tue 2016-05-24 12:53:06 AEST; 16s ago >>>>> Docs: man:systemd-sysv-generator(8) >>>>> Process: 10986 ExecStart=/etc/init.d/puppetserver start >>>>> (code=exited, status=0/SUCCESS) >>>>> CGroup: /system.slice/puppetserver.service >>>>> └─10992 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p >>>>> -Djava.security.egd=/dev/urandom -Xms256M -Xmx256M -XX:MaxPermSize=256m >>>>> -cp >>>>> /opt/puppetlabs/server/apps/puppetserver/puppet-se... >>>>> >>>>> May 24 12:52:22 jim-Ubuntu1504 systemd[1]: Starting LSB: >>>>> puppetserver... >>>>> May 24 12:53:06 jim-Ubuntu1504 systemd[1]: Started LSB: puppetserver. >>>>> root@jim-Ubuntu1504 >>>>> :/etc/puppetlabs/code/environments/production/manifests# >>>>> >>>>> Any advice would be much appreciated. >>>>> >>>>> Cheers >>>>> Jim >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Puppet Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/puppet-users/60c8e923-10ca-4d01-a09c-7e87a5036fdb%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/puppet-users/60c8e923-10ca-4d01-a09c-7e87a5036fdb%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/375e3af0-4592-4ac8-b87d-473b85107b70%40googlegroups.com >>> <https://groups.google.com/d/msgid/puppet-users/375e3af0-4592-4ac8-b87d-473b85107b70%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Trevor Vaughan >> Vice President, Onyx Point, Inc >> (410) 541-6699 x788 >> >> -- This account not approved for unencrypted proprietary information -- >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/97f14473-d7cc-4cab-be6f-9e726a686dad%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/97f14473-d7cc-4cab-be6f-9e726a686dad%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CACD%3DwAeQLPntPjZrz6R65PiQRQ_LFOVWSuWL1pMjMx154GAOOw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
