Hi Matthaus,
My hostname is misleading. I upgraded at the time of posting to Ubuntu
15.10. Now I have upgraded to 16.04.
I ran the command you suggested instead of service puppetserver start and I
don't see OutOfMemory errors process.
root@jim-Ubuntu1504:~# ps -aef|grep -i puppet|grep -v grep
root 1116 1 0 19:34 ? 00:00:02
/opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent
--no-daemonize
root 1719 1 0 19:38 ? 00:00:08
/opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived
--config=/etc/puppetlabs/mcollective/server.cfg
--pidfile=/var/run/puppetlabs/mcollective.pid --daemonize
root 2644 1544 0 22:35 pts/0 00:00:00 bash
/opt/puppetlabs/server/apps/puppetserver/cli/apps/foreground
root 2650 2644 0 22:35 pts/0 00:00:00 runuser puppet -s /bin/bash
-c /usr/bin/java -Xms512m -Xmx512m -XX:MaxPermSize=512m
-Dlogappender=STDOUT -cp
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar
clojure.main -m puppetlabs.trapperkeeper.main --config
/etc/puppetlabs/puppetserver/conf.d --bootstrap-config
/etc/puppetlabs/puppetserver/bootstrap.cfg
puppet 2651 2650 55 22:35 ? 00:00:42 /usr/bin/java -Xms512m
-Xmx512m -XX:MaxPermSize=512m -Dlogappender=STDOUT -cp
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar
clojure.main -m puppetlabs.trapperkeeper.main --config
/etc/puppetlabs/puppetserver/conf.d --bootstrap-config
/etc/puppetlabs/puppetserver/bootstrap.cfg
root@jim-Ubuntu1504:~#
Also when I do see that error as like below when I do Ctrl + C and execute
service puppetserver start I get the OutOfMemory error like below.
root@jim-Ubuntu1504:~# ps -aef|grep -i puppet|grep -v grep
root 1116 1 0 19:34 ? 00:00:02
/opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent
--no-daemonize
root 1719 1 0 19:38 ? 00:00:08
/opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived
--config=/etc/puppetlabs/mcollective/server.cfg
--pidfile=/var/run/puppetlabs/mcollective.pid --daemonize
puppet 3386 1 66 22:41 ? 00:00:45 /usr/bin/java
-XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom -Xms512m
-Xmx512m -XX:MaxPermSize=512m -cp
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar
clojure.main -m puppetlabs.trapperkeeper.main --config
/etc/puppetlabs/puppetserver/conf.d -b
/etc/puppetlabs/puppetserver/bootstrap.cfg
root@jim-Ubuntu1504:~#
I wonder if I am doing start-up incorrectly ? Either even in this state
where it sayOutOfMemoryError my puppetserver still serves all of its
agents. The reason I say so is I changed motd file on all agents +
installed vim package on agents.
I am not sure which is the right method to start the server ? Also instead
of running in foreground can I run it in background using nohup ?
Regards,
Jim
On Friday, 27 May 2016 02:46:14 UTC+10, Matthaus Litteken wrote:
>
> Jim,
> For debugging failures on ubuntu 15.04, I would recommend trying
> `/opt/puppetlabs/bin/puppetserver foreground`, it may show you the errors
> that are causing the service to shutdown. Ubuntu 15.04 is a bit difficult
> to debug on, because while it has systemd support, the puppetserver
> packages for that platform still ship sysv init scripts. In my experience,
> the sysv-systemd integration seems to swallow some startup errors and
> messages that would usually go to journalctl on a systemd platform. We
> started shipping systemd service files for puppetserver starting in Ubuntu
> 16.04.
>
> HTH
>
> On Thu, May 26, 2016 at 6:56 AM, Jim <ameya...@gmail.com <javascript:>>
> wrote:
>
>> Hi Trevor,
>>
>> No luck.
>>
>> 2016-05-26 23:46:26,745 INFO [Thread-1] [p.t.internal] Shutting down due
>> to JVM shutdown hook.
>> 2016-05-26 23:46:26,747 INFO [Thread-1] [p.t.internal] Beginning
>> shutdown sequence
>> 2016-05-26 23:46:26,761 INFO [clojure-agent-send-pool-1]
>> [p.s.j.jruby-puppet-agents] Flush request received; creating new JRuby pool.
>> 2016-05-26 23:46:26,765 INFO [clojure-agent-send-pool-1]
>> [p.s.j.jruby-puppet-agents] Replacing old JRuby pool with new instance.
>> 2016-05-26 23:46:26,765 INFO [clojure-agent-send-pool-1]
>> [p.s.j.jruby-puppet-agents] Swapped JRuby pools, beginning cleanup of old
>> pool.
>> 2016-05-26 23:46:26,770 INFO [clojure-agent-send-pool-1]
>> [p.s.j.jruby-puppet-internal] Cleaned up old JRuby instance with id 1.
>> 2016-05-26 23:46:26,772 INFO [Thread-1] [p.t.s.w.jetty9-service]
>> Shutting down web server(s).
>> 2016-05-26 23:46:26,781 INFO [Thread-1] [p.t.s.w.jetty9-core] Shutting
>> down web server.
>> 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.ServerConnector]
>> Stopped ServerConnector@79303966{SSL-HTTP/1.1}{0.0.0.0:8140}
>> 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler]
>> Stopped o.e.j.s.h.ContextHandler@2b87522b{/,null,UNAVAILABLE}
>> 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler]
>> Stopped
>> o.e.j.s.h.ContextHandler@1f2ffb51{/puppet-admin-api,null,UNAVAILABLE}
>> 2016-05-26 23:46:26,824 INFO [Thread-1] [o.e.j.s.h.ContextHandler]
>> Stopped o.e.j.s.h.ContextHandler@609b2b8a{/puppet,null,UNAVAILABLE}
>> 2016-05-26 23:46:26,825 INFO [Thread-1] [o.e.j.s.h.ContextHandler]
>> Stopped o.e.j.s.h.ContextHandler@5b1cbf6d{/puppet-ca,null,UNAVAILABLE}
>> 2016-05-26 23:46:26,827 INFO [Thread-1] [p.t.s.w.jetty9-core] Web server
>> shutdown
>> 2016-05-26 23:46:26,837 INFO [Thread-1] [p.t.internal] Finished shutdown
>> sequence
>> 2016-05-26 23:49:33,878 INFO [main] [o.e.j.u.log] Logging initialized
>> @14524ms
>> 2016-05-26 23:49:41,413 INFO [async-dispatch-2]
>> [p.s.v.versioned-code-service] No code-id-command set for
>> versioned-code-service. Code-id will be nil.
>> 2016-05-26 23:49:41,414 INFO [async-dispatch-2]
>> [p.s.v.versioned-code-service] No code-content-command set for
>> versioned-code-service. Attempting to fetch code content will fail.
>> 2016-05-26 23:49:41,417 INFO [async-dispatch-2] [p.t.s.w.jetty9-service]
>> Initializing web server(s).
>> 2016-05-26 23:49:41,457 INFO [async-dispatch-2]
>> [p.s.j.jruby-puppet-service] Initializing the JRuby service
>> 2016-05-26 23:49:41,457 WARN [async-dispatch-2]
>> [p.s.j.jruby-puppet-service] The 'jruby-puppet.use-legacy-auth-conf'
>> setting is set to 'true'. Support for the legacy Puppet auth.conf file is
>> deprecated and will be removed in a future release. Change this setting to
>> 'false' and migrate your authorization rule definitions in the
>> /etc/puppetlabs/puppet/auth.conf file to the
>> /etc/puppetlabs/puppetserver/conf.d/auth.conf file.
>> 2016-05-26 23:49:41,500 INFO [clojure-agent-send-pool-0]
>> [p.s.j.jruby-puppet-internal] Creating JRuby instance with id 1.
>> 2016-05-26 23:49:58,448 INFO [clojure-agent-send-pool-0] [puppet-server]
>> Puppet Puppet settings initialized; run mode: master
>> 2016-05-26 23:49:59,624 INFO [clojure-agent-send-pool-0]
>> [p.s.j.jruby-puppet-agents] Finished creating JRubyPuppet instance 1 of 1
>> 2016-05-26 23:49:59,636 INFO [async-dispatch-2]
>> [p.s.c.puppet-server-config-core] Initializing webserver settings from core
>> Puppet
>> 2016-05-26 23:49:59,672 INFO [async-dispatch-2]
>> [p.p.certificate-authority] CA already initialized for SSL
>> 2016-05-26 23:49:59,674 INFO [async-dispatch-2]
>> [p.s.c.certificate-authority-service] CA Service adding a ring handler
>> 2016-05-26 23:49:59,749 INFO [async-dispatch-2]
>> [p.p.certificate-authority] Master already initialized for SSL
>> 2016-05-26 23:49:59,750 INFO [async-dispatch-2] [p.s.m.master-service]
>> Master Service adding ring handlers
>> 2016-05-26 23:49:59,813 INFO [async-dispatch-2]
>> [p.s.p.puppet-admin-service] Starting Puppet Admin web app
>> 2016-05-26 23:49:59,844 WARN [async-dispatch-2]
>> [o.e.j.s.h.ContextHandler] Empty contextPath
>> 2016-05-26 23:49:59,849 INFO [async-dispatch-2] [p.t.s.w.jetty9-service]
>> Starting web server(s).
>> 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core]
>> webserver config overridden for key 'ssl-cert'
>> 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core]
>> webserver config overridden for key 'ssl-key'
>> 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core]
>> webserver config overridden for key 'ssl-ca-cert'
>> 2016-05-26 23:49:59,868 INFO [async-dispatch-2] [p.t.s.w.jetty9-core]
>> webserver config overridden for key 'ssl-crl-path'
>> 2016-05-26 23:50:00,249 INFO [async-dispatch-2] [p.t.s.w.jetty9-core]
>> Starting web server.
>> 2016-05-26 23:50:00,260 INFO [async-dispatch-2] [o.e.j.s.Server]
>> jetty-9.2.z-SNAPSHOT
>> 2016-05-26 23:50:00,360 INFO [async-dispatch-2]
>> [o.e.j.s.h.ContextHandler] Started
>> o.e.j.s.h.ContextHandler@62f22fd{/puppet-ca,null,AVAILABLE}
>> 2016-05-26 23:50:00,360 INFO [async-dispatch-2]
>> [o.e.j.s.h.ContextHandler] Started
>> o.e.j.s.h.ContextHandler@dc925ec{/puppet,null,AVAILABLE}
>> 2016-05-26 23:50:00,360 INFO [async-dispatch-2]
>> [o.e.j.s.h.ContextHandler] Started
>> o.e.j.s.h.ContextHandler@55c827e3{/puppet-admin-api,null,AVAILABLE}
>> 2016-05-26 23:50:00,360 INFO [async-dispatch-2]
>> [o.e.j.s.h.ContextHandler] Started
>> o.e.j.s.h.ContextHandler@3b82279{/,null,AVAILABLE}
>> 2016-05-26 23:50:00,472 INFO [async-dispatch-2]
>> [o.e.j.s.ServerConnector] Started ServerConnector@2fcb900d{SSL-HTTP/1.1}{
>> 0.0.0.0:8140}
>> 2016-05-26 23:50:00,473 INFO [async-dispatch-2] [o.e.j.s.Server] Started
>> @41120ms
>> 2016-05-26 23:50:00,486 INFO [async-dispatch-2] [p.s.m.master-service]
>> Puppet Server has successfully started and is now ready to handle requests
>> 2016-05-26 23:50:00,488 INFO [async-dispatch-2]
>> [p.s.l.legacy-routes-service] The legacy routing service has successfully
>> started and is now ready to handle requests
>> 2016-05-26 23:50:00,602 INFO [clojure-agent-send-off-pool-0]
>> [p.d.version-check] Newer version 2.4.0 is available! Visit
>> https://github.com/puppetlabs/puppetserver/tree/stable for details.
>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
>> service puppetserver status
>> â— puppetserver.service - LSB: puppetserver
>> Loaded: loaded (/etc/init.d/puppetserver)
>> Active: active (running) since Thu 2016-05-26 23:50:01 AEST; 4min 55s
>> ago
>> Docs: man:systemd-sysv-generator(8)
>> Process: 4397 ExecStart=/etc/init.d/puppetserver start (code=exited,
>> status=0/SUCCESS)
>> CGroup: /system.slice/puppetserver.service
>> └─4403 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p
>> -Djava.security.egd=/dev/urandom -Xms512m -Xmx512m -XX:MaxPermSize=512m -cp
>> /opt/puppetlabs/server/apps/puppetserver/puppet-ser...
>>
>> May 26 23:49:19 jim-Ubuntu1504 systemd[1]: Starting LSB: puppetserver...
>> May 26 23:50:01 jim-Ubuntu1504 systemd[1]: Started LSB: puppetserver.
>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
>> cat /etc/default/puppetserver
>> ###########################################
>> # Init settings for puppetserver
>> ###########################################
>>
>> # Location of your Java binary (version 7 or higher)
>> JAVA_BIN="/usr/bin/java"
>>
>> # Modify this if you'd like to change the memory allocation, enable JMX,
>> etc
>> JAVA_ARGS="-Xms512m -Xmx512m -XX:MaxPermSize=512m"
>>
>> # These normally shouldn't need to be edited if using OS packages
>> USER="puppet"
>> GROUP="puppet"
>> INSTALL_DIR="/opt/puppetlabs/server/apps/puppetserver"
>> CONFIG="/etc/puppetlabs/puppetserver/conf.d"
>> BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetserver/bootstrap.cfg"
>> SERVICE_STOP_RETRIES=60
>>
>> # START_TIMEOUT can be set here to alter the default startup timeout in
>> # seconds. This is used in System-V style init scripts only, and will
>> have no
>> # effect in systemd.
>> # START_TIMEOUT=120
>>
>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
>>
>> Cheers
>> Jim
>>
>> On Wednesday, 25 May 2016 19:56:43 UTC+10, Trevor Vaughan wrote:
>>>
>>> Hi Jim,
>>>
>>> Your logs should be in /var/log/puppetserver/puppetserver.log.
>>>
>>> Also, I'm guessing that you need to change the Xmx java option in your
>>> puppetserver start script.
>>>
>>> Trevor
>>>
>>> On Tue, May 24, 2016 at 8:51 AM, Jim <ameya...@gmail.com> wrote:
>>>
>>>> Any idea which log file to look at when I run the service puppetserver
>>>> start command?
>>>>
>>>>
>>>> On Tuesday, 24 May 2016 22:44:03 UTC+10, Jim wrote:
>>>>>
>>>>> Hi Johan,
>>>>>
>>>>> I did try 512m option not just 256m before posting. and it didn't
>>>>> work. The link which you have provided talks basically about
>>>>>
>>>>> 1. Changing parameters for java options
>>>>> 2. max active instances(Its commented out by default)
>>>>> 3.num-cpus configuration. I did recursive grep but couldn't find it.
>>>>> Puppetlabs says that they ship it anyway with low value so I don't think
>>>>> that should be cause or concerb.
>>>>>
>>>>> I would probably get hold of logs and see if I can find any valuable
>>>>> information.
>>>>>
>>>>> Kind Regards,
>>>>> Jim
>>>>>
>>>>> On Tuesday, 24 May 2016 15:39:05 UTC+10, Johan De Wit wrote:
>>>>>>
>>>>>> Hi Jim,
>>>>>>
>>>>>>
>>>>>> You should find some answers here :
>>>>>> https://docs.puppet.com/puppetserver/2.3/tuning_guide.html
>>>>>>
>>>>>>
>>>>>> per jruby instance, we assign 512mb tot the VM, so 256mb is a little
>>>>>> too less.
>>>>>>
>>>>>>
>>>>>> hth
>>>>>>
>>>>>>
>>>>>> Johan
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original message-----
>>>>>> *From:* Jim <ameya...@gmail.com>
>>>>>> *Sent:* Tuesday 24th May 2016 4:54
>>>>>> *To:* Puppet Users <puppet...@googlegroups.com>
>>>>>> *Subject:* [Puppet Users] Outofmemory error for puppetserver process
>>>>>>
>>>>>> Hey Guys,
>>>>>>
>>>>>> I am wondering what is the problem with the following puppetserver
>>>>>> process and how to have it resolved.
>>>>>>
>>>>>> Interesting thing is puppetserver is still running ! and is able to
>>>>>> communicate it with agents so I am not sure what is the problem !
>>>>>>
>>>>>> I still have 2GB of free memory on my VM
>>>>>>
>>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
>>>>>>
>>>>>> ps -aef|grep -i puppet|grep -v grep
>>>>>> root 742 1 0 09:39 ? 00:00:02
>>>>>> /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent
>>>>>> --no-daemonize
>>>>>> root 1369 1 0 09:39 ? 00:00:08
>>>>>> /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/mcollectived
>>>>>> --config=/etc/puppetlabs/mcollective/server.cfg
>>>>>> --pidfile=/var/run/puppetlabs/mcollective.pid --daemonize
>>>>>> puppet 8044 1 0 09:53 ? 00:01:09 /usr/bin/java
>>>>>> -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/dev/urandom
>>>>>> -Xms256M
>>>>>> -Xmx256M -XX:MaxPermSize=256m -cp
>>>>>> /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar
>>>>>> clojure.main -m puppetlabs.trapperkeeper.main --config
>>>>>> /etc/puppetlabs/puppetserver/conf.d -b
>>>>>> /etc/puppetlabs/puppetserver/bootstrap.cfg
>>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
>>>>>>
>>>>>> free -g
>>>>>> total used free shared buffers
>>>>>> cached
>>>>>> Mem: 3 1 2 0 0
>>>>>> 0
>>>>>> -/+ buffers/cache: 1 2
>>>>>> Swap: 3 0 3
>>>>>>
>>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
>>>>>>
>>>>>>
>>>>>>
>>>>>> As you can see I have set puppetserver memory to as low as 256MB. I
>>>>>> tried to change it to 512MB as well but still it reports "OutOfMemory"
>>>>>> so
>>>>>> kept it on 256MB anyway.
>>>>>>
>>>>>> Here is the output which confirms that puppetserver is running.
>>>>>>
>>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
>>>>>>
>>>>>> service puppetserver status
>>>>>> â— puppetserver.service - LSB: puppetserver
>>>>>> Loaded: loaded (/etc/init.d/puppetserver)
>>>>>> Active: active (running) since Tue 2016-05-24 12:53:06 AEST; 16s
>>>>>> ago
>>>>>> Docs: man:systemd-sysv-generator(8)
>>>>>> Process: 10986 ExecStart=/etc/init.d/puppetserver start
>>>>>> (code=exited, status=0/SUCCESS)
>>>>>> CGroup: /system.slice/puppetserver.service
>>>>>> └─10992 /usr/bin/java -XX:OnOutOfMemoryError=kill -9
>>>>>> %p -Djava.security.egd=/dev/urandom -Xms256M -Xmx256M
>>>>>> -XX:MaxPermSize=256m
>>>>>> -cp /opt/puppetlabs/server/apps/puppetserver/puppet-se...
>>>>>>
>>>>>> May 24 12:52:22 jim-Ubuntu1504 systemd[1]: Starting LSB:
>>>>>> puppetserver...
>>>>>> May 24 12:53:06 jim-Ubuntu1504 systemd[1]: Started LSB: puppetserver.
>>>>>>
>>>>>> root@jim-Ubuntu1504:/etc/puppetlabs/code/environments/production/manifests#
>>>>>>
>>>>>>
>>>>>> Any advice would be much appreciated.
>>>>>>
>>>>>> Cheers
>>>>>> Jim
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Puppet Users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to puppet-users...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/puppet-users/60c8e923-10ca-4d01-a09c-7e87a5036fdb%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/puppet-users/60c8e923-10ca-4d01-a09c-7e87a5036fdb%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Puppet Users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to puppet-users...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/puppet-users/375e3af0-4592-4ac8-b87d-473b85107b70%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/puppet-users/375e3af0-4592-4ac8-b87d-473b85107b70%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> Trevor Vaughan
>>> Vice President, Onyx Point, Inc
>>> (410) 541-6699 x788
>>>
>>> -- This account not approved for unencrypted proprietary information --
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to puppet-users...@googlegroups.com <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/97f14473-d7cc-4cab-be6f-9e726a686dad%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/puppet-users/97f14473-d7cc-4cab-be6f-9e726a686dad%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/da1a5527-cd82-4b6d-adfd-5789b96ab32f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
