Re: [Shinken-devel] Problem with timeouts

[David Good]

Does it hurt to have more than I need?  All of the Shinken servers have plenty of resources to spare.  I'd rather they have more threads than they need than too few.  Unless that creates other problems. I've removed the extra schedulers and pollers I had added thinking it would help (now a total of 4 each plus a spare) and raised the thread pool size on all daemons to 50 and I'm still having issues with timeouts. On 5/12/15 2:34 AM, Felipe openglx wrote: Those timeout issues happened to me on a similar sized structure. One of my realms was never getting stable and I discovered it wasn't timeout but actually number of threads on Cherrypy due number of other Shinken components. Try allowing more threads on your scheduler (as it is the most central component): daemon_thread_pool_size=12 Twelve threads was good enough for me but your number will be higher. So you have: * 2 "master": 1 arbiter, 1 broker, 1 reactionner, 1 receiver, 2 schedulers, 2 pollers * 3 nodes: 2 schedulers, 2 pollers That means that you have 2*8 + 3*4 = 28 daemons. Maybe bring your thread pool size to 30 ? I'd suspect root cause is your setup having way too many schedulers for no apparent reason. If you are having them distributed please take some read on setting up some realms. Regards On 12 May 2015 at 05:29, David Good <dg...@willingminds.com> wrote: We're still running 2.2 currently (we're on a tight schedule to get Shinken into production).  We just revamped our host and service configuration, adding about 1000 hosts (currently 3351 hosts, 1839 hostgroups, 30404 services).  Since loading that configuration we've had trouble with connections from the arbiter timing out so it's constantly redispatching the configuration only to have the daemon that timed out recover.  I've increased ping timeouts to 6 seconds and data timeouts to 120 seconds but it still times out.  I though maybe adding more schedulers and pollers would help as it would cut the config into smaller pieces that could be processed more quickly, but it's still constantly dispatching and re-dispatching configurations. We're running on 5 servers, each with 40 CPUs and 64 GB of RAM.  One of the servers is the master and runs all daemons plus an extra poller and scheduler.  Three other servers are running two schedulers and two pollers each.  The last is a spare setup the same as the master.  None of the servers are showing significant CPU, I/O, Memory or network usage. Any ideas?  Would upgrading to 2.4 help? ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ Shinken-devel mailing list Shinken-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shinken-devel ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ Shinken-devel mailing list Shinken-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shinken-devel

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Shinken-devel mailing list
Shinken-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shinken-devel