We are running a PuppetDB instance with ~8000 clients using a 30 minute runinterval. We have hit some scaling issues in the past but the latest PuppetDB version (2.3.5) is running nicely for us. This is with stringify_facts=false, which does create some extra load compared with stringify_facts=true.
We have separated the PuppetDB service hosts and the postgres databases to different hosts. For postgres we have a write master with a hot standby and four read-only slaves (one for each PuppetDB host). This was scaled up when we actually had some performance issues with earlier PuppetDB versions and can likely be scaled down to just two service hosts and read-only slaves without issues now. The hardware is pretty beefy, the service hosts are Dell PowerEdge R620 with 24 CPU cores @ 2.00GHz and 32GB RAM. The databases are all Dell PowerEdge R620 with 32 CPU cores @ 1.80GHz and 64GB RAM. For disks they have a 3TB Fusion-IO ioDrive2 each, they have pretty impressive performance that you can read about here: http://www.fusionio.com/data-sheets/iodrive2 So, this is probably on the higher end of scaling up a PuppetDB cluster and I'm fairly confident it could handle a lot more than 8000 clients, but haven't done any load test on it so far to see what the limit is. We have a smaller PuppetDB installation as well which is around 2000 clients, and there we can manage with just a single of these database hosts and PuppetDB service hosts. On Fri, 26 Jun 2015 at 15:25 Christopher Wood <[email protected]> wrote: > There's a load balancer in front of the puppetmasters (which have SAN > certs) but a single puppetdb host. Nothing save puppetmasters and my own > curiosity depends on puppetdb (2.2.2) right now. (My disaster recovery plan > is to rebuild the host and let the data refill by itself.) > > If I need another scale-up or more things rely on puppetdb I might add > database replication for PostgreSQL and a few extra puppetdb instances > behind the load balancer. Given current usage I have no idea when I'd hit > that point. I have 2GB max heap set and it spikes to 1.4GB usage quite > rarely. > > On Fri, Jun 26, 2015 at 07:32:18AM -0400, Tom Tucker wrote: > > Chris, > > Thanks for the detailed response. BTW did you deploy HA, load > balancing, > > etc on the Postgres side? > > Cheers > > On Thu, Jun 25, 2015 at 11:11 PM, Christopher Wood > > <[1][email protected]> wrote: > > > > Somewhere past 700 nodes (still puppetizing) our 1-core, 2GB-RAM > > puppetmasters and 2-core, 2GB-RAM puppetdb host started showing > signs of > > overload (ssl/connect errors in agent logs, catalog/report > mismatches in > > puppetdb). I augmented the VMs with "hardware" to stop the > complaints > > and later on went off tuning. I moved the puppetmasters up to 4-core > > 8GB-RAM and the puppetdb host is now 4-core 16GB-RAM. Definitely > > rattling around in them now but there's lots of room for growth. > > > > For scaling/tuning, among others: > > > > [2] > https://ask.puppetlabs.com/question/13433/how-should-i-tune-passenger-to-run-puppet/ > > > > [3] > https://docs.puppetlabs.com/puppetdb/latest/scaling_recommendations.html > > > > [4]https://docs.puppetlabs.com/guides/scaling.html > > > > [5]http://activemq.apache.org/scaling-queues.html > > > > [6]http://activemq.apache.org/javalangoutofmemory.html > > > > After all that I analyzed catalogs across the deployment and found > that > > the datacat usage in the mcollective module (now > > [7]https://github.com/puppet-community/puppet-mcollective) was an > > abominable percentage of the total number of resources. The firewall > > type ([8]https://github.com/puppetlabs/puppetlabs-firewall) was 3% > of > > the total resources. Since it takes less horsepower to puppet up > fewer > > things I figure there will be a benefit in some judicious > refactoring > > here and there. (Templates instead of file_line, iptables config > instead > > of firewall resources, et cetera.) > > > > However, I figure there's a benefit in a conversion effort to cram > > things into puppet first and sort them out later. I had a good time > just > > throwing hardware at the problem to start and then tuning after the > bulk > > of hosts were converted to puppet management. People at companies > where > > incremental hardware use is expensive may want to tune early and > shrink > > manifests more aggressively. > > On Thu, Jun 25, 2015 at 08:16:31PM -0400, Tom Tucker wrote: > > > Assuming 2,500 Linux clients running Puppet community edition > 3.8. > > Any > > > sizing recommendation for a PuppetDB system in regards to disk > size > > for > > > the DB, CPU, Memory, etc. > > > Thank you for your time and feedback. > > > > > > -- > > > You received this message because you are subscribed to the > Google > > Groups > > > "Puppet Users" group. > > > To unsubscribe from this group and stop receiving emails from > it, > > send an > > > email to [1][9][email protected]. > > > To view this discussion on the web visit > > > > > [2][10] > https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com > . > > > For more options, visit [3][11] > https://groups.google.com/d/optout. > > > > > > References > > > > > > Visible links > > > 1. mailto:[12][email protected] > > > 2. > > [13] > https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com?utm_medium=email&utm_source=footer > > > 3. [14]https://groups.google.com/d/optout > > -- > > You received this message because you are subscribed to the Google > > Groups "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, > send > > an email to [15][email protected]. > > To view this discussion on the web visit > > [16] > https://groups.google.com/d/msgid/puppet-users/20150626031113.GA832%40iniquitous.heresiarch.ca > . > > For more options, visit [17]https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, > send an > > email to [18][email protected]. > > To view this discussion on the web visit > > [19] > https://groups.google.com/d/msgid/puppet-users/CAGymF1CXG4cfFfFDEv4vRkko558un3Duo89LpqTTaHd1VmkQRQ%40mail.gmail.com > . > > For more options, visit [20]https://groups.google.com/d/optout. > > > > References > > > > Visible links > > 1. mailto:[email protected] > > 2. > https://ask.puppetlabs.com/question/13433/how-should-i-tune-passenger-to-run-puppet/ > > 3. > https://docs.puppetlabs.com/puppetdb/latest/scaling_recommendations.html > > 4. https://docs.puppetlabs.com/guides/scaling.html > > 5. http://activemq.apache.org/scaling-queues.html > > 6. http://activemq.apache.org/javalangoutofmemory.html > > 7. https://github.com/puppet-community/puppet-mcollective > > 8. https://github.com/puppetlabs/puppetlabs-firewall > > 9. mailto:puppet-users%[email protected] > > 10. > https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com > > 11. https://groups.google.com/d/optout > > 12. mailto:puppet-users%[email protected] > > 13. > https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com?utm_medium=email&utm_source=footer > > 14. https://groups.google.com/d/optout > > 15. mailto:puppet-users%[email protected] > > 16. > https://groups.google.com/d/msgid/puppet-users/20150626031113.GA832%40iniquitous.heresiarch.ca > > 17. https://groups.google.com/d/optout > > 18. mailto:[email protected] > > 19. > https://groups.google.com/d/msgid/puppet-users/CAGymF1CXG4cfFfFDEv4vRkko558un3Duo89LpqTTaHd1VmkQRQ%40mail.gmail.com?utm_medium=email&utm_source=footer > > 20. https://groups.google.com/d/optout > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/20150626132535.GA4476%40iniquitous.heresiarch.ca > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAAAzDLf6X_%3DKqGRjREo6uR51roKnkdj%2B6UShhjZCf3LxGxak%3Dw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
