On 16/03/09 18:30, Mark Plaksin wrote: >> Here are several general tips: > > Thanks!! > > We're using all of 24.8rc1. > >> * use InnoDB for better concurrency, and data persistance >> * use a recent mysql version (ie 5.0 or 5.1) >> * properly size your innodb_buffer_pool to the rough size of your >> on-disk database to not be I/O bound > > We need to fix this one. But that won't stop all of the UPDATEs :)
Of course but they will be latched in RAM and later batched to disk. >> * use a dedicated server for mysql (and not a VM or the same host as >> your master). Use decent hardware. >> * put your databases files on a RAID10 (or RAID1) with Battery Backed >> Cache (that's actually really important to absorb the fsyncs without >> being detrimental to the performance) >> * use as more spindles as you can for better concurrency I forgot: make sure your db host is not swapping. > Our database is currently on an internal hardware RAID1 controller > with two disks. We've talked about moving the database to our SAN but > the disks aren't *that* slow. Our stats say that when storeconfigs is > on there's very little iowait--just lots of CPU. Beware with SANs, they can have a better throughput but a really worse latency which is what you want to minimize to have fast fsyncs. Make sure your HW RAID has a BBU unit, that's the real important point. Now that I think about it, how many puppetmaster active connections do you have on your sql engine? And how many cpu (and cores) do your mysql server has? I'm asking this because MySQL doesn't really scale with the number of cpu/cores in term of concurrent clients. If that's the case, you might want to make sure you don't have that many active connections at the same time or limit the mysqld executable to a small (ie 4) number of core. The alternative is to use a patched MySQL (ie Percona or Ourdelta variants) version in combination with TCMalloc (Google Perftools). >>> FWIW, we turned on query logging >>> and there are 17k lines in the log associated with a single host. >> Which is not normal. With the perf path I contributed, you should see at >> most for one host, if nothing has changed: >> * 1 SELECT request for the querying the host >> * n SELECT request for exported resources (n == the number of exported >> resources) >> * a few request to get the facts >> * 1 SELECT request to retrieve parameters >> * 1 SELECT request to retrieve tags >> * a few update >> >> If you made changes to the manifests, then that's normal to see more >> insert requests. >> >>> About half are SELECTs and the other half UPDATEs. We have about 400 >>> hosts and, according to our stats based on puppet reports, we have >>> about 1.5 million resources total. >> You can >> select count(*) from resources group by host_id >> >> to get the number of resources per host. > > Cool. Most of the 17 hosts that made it into the DB before we turned > storeconfigs back off have about 750 resources. One has 5800 :) > >> On which tables do you see the UPDATES? > > It's only the resources table. Perhaps our entire problem is getting > over the initial hump. That is, we're starting from scratch so every > host needs to UPDATE all of its info. Actually, that's strange, you should see INSERTs don't you? I suspect something bad happening if you actually get UPDATEs. But I think you are right about your diagnostic. The basic issue is that tons of hosts are checking in at the same time for the first time in your masters, so the write load is huge on your database. In which case you should tune your database for a write load, that means: * increase the size of your innodb logs (if possible to some large values, like 892MB). * disable binlog and binlog_sync * innodb_flush_log_at_trx_commit = 2 * make sure innodb_flush_method = O_DIRECT if you're on linux * set innodb_max_dirty_pages_pct = 90 The idea to absorb this load would be to have a dedicated master setup (running on another port for instance) with storeconfigs enabled. Then manually move some hosts to this master by group of 10s or 20s. Then move them back to their regular masters. That would smooth the load on your database engine. Hope that helps, -- Brice Figureau Days of Wonder http://www.daysofwonder.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en -~----------~----~----~----~------~----~------~--~---
